Beyond the Model: 9 Practical Engineering Hurdles in Real-Robot Imitation Learning (SO-101 + ACT)

Introduction

The author built a project as the final assignment for a course at the University of Tokyo. The goal was to use imitation learning on a small desktop robot arm to pick a sponge, carry it to a target sheet of paper, and place it gently. The robot was tested on the real hardware, not just in simulation.

Challenges

The author faced several engineering hurdles while bringing the robot arm to autonomous motion:

1. Power and wiring issues

2. Desk layout inconsistencies

3. Data quality control

4. Policy training and selection

5. Real-robot evaluation and problem finding

Hardware wiring and power

The author initially struggled with connecting the leader and follower serial links and mixing up power supplies. They solved this by carefully plugging in USB serial adapters one at a time, labeling cables, and creating a power-up checklist. This eliminated communication errors and servo-related problems caused by swapped ports and missing power.

Desk layout standardization

Early in the project, the arm would sometimes touch the stand's leg as it approached the target. The author redesigned the desk layout to keep the arm's range and camera framing consistent. They also changed the sponge and paper target dimensions to create a uniform coordinate system for the datasets.

Data quality control

The author recorded demonstrations using the leader arm and inspected joint angles and camera video to remove outliers and unneeded idle time. This ensured the training data was high-quality and relevant for the imitation learning process.

Policy training

Using a MacBook Pro with Apple Silicon and PyTorch MPS, the author trained an Action Chunking with Transformers (ACT) policy. They avoided renting cloud GPUs and trained the policy locally, achieving a success rate of 3/3 on the evaluation set. The policy was further refined by selecting weights based on real-robot behavior and reach, not just validation loss.

Real-robot evaluation and problem finding

The author conducted autonomous tests in a standardized setup, identifying the causes of tremors and missed grasps. They then fed this information back into the loop for further improvements.

Overall, the author's project demonstrates the importance of addressing both digital and physical challenges in real-robot imitation learning. By systematically working through each stage and finding solutions, they were able to achieve successful grasp-and-place runs on the L2 evaluation set.