Creating a Robot Sensor Data Recording and Replay System

Building a comprehensive robot sensor data recording and replay system is essential for effective robot learning workflows. Recording enables dataset creation and post-session debugging, while replay allows for the reuse of recorded sensor streams in various applications without the need for a physical robot, accelerating development and debugging. This tutorial outlines the process of building both recording and replay components as a unified system.

The significance of a robust replay system cannot be overstated. While recording may appear similar to simple logging, a well-designed replay system offers substantial benefits:

Debugging without hardware: Reproducing bugs from previous sessions becomes effortless, without the need to allocate robot time.

Regression testing: Recording a fixed set of sessions and replaying them through an updated perception pipeline allows for easy comparison of outputs, ensuring the stability of the system.

Dataset iteration: Previously recorded sessions can be reprocessed, re-labeled, or resampled as the learning pipeline evolves, all without the need to re-record data.

Simulation grounding: Comparing simulated sensor outputs against real recorded sensor data for the same trajectory aids in validating the system's performance.

Core design: A session consists of multiple synchronized streams, each containing its own timestamped samples. These streams include camera feeds, joint states, IMU data, force-torque information, and more. By storing each stream separately with accurate timestamps, it becomes possible to query the state of every stream at any given time T.

To accommodate sensors with varying sampling rates, the system records each stream at its native rate, preserving accurate timestamps. During replay, streams can be resampled to match the desired rate of downstream components (e.g., training pipelines, replay tools) without losing valuable information.

The recorder's responsibility is straightforward: subscribe to every sensor stream, timestamp each sample, and write the data to disk without blocking the control loop. By utilizing a dedicated writer thread per stream, the recorder ensures that disk I/O does not interfere with the control loop, preventing common issues like juddering movements during robot operation.

Recording is performed asynchronously, with sensor callbacks operating in separate threads. When called, the `record()` function must return immediately, passing the timestamped data to a queue for later processing. A dedicated writer thread handles the actual disk I/O, ensuring minimal impact on the control loop's performance.

To enable efficient querying of recorded data, each stream is associated with a timestamped data array. When retrieving data at a specific query time, the system identifies the nearest sample within the stream, ensuring precise alignment of sensor data for replay purposes.

The replay engine reconstructs a session and plays it back through the same interfaces used during live operation, allowing downstream code to remain unchanged. By loading the session data from disk, the replay engine reconstructs the individual streams and establishes their temporal relationships. This enables the replay system to mimic the live robot's sensor interface, ensuring seamless integration with existing software components.

In summary, a well-designed robot sensor data recording and replay system is indispensable for efficient robot learning workflows. By separating recording and replay concerns, accommodating sensors with varying sampling rates, and providing a cohesive replay engine, developers can accelerate development, streamline debugging, and improve the overall reliability of their robot systems.