Using VR Controllers for Robot Teleoperation

VR controllers, such as Meta Quest, HTC Vive, and Valve Index, have emerged as a favored teleoperation input for robot learning due to their comprehensive 6-DOF pose tracking, intuitive hand-based control, and integrated buttons/triggers that align conveniently with gripper operation. Operating within a budget-friendly, consumer-grade package, these controllers offer an appealing solution.

The advantages of utilizing VR controllers over a gamepad or keyboard are manifold: they provide continuous 6-DOF pose tracking, updating swiftly and intuitively mapped to end-effector pose control. The natural hand motion allows for trajectories that resemble genuine human manipulation, deviating from the typical artificial waypoint sequences.

Moreover, the built-in analog trigger is ideal for continuous gripper open/close control, eliminating the need for a binary toggle. Additionally, some controllers offer haptic feedback, offering valuable signals regarding contact or constraint violations directly back to the operator. To harness controller pose data, most VR SDKs, including OpenXR, Meta's Oculus SDK, and SteamVR, supply controller pose as a position and quaternion relative to the headset's tracking origin, frequently updated at high frequencies (ranging from 60 to 120 Hz).

A basic OpenXR-style polling loop can be structured as follows: a class called VRControllerInput, initialized with the xr_session and controller_path, defines a read method that retrieves the controller pose via session.get_controller_pose(controller_path), obtains the trigger value using session.get_input_value(controller_path, trigger), and retrieves the grip button status via session.get_input_value(controller_path, grip).

The method then returns a dictionary containing the position (x, y, z), orientation (quaternion x, y, z, w), trigger value (0.0 - 1.0 mapping to gripper control), and grip button status (engage/disengage teleop). For Unity or Unreal developers integrating VR side streaming into a Python robot controller, this pose data is typically transmitted over a lightweight transport method like ZeroMQ, gRPC, or WebSocket, given that VR engines and robotics stacks (ROS, Python control loops) generally reside in separate processes.

The primary challenge lies in mapping the controller's pose to the robot's end-effector pose, as the coordinate frames and workspace scales frequently diverge. A common strategy involves establishing a reference frame when the operator engages the grip button, recording the controller's current pose as the origin. For subsequent frames, the controller's pose is computed relative to this origin, rather than its absolute pose.

A scale factor is then applied to account for the disparity between human arm movement ranges and the robot's workspace capabilities. This adjusted relative motion is added to the robot's pose at engagement time, generating the new target pose.