Code for the Real-World Robotics class (RWR 2026) with the ORCA v2 hand:
- Reinforcement learning: run a policy trained in simulation on the real hand (cube spin), with camera-based cube detection.
- Imitation learning: teleoperate the hand (Rokoko glove) and the Franka arm, record demonstrations, and run the trained policy.
The code of the RWR 2024 class is kept under the git tag rwr2024.
| Folder | ROS package | Content |
|---|---|---|
src/hand_control |
hand_control |
ORCA v2 hand controller (through orca_core), Franka gripper, tensioning |
src/ingress |
ingress |
Sensor input: OAK-D and ZED cameras, Rokoko glove, webcam hand tracking (MediaPipe) |
src/retargeter |
retargeter |
Maps the human hand keypoints to the hand's joint angles, with tools to tune and calibrate it |
src/logger |
logger |
Records ROS topics during demonstrations and converts them to HDF5 |
src/viz |
viz |
Robot models (ORCA v2, Franka, cube), RViz configs, joint visualization |
src/common |
Shared Python utilities | |
experiments |
experiments |
Launch files, configs and scripts for the tasks |
tests |
faive_test |
Retargeter replay test, ORCA v2 chirp tests |
third_party/franka_msgs |
franka_msgs |
Franka message definitions |
The Python package is called faive_system (e.g. from faive_system.src.common.utils import ...).
- Create the
srcfolder of your ROS 2 workspace and clone the repository into it:mkdir -p ~/ros_ws/src cd ~/ros_ws/src git clone https://github.com/srl-ethz/rwr_system.git
- Set up a ROS 2 Jazzy environment, e.g. with pixi following Getting Started with RoboStack. With pixi, prefix the commands below with
pixi runand run them from~/ros_ws. - Install the ROS dependencies:
If
pixi add pip ros-jazzy-rosidl-typesupport-c ros-jazzy-joint-state-publisher-gui ros-jazzy-xacro pixi run which pip # should print ~/ros_ws/.pixi/envs/default/bin/pip or similarwhich pipprints another pip, comment out the conda/mamba setup in~/.bashrcor~/.zshrc. - Install the Python dependencies:
pixi run pip install -e './src/rwr_system[all]' # the ORCA v2 driver. --no-deps: orca_core asks for a newer numpy than this package pins, but works with it pixi run pip install --no-deps orca_core==0.5.2
- Build the workspace and source it:
With pixi, add the source step to
cd ~/ros_ws pixi run colcon build --symlink-install source install/setup.bash
~/ros_ws/pixi.tomlinstead:If the build fails after changes, remove[target.unix.activation] scripts = ["install/setup.bash"]
build/,install/andlog/and build again. - For the RL task, download the cube detector's model files once:
./src/rwr_system/experiments/rl_inference/pose_estimation/download_models.sh
Task 2 uses ROS 2 Humble. You can set it up in your Task 1 workspace, which replaces its Jazzy environment: remove pixi.toml, pixi.lock, .pixi/, build/, install/ and log/ there, then start at step 2. Or keep Jazzy and use a separate workspace (e.g. ~/rwr_humble_ws), starting at step 1:
- Clone the repository into
~/rwr_humble_ws/srcas in step 1 above. - Set up the Humble environment. Python is pinned to 3.12: pixi would otherwise install Python 3.14, which some of our dependencies do not support yet.
cd ~/rwr_humble_ws pixi init -c https://prefix.dev/robostack-humble -c https://prefix.dev/conda-forge pixi add "python=3.12" ros-humble-desktop ros-dev-tools pip ros-humble-rosidl-typesupport-c ros-humble-joint-state-publisher-gui ros-humble-xacro "pytest<9"
- Add to
~/humble_ws/pixi.toml:[activation.env] VIRTUAL_ENV = "" # a virtual env active in your terminal (e.g. set by VS Code) would otherwise break the build [target.unix.activation] scripts = ["install/setup.bash"]
- Follow steps 4 to 6 above in
~/rwr_humble_ws.
Run nodes with ros2 run <package> <file> and launch files with ros2 launch <package> <launch file>.
Tension the tendons, calibrate the hand and move the joints: see src/hand_control/README.md.
To check the hand without teleop or a policy, experiments/fixed_command.py publishes fixed joint poses (edit the poses in the script), and ros2 run experiments command_gui.py -n 17 -t policy_output moves each joint with a slider.
Run a policy trained in faive_lab on the hand, detect the cube, and calibrate the camera: see experiments/README.md.
For an isolated Linux setup, SSH-friendly launch arguments and raw/processed data validation, see the RWR 2026 collection guide.
The hand follows the Rokoko glove through the retargeter, and the Franka arm follows the glove's wrist. The Franka arm controller is not part of this repository: it subscribes to /right/arm/end_effector_pose_cmd and publishes /right/arm/end_effector_pose.
-
Cameras. Recording uses
cube_sorting_task.yamlby default. Check that each OAK-D MXID matches its physical view. For a different camera mapping, pass its full path asoakd_config:=/path/to/cameras.yaml. Passzed_config:=/path/to/zed.yamlfor a ZED camera. Incontrollers_and_cameras.launch.py, select the files usingOAKD_CAMERA_CFG_NAMEandZED_CAMERA_CFG_NAME. Measure camera extrinsics withexperiments/calibrate_camera_pose.py. -
Retargeter. Tune the mapping from your hand to the robot hand, see src/retargeter/README.md. The ORCA v2 settings are in
experiments/cfgs/retargeter/. -
Test the teleop (no recording):
ros2 launch experiments teleop_rokoko.launch.py
A
keyboard_runnerterminal opens: press Enter to start, pause and resume the teleop. Foot pedals that type keys work the same way. -
Record demonstrations. The dataset name defaults to
cube_sorting. Passdataset_name:=NAMEto choose another name.ros2 launch experiments record_demonstration.launch.py
In the
keyboard_runnerterminal, Enter starts the teleop, starts the recording and stops it. Each demonstration is saved as an.h5file intmp/<DATASET_NAME>/(relative to where you launched). The default recorded topics are listed inexperiments/cfgs/logger/datacollection_right.yaml. Passlogger_config:=/path/to/logger.yamlto record a different set of topics. -
Process the data. Resample all topics of the recordings in a folder to a common rate:
ros2 run logger syncronize_data.py tmp/<DATASET_NAME> --sampling_freq 10 \ --cfg_file "$(ros2 pkg prefix --share experiments)/cfgs/logger/datacollection_right.yaml"
Check the demonstrations with
experiments/scripts/plot_trajectories.pyandexperiments/scripts/visualize_videos.py. -
Train a policy, e.g. with srl_il.
-
Run the RWR26 cube sorting policy. Bring up the Franka feedback and controller separately. The launch starts the cameras and the ORCA v2 driver. The ORCA v2 driver moves the hand to neutral on startup.
ros2 launch experiments run_policy_inference.launch.py policy_ckpt_path:=/path/to/best_model.pth
Send
starton/model_inference_commandto begin policy execution. All policy outputs are published on/policy_inference/actions_armand/policy_inference/actions_hand. Live mode relays them to the robot controllers. Usepreview:=trueto keep only the policy outputs.For optional Franka initialization, set
allow_arm_init:=trueand pass an operator-confirmed pose toarm_init_pose:=as a list of seven numbers[x, y, z, qx, qy, qz, qw]in the configured arm command frame. Sendinit_armwhile the policy is idle. Wait forArm reached init pose. Ready for start.before sendingstart. Usestopto stop policy output or abort initialization. Theresetcommand clears policy state.
| Tool | Use |
|---|---|
experiments/gesture_runner.py |
Enables the retargeter only while one wrist is held more than 10 cm higher than the other (needs both gloves), so the hand stops following when you lower it |
experiments/run_keyboard_teleop_franka.py |
Moves the Franka end effector with the keyboard |
experiments/calibrate_franka_load.py |
Calibrates the load mounted on the Franka |
src/viz/visualize_static_frames.py |
Publishes fixed frames (e.g. cameras) from a YAML file such as experiments/cfgs/tf/static_frames_bimanual.yaml |
experiments/orca/ |
ORCA v2 chirp data acquisition for system identification, see experiments/orca/README.md |
To contribute, fork the repository and open a pull request.