oak_camera_calibration

ROS 2 helpers for hand-eye calibration with an OAK 4 Pro AF mounted on a UR10e end effector.

First step: acquire calibrated RGB images

Build the package from the workspace root:

cd /home/rosmatch/colcon_ws
source /opt/ros/jazzy/setup.bash
colcon build --packages-select oak_camera_calibration --symlink-install
source install/setup.bash

If ROS prints warnings about switching between one and jazzy, start a fresh terminal or clear the old ROS environment before sourcing Jazzy:

unset ROS_DISTRO ROS_VERSION ROS_PYTHON_VERSION ROS_ROOT ROS_PACKAGE_PATH
unset AMENT_PREFIX_PATH COLCON_PREFIX_PATH CMAKE_PREFIX_PATH
unset LD_LIBRARY_PATH PYTHONPATH
source /opt/ros/jazzy/setup.bash
source /home/rosmatch/colcon_ws/install/setup.bash

Start the OAK via the installed DepthAI ROS 2 V3 driver. The default parent frame is the right UR TCP, mur620/UR10_r/tool0:

ros2 launch oak_camera_calibration oak4_pro_af_4k.launch.py

This uses config/oak4_pro_af_4k.yaml and requests a 3840x2160 RGB stream. It also publishes the calibrated camera TF from mur620/UR10_r/tool0 to the DepthAI oak base frame. The direct hand-eye result is documented in config/mur620_ur10_r_oak_handeye.yaml as mur620/UR10_r/tool0 -> oak_rgb_camera_optical_frame; the launch defaults are the equivalent tool0 -> oak parameters expected by the DepthAI driver. The higher-resolution profile is also available:

ros2 launch oak_camera_calibration oak4_pro_af_rgb.launch.py

Save one image together with the factory calibration published as CameraInfo:

ros2 run oak_camera_calibration capture_oak_snapshot

By default the snapshot tool listens to:

• /oak/rgb/image_raw
• /oak/rgb/camera_info

and writes to ~/oak_handeye_samples. It ignores the first frames while auto-exposure and autofocus settle, skips very dark frames, then stores the sharpest frame from a short burst. It writes the raw image, the received CameraInfo, image metrics, and a rectified PNG computed from that CameraInfo. For hand-eye calibration it is usually best to keep the raw image plus distortion coefficients and let the calibration/detection code use both explicitly.

You can override topics or output path like this:

ros2 run oak_camera_calibration capture_oak_snapshot --ros-args \
  -p image_topic:=/oak/rgb/image_raw \
  -p camera_info_topic:=/oak/rgb/camera_info \
  -p output_dir:=~/oak_handeye_samples \
  -p save_rectified:=true

Useful snapshot parameters while tuning image quality:

ros2 run oak_camera_calibration capture_oak_snapshot --ros-args \
  -p warmup_frames:=20 \
  -p select_frames:=20 \
  -p min_mean_intensity:=10.0 \
  -p timeout_sec:=60.0

If images remain blurry after warmup, inspect live focus first:

ros2 run rqt_image_view rqt_image_view /oak/rgb/image_raw

For calibration captures, fixed lighting and a fixed working distance are preferable. Once the target distance is known, manual focus via the DepthAI driver parameters can be tested instead of relying on autofocus.

GUI sample capture

For repeated hand-eye samples, keep the camera driver running and start the GUI:

ros2 run oak_camera_calibration oak_sample_gui

The GUI uses the 10x7 ArUco GridBoard with DICT_4X4_250, 30 mm markers, and 10 mm separation. It detects the board with a downscaled full-image pass, then refines inside an ROI so the full 8k image is never passed directly to cv2.aruco.detectMarkers().

By default each saved sample also stores the current TF from base_link to tool0. If your UR TCP frame has a different name, pass it explicitly:

ros2 run oak_camera_calibration oak_sample_gui --ros-args \
  -p robot_base_frame:=base_link \
  -p robot_tcp_frame:=tool0

The HUD shows whether this TF is available. With the default require_robot_pose:=true and require_detection_pose:=true, pressing s only writes a sample if the image, CameraInfo, marker pose, and robot TF are all available.

Keys:

• s: save one sample
• d: toggle marker detection
• q or Esc: close the GUI

Samples are appended to ~/oak_handeye_samples as sample_000, sample_001, and so on. Closing the GUI never deletes samples; delete files manually only when you really want to discard them.

Each sample contains:

• raw PNG
• optional rectified PNG
• annotated PNG
• CameraInfo YAML
• JSON metadata with board model, marker IDs, ROI, pose, reprojection error, and the UR base-to-TCP transform

Compute hand-eye transform

After capturing multiple poses, compute the camera transform relative to the TCP:

ros2 run oak_camera_calibration compute_handeye --samples-dir ~/oak_handeye_samples

The tool uses OpenCV's eye-in-hand calibration and prints tcp <- camera, per-sample translational/rotational consistency, and the marker reprojection error. Samples without robot TF are skipped. Large residuals are a good hint that a sample should be inspected and deleted manually before recomputing.

Semi-automatic ChArUco capture

For a guided hardware session, start the robot first, then launch the OAK driver and the semi-auto GUI together:

ros2 launch oak_camera_calibration mur620_oak_handeye.launch.py

The launch defaults are for the OAK mounted on the right arm:

• arm:=r
• camera_launch_file:=oak4_pro_af_4k.launch.py
• camera_parent_frame:=mur620/UR10_r/tool0
• calibrated camera driver TF: camera_cam_pos_x:=0.0068564203, camera_cam_pos_y:=-0.0892312561, camera_cam_pos_z:=0.1018930213, camera_cam_roll:=-3.0221294847, camera_cam_pitch:=-1.5221462887, camera_cam_yaw:=-1.7026932502
• robot_base_frame:=mur620/UR10_r/base_link
• robot_tcp_frame:=mur620/UR10_r/tool0
• action_name:=/mur620/jparse_move_r
• jog_twist_topic:=/mur620/jparse_velocity_controller_r/twist_cmd
• jog_frame:=UR10_r/base_link
• jog_gui_enabled:=true
• move_enabled:=false
• keyboard_jog_enabled:=false
• dictionary:=DICT_4X4_250
• board_id_order:=column_major
• squares_x:=6
• squares_y:=9
• square_length_m:=0.065
• marker_length_m:=0.048
• use_camera_tf_initial_guess:=true
• camera_look_axis:=plus_z
• camera_roll_reference:=current
• target_max_camera_z_above_start_m:=0.01
• max_linear_velocity:=0.025
• max_angular_velocity:=0.10
• target_max_tcp_delta_m:=0.25
• target_max_camera_delta_m:=0.30
• target_min_camera_delta_m:=0.04
• target_max_rotation_deg:=35.0
• split_target_motion:=true
• split_rotation_step_deg:=25.0
• center_camera_xy_only:=true
• draw_board_center_overlay:=true
• target_pattern:=spiral_hemisphere
• hemisphere_axis_source:=board_normal
• samples:=18
• sphere_polar_span_deg:=50.0
• sphere_spiral_turns:=1.25

The session estimates the board pose from the first usable view, generates a 3D spiral on a hemisphere around the board center, and writes the same sample_*.json files used by compute_handeye. By default it writes to ~/oak_charuco_6x9_column_major_handeye_samples so older samples captured with the previous wrong board geometry are not mixed into the solve.

If the camera driver is already running, keep it running and start only the session:

ros2 launch oak_camera_calibration mur620_oak_handeye.launch.py launch_camera:=false

When the generated poses look plausible, enable automatic J-PARSE motion:

ros2 launch oak_camera_calibration mur620_oak_handeye.launch.py move_enabled:=true

With split_target_motion:=true, automatic target moves first drive to the new TCP position while keeping the current orientation, then rotate toward the final target in smaller split_rotation_step_deg increments.

The initial camera pose is treated as the top of the hemisphere. Proposed camera targets are filtered so they do not move more than target_max_camera_z_above_start_m above that start height.

The z GUI action computes the lateral offset between the optical axis and the estimated board center, keeps the current camera orientation and height, and moves only in base XY by default. Set center_camera_xy_only:=false to allow the full image-plane correction.

If a proposed target moves to a plausible position but rotates the camera away from the board, press v in the GUI and then n to regenerate the target. If that fixes the sign, restart with camera_look_axis:=minus_z.

The target orientation keeps the selected optical axis pointed at the board center and uses camera_roll_reference to choose the roll around that view axis. The default current keeps the current/start image roll, which is useful for the wide OAK image sensor. Useful alternatives are board_x_to_camera_x, minus_board_x_to_camera_x, board_y_to_camera_y, and minus_board_y_to_camera_y.

In the GUI, use:

• n: propose the next sphere target and show TCP/camera deltas
• g: send the shown target to J-PARSE if move_enabled:=true
• z: center the current camera view over the board center with lateral motion
• p: point the current camera orientation at the board center without translating
• b: move back to the first valid TCP pose captured after session start
• c: save the current sample
• v: flip the camera look axis for target generation, then press n again
• r: cycle the camera roll reference, then press n again
• manual Cartesian jog is handled by the separate mur620 robot jog window
• m: toggle translation/rotation jog mode
• q or Esc: close the GUI

When a ChArUco pose is available, the live image overlays board geometry: magenta marks the board center, blue marks the board origin, cyan marks the image center, orange marks the assumed full board outline, green marks the inner ChArUco-corner area, and the yellow line shows the pixel offset to center.

In the separate jog GUI, hold the on-screen buttons to jog the robot. Keyboard fallbacks are arrow keys for XY, PgUp/PgDn for Z, m to toggle rotation mode, i/k, j/l, u/o for direct rotation, the house button or h to move to the stored home pose, space or . to stop, and q to close only the jog window. The default home pose is stored in config/mur620_ur10_r_home.yaml.

The terminal prompts before every automatic move and before every saved sample. After at least four usable samples it updates the current tcp <- camera estimate and stores the latest session state in semi_auto_session_state.yaml.

The saved ChArUco samples can be solved explicitly with:

ros2 run oak_camera_calibration compute_handeye \
  --samples-dir ~/oak_charuco_6x9_column_major_handeye_samples