To teach a robot how humans perform manual tasks, we need a 3D record of where the hands moved and when. Our research team ran an experiment to extract those coordinates from first-person video captured with a single GoPro. The result reconstructs both the camera trajectory and hand motion from the same video.

Input: A Single Camera Worn by the Operator

The input is first-person video captured with a GoPro. The footage was recorded while an operator wore the camera and performed real tasks, with no simulated renders involved. It was captured at 1920×1080 resolution and 60 frames per second.

We chose this type of footage because it is one of the easiest forms of data to collect in the field. No dedicated studio or markers are required. The operator only needs to wear a single camera. Converting this video into spatial coordinates is an early step in building training data for Physical AI.

Processing: Two Pipelines from a Single Video

We extract two types of information from the same video. RGB and IMU data are used to estimate camera odometry. RGB data is used to reconstruct 3D hand motion. The two tracking pipelines operate independently, so each can produce results on its own. This demo explores what becomes possible when the two outputs are brought together in the same scene.

Odometry estimates where and how far the camera has moved by accumulating changes across consecutive frames. In this demo, the inputs are the video itself and IMU measurements recorded by the GoPro. For odometry, the video is downsampled to 960×540. Hand-motion reconstruction uses the original resolution.

Pipeline diagram. GoPro first-person capture (RGB + IMU) branches into camera odometry and 3D hand-motion reconstruction, with both rendered together inside a Gaussian Splatting reconstruction of the environment.

Output: Camera Trajectory and Hand Motion in the Same Reconstructed Space

The workspace is reconstructed as a 3D environment using Gaussian Splatting. The estimated camera poses and hand motion are then rendered together inside that space.

Bringing both into the same space is the key point of this demo. When the camera trajectory and hand motion share the same coordinate system, we can preserve a single record of what the hands did and where in the environment each action took place.

The video display has three views. The lower-left panel shows the original GoPro footage. The lower-center panel shows the reconstructed environment rendered from the estimated camera pose. The main view shows the camera trajectory and hand motion together inside the reconstructed space.

The background in the main view is virtual. The camera trajectory and hand motion shown within it are estimated from the actual recorded footage.

The lower-center panel serves as a validation view. If the scene rendered from the estimated pose aligns with the original frame from the same moment, the pose estimate is consistent. If the two diverge, tracking is less stable in that segment.

What We Measured

The median camera position error was approximately 1.4 cm. In more difficult tracking segments, the error increased to around 3 cm.

The reference for comparison was generated by processing the same footage offline. Because that process also incorporated external-view images covering the entire environment and jointly aligned them, we treated the result as a more accurate reference, or pseudo-ground truth. Odometry inference ran at 27 FPS.

27 FPS means that the odometry pipeline estimates 27 camera poses per second. It does not correspond one-to-one with the original capture rate of 60 FPS. In video inference, real-time processing is commonly considered to mean at least 15 frames per second with latency within approximately one to two seconds. By that criterion, 27 FPS qualifies as real time.

Using Camera Trajectory Tracking on Its Own: Reconstructing a Path Through Space

We also tested what the camera-trajectory branch can provide when used independently for spatial movement tracking. A first-person video recorded while walking through the environment is aligned with the reconstructed 3D space, allowing the camera’s position and orientation to be estimated frame by frame.

Each yellow dot in the video represents an estimated camera position. The dots shown on screen are sampled points along the path; the actual estimated trajectory is denser.

Hand-level task tracking and environment-scale trajectory reconstruction operate at different scales. What they share is the ability to recover a 6DoF trajectory from video captured with a single camera.

We covered the process of reconstructing the 3D environment itself in our previous post, “Not a Photo, but a Space You Can Walk Into: Reconstructing 50 Korean Homes with 3DGS.”

Test Conditions and Limitations

We believe the measurement conditions matter more than the headline numbers, so here is the exact scope within which these results were confirmed.

• Testing was performed on a single NVIDIA A6000 GPU.
• The input was first-person GoPro footage at 1920×1080 and 60 FPS. Odometry was processed at 960×540, while hand-motion reconstruction used the original resolution.
• The reported 27 FPS is the odometry inference speed. We used the common video-inference criterion of at least 15 FPS and latency within one to two seconds as the definition of real time.
• Test videos were approximately one minute long. Video duration does not affect processing speed.
• The median position error of 1.4 cm is a relative measurement against the offline-processing result, which included alignment with external-view images and was treated as pseudo-ground truth. It was not measured against an external ground-truth system such as motion capture. Error increased to around 3 cm in difficult segments, and the worst individual frames exceeded that range.
• We do not yet have finalized quantitative metrics for the trajectory-only demo, so none are reported here.
• Both videos show results from internal demos. The measurements were not produced using an external benchmark or public dataset.
• Some scene conditions remain difficult for tracking. When the camera moves while a large portion of the scene is also moving, it can be difficult to distinguish camera motion from scene motion. IMU acceleration alone is also insufficient for accurately estimating changes in position.

If you are exploring how to build action data for robot training, tell us about your use case below. A Superb AI expert will get in touch.

Research by Chan Young Song at Superb AI

Superb AI is a Vision Intelligence company that transforms visual data from industrial environments into actionable intelligence for enterprises. We build data pipelines that convert human actions captured in first-person video into spatial coordinates for Physical AI training.