When people think of robot training data, they often picture a person standing beside a physical robot and demonstrating actions one by one. But that approach alone makes it difficult to collect the volume of data needed while varying environments and objects.

Our research team is experimenting with three approaches in parallel: generating data in simulation, collecting data from a physical robot, and having a robot collect data autonomously as it patrols. This post documents the setups and conditions used in these three demos.

In a previous post, we covered how images and ground-truth labels can be generated together inside a simulator. This post moves to the next step: robot behavior data—the episodes used to train Physical AI models.

Read more → Building Superb AI’s Synthetic Data Pipeline with NVIDIA Isaac Sim

Path 1: Generate Episodes in Simulation (SIM)

As a robot performs a task in Isaac Sim, its observations and actions are recorded as an episode. Repeating this process while varying the environment, task, and objects builds up training data. The rapidly changing scenes in the first part of the video show episodes generated in this way.

Figure 1. Grid of simulation scenes showing the same robot performing tasks across different combinations of environments and objects

The final scene in the video shows an evaluation episode in which a GR00T model successfully completes a pick-and-place task. This model was trained at an earlier stage, before the pipeline described above. It was fine-tuned for 100k steps on 1,000 episodes from a single environment containing one red cube, with no background or floor textures.

The pipeline that generates data while varying environments, tasks, and objects is the next step. A model trained only on data from this single, empty environment is unlikely to generalize far beyond the red cube.

Our goal is to use the generation pipeline to create hundreds of thousands of demonstrations and train a policy that continues to work as objects and environments change. We will report the success rate together with a before-and-after comparison once the multi-environment training results are available.

Path 2: Human Demonstrations on a Physical Robot (REAL)

With the physical robot, a human operator creates demonstrations through teleoperation. Video from a head camera and wrist camera is recorded in sync with the robot’s joint states. Each episode is paired with one language instruction, such as “Pick black small cylinder into white bin.” Together, these elements make up one training episode. We have collected 2,500 episodes to date.

Figure 2: Diagram showing the head camera, wrist camera, joint states, and a language instruction combined into a single episode. Do not use screenshots from the actual data-collection interface

We changed the camera configuration several times. The head camera was attached using a mount designed in-house and angled approximately 30 degrees downward toward the floor. Objects on the ground are captured primarily by the head camera, while elevated objects are captured mainly by the wrist camera.

Not every recorded episode is used. If footage is missing from either camera, the episode is discarded. An episode is also rejected if the location where the object is placed falls outside the head camera’s field of view.

The commanded joint values and actual joint values do not match exactly. The actual joint values are produced by the robot’s low-level controller, which follows commands subject to several constraints. One of those constraints is a joint-velocity limit imposed for safety.

By keeping the same controller settings during inference, the data distributions during training and inference remain aligned. We therefore continued collecting data with this difference between commanded and actual joint values left intact.

Path 3: The Robot Collects Data as It Patrols (SIM)

We also tested a setup in which data can accumulate without human demonstrations. Inside Isaac Sim, a quadruped robot patrols a predefined route and saves images at each waypoint. These images are used as camera data collected by the patrol robot. In this setup, the patrol itself is the data-collection process.

Locomotion is controlled by a policy trained with reinforcement learning. The policy and the physics simulation work together to produce the robot’s movement.

The scene in which the arm picks up a bottle and a cup at the destination is not the result of learned manipulation. Given a target pose, cuRobo motion planning computes the arm trajectory. The target poses and dataset were taken from RoboTwin.

Because the video alone could make it appear that the picking behavior was also learned, we explicitly distinguish between the learned and computed components here.

Where the Three Paths Meet

The three demos approach the same question from different directions. Simulation provides scale, physical robots provide real-world data, and autonomous patrol enables collection without continuous human demonstrations.

To use simulation data and real-robot data in the same training process, their distributions need to be aligned. Applying the same constraints in simulation—for example, the same joint-velocity limits used on the physical robot—is one part of that work.

Superb AI is continuing to add this type of robot behavior data to its more than 130 million industrial visual data samples.

Conditions and Limitations

• Simulation data generation and autonomous patrol are simulation results. Only the teleoperation data collection was performed with a physical robot. Each section is labeled SIM or REAL accordingly.
• The GR00T model in Path 1 is a baseline model trained on 1,000 episodes from a single environment with a red cube and no background. We do not yet have results from training on the multi-environment generated dataset. We will report the success rate when those results are available.
• The arm’s pick-and-place motion is the result of motion planning. The learned policy is used only for locomotion.
• We have collected 2,500 episodes on the physical robot. No other quantitative figures are included because we do not yet have finalized values. We will report additional numbers using the same criteria once they are confirmed.
• We are not disclosing the value of the joint-velocity limit. The robot model is also not identified.
• File names, data-collection interface screenshots, and operator information from the teleoperation dataset are not disclosed.

If you are exploring how to build robot behavior data for training, tell us about your use case below. A Superb AI expert will get in touch.

Superb AI is a Vision Intelligence company that transforms visual data from industrial environments into actionable intelligence for enterprises. We build robot behavior datasets in both simulation and real-robot environments.