“The Video Keeps Piling Up, but the Training Data Doesn’t”: Building a Keypoint Labeling Workflow for Behavior Analysis AI

From customer-extracted frames to keypoint annotation: how dataset development responsibilities are divided (reconstructed illustration)

More than 80% of the data generated worldwide is visual data such as images and video. Yet less than 1% of it is converted into business value. This gap is especially pronounced for teams developing AI from video.

Video accumulates quickly in storage, but usable training data does not. A one-minute video recorded at 30 frames per second contains 1,800 frames, yet most look nearly identical to the frames immediately before and after them. That is why video AI teams often begin by selecting and extracting only the frames that are useful for training.

A video-based behavior analysis AI company had already built its own system for this first step. The company develops AI that analyzes video captured in research and experimental environments to automatically identify subjects’ movements and behaviors—automating observational work that previously required researchers to review footage and record what they saw manually.

Demand for this category of technology, which estimates movement and pose directly from video without physical markers, is growing rapidly. Grand View Research projects the global markerless motion capture market to grow from USD 62.4 million in 2023 to USD 207.9 million by 2030, representing a CAGR of 17.1% (Grand View Research, Markerless Motion Capture Market Report).

The company uses its own model-based logic to extract training frames from video. Its challenge came next: turning those extracted frames into labeled training data through a workflow that a small research team could realistically sustain. To address this, the company adopted Superb Platform and Superb AI Data Labeling Services.

Challenge: Turning Extracted Frames into Training Data

The company works with frames extracted from observational video captured using fixed cameras. Its internal system had already solved the problem of selecting which frames to use for training. Converting those frames into training data, however, created a different set of operational challenges.

Keypoint Labeling Is Far More Labor-Intensive Than Bounding Boxes

Recognizing behavior requires more than knowing where a subject is. The model also needs to understand the subject’s pose. That is why the company labeled multi-joint skeleton keypoints alongside bounding boxes.

Keypoint annotation requires annotators to place each joint individually. When a subject curls up or parts of the body are occluded, annotators must also determine how to label joints that are not directly visible. As a result, each image requires substantially more work than conventional bounding-box annotation.

When Guidelines Changed, Existing Data Had to Be Reworked

As model development progressed, the labeling criteria evolved as well. Rules for handling occluded joints and determining which frames should be excluded from training were adjusted based on experimental results.

When those criteria changed, previously labeled data had to be updated to meet the new standards. Model experimentation could be delayed while the team reworked existing annotations.

Labeling Volume Was Consuming Research Time

As the frame-extraction pipeline continued to generate new data, the backlog waiting for annotation kept growing. Labor-intensive keypoint labeling, combined with rework caused by changing guidelines, increasingly tied up the small research team that should have been focused on model development. The same researchers were also responsible for reviewing annotation quality.

Solution: Standardize the Workflow in Superb Platform and Scale Labeling with Data Services

Superb Platform connects the AI development process—from data collection and labeling to model training and evaluation—within a single Vision Intelligence platform.

Once the frames extracted through the customer’s internal logic are uploaded to Superb Platform, the labeling and quality-management workflow is handled through Superb Platform and Superb AI Data Labeling Services.

Workflow from customer-side frame extraction to labeling and workload distribution (reconstructed illustration)

1. Managing Keypoints and Bounding Boxes in a Single Project

Bounding boxes for object detection and keypoints for pose estimation were annotated together within the same project.

The joint configuration and connections were defined as a skeleton, while visibility states for individual joints were recorded as attributes. Because these criteria were defined within the project rather than left to individual annotator judgment, the same standards could be applied consistently even when different people worked on the data.

Since both annotation types were created on the same images, the company also avoided having to manage separate datasets for each model.

2. Delegating High-Volume Labeling to Data Labeling Services

High-volume work—including relabeling existing data after guideline changes and annotating newly collected data—was handled through Superb AI Data Labeling Services.

Dedicated annotators performed labeling and quality review according to the guidelines defined by the customer, while the customer reviewed the completed results. This allowed the research team to spend less time on repetitive data work and more time on model development.

Benefit: Keeping Dataset Development from Holding Back Research

The biggest change was not simply faster labeling. It was a shift in the role dataset development played within the team.

From researchers handling labeling themselves to a workflow shared across Superb Platform and data labeling services (conceptual illustration)

• Keypoint labeling moved from individual expertise to a repeatable system: Skeleton structures and visibility criteria are defined within the project, helping maintain dataset consistency even when annotators change.
• Changes in labeling criteria became easier to manage: When guidelines evolve, rework can be handled with external labeling resources so dataset maintenance does not disrupt the model development schedule.
• A small research team can stay focused on research: With a significant portion of data operations handled through Superb Platform and data labeling services, researchers can devote more of their time to model experimentation.

The company has continued using this workflow over multiple years, expanding its datasets alongside each round of model improvement.

Keypoint Dataset Quality Depends on Labeling Operations

For tasks such as pose estimation, where performance depends on joint-level precision, dataset quality is shaped not only by annotation accuracy but also by the operational system behind it: consistent labeling criteria, reliable quality review, and the ability to handle rework when those criteria change.

This case shows where even teams with an established in-house frame-extraction pipeline can encounter bottlenecks in labeling operations—and what changes when the labeling framework and high-volume annotation work are supported through Superb Platform and data labeling services.

With experience handling more than 130 million industrial visual data samples, Superb AI supports the full process of turning visual data into intelligence, from data labeling to model development and operation.

Frequently Asked Questions

Q. Can video data be labeled?

Yes. A common approach is to extract individual frames from video and upload those images for annotation. In this case, the customer used its own internal logic to select and extract training frames, then uploaded them to Superb Platform for labeling.

Q. Does Superb AI support keypoint annotation for pose estimation?

Yes. Keypoint templates can be customized to the structure required by each project. Points and the connections between them can be defined as a skeleton, and visibility attributes—such as whether a keypoint is visible or occluded—can also be recorded. Keypoints can be used alongside other annotation types, including bounding boxes, within the same project.

Q. What happens if the labeling guidelines change during the project?

Existing data may need to be reworked to reflect the new criteria. In this case, Superb AI Data Labeling Services handled both updates to previously labeled data and annotation of new data, helping keep the customer’s research team from being pulled into rework.

Q. How is data security managed when labeling is outsourced?

Data is used only within the agreed scope and purpose of the project, and annotators are granted access on a project-by-project basis. Superb Platform also supports access controls such as two-factor authentication (2FA) and IP allowlisting. For customers with additional security requirements, Superb AI also supports closed environments, including on-premises deployment.