From Seeing to Doing: Why 4D Labs Thinks Physical AI Needs a New Data Layer

Written by Silvia Pavelli

Language models can absorb text and image models can classify a scene, but neither modality alone tells a machine what happens when a soft object yields, a surface slips or a tool meets resistance. The 4D Labs whitepaper argues that systems trained chiefly on digital material do not fully encounter the continuous physical phenomena that govern action: gravity, friction, deformation and contact force, among them.

That distinction is the company’s starting point. A robot that recognizes a cup is not necessarily able to grasp it gently, set it down on an uneven surface, or recover when its first movement fails. Those are not simply perception problems; they involve timing, material response and the link between an action and its consequence.

For 4D Labs, the missing layer is embodied data: high-fidelity records of physical interaction rather than a larger archive of pixels alone. Its whitepaper is not chiefly a plan to operate robot fleets. It proposes infrastructure that makes real-world interaction collectible, governable and reusable as training material.

Data is the first rung of an embodied ladder

The company organizes its thesis as a progression from Data to Embodied to AGI. Data combines vision, touch and force. Embodied adapts that learning to produce action through a robot body. AGI, the longer-term claim, becomes more general only when it can reason and act within physical laws.

That sequence places a useful constraint on the conversation about general intelligence. In 4D Labs’ framing, generality is not just broader language or larger parameter counts. It must be tested where objects collide, deform and behave differently from one setting to the next. A model that can describe a task may still lack the feedback needed to execute it.

The whitepaper’s test for the underlying data has four parts: it should be real, multimodal, scalable and diverse. Simulation can generate volume but may diverge from actual conditions; laboratory collection can be precise but narrow; vision-only datasets omit contact signals; conventional teleoperation can be costly and difficult to extend. 4D Labs proposes that satisfying all four together—not optimizing one in isolation—is the hard problem.

Making routine work into training material

Its answer is to make the real world a training ground. The company materials describe human experts’ everyday or professional tasks as sources of first-person demonstrations. Robot tasks captured through teleoperation can become examples for later training, while customer workflows can serve as sites for data creation, validation and scenario reuse.

The proposed collection system has several paths. The Ego Suite combines RGB and IMU glasses with tactile gloves or e-skin and an EMG wristband for first-person capture. A UMI-style gripper adds a camera and force sensor for repetitive grasping work. Teleoperation via exoskeletons, VR headsets and force-feedback devices is positioned for complex tasks. Smartphone 3D scanning is intended to widen spatial-scene collection.

Capture, however, is not the end product. 4D Labs describes a path from collection through cleaning, synchronization and labeling to training-ready data, with traceability across the workflow. Its intended Collect · Train · Deploy flywheel uses model feedback to inform what should be captured next.

Embodiment is the transfer test

Moving from data to embodied action is the more demanding step. Human hands and robot arms do not share the same joints, constraints or control logic. 4D Labs proposes Cross-Embodiment Mapping, a nonlinear approach intended to align human demonstrations with different robotic forms while accounting for dynamics, self-collision and task constraints. It is a proposed method, not an independently validated universal transfer mechanism.

The whitepaper also describes ViTam, a vision–tactile fusion approach meant to preserve temporal consistency while reconstructing hand–object interactions, especially when deformation or occlusion makes a camera-only record incomplete. Alongside it, AXON is presented as a closed loop of data calibration, counterfactual generation and neural simulation. The company says this loop could extend real captures while checking generated examples against learned physical priors; such capability claims remain the company’s own.

The wider point is less about any single module than about the order of operations. 4D Labs contends that physical AI must begin with contact-rich evidence, then learn to adapt that evidence across bodies, before it can plausibly claim more general competence. Whether that architecture delivers at scale remains an open execution question. But it identifies the central transition: moving from systems that see the world to systems that can do something in it.

News
Silvia Pavelli

Silvia Pavelli

Silvia Pavelli is an Italian journalist and AI correspondent based in Rome. She covers how artificial intelligence is reshaping business, policy, and everyday life across Europe. When she's not chasing a story, she's probably arguing about espresso.