Original Feishu Document · Source Revision 15

💡

Systems Lab: Using an open-source VLA training stack such as OpenPI, complete the full workflow of “data contract → training → evaluation → inference service → control interface → monitoring → failure feedback.” The focus is not on memorizing commands, but on understanding the responsibility boundaries and evidence required at each layer.

Learning Objectives

After completing this lesson, you should be able to identify the data, model, and inference boundaries in an OpenPI-style codebase; establish a minimally reproducible training workflow; validate action normalization, temporal alignment, and control frequency; design offline, replay, shadow, and real-robot gates; and make each failure traceable, reproducible, and repairable.

1. System Layers

Any layer can cause a system to be “correct offline but wrong on the real robot.” The lab requires independent tests and version identifiers for every layer.

2. What Happens Before Data Enters the Model

Typical transformations include episode slicing, image resizing/cropping, language tokenization, embodiment selection, action coordinate conversion, action normalization, and action chunk construction. A training sample can be written as:

Interpretation: The model input at time t consists of images from the most recent k steps, robot state, and a language instruction. The supervision target is the H-step action chunk starting at t.

Derivation: A single-step policy learns only the mapping from the current state to the current action. An action chunk policy treats the next H actions jointly as one supervision target, reducing step-by-step sampling overhead and learning short-term action correlations. Here, t must be a physical time on a unified clock rather than an array index in each file.

2.1 Temporal Alignment Is a Causal Contract

Images, joint states, and actions are often recorded by different threads, at different frequencies, and with different device clocks. For an observation time t_o, select the action whose actual effective time is closest within the permitted window:

Interpretation: Select the action sample whose timestamp is closest to the current observation, and require the time difference between them to be no greater than the tolerance epsilon.

Derivation: Supervised learning assumes that the input precedes the physical response corresponding to its label. First performing nearest-neighbor matching by absolute time difference and then using epsilon to reject samples that are too far apart prevents queue latency, dropped frames, or clock drift from being mistaken for policy behavior. More rigorous systems must also compensate for camera exposure, network, and actuator activation latency.

Validation cannot rely only on array lengths. Plot images, states, candidate actions, and actions that actually took effect on the same timeline, then inject known motions and verify the temporal ordering between action direction and object response.

3. Normalization Is a Deployment Contract

If training uses:

Interpretation: Subtract the training-set mean from the j-th action dimension, then divide by that dimension’s standard deviation to obtain the normalized action.

Derivation: If action dimensions have different units and numerical ranges, an unscaled squared error will be dominated by large-scale dimensions. Centering removes positional bias, while division by the standard deviation brings dimensions to approximately comparable scales. The mean and standard deviation must be frozen for each data version, embodiment, and action semantics.

Deployment must apply the inverse transformation using the same statistics:

Interpretation: Multiply the model output by the standard deviation used during training, then add back the mean to recover the physical quantity of the j-th action dimension.

Derivation: Multiplying both sides of the normalization definition by sigma_j and then adding mu_j yields the inverse transformation. It is the inverse of the original transformation only when the dimension order, units, robot embodiment, and statistics exactly match those used during training. A mismatch in any one of these can produce numerically valid but physically incorrect commands.

Quantile scaling is often used for heavy-tailed actions. Statistics must be bound to the data version, robot, and action dimensions. Incorrect units, dimension ordering, or gripper direction can produce valid tensors while causing dangerous motions.

4. Minimal Training Reproduction

1. Pin the code commit, container, pretrained weights, and data manifest.
2. First overfit on a very small dataset to confirm that the loss decreases and that actions can be denormalized.
3. Fix the batch configuration and verify effective-batch semantics for single-GPU and multi-GPU training.
4. Save the random seed, optimizer, learning rate, gradient norm, and checkpoint hash.
5. Reconstruct inference from the checkpoint in a clean environment and compare outputs for fixed inputs.

5. Where Does the Action Output Actually Become a Robot Command?

Boundary	Validation Question
Model output	Continuous action, token, noise, or flow state
Sampler	Number of steps, stochasticity, temperature, and numerical integration
Denormalization	Are the statistics, units, and dimensions consistent?
Robot adapter	Absolute/incremental semantics, coordinate frames, limits, and gripper semantics
Controller	Frequency, interpolation, and whether only the first few steps of the chunk are executed

5.1 Coordinate Transformations Must Form a Closed Loop

If the model predicts a target end-effector pose in the camera frame while the controller requires commands in the base frame, then:

Interpretation: The commanded pose from the base to the target end effector equals the calibrated transform from the base to the camera multiplied by the pose from the camera to the predicted end effector.

Derivation: Homogeneous transforms are multiplied in path order: first from the base to the camera, then from the camera to the target end effector, yielding the transform from the base to the target end effector. An error in left/right multiplication order, coordinate-frame direction, or meters versus millimeters can preserve the correct matrix shape while causing the robot to move in the wrong direction.

5.2 Action Chunk Length Implies a Control Duration

Interpretation: The actual duration covered by an action chunk equals the number of action steps executed in this cycle divided by the control frequency.

Derivation: The control period is one divided by the control frequency, so the total duration of executing H_exec consecutive steps is the number of steps multiplied by the period. If the model is trained at 10 Hz but the controller replays actions at 20 Hz, the physical duration of the same action chunk is halved. The step count, frequency, and replanning period must therefore be versioned as a single deployment contract.

6. Latency Budget

Record P50/P95/P99:

Interpretation: End-to-end latency equals the sum of acquisition, preprocessing, inference, postprocessing, transmission, and controller wait times.

Derivation: An action must pass sequentially through these stages before it can affect the robot, so the durations on the critical path are additive. If some stages run in parallel, use the critical path on the actual timeline rather than mechanically summing all durations. Deployment gates should check P95, P99, and worst-case jitter rather than only the average.

6.1 Stale Actions Must Expire Before Execution

An action generated from an observation can represent the current world for only a limited time. Before executing action step r, the following condition should hold:

Interpretation: The time elapsed since the observation, plus the estimated time until step r of the action chunk is executed, must not exceed the validity period of the action plan.

Derivation: Actions later in the chunk depend on increasingly stale observations, so action age includes not only current communication and inference latency but also the queueing time within the chunk. Continuing to retry after expiration can turn an action that was correct for an old state into an incorrect action for the new state. The system should discard it, perceive again, or switch to a safety controller.

7. Four-Level Validation Path

1. Unit: Schema, transforms, normalization, sampler, and adapter.
2. Offline: Loss, action error, fixed input/output pairs, and held-out episodes.
3. Replay/Shadow: Run in real time on actual logs without taking control.
4. Robot: Use a safe environment, restrict actions, provide human takeover, and gradually expand the task scope.

Offline action MSE can validate the pipeline but is insufficient to demonstrate closed-loop success because multimodal tasks, distribution shifts, and recovery behaviors cannot be adequately represented by single-step errors.

8. Key Ablations for an OpenPI-Style System

• Hold the data fixed and change only the action representation or number of sampling steps.
• Hold the model fixed and change only the executed action chunk length.
• Hold the policy fixed and change only the control frequency and latency.
• Remove data by source, robot, and task to measure cross-embodiment benefits.
• Compare the original checkpoint, the version post-trained for the target scenario, and the version produced after failure feedback.

9. How a Failure Becomes a Training Fact

Bind every online episode to a deployment ID, model hash, config hash, robot/firmware version, data version, and monitor events. After a failure enters triage, do not merely label it as “failed.” Also identify the stage, recoverability, physical severity, probable root cause, and required type of supervision.

10. Lab Deliverables

Lab objective. Use a small task to complete the full data, training, inference, control, monitoring, and failure-feedback workflow. Every result must be traceable from the deployment ID to the code, model, data, configuration, statistics, robot, and firmware.

Stage	Minimum Operation	Passing Evidence	Fault Injection
Data contract	Validate the schema, timestamps, coordinate frames, units, and action dimensions	Random-sample visualizations and validator report	Swap dimensions, reverse the gripper sign, and introduce temporal misalignment
Minimal training	Overfit on a small dataset, then evaluate on independent held-out episodes	Loss, denormalized trajectories, seed, and checkpoint hash	Remove normalization stats and change the effective batch
Inference reconstruction	Start the service from the manifest in a clean container	Matching fixed input/output pairs, dependency lockfile, and model configuration	Change the crop, tokenizer, or default configuration
Action adaptation	Validate coordinates, units, absolute/incremental semantics, and limits layer by layer	Synthetic pose tests and zero-action tests	Introduce millimeter/meter mismatch and reverse matrix order
Real-time replay	Run replay/shadow at the actual frequency	P50/P95/P99, action age, and discard rate	Network jitter, timeouts, retries, and GPU downclocking
Controlled real robot	Limit speed and workspace, enable human takeover, and expand scope incrementally	Task outcome, peak force, stopping distance, and recovery rate	Move the target, freeze sensors, and introduce contact disturbances
Failure feedback	Convert one failure into a persistent data slice and regression test	Event timeline, root cause, fix commit, and gate results	Roll back the model while retaining the incorrect configuration, then verify that the mismatch is detected

Minimum delivery package. The episode schema, data manifest, training manifest, checkpoint, normalization stats, inference image, action-adapter tests, latency report, shadow results, real-robot records, deployment manifest, and failure lineage are all mandatory.

11. Common Failure Modes

Failure Mode	Why It Is Hard to Detect	Decisive Check	Fix
Different crops during training and deployment	The tensor shape is unchanged, and the offline service can still produce outputs	Save the transformed pixels and compare them pixel by pixel	Include the transform configuration in the manifest and hash
Normalization stats are missing or belong to the wrong embodiment	The output range appears normal, but the physical scale is incorrect	Normalizing and then denormalizing must reconstruct every dimension	Atomically package the statistics with the data, embodiment, and action schema
Training at 10 Hz and deployment at 20 Hz	The action direction is correct, but duration and speed are doubled	Compare the number of seconds covered by the action chunk, not only the step count	Jointly version the frequency, executed step count, and replanning period
Network retries execute stale actions	The request eventually succeeds, making service availability appear higher	Record the observation time, generation time, and execution age of every step	Use action expiration, idempotent sequence numbers, and renewed perception after timeout
Configuration defaults are excluded from the artifact hash	The same checkpoint behaves differently across environments	Reconstruct in a clean environment and compare the complete resolved configuration	Freeze the resolved configuration and include it in the deployment manifest
Only successful videos are retained	The monitor events, actions, and system context cannot be replayed	Determine whether a randomly selected failure can be reconstructed as a complete timeline	Save raw observations, actions, monitor events, versions, and human interventions

12. Exercises

1. Diagram every coordinate and unit transformation from a dataset sample to a motor command.
2. Design a unit test that can detect a reversed sign in gripper actions.
3. Explain why matching fixed input/output pairs still cannot guarantee closed-loop reproducibility.
4. Design a watchdog and degraded-mode action for P99 inference timeouts.
5. Decompose a “drop after grasping” failure into diagnostics at the data, policy, control, and monitoring layers.

Papers and Engineering References

Work	Paper or Engineering Fact	Authors’ Explanation	Course Assessment
OpenPI	The open-source repository provides π-series models, training configurations, data transformations, policy services, and examples of robot integration.	The project places model implementations together with data, inference, and adaptation code in a single training stack.	The true unit of reproduction is not the checkpoint, but the complete combination of code, configuration, statistics, data contracts, and service interfaces.
π0	π0 combines a pretrained vision-language model with a flow-matching action expert to generate continuous action chunks.	The authors use general-purpose semantic representations to connect multitask robot data with continuous control.	Deployment audits must simultaneously inspect VLM input processing, the action sampling process, and the actual control semantics of action chunks.
FAST	FAST uses frequency-domain transformations and discretization to compress continuous robot actions, improving the efficiency of autoregressive action tokens.	The authors compress high-frequency actions into token sequences that are easier for language models to model.	The tokenizer is a deployment artifact; mismatches in its version, decoding process, or action statistics directly alter physical commands.
Open X-Embodiment / RT-X	This work unifies data from multiple institutions, robot embodiments, and tasks, and trains cross-embodiment policies.	The authors expand transferable experience by standardizing data formats and embodiment-specific action spaces.	Cross-embodiment benefits depend on schema and adaptation boundaries; matching concatenated tensor shapes must not be mistaken for matching physical semantics.
DROID	DROID provides large-scale real-world robot manipulation data, standardized collection hardware, and task distributions spanning diverse scenarios.	The authors emphasize scalable collection workflows and diversity in real-world environments.	Data scale produces engineering value only when temporal synchronization, device calibration, failure retention, and version lineage are reliable.
LeRobot	LeRobot provides an open-source toolchain for robot datasets, policy training, evaluation, and hardware interfaces.	The project aims to lower the barrier to reproducing robot-learning experiments and integrating hardware.	A toolchain can reduce boilerplate code, but coordinate frames, frequencies, limits, latency, and safety layers must still be explicitly validated for every robot.

13. Cross-References

Read together with Track A. π0, FAST, and action generation determine what the model outputs; G4 asks what physical commands those outputs actually become after sampling, denormalization, and adaptation.

Read together with Tracks E and F. Cross-embodiment data requires a unified schema, but coordinate frames, frequencies, dynamics, and controllers must still be explicitly reconstructed at the robot boundary.

Read together with G1–G3. Statistical evaluation determines whether the evidence is credible, data lineage determines whether results can be reproduced, and runtime safeguards determine how much deployment authority the model receives.