← All writing

Batch-Invariant VLAs, Vol. 2: When Numerical Differences Change Robot Behavior

From numerical batch dependence to closed-loop robot behavior, and the operators that make inference consistent.

Initial draft. Evidence snapshot: operator library b734818 and behavioral experiments 7c0caec. Repository links below are pinned to those snapshots; measurements retain their own implementation and environment revisions. Behavioral claims apply to the campaign's frozen operator stack; newer library features have separate numerical evidence.

1. From Numerical Differences to Behavioral Differences

In Vol. 1, I traced batch-dependent numerical differences through π0: first a SigLIP patch projection, then attention multiplication that bypassed the operator I thought I had replaced. The remaining question was more interesting than another nonzero tensor diff: does any of this change what the robot actually does?

I tested that with paired closed-loop SIMPLER rollouts using the pretrained Fractal-Beta policy. For each task initialization, I ran four conditions: native singleton, native dynamic batching, invariant singleton, and invariant dynamic batching. Dynamic inference cycles through batch sizes 1, 2, 4, and 8, moves the target request between positions, and adds unrelated replay requests. Policy noise is indexed by task, initialization, and policy call, rather than drawn from a batch-shaped random stream. The evaluator implementation records those identities.

The drawer tasks give the cleanest answer:

Task, 50 paired initializations eachNative singleton successNative dynamic successNative outcome disagreementsInvariant singleton successInvariant dynamic successInvariant outcome disagreements
Open drawer50%54%6/5044%44%0/50
Close drawer72%74%3/5076%76%0/50

“Disagreement” means the same initialization succeeded under one batching condition and failed under the other. It is not the difference between average success rates: successful and unsuccessful flips can cancel. These values come from the retained results table and paired analysis.

Both drawer tasks have byte-identical returned reset records across all four conditions in all 50 initialization blocks. The initial robot state also matches exactly. That makes these a controlled demonstration that changing inference batching can change the subsequent closed-loop outcome, with the checkpoint, initial conditions, and request-indexed noise held fixed. Native paired trajectories reached maximum end-effector position differences of approximately 0.234 m for opening and 0.231 m for closing. Invariant singleton/dynamic position differences were zero; quaternion-distance calculations retained tiny numerical residuals. See the experiment report and reset audit.

This is evidence for behavioral consistency, not a claim of improved average task success. Open-drawer success actually decreases from native singleton's 50% to invariant singleton's 44%. The native dynamic-minus-singleton success differences have paired 95% bootstrap intervals of −6 to +14 percentage points for opening and −4 to +10 for closing. Fifty pairs do not establish universal behavioral invariance.

The complete campaign contains 800 episodes across four tasks. Pick-can and move-near have small post-reset object-pose differences despite matching seeds, so their outcomes retain a physics-reset confound. In particular, invariant inference still shows disagreements there. I do not use those tasks to claim that batching alone caused their behavioral changes.

Figure 1 — TODO: Paired rollout videos. Render native singleton/dynamic and invariant singleton/dynamic drawer rollouts side by side. Open-drawer initialization 8 is a recorded native success-to-failure pair; close-drawer initialization 7 is another. Use the retained episode archive, preserve request-noise identity, and label conditions. No rollout videos are committed in this experiment snapshot.

Figure 2 — TODO: Behavioral disagreements and success rates. Plot the table above as separate disagreement and success-rate panels. Use the stored paired bootstrap intervals; do not treat agreement as a success-rate improvement.

2. How Widespread Is This Problem?

I expanded the investigation beyond π0 to more than 20 robotics model families: VLAs, diffusion policies, and models jointly generating video and actions. The cross-model plan enumerates 23 additional runnable families and 29 distinct execution configurations, excluding the completed π0 reference campaign. This is architectural coverage, not an unbiased estimate of prevalence across all robotics models.

The numerical harness fixes the target observation, preprocessing, weights, precision, and randomness; verifies singleton repeatability; then changes batch size and companions. Duplicate companions isolate shape dependence. Unrelated companions test composition dependence. The usual sweep includes awkward sizes around thresholds—3, 7, 15, 17, 31, and 33—not just powers of two.

Representative completed numerical investigations are below. “Exact” means agreement with the repaired singleton under the specified tests, not agreement with native singleton or proof of task success.

Model / tested pathFirst divergent operation; later cause where relevantResult with library and documented adapter controls
GR00T N1.7Action-encoder bmm; then attention-key addmmComplete action exact through B=64
SmolVLAModality-connector mm; then state-projection addmmComplete action exact through B=64
X-VLAFlorence downsample convolution; then projector mmComplete action exact through B=64
Cosmos 3 NanoProjection differences; then dense/packed attention switchActions and latents exact through B=64; decoded video tested at B=2
Cosmos 3 EdgeDense/packed attention; later decoder linalg_vector_normActions and latents exact through B=64; decoded video tested at B=2
RDT-1BImage cross-attention SDPA; later language-adapter addmm at B=64Complete action exact through B=64 on the stated numerical fixtures
UVACLIP MLP addmm; then VAE downsample convolutionAction exact through B=17 in both compositions and duplicate B=31; larger tested cases OOM. Joint decoded video/action exact at B=2
LingBot-VLA 2.0Released atomic MoE is not singleton-repeatable; controlled baseline first diverges at state addmmComplete FP32 action exact through B=64 with deterministic MoE fallback and addmm/mm/bmm repairs

These results belong to pinned checkpoints, fixtures, and software stacks. Some fixtures are explicitly off-domain, including RDT's real-image numerical inputs. They do not measure corresponding task success.

The broader evaluation campaign is unfinished. Its plan explicitly retracts an earlier “measurement core complete” claim: reduced pilot schedules do not satisfy the independent full-coverage audit. No path had passed that full gate in this snapshot. Completed numerical tests and individual timing sub-audits must remain separate from pending diagnostic, held-out, serving, and paired behavioral campaigns.

Figure 3 — TODO: Cross-model heatmap. Expand the representative table using committed research/*/fixed.json results. Distinguish exact, failing, OOM, untested, and incomplete full campaigns; do not color them all as equivalent passes.

3. Different Models, Common Numerical Problems

The model architectures differ substantially. The numerical failure modes repeat.

GEMM and BMM: the reduction schedule is part of the result

In GR00T, the first divergent operation is the category-conditioned action projection:

[B, 40, 3072] @ [B, 3072, 1536] -> [B, 40, 1536]

At B=1, cuBLASLt selects a split-K kernel and a separate reduction. At B=2, it selects a non-split-K kernel. The target inputs are exact, but two BF16 output elements differ, with maximum absolute difference 0.00048828125. Repairing BMM reveals another batch-dependent projection using addmm. The profile and tensor hashes make this a specific dispatch diagnosis rather than an inference from final actions.

SmolVLA adds a useful warning: different kernel names are not required. Its modality connector uses the same named split-K kernel family, but changing flattened M from 64 to 128 changes launch geometry and reduction grouping. The subsequent state projection switches from GEMV at B=1 to a small-N GEMM at B=2. Both boundaries require repair.

Convolution: this is not just a patch-embedding problem

X-VLA first diverges at a Florence 3×3 downsample convolution. Two camera views make policy B=1 and B=2 become convolution batches of two and four. cuDNN switches between 64×64 segment-K-on and 128×128 segment-K-off schedules. The first target output differs in 32,703 FP32 elements, with maximum difference about 4.24e-5. A regular convolution can therefore be batch-dependent too; the behavior observed for particular small convolutions in Vol. 1 was not a general guarantee. Source.

Attention: exact Q, K, and V are not enough

Cosmos Nano uses dense cuDNN attention for a singleton and packed variable-length CUTLASS attention for multiple samples. After projection repairs make Q/K/V exact, the first causal attention result still differs in 14 BF16 elements. The branch repeats in cached text-KV attention during later flow evaluations. The repair aligns both paths to the same logical sequence geometry and fixed-reduction attention implementation.

Normalization: exact latents are not the end of the pipeline

Cosmos Edge initially produced exact actions and future-video latents but still changed 15,927 decoded RGB values at B=2. After 687 matching leaf-module calls, the first mismatch appeared in the Wan decoder's channel-wise RMS normalization. F.normalize reached aten::linalg_vector_norm, whose reduction depended on batch shape. A fixed reduction tree per output vector repaired the decoded video. Source.

Atomics: first make repeated execution repeatable

LingBot's released MoE kernel uses relaxed atomic additions for route packing and route accumulation. Two identical B=1 calls already differ in all 2,750 action values. That is a separate problem from changing B.

The deterministic fallback evaluates experts with fixed-schedule BMMs, gathers selected routes, and accumulates the top-k contributions in a fixed order. It preserves the routing semantics, not bitwise identity to a released kernel that does not equal itself. Only after controlling that non-repeatability can I isolate the remaining batch-dependent matrix operations. Investigation.

This is why deterministic execution and batch-invariant execution are different contracts. A runtime can consistently choose schedule A for B=1 and schedule B for B=2. Both executions are repeatable; their answers can still differ. The π0 campaign's native_deterministic mode still fails all 18 arrangements in the separate BF16 ablation. Source.

Figure 4 — TODO: First-divergence kernel schedules. Visualize GR00T's split-K-plus-reduction versus non-split-K execution using the recorded profile. Show identical target operands and the first differing output elements; avoid inventing a specific internal summation tree beyond what the profiler establishes.

4. Building robotics-batch-invariant-ops

Vol. 1 ended with a correctness-first workaround: force individual matrix multiplications through the invariant mm path. The reusable operator library extends the original Thinking Machines approach so models can retain their ordinary PyTorch layers while changing the arithmetic underneath them.

The integration uses torch.library.Library("aten", "IMPL") to register accelerator implementations for mm, addmm, linear, bmm, convolution, SDPA, softmax, and selected reductions. The context manager installs those overrides and restores the previous mode afterward. The implementation is the actual API boundary, not a promise to intercept every custom fused kernel.

The persistent matrix kernels use dtype-specific tile configurations, FP32 accumulation, and a fixed traversal of K. Persistent BMM assigns each matrix its own logical grid and never accumulates across batch members. It executes independent arithmetic together, instead of paying for a Python loop of separate launches. Adding requests can change how much work is scheduled without changing an existing output's reduction schedule.

Convolution lowers patches into grouped matrices and uses that same BMM path. Attention combines fixed-schedule QK and probability-value products with row-wise softmax. Packed attention preserves each sequence's boundaries and groups equal geometries into bounded launches; the implementation bounds each launch's score matrix to 512 MiB rather than materializing the whole pack at once. Normalization needs its own fixed per-vector reductions.

After installing the package in a CUDA/PyTorch/Triton environment, this minimal example exercises the public context and nn.Linear path:

import torch
from batch_invariant_ops import set_batch_invariant_mode

torch.manual_seed(0)
projection = torch.nn.Linear(64, 32).cuda().eval()
inputs = torch.randn(4, 64, device="cuda")

with torch.inference_mode(), set_batch_invariant_mode():
    alone = projection(inputs[:1])
    batched = projection(inputs)[:1]

assert torch.equal(alone, batched)

The context and exported operators are documented in the package exports. This small example checks an operator path; a complete robot policy still needs explicit noise, controlled preprocessing and caches, and end-to-end comparison.

Some integrations need adapter work. Cosmos has custom grouped projections outside the patched MM/BMM dispatcher, so its adapter keeps those request-local and routes both ordinary and cached attention through the reusable packed operator. Its recorded numerical tests use eager execution. LingBot explicitly installs the deterministic MoE fallback. Those controls are part of the result, not details to omit from a “just enable the library” claim.

My debugging loop remains: find the first unequal tensor, replay its operator with identical target operands, inspect dispatch, replace the arithmetic, and repeat until the complete output is invariant. The reusable library makes each repair available to the next model.

Figure 5 — TODO: Library architecture diagram. Show model/adaptor calls → ATen dispatcher → invariant matrix/convolution/attention/reduction operators → fixed Triton schedules, with explicit side paths for Cosmos packed attention and LingBot MoE. Mark external fused kernels as requiring integration.

5. Common Pitfalls and Lessons Learned

Replacing mm does not replace every matrix multiplication. Higher-rank matmul can reach bmm; biased projections can reach addmm; linear is another dispatch boundary. GR00T requires BMM and addmm repairs. SmolVLA needs both its bias-free connector and biased state projection fixed. Follow profiler events, not Python spelling.

Accuracy is not invariance. Two kernels can both approximate the real-valued operation well while rounding differently. Conversely, an invariant replacement can consistently produce different bits from native singleton inference. Across the π0 campaign's 2,400 held-out request/noise pairs, none of the invariant singleton outputs was bit-identical to native singleton; the maximum absolute difference was about 2.93e-4. Singleton fidelity and batch invariance are separate measurements. Source.

Repairing the first divergence exposes the next one. X-VLA's repaired convolution leaves a projector mismatch. GR00T's repaired BMM leaves an addmm mismatch. The pretrained FP32 π0 investigation also begins at proprio_encoder, not necessarily the SigLIP boundary from the earlier configuration. Its explicit-per-matrix attention ablation still leaves a locally batch-sensitive output projection. Exact inputs to that projection do not force exact outputs. The local replay records both facts.

A seed is not a per-request randomness contract. A batch-shaped random draw can change a target's noise even after resetting the same global seed. UVA explicitly carries posterior noise, initial noise, and step noise. X-VLA's released inference path accepts a noise argument but does not consume it; its adapter injects controlled noise into the executed body. LingBot mutates supplied noise in place, so the adapter clones it. These controls are documented in their investigations.

Preprocessing and state can change the experiment. Broader unrelated-task replay exposed dynamic prompt-padding dependence in SmolVLA and GR00T; their repaired paths use fixed token extents of 48 and 156, respectively. An optimized InternVLA CUDA-graph cache failed on B1→B2→B1, requiring per-call recapture whose cost must be measured. LingBot caches shape metadata from the first request. These are execution-configuration problems alongside numerical kernels. Campaign notes.

Measure the final artifact, then check the environment. Cosmos Edge's exact latent did not imply exact decoded RGB. Likewise, a matching reset seed did not imply identical object poses in SIMPLER. The rejected X-VLA paired campaign had initialization mismatches caused by evaluator auto-reset behavior; its results cannot be promoted into paired evidence. A plausible success-rate number is not enough.

In the completed π0 held-out replay, native inference differs in all 50,400 non-singleton arrangements; the invariant path differs in zero. Those arrangements reuse 2,400 request/noise pairs, rather than representing 50,400 independent scenes. Across comparisons, native maximum action-state error rises from approximately 1.89e-4 after the first flow update to 0.14864 after the tenth. These are aggregate maxima, not a single trajectory's growth curve. Report.

Existing held-out flow-step error figure

Figure 6 — Existing experiment figure: median and p95 of per-arrangement maximum action-state error across flow steps. The invariant series is zero. Generator and summary. TODO: Finalize publication styling while preserving the distinction between these statistics and the single-arrangement diagnostic figure.

6. Performance and Trade-offs

The invariant path is slower in the measured complete-policy benchmark. On the recorded H100 NVL, with PyTorch 2.8.0, CUDA 12.8, Triton 3.4.0, and FP32 Fractal-Beta inference, processing four logical requests gives:

Execution modeMedian time for four requestsRequests/s
Native dynamic, physical B=4371.64 ms10.76
Invariant dynamic, physical B=4958.73 ms4.17
Four native singleton calls, serial944.00 ms4.24
Native fixed-size padding, physical B=8528.89 ms7.56

These are complete-policy timings on fixed generated inputs, not closed-loop controller latencies or a cross-task model benchmark. The runner uses five sessions, each with 20 warm-ups and 200 timed calls. The summary reports invariant dynamic inference at approximately 2.58× native latency. The paired simulator campaign runs separately on an RTX PRO 6000 Blackwell; the two environments must not be conflated.

Persistent BMM nevertheless pays down much of Vol. 1's launch overhead. At request B=8, the attention PV benchmark multiplies [64,4,281] by [64,281,256]. The explicit invariant loop takes 2.5015 ms and 65 profiled device launches; persistent invariant BMM takes 0.07234 ms and one launch—about 34.58× faster. Native BMM still takes only 0.04102 ms. The improvement is over the explicit invariant workaround, not a speedup over native. Kernel measurements.

Other shapes are substantially more expensive. SmolVLA's connector replacement is 22.68× stock latency at B=1, while the optimized X-VLA convolution remains 14.12× stock at B=1. LingBot's dense deterministic MoE fallback evaluates every expert and costs 8.22–14.04× the released atomic implementation across B=1/2/8. These are isolated operator measurements from their respective environments, not whole-model slowdowns.

The reasons are visible in the implementation: conservative fixed tiles waste work on small matrices; im2col creates temporaries; explicit attention materializes scores; FP32 IEEE matrix arithmetic restricts faster alternatives; dense MoE trades sparse efficiency for a fixed schedule. Fewer kernel launches alone do not guarantee lower total latency.

Fixed-size padding and serial singleton execution are useful alternatives. Padding keeps physical batch geometry fixed but wastes capacity and needs controlled language extents and a validated reference for that execution shape. Serial singleton removes cross-request batching but sacrifices shared execution. In the separate queue experiment, both alternatives and invariant dynamic execution satisfy their tested numerical contracts at all five offered loads; native dynamic satisfies none. Queue latency and throughput still depend strongly on load.

Optimization opportunities include invariant shape-specific tiles, less temporary storage, fused attention with a fixed reduction schedule, and deterministic sparse MoE routing. Each optimization needs renewed numerical qualification; changing the schedule is exactly what started this investigation.

Existing serving trade-off figure

Figure 7 — Existing arrival-to-completion p95/throughput serving measurements. TODO: Add a separate complete-policy latency panel from the table above; retain offered-load labels and do not present queue latency as kernel time.

7. Implications and What's Next

A shared robot inference service should be able to batch requests without changing the numerical policy assigned to an individual robot. Otherwise queue occupancy, companion prompts, and batching partitions become hidden inputs to the controller. The drawer rollouts demonstrate that this can reach task outcomes; the cross-model investigations show several ways it can enter the computation.

Distributed and asynchronous robot RL make the question more consequential. If rollout workers and a learner evaluate nominally identical policies with different batching, they may not execute the same numerical function. That could affect replay, debugging, and attribution of training changes. This is a hypothesis about training systems, not a result established by these inference experiments.

Sampler–learner numerical parity is a next research question: compare matching requests under actual rollout and learner execution configurations, including precision, caches, compilation, and operator paths. Invariant inference alone does not establish equality with training-mode computation, gradients, or distributed reductions.

The immediate work is to finish the independently audited cross-model campaigns, expand paired behavioral evidence with exact resets, and optimize expensive operators while preserving their arithmetic contract. The present guarantee remains tied to tested checkpoints and hardware/software revisions, not universal bitwise equality across devices or framework versions.

TODO — Pending data: Add full cross-model campaign results and sampler–learner measurements when independently validated. No completed result for either is asserted in this draft.

A robot's output should not depend on unrelated requests sharing its batch.