A technical report — seven falsified metrics, seven closed roads

Interactive world models on commodity GPUs: a report on measurements that were wrong

We serve a block-causal latent video diffusion model as a live interactive system on two 24 GB consumer cards. Any still image is an entry point; eight keys drive the camera; a text condition swap changes weather and light mid-session. Nothing is pre-rendered and no 3D assets exist anywhere in the stack. This report is not about that. It is about the seven measurements that were internally consistent, reproducible across seeds, and false — and about the discipline we now run to catch the eighth.

2 × 24 GB CONSUMER GPUs · LIVE 480×832 / 2 SAMPLING STEPS · OFFLINE 1280×704 / 4 STEPS
ALL FIGURES MEASURED, NONE ESTIMATED

Fig. 1 — The system, running

1280×704 native · four sampling steps · no grading · keyboard overlay parsed from capture filenames

↓ DOWNLOAD THE REEL · 63 s · 33 MB

The key overlay is parsed from the capture filenames — it is the input that drove each block, not an annotation added afterwards. A world-model reel that hides its input is indistinguishable from any text-to-video clip. The closing 19 s is one continuous generation, uncut.

What this report claims

An empirical law for self-conditioned rollouts. Motion retention decreases monotonically with attention context length. Six window sizes, shared-prefix control (the first seven blocks are bit-identical across all six), and a pre-registered 3-seed acceptance. The trained default sits on the falling side of the curve; moving it doubles retention with a one-line inference change, no retraining, and improves revisit similarity rather than trading it away. This runs against the field's default instinct that more context is better.
Exact-rollback speculative execution. To evaluate a candidate action by generating a block and undoing it, three things must be restored: the KV span the block overwrites, the index counters, and the global RNG state. Verified bit-exact (max |Δ| = 0). The RNG term is the non-obvious one and it is not optional: without it, two candidate branches differ by their random draw rather than by their action, and the resulting comparison is meaningless while the logs look entirely reasonable.
A validation protocol for selection metrics, with seven counterexamples. No measure is allowed to decide anything until it ranks a set of known-answer samples correctly. Every counterexample below is drawn from our own decision logs, not constructed.
A boundary for text-conditioned control mid-rollout. Distributed appearance properties — weather, time of day, palette — transfer cleanly under a condition swap. Instantiating a new entity does not: the model reads the spatial language of the prompt as a camera command and tears the frame. The boundary is structural, not a matter of prompt quality.

§1

What is being served

The generator is a 5B-parameter causal autoregressive latent video diffusion transformer, step-distilled to two sampling steps for the live path. Generation is block-wise: each block covers three latent frames, decoded to twelve pixel frames. Attention inside a block is causal; across blocks it reads a KV cache holding a rolling window of past frames plus a small pinned prefix (an attention sink). Eight discrete key states enter through an additive adapter applied to the transformer features. Text conditioning can be replaced between blocks.

L_input = t_rem + t_block + Fr ≈ 0.38 + 0.75 + 12/16 ≈ 1.5 s (1)

Input latency decomposes into the remainder of the block in flight, the generation of the next block, and the playout of its F = 12 frames at rate r. Only the middle term is compute. We reported 442 ms for a long time; that was t_block alone, and it is not what a player feels. The dominant term is block granularity, and block size is not reducible: at two latent frames per block the decoder emits zeros, at one it emits NaN — both without raising an error.

§2

Seven metrics that lied

Each entry below was, at the time, our decision criterion. Each produced numbers that were self-consistent, reproducible, and stable across seeds. Each was wrong in the same structural way: the measure correlated with the failure it was supposed to detect.

J_b = ‖ m_b − m_b−1 ‖₂‖ m_b−1 ‖₂ R = (Σ λ_i)²C Σ λ_i² ρ = ⟨J⟩_tail-5⟨J⟩_head-5 (2)

J is inter-block motion: the normalised L2 between consecutive block mean states m_b in latent space. R is the participation ratio of the channel covariance spectrum over C channels — a structural rank proxy. ρ is motion retention across a run. J and R are the two quantities that the rest of this section is about.

Criterion	What it said	What was true
`R` alone	Structure intact, channel rank healthy — long-horizon consistency met.	A completely frozen world scores highest. Surfaced only by a player reporting “I press keys and nothing moves.”
Laplacian sharpness retention	Window 21 is 1.9× sharper than window 12.	The high-frequency energy is motion smear and clipped highlights. At block 39 the pillars in window 21 are streaks.
Per-pixel gradient detail	480p scores above 1280×704 (0.0281 vs 0.0217).	The same edge spread over more pixels lowers the mean. The quantity is not comparable across resolutions.
Hand-built shot objective	Weighted horizon position, tilt, foreground planarity, detail.	Ranked a camera buried in grass highest — grass has the most texture. Rated a level meadow as rolling and a lighthouse frame as having no sky.
Depth gradient	Detail should fall with height if the camera holds the horizon.	Usable frames spanned −0.17…+0.95, unusable −0.69…+0.36. Complete overlap.
Sky-mass drift rate	Baseline sky 0.432→0.074 (collapsed); intervention 0.415→0.452 (held).	Foam and breaking water carry strong texture and were classified as “not sky”. The baseline frame is a clean storm seascape. The entire collapse narrative was an artefact of this measure.
In-run critic scores	Candidate branches 0.956 / 0.903 / −0.250 — a plausible spread.	The same frame scored 0.191 when re-scored offline. Cause not yet isolated; the branch built on these scores is withheld from this report.

The expensive case — two wrong measures composing one coherent false result

The last two entries occurred on the same day and pointed in opposite directions. One said the baseline had collapsed; the other said an intervention had fixed it. Together they produced a clean, self-consistent, cross-checked conclusion that we announced. The truth was that the baseline was the good run, and the intervention had turned an excellent storm seascape into featureless haze.

It was not caught by tighter statistics. It was caught by opening one frame and looking at it.

The rules that came out of it

playable(run) ⇔ ¬collapse(R) ∧ ¬decay(J) (3)

A single criterion can always be satisfied by some other failure mode. Two criteria that fail in orthogonal directions are the minimum. This one rule invalidated every prior “pass” and forced a full re-run.

Second: any measure used for automatic selection must first be validated on a set of samples whose answer is known. We now score candidate frames with an image–text similarity critic, s(x) = Σ_t∈G p_t(x) − Σ_t∈B p_t(x), where p is the softmax over a fixed prompt set split into acceptable (G) and failure (B) descriptions written from real broken frames. Validated before deployment on nine frames whose defect was known by eye: nine of nine ranked correctly, worst acceptable +0.948 against best unacceptable +0.745. It is the first criterion in this project that passed acceptance before it was allowed to decide anything.

§3

Levers that survived

Attention window is the motion knob, and the trained default sits on the wrong side of it

Same seed, same action script, 22 blocks, only the local attention window changed. The first seven blocks are bit-identical across all six configurations — the window is not yet full, so its size cannot matter — which also establishes that the runs are comparable.

Fig. 2 — Inter-block motion J by attention window

one seed · 22 blocks · shared prefix · toggle a window to isolate it · hover for values

How to read it: the curves share their first seven blocks by construction. Where they separate is where the window size starts to bind — and the wider the window, the harder motion falls. Window 12 holds; the trained default at 21 loses 40 % of its motion; 33 and 45 collapse.

window (frames)	ρ (retention)	cliff	R (tail)
12	0.91	none	0.136
15	0.90	none	0.134
18	0.85	b15	0.126
21 (default)	0.59	b11	0.132
33	0.21	b7	0.113
45	0.17	b7	0.113

Acceptance, pre-registered and run at 3 seeds × 40 blocks of sustained forward motion: ρ = 0.83 / 0.78 / 0.94 at window 12 against 0.38 / 0.40 / 0.40 at window 21 — 3 of 3 seeds, roughly 2×. The risk we expected, that a shorter window means less memory and therefore worse behaviour on turning back, did not materialise: revisit similarity improved as well (0.84 against 0.44).

It also falsified our own mechanism

The hypothesis was that decay begins when the window starts evicting; widening it should push the cliff later. Widening killed motion outright, and the cliff position is anti-correlated with the block at which the window fills (window 21 fills at block 7 and breaks at 11; window 18 fills at 6 and breaks at 15). The relation is monotone in context length: the more past the model can attend to, the more evidence it has for holding still.

Ninety per cent of generated frames were being discarded in the browser

Measured inter-arrival times over the socket. Twelve frames land inside ~100 ms, then the channel is silent for 700–980 ms while the next block generates. The client, written to minimise latency, drew only the newest frame and dropped the backlog.

Fig. 3 — Frame inter-arrival, one session

each bar is one frame gap · height is compressed (gap^0.42) so the 7 ms bars stay visible next to the 978 ms ones

0 msred: block boundary (≈700–980 ms)1000 ms

Server-side ≈16 fps; on screen ≈1.2 fps. Pacing playout across the block period restored 12.0 fps at zero compute cost. The canvas was also fixed at 512×320 while the server sent 832×480 — weeks of work moving from 320p to 480p, then 60 % of the pixels thrown away in the browser. Every latency and quality investigation had been run server-side; none had ever measured the client.

Text conditioning: global appearance transfers, object instantiation does not

Controlled swap at block 10, shared seed, identical first ten blocks (per-frame correlation 1.0000 before the swap, −0.11 to −0.13 after).

Instruction	Outcome
“violent thunderstorm, dark sky, heavy rain”	Same geometry, same camera position; sky darkens, rain streaks appear, lighting turns wet and cold. Semantically correct and clean.
“a huge creature blocks the path, towering over the ruins”	No creature. The model read “towering over” as a violent pitch command and tore the frame — the upper third jumped to a different viewpoint.

The distinction is structural. Weather, time of day and palette are distributed properties of the output, and the text condition can shift them. Instantiating a new entity with its own geometry is a compositional operation the model was never trained to perform mid-rollout. The product boundary follows directly: ship live atmosphere control, do not promise creatures.

Speculative blocks: what must be restored for an exact undo

A block-causal generator is not only a renderer. Given a state and an action it predicts the next second of the world, which makes it usable as its own forward model: propose candidate inputs, generate one block for each, score the previews, keep one and undo the rest. The primitive that makes this possible is an exact undo, and getting it right is not obvious.

S = ( K[s : s+w], V[s : s+w], i_g, i_l, f₀, σ_cpu, σ_gpu ) (4)

w = n_fpb · ℓ_seq

The cache is a ring, and a block writes exactly w = n_fpb · ℓ_seq entries from the current local index s. Only that span, the two index counters and the frame cursor are dirtied; copying the whole cache costs about 4.9 GB at 1280×704 across thirty layers and does not fit beside the model on a 24 GB card. σ is the global RNG state, and it is the term that is easy to miss.

Verification is a two-line experiment and it should be run before any planner is built on top: generate a block from noise ε, snapshot, restore, generate again from the same ε, and require the two latents to be identical. With the cache and counters alone the result was max |Δ| = 2.85. The block draws internally from the global generator, so the second call sampled differently. With σ included, max |Δ| = 0.

Why the RNG term is load-bearing

Without it, candidate A and candidate B are not two actions evaluated under one draw; they are two draws. Any ranking between them is then dominated by sampling noise, and nothing in the logs would look wrong — the scores stay in a plausible range and the selected actions stay sensible. This is the same failure shape as the metric table above, one level down in the stack.

The snapshot is held in host memory; the copy costs a second or two per candidate, which is irrelevant for offline rendering and would need a different design for the live path. We report the primitive and its verification here. The planner built on top of it is not reported: its first run was driven by a critic whose in-run scores did not reproduce offline, which is the last row of the metric table, and we do not publish a result whose measuring instrument is still under investigation.

§4

Seven roads that ended

Listed because knowing what does not work is as much an asset as knowing what does, and it is considerably harder to fabricate. Each has measured data behind it.

Distillation objective causes motion decay3 seeds

Controlled comparison against published checkpoints of the same backbone family at the same resolution and block structure, where the only variable is whether a distribution-matching term is stacked on the consistency objective. Slopes −0.039 ± 0.007 without against −0.032 ± 0.018 with. The difference is inside seed noise, and the direction is marginally in favour of the term we suspected. The premise for a costly retrain evaporated.

Gradient flow through the KV write improves long-horizon consistency5 seeds × 220 blocks

Our first implementation was the naive variant that the source paper explicitly reports as running out of memory. Reimplementing the actual method brought the overhead to zero (10.11 GB peak, identical to baseline). The algorithm then showed no significant difference on any of four metrics and was directionally slightly worse. Engineering success is not algorithmic success, and the two should be booked separately.

Longer drift exposure teaches self-correction3 seeds

All three seeds worsened; the worst degraded by 14×. The reason is a category error: exposure to drift is not the same as supervision toward recovery. Conditioned on eleven blocks of accumulated drift, the teacher produces an output that is self-consistent with that drift, so the student learns to continue smoothly into the degradation. The missing half is a clean target, not a longer rollout.

Geometric supervision from photogrammetry fixes action→displacementheld-out scene

The pipeline worked: 113/113 registration at 1.099 px reprojection error, PSNR 24.2, and novel-trajectory revisit similarity 1.000 against 0.284 for the model. The fine-tuned adapter was then worse on both a trained and a held-out scene; the held-out scene retained 8 % of its high-frequency energy by the end of a sustained run. The supervision signal carries the renderer's appearance along with its geometry, and appearance is the one thing the model still does well.

Integer quantisation reduces latencyend-to-end

Isolated GEMMs at the real shapes ran 2.1–3.7× faster. End to end the pipeline was 126 ms slower (540 against 414). Dequantisation passes are memory-bound at these shapes, and a fused kernel path did not exist for this architecture generation. The same reasoning applies to recent calibration-free 4-bit work: the papers report memory and simulated overhead, not wall-clock, and state that a fused kernel is still required. We track it for the memory axis, not for latency.

The action basis can be linearly recalibrated4 α levels

The adapter output is additive, so the action space is linear and key values need not be binary. Adding a constant look-up component to sustained forward motion produced a monotone response at one resolution and a non-monotone one at the resolution where the failure actually occurs. The reason is upstream: this system has no controllable camera. There is no action inverse, a turn key moves roughly 0.3° per block, and free rollouts never revisit a previously occupied pose. Downward drift is a property of the learned continuation distribution, not an orientation offset that a linear combination can cancel.

Block granularity is tunable3 settings × 200 blocks

Two latent frames per block emits all zeros; one emits NaN with a jump statistic of 6.2 × 10⁶. All three settings completed 200 blocks without raising an error and produced attractive latency numbers. Shipping on the latency table alone would have delivered a black screen at twice the frame rate.

The most expensive lesson has nothing to do with statistics

Six consecutive evaluation rounds ran with the attention sink disabled while the served system had it enabled. Multi-seed protocols, pre-registered criteria and mechanistic explanations were all in place — every one of them operating inside the experiment. Nothing checked that the configuration under test was the configuration being shipped. Under the correct configuration the slope is positive on all five seeds and no collapse occurs. Those six rounds were not failures; they were repairs to a problem that did not exist.

§5

What we cannot do

The latency budget is exhausted at the algorithm level. Sparse re-caching breaks consistency; dropping to 256×384 saves 44 ms (a 40 % token reduction buying 11 %, which places the bottleneck in model depth rather than sequence length); asynchronous decode saves 47 ms; integer quantisation is net negative; tensor-parallel splitting loses to its own all-to-all. Reaching a 33 ms frame budget requires roughly an order of magnitude, and the paths are a hardware generation with native low-precision support or a substantially smaller backbone.

At the algorithm level we hold a reproducible phenomenon and a working knob, and no mechanistic account of why self-conditioned contraction occurs, why it is seed-dependent, or why it is monotone in context length. We do not claim one.

§6

Conventions

J is computed on block mean states in latent space and is the only quantity compared across backbones, since R normalises by channel count and latent channel dimension differs between architectures. Retention ρ uses the trailing five and leading five blocks of a run. All rollouts use a fixed seed with a deterministic pipeline; an earlier library call silently overrode the seed argument, so every batch was in fact the same trajectory — all conclusions from that period were discarded and re-run. Reel footage is 1280×704 native at four sampling steps with no grading; the live demo is 480×832 at two steps.

Comparisons in this report are described by capability rather than by name — the open backbone, the source paper behind the KV-write method, the recent calibration-free 4-bit work. We intend to release the evaluation harness, the metric definitions and the configuration behind every figure here; the identities and the code needed to rerun the comparisons come with that release.

All figures measured on two 24 GB consumer GPUs, not estimated. Falsified conclusions are retained in the text rather than deleted, because they are the part of this system that is hardest to rebuild.