Leaderboard

Evergreen (renamed from “Ledger”, owner steering 2026-08-07 10:04Z): the best banked score for every model family × decode config, in one place, updated as endpoints land. Numbers only compare within one frame set; frozen panels are immutable; flow results state their noise draws; deployment vs unconstrained never mix (docs/architecture.md §7).

The scoreboard — community panel v1, deployment class

All rows: bijou.eval on community_curated_v0 holdout, plans/holdout_curated_v0_k4l2.json — 25,800 frames scored, 17,204 core frames pooled, identical rows for every entry. Sorted by panel MAE. Breakthrough bars (charter §2): ☆ ≤ 5.0 · ☆☆ ≤ 4.5 or first_mae ≤ 1.6 · ☆☆☆ mainline adoption.

Row 8 added 2026-08-09 (60k continuation read): +20k fresh-data steps on the Molmo2 trunk, paired Δ(60k−40k) −0.1388 [CI95 −0.194, −0.090] on 17,204 core frames — IMPROVED; the AR-100k greedy bar (row 6) is NOT yet passed (+0.058 chunk, cross-trunk unpaired; first_mae 2.0719 is already below the 100k’s 2.1431). Decode cost columns inherited from the 40k rows (same architecture and decode config, ⁴).

Row 2 re-seated 2026-08-08 (noise-ladder seating read, paired per-frame): the mean-of-top-10-tickets ensemble replaces the random mean-of-10 (5.3645/1.4242) it was measured against — paired Δ −0.174 [CI95 −0.196, −0.152] on 17,204 core frames, clustered CI agrees (analysis). ⁵ cost cells inherited from the random mean-of-10 row — identical decode config (10 draws × Heun-30), only the noise source differs. The ☆☆ first-mae arm (≤ 1.6) is crossed — 1.3831 is the best first-step accuracy banked (student: 1.5927). The ☆ chunk bar (≤ 5.0) is open: current best 5.1847, gap 0.18 (was 0.37 before the re-seating). The student-vs-teacher compute story (rows 1 vs 2): the 30×-cheaper student now trails the teacher’s best ensemble by 0.18 — the distillation target moved.

Row 5 landed 2026-08-07 (the draws10_t1 boundary, all three pre-registered expectations met): AR mean-of-10 buys −0.145 [CI95 −0.182, −0.109] — real but ~9× smaller than the flow families’ draws gain, the pre-registered mean-collapse shape (greedy AR decode already sits near the predictive mean). Row 9 landed 2026-08-08 (endpoint chained eval; frozen Read 1 = BEATS its own-topology E2B control 7.7966 by paired −1.717 [CI −1.80, −1.63] → phase-2 flow-trunk candidate): the Molmo2 trunk at 40k sits 0.21 behind AR-100k’s greedy at 2.5× fewer steps. Row 7 landed 2026-08-08 (golden-ticket screen R2 = REAL): a single sha-pinned noise vector (ticket 33, searched over a 64-candidate bank on probe rows, judged on 14,746 complement rows: paired −0.924 [CI −0.985, −0.866] vs stable-key) captures ~75% of the mean-of-10 gain at 1/10th the draws; keying ticket, effect directional not norm (norm rank 29/64). ³ cost cells inherited from the stable-key single-draw row — identical decode config, only the noise source differs. Row 10 landed 2026-08-08 (#19 molmo2 draws arm, all pre-registered expectations met): molmo2 mean-of-10 buys Δ_AR −0.154 [CI95 −0.195, −0.113] — the same mean-collapse shape as AR-100k’s −0.145, replicated on a second AR trunk; no overtake of the flow draws band (5.365). ⁴ molmo2 cost cells measured 2026-08-08 on the box H100 (same harness, flags byte-matched to the panel stems, record-only extension of the pre-reg’s registered set — other rows were measured on the local 1×H100; same GPU model, cross-machine deltas are directional): analysis__leaderboard_decode_microbench_molmo2.json. The mtime caveat on row 9 is retired. The T-sensitivity rungs (T ∈ {0.5, 0.7, 1.3}) are record-only by pre-registration and never enter the leaderboard — dT diagnostic only.

Reading the compute column

¹ Both ⏱ columns are the same-harness micro-benchmark (pre-reg, results in the main-sync post, data reports/analysis__leaderboard_decode_microbench*.json), measured 2026-08-07 on the local 1×H100 on the post-merge tree (batched noise-draw ensembling in): identical frames per mode across every row, decode flags byte-matched to the banked panel stems. eval ms/f = batched-eval throughput (b32/w20, N=320) — the cost of running the panel. b=1 ms = single-stream latency (b1/w4, N=50) — the deployment-facing read (#16 hook). evals/frame stays as the structural column (draws × solver evals; AR decodes are token-serial — no eval count captures them, hence “(serial)”). These replace the earlier mtime-derived ≈ estimates and the two heterogeneous ⏱ wall-clocks; cross-row deltas are now apples-to-apples. AR singles were measured pre-merge (the merge does not touch the AR decode path); the flow draws=1 pre/post control pairs reproduce to ≤0.3%.

The structural story, post-merge (batched draws): mean-of-N now costs single-draw latency — student mean-of-10 111 ms vs single 100 ms; teacher mean-of-10 1,245 ms vs single 1,234 ms (was 11,284 sequential: 9.1×). The student’s 10 draws cost 11% extra latency for a −0.24 panel gain (rows 1 vs 4); the AR family pays serially either way (2.2 s greedy → 8.0 s draws-10 single-stream) for a −0.145 gain — mean-of-draws is a flow-family superpower, not a universal one (row 5’s readout).

² The student’s mean-of-10 first_mae (1.5927) crosses the ☆☆ first-mae bar (≤ 1.6); the teacher’s top-10-ticket 1.3831 is the best first-step accuracy banked (the random mean-of-10’s 1.4242 held this title until the 2026-08-08 re-seating).

The instrument

Headline metric: community panel MAE — bijou.eval --sample-plan plans/holdout_curated_v0_k4l2.json on community_curated_v0, --episodes holdout --holdout-episodes 0.1 --split-seed 0 --fps 30 --camera-counts 1 2, deployment-class decoding stated per row. Deterministic per checkpoint (flow rows: stable noise keying, draws stated).

Confirmation: the sealed panel (plans/holdout_curated_v0_k4l2_sealed.json, plan seed 1) — scored only on claimed bests, at most ~weekly.

Own-instrument verification (charter §10.5): DONE — the AR-100k baseline re-scored locally reproduces 5.8026/2.1431 exactly (banked npz + report in reports/; the draws10_t1_results.py and selection_ceiling_results.py oracles re-derive both numbers from the raw npz on every run). Sealed-panel anchors land with the integrity kit.

Critical-frame robustness (2026-08-07, pre-reg + results): every published ranking holds when the panel is re-pooled over task-critical frames only (judge-labeled subgoal boundaries, holding transitions, events — the CI-MSE 2606.29898 concern, tested with our own labels at zero GPU cost). All 10 pairwise gaps keep their sign with CI95 excluding 0, and the model-vs-state-copy separation widens on critical frames — the board’s ordering is not an easy-frame artifact. Offline-vs-rollout remains open until a rig benchmark exists (#16).

Panel-row integrity (2026-08-09, continuity screen + wrap census): 8 panel episodes (≤ 32 of 25,800 rows) come from the two structurally non-conforming repos (kevin510 ±180° wrap seam, willnorris raw encoder counts). Pooled numbers are robust — the census measured the whole class at +0.072 and the bounded worst case is ~0.05 — but per-repo or max-row diagnostics touching those two repos are not trustworthy; standing caveat wherever k4l2 anchors are sliced fine.

Anchors (mainline-measured, inherited 2026-08-05)

Own-topology results — deployment class

Frame set: k4l2 community panel v1, greedy AR, 17,204 core frames. Topology caveat (§2): eff-10 1×H100-slice arms — cross-topology vs the mainline anchors is directional only; paired reads within the batch are clean.

Own-topology results — unconstrained class

(empty — no runs yet)