Keyboard shortcuts

Press or to navigate between chapters

Press S or / to search in the book

Press ? to show this help

Press Esc to hide this help

Accuracy by field: Molmo2’s aux head reads the scene far better — visible slots 0.32 → 0.82

2026-08-09 01:0xZ. The registered fields panel (pre-reg, owner ask 10:08Z 08-08) closed rc=0 at 00:49Z on the box: the 60k Molmo2 endpoint’s chained-eval command re-run byte-identical with the narrated pass riding (bijou@60000+fields decodes every trained aux field, then its actions). ~3.1 GPU-h of the 6 gate (00:03:30→00:49:43Z × 4 GPUs). All reads record-only per the pre-reg; the launcher printed them mechanically at rc=0.

Read 3 first (validity oracle): green. The fields run’s base bijou@60000 arm reproduces the chained eval’s chunk MAE to full JSON precision (5.86022663460471) — same instrument, same 4-rank sharding, so the narrated numbers below sit on a verified base.

Read 1 — the accuracy-by-field table (the owner deliverable)

fieldmetricAR-100k anchorMolmo2 60kframes (M2)
holdingaccuracy0.8070.8978,987
progressMAE (lower better)0.0620.0598,987
eventpresence accuracy0.8780.8808,987
visibleslot-set accuracy0.3190.8198,981

The headline is visible: 0.319 → 0.819 (+0.50) on the strictest metric in the table (exact set-equality of the parsed object slots). Two details make it hard to dismiss:

  • No parse-selection excuse. AR-100k only produced scoreable visible lines on 8,260 frames; Molmo2 parsed 8,981 — it scored more frames and got the sets right 2.6× as often.
  • It rhymes with the trunks’ pedigrees. Molmo2’s VLM was trained with dense pointing/grounding supervision; “which objects are visible” is the most vision-grounded field we narrate. The action-adjacent fields (event presence, progress) barely move — the gap is specifically in scene reading, not narration fluency.

Holding also steps up meaningfully (0.807 → 0.897, n=8,987); progress improves at the label-noise floor. Standing caveats travel with all of it: the judge labels are weak (~80% inter-judge holding agreement, ±15% progress MAE), so the flat fields may simply be at their label ceilings, and columns are not comparable to each other. Anchor robustness: the AR-100k anchor is the pre-registered panel_k4l2 table; the same-panel panel_curated_v0 values agree to ≤0.007 on every field, so nothing hangs on the panel choice.

Read 2 — does narration help actions? (still no)

Paired on the same frames, +fields costs chunk MAE on Molmo2: pooled 5.9467 vs 5.8602 (+0.0865; paired mean Δ +0.083, win rate 44%) vs the AR-100k anchor +0.054. Slightly worse than the Gemma trunk, same sign. The by-outcome slice adds a wrinkle we’ve seen before: the narration cost concentrates on failure-labeled frames (+0.50 there vs +0.09 on success) — conditioning on decoded text hurts most exactly where the model’s own scene reading is likely wrong.

So the fields head is a diagnostic asset, not a decode-time win: consistent with the whole #6/fieldcond thread (the aux head is load-bearing for representation quality at training time; making the model read its own narration at decode time never pays).

Where this lands

  • The owner table now exists for both trunks; this was the last pending input for the consolidated field-conditioning + subgoal meta-report (fieldcond-subgoal-meta-report), which can now compose.
  • The visible-field jump is the first field-level evidence for the vision-side story in the Molmo2 bet (#17): the trunk’s grounding shows up exactly where grounding is measured. It also sharpens the scorer-escalation map from rung (b′): any future learned subgoal/candidate scorer on Molmo2 gets a much better scene reader for free.
  • Nothing here gates the attach chain: the K-smoke ladder re-run at the 60k warm start remains the next box item after perf-pass1.

Artifacts: reports/eval__fontaine_molmo2_ar_60k_ddp4__step_060000__panel_curated_v0_k4l2_fields.{npz,json,html} on the box (json mirrored locally for the chart); run log + mechanized read block in the unit journal (fontaine-fields-panel-60k, rc=0).