Keyboard shortcuts

Press ← or → to navigate between chapters

Press S or / to search in the book

Press ? to show this help

Press Esc to hide this help

Pre-registration: E4B trunk-swap screen — matched-params AR-100k on 4×H100

2026-08-05 ~22:4xZ. Posted before launch (charter §4). Owner pick 21:57Z: the E4B screen is the next pre-reg; the freed 4×H100 (192.222.55.210) goes here once tonight’s box-batch arms + reads land. Launch is gated on the pre-launch checklist below and on a short finalization amendment that fills the E5 seed-noise constant from tonight’s replicate reads.

Question

Attribution front, question 1 (owner 21:43Z: “bigger trunk?”): does trunk scale — Gemma 4 E2B → E4B (2.3B → 4.5B effective; text decoder 35×1536 → 42×2560, ~2.2× text params) — move comm-holdout MAE under the identical AR recipe? This is survey rank 1 in ideas #17 (the zero-port-cost in-family rung): if scale at matched compute-class doesn’t pay, the front moves to grounding (#11) and the video-trained trunks (Molmo2-4B, survey rank 2).

Design — one run, matched parameters

The mainline E2B reference is bijou_arb_rcond_100k_ddp4 (docs/architecture.md § experiment reports; panel chunk_mae 5.8026 @100k, the banked anchor). The screen re-runs its verbatim recipe with exactly one science change:

  • --backbone google/gemma-4-e4b-it (E2B default → E4B).

Verified from code this session: the ar_backbone path is fully config-driven — full-depth trunk (BackboneDepth.FULL), FAST block tail-anchored at vocab_size − vocab_total (same 262,144 vocab ⇒ same block base), no expert/stream surface involved. bijou/gemma4/ implements E4B (e4b_config, parity harness covers it).

Everything else matched to the reference: --decoder ar_backbone, fast_tokenizer_v2, aux fields subgoal/holding/progress/event/visible @ weight 0.5, --aux-dropout 0.0 --field-dropout 0.1, conditioning subgoal/outcome/smoothness --condition-dropout 0.1 --subgoal-dropout 0.5, --instruction-augment 0.5 --camera-kind-dropout 0.1, --decoder-lr 1e-4 --backbone-text-lr 2e-5 --grad-clip 100, 100k steps, warmup 1k, batch 12/GPU × DDP4 = effective 48, workers 20, prefetch 4, --eval-samples 256 --eval-every 500 --save-every 2500 --log-every 20, --seed 0 --split-seed 0 --holdout-episodes 0.1, corpus community_curated_v0 @ --fps 30 --camera-counts 1 2 (box copy, frozen; cleanup boundary still in force). Run name fontaine_arb_rcond_e4b_100k_ddp4, project fontaine.

The two seams, stated up front

  1. The E2B reference’s own batch seam. The reference ran eff-48 to 20k, OOM’d (77.5 GiB at B12), and finished at B10/eff-40 — so E4B held at eff-48 throughout sees ~+15% samples by 100k (4.80M vs 4.16M). Owner pick (21:57Z, “owner remembered 10” — the recipe’s launch command says 12): match the recipe, eff-48, never change batch semantics mid-run. Consequence for reading results: gates at ≤20k are seam-free matched; post-20k the seam favors E4B, so a kill (“E4B not ahead despite ≥ samples”) is conservative-valid, while an endpoint adopt carries the +15% caveat in the writeup.
  2. Probe corpus seam (small). The mainline curve was measured on the owner’s corpus copy (42,853 episodes, stamp 9b796de); the box copy selects 42,872 (Δ19 episodes, 0.04%, already E1-verified identical across tonight’s four box arms). The 256-frame in-run probe may therefore differ slightly in composition from the mainline probe. Mitigation: probe deltas are read against a ±0.5 noise floor (observed inter-eval scatter ±0.3 late, ±0.5–0.8 early, plus this seam); the panel (frozen plan panel_curated_v0_k4l2, 25.8k frames — the exact plan the 5.8026 anchor was scored on, from this box copy, by the owner, today) is seam-free and is the decision instrument wherever a checkpoint exists.

Memory reality and the pre-registered fallback ladder

E4B at B12/GPU will likely OOM: the E2B reference peaked 77.5/79.2 GiB at B12, and E4B’s text trunk is ~2.2× the parameters (trained at 2e-5 ⇒ optimizer state scales with it). bijou.train today has no gradient accumulation (single loss.backward() per step). Pre-registered ladder, decided at the pre-launch memory smoke — never mid-run (the reference’s batch roulette is the lesson):

  1. B12 direct if the smoke fits with ≥3 GiB headroom.
  2. Else chunked backward at loader batch 12: the per-rank batch stays 12 (identical per-step sample composition), forward/backward split into equal chunks (2×6 → 3×4 → 4×3, first that fits) with gradient averaging — mathematically the B12 gradient up to fp reduction order (equal chunks ⇒ mean of chunk-means = batch mean), DDP no_sync on all but the last chunk. This is a small bijou.train change to be landed before launch with: the three CPU loss oracles bit-exact with chunking OFF, and a chunked-vs-unchunked gradient-equivalence test (tolerance-level, CPU) with chunking ON. Effective batch 48 and every LR/schedule constant are invariant at every rung; the chosen rung is recorded in the finalization amendment.
  3. If even 4×3 doesn’t fit: do not launch; post the finding (E4B doesn’t fit this recipe on 80 GB — itself an attribution datum) and take the follow-on decision to the owner.

Expectations & gates

  • E1 startup (hard gate): selection line 878 datasets / 42,872 episodes / dims 6/6 (box copy, identical to tonight’s four arms); model line shows the E4B geometry (42 layers / hidden 2560) and decoder head sized off it; block base = 262,144 − vocab_total, same value as E2B. Any selection deviation ⇒ abort before step 1.
  • E2 first poll (util rule): record s/step and peak VRAM. Expected 0.9–1.1 s/step at B12-equivalent (~2.2× the reference’s 0.46–0.49; chunked backward adds a little); slowness is data, not a kill. Starving util ⇒ input-pipeline fix at a safe boundary, logged. Wall estimate 26–31 h; the 30k decision gate bounds a losing run to ~9 h.
  • E3 probe curve vs the banked E2B curve (256-frame in-run probe, matched cadence, ±0.5 floor). Reference points: E2B 9.43@5k, 7.54@10k, 7.33@20k, 6.57@30k, 6.03@40k, 5.79@50k, 5.55@100k (best 5.29@99.5k). Pre-registered readings:
    • @10k: record only. No kill except divergence (probe >15 with a falling-then-rising shape, or NaN). Bigger trunks may descend slower early — tonight’s aux-off arm is a fresh lesson that early dynamics mislead.
    • @30k: DECISION. Kill if E4B probe > 7.07 (E2B 6.57
      • 0.5) and the 25k panel read (below) does not contradict it. At matched steps and ≥ samples, a 2.2×-text-params trunk showing no probe advantage by 30k means scale is not the cheap lever at this budget — bank the negative, free the box for grounding arms. If probe and panel disagree, continue to 50k.
    • @50k: re-check. Kill if E4B probe > 6.29 (E2B 5.79 + 0.5) with the same panel cross-check.
  • E4 mid-run panels (decision instrument): checkpoints at 25k and 50k rsync to the local box; panel eval (k4l2 plan, --dump-predictions, 1×GPU, ~1.7 h) runs at the first quiet local boundary after each lands. Anchors: E2B’s only panel point is 5.8026 @100k; a probe→panel offset estimate (+0.25, from E2B’s 100k probe 5.55 vs panel 5.8026, single-pair, approximate) puts E2B’s matched-step panel @~25–30k near ~6.8. Readings: E4B@25k panel ≥ 6.9 corroborates a probe kill; E4B@50k panel < 5.8026 (beating E2B’s endpoint at half the steps) is a strong adopt signal and gets posted immediately.
  • E5 endpoint (primary read): E4B@100k panel chunk_mae vs 5.8026, matched eval command (4-GPU sharded, --dump-predictions so per-frame paired analysis works). Adopt iff E4B beats 5.8026 by more than max(3·σ_seed, 0.15), where σ_seed = the pairwise replicate panel spread from tonight’s E5 noise-floor read (A-s0/s1/s2 @40k) — the constant is filled by the finalization amendment before launch, not invented here. Also read: first_mae (E2B 2.1431 — the grounding-sensitive column) and the per-repo delta distribution (coherent vs single-repo-driven).
  • E6 hygiene: loader substitutions / value-budget fallbacks / cuDNN asserts counted; >2 substitutions or any assert ⇒ noted in the results post.

Decision semantics (what this changes)

  • Adopt ⇒ E4B becomes the trunk candidate: the follow-on ablation arm is the image-embedding budget on E4B (owner 21:58Z, one variable per rung), and stage-2 flow-expert work re-targets E4B (streams (5,11,17,23) — four --stream-counts entries; noted, out of scope here).
  • Kill/tie ⇒ trunk scale is not the cheap lever; the box goes to #11 grounding arms and the Molmo2-4B port moves up (survey rank 2). Either way the screen answers the owner’s attribution question 1 with one pre-registered run.

Pre-launch checklist (blocks launch, not this post)

  1. Box free (all four arms + panel evals done, results post out).
  2. Checkpoint present: google/gemma-4-e4b-it is not in the box HF cache (checked 22:2xZ; only e2b) — download (~16 GB).
  3. Parity spot-check on the box: python -m bijou.gemma4.verify_parity for E4B (greedy tokens must match HF; the harness documents E4B ULP-tie behavior).
  4. Memory smoke: 1×GPU, E4B, this recipe, ~50 steps at B12; record peak; pick the ladder rung. If rung 2: land the chunked backward change + oracles first (CPU work item, next GPU-busy window).
  5. Finalization amendment: σ_seed from tonight’s replicate panels, the chosen ladder rung, measured smoke peak, disk check (7.2T free today; 40 saves × ~35–40 GB ≈ 1.4–1.6 T fits; owner checkpoints untouched as ever).
  6. rsync-back loop extended to the new run’s log + latest two saves.

Cost

One 4×H100 run, 26–31 h wall if it goes the distance, ~9 h if the 30k gate kills it; two 1×GPU local panel evals (~3.5 h) mid-run; the endpoint panel (~30 min sharded). Charter §3: the run answers an owner-ranked attribution question with a pre-registered kill that bounds the downside.

Amendment 1 (2026-08-05 ~23:0xZ, before the memory smoke): chunked backward LANDED — with one mechanism correction

Checklist item 4’s conditional impl is now unconditional and done (--backward-chunks N in bijou.train, default 1 = byte-identical path), so an OOM at the smoke costs zero launch delay. Landing it surfaced one error in this pre-reg’s mechanism sketch, corrected here before any E4B data exists:

The sketch said “equal chunks ⇒ mean of chunk-means = batch mean.” That is false for this objective. ar_backbone’s CE pools over valid TOKEN positions (sum/count), and FAST token counts differ per sample — equal-sample chunks still carry unequal token counts, so a mean of chunk means weights tokens unequally (same for the aux ratio, which is a global sum/count). The implementation therefore does something strictly stronger than the sketch: each chunk backwards its sum-form loss normalized by the FULL step’s counts (computed data-only before any forward; aux term over the global aux count), which reproduces the unchunked gradient exactly — unequal counts and all — up to fp reduction order. DDP syncs on the last chunk only; static_graph is dropped when chunking (plain DDP is the well-trodden accumulation path); sample composition, effective batch 48 and every schedule constant remain invariant as pre-registered.

Oracles, all run before this amendment posted:

  • Chunking OFF (the running lineages’ path): all three CPU loss oracles bit-exact — flow 2.7903/1.9152, ar_fast 4.9232/4.8631, ar_backbone 27.8262/27.7701.
  • Chunking ON, ar_fast CLI A/B (2×1 vs B2): loss AND grad_norm bitwise at printed precision on both steps.
  • Chunking ON, ar_backbone CLI A/B: loss identical (27.8262), grad_norm 41.576 vs 41.459 (0.28%). Diagnosed, not waved off: with bit-identical prefix memory the chunk decomposition reproduces gradients to rel ~5e-7 (the math is exact); the residual comes from per-chunk collation width shifting the prefix-encode fp reduction order, amplified through the RANDOM tiny fixture’s saturated 262k softmax (forward matches to 1e-6). Same math, different fp realization — within the pre-registered “up to fp reduction order” contract, and far below the bf16 autocast noise the real run carries anyway.
  • Gradient-equivalence test committed (tests/test_chunked_backward.py, 7 tests): the aux case with UNEQUAL per-chunk aux counts (8 vs 0) asserts chunked ≡ unchunked at rel < 1e-5 — the exact case the original sketch got wrong. check.py green (191).

The finalization amendment still records the chosen rung after the smoke; rung semantics (B12 direct / 2×6 / 3×4 / 4×3 / no-launch) are unchanged.

Amendment 2 — finalization (2026-08-06 ~05:4xZ): σ_seed filled, ladder EXHAUSTED — NO-LAUNCH

Checklist item 5, posted after the box-batch replicate reads and the B12 memory smoke. Every constant below is measured, none invented. The pre-registered ladder’s terminal branch fired: all four rungs OOM, so per the pre-reg (“if even 4×3 doesn’t fit: do not launch”) E4B does not launch under this recipe. The finding and the follow-on decision go to the owner (finding post).

E5 adopt band (σ_seed filled). The three replicate panels (A-s0/s1/s2 @40k) pooled chunk_maes 7.7966 / 7.8052 / 7.7355 ⇒ σ_seed(chunk) = 0.038 (ddof=1, computed by box_batch_results.py, reports/analysis__box_batch_40k_k4l2.json). Adopt rule as pre-registered: E4B@100k panel chunk_mae must beat 5.8026 by more than max(3·σ_seed, 0.15) = max(0.114, 0.15) = 0.15 — the floor binds. Adopt threshold: panel chunk_mae < 5.6526. Context: the largest pairwise replicate pooled |Δ| was 0.0697; σ_seed(first) = 0.0992 is recorded for the first_mae secondary read (descriptive, no gate). These constants are recorded for any future E4B-class screen even though this launch does not happen; the band derivation is recipe-independent.

Ladder rung (memory smoke). smoke_e4b_b12.sh — 1×H100, the exact recipe, 60 steps at loader B12, 2-s VRAM sampler, run once per rung until one fits. None fit:

  • Rung 1 (B12 direct): OOM — 04:31–04:38Z, sampler peak 81,035 MiB of 81,559, torch.OutOfMemoryError at a 78 MiB allocation before the first logged step, with expandable segments on. Not close to the ≥3 GiB headroom bar.
  • Rung 2 (2×6 chunked backward): OOM — 04:54–05:00Z, peak 81,059 MiB; died in the forward SDPA of the first train_step with 78.16 GiB torch-allocated (20 MiB free, 20 MiB requested).
  • Rung 3 (3×4): OOM — ~05:05Z, peak 81,035 MiB; died in the backward of the first train_step (44 MiB free, 100 MiB requested).
  • Rung 4 (4×3): OOM — ~05:19Z, peak 81,049 MiB; died in the backward of the first train_step with 78.05 GiB torch-allocated (30 MiB free, 50 MiB requested).

Each rung’s log carries the correct chunked-backward banner (2×6 / 3×4 / 4×3, loader batch 12 unchanged), so the ladder was exercised as registered, on box code 9ddcfe3 (includes the Amendment-1 chunked-backward impl + oracles). No rung completed a single optimizer step — every traceback starts at the first train_step call — so Adam’s fp32 exp_avg/exp_avg_sq for the 3,975.3M live text params (~2 × 15.9 GiB ≈ 31.8 GiB) were never even allocated. The deficit is therefore not the ~0.1 GiB the OOM margins suggest: steady-state training needs roughly ≥110 GiB/rank under this recipe (fp32 masters + fp32 grads + bf16 weights + frozen tables ≈ high-40s GiB, + ~32 GiB Adam, + activations that alone overflow the remainder even at 3-sample chunks). Chunking the batch further cannot close a fixed-cost gap of that size — the ladder’s terminal branch is the correct read, not bad luck at rung 4.

Smoke E1 lines (match the four box arms, identical across all four rungs). Selection: 878 datasets, 38,571 train + 4,301 holdout = 42,872 episodes, dims 6/6. Model: E4B geometry confirmed — 42 layers, text 3,975.3M params (~2.2× E2B), fp32 masters + bf16 autocast, fp32 ar_backbone decoder.

Checklist state at termination: (1) box free ✓; (2) e4b checkpoint in box cache ✓ (snapshot ee0ef60); (3) parity spot-check passed ✓ (~/e4b_parity.log); (4) smoke ✓ — ladder exhausted; (5) this amendment — outcome: NO-LAUNCH per the pre-registered terminal branch; (6) rsync-back loop extended ✓ (E4B rotation now dormant). The follow-on decision — whether to re-enter E4B under a changed memory recipe (which would be a NEW pre-reg, not an amendment) or to redirect the box — is the owner’s, posted with options in the finding post.