Developmental checkpoints: direction questions

Q1 open Base models cannot follow instructions

Pythia checkpoints are base models, and the introspection and self-report protocols need instruction following. Default: the representation-level measures (M1 to M3) run on Pythia and the behavioural measure (M4) runs on the OLMo 2 post-training stages. A measure with no representation-level form waits for OLMo.

Q2 open Checkpoint storage

2.8B x 20 checkpoints at fp16 is about 110 GB. This box had 2.6 GB free on 2026-09-30 and 9 GB on 2026-10-30. Default: stream each checkpoint to the rented GPU, export the tables, then delete it. Weights are never cached here.

Q3 open Which OLMo release

We have confirmed that OLMo 2 publishes stage checkpoints and data. A newer release (OLMo 3 or a successor) may do the same with better coverage, and we need to check before building tooling against any one version. Record the answer in reading.md.

Q4 open Change-point detection on ~20 points

Twenty checkpoints is a coarse grid. Pythia has 154, dense in the early steps. Default: if a measure shows a candidate onset on the coarse grid, rerun it on all 154 for the two smallest sizes.

Q5 open The loss control in analysis question (b) runs hot

atlas.py tests whether a measure tracks loss beyond step by correlating the residuals of each after a linear fit on log step. Loss is convex in log step, so curvature left in both residuals can look like shared signal. On synthetic series that depend on step only, the self-test sees 5 false positives in 40 at the 0.05 level, where about 2 are expected. Default: before the atlas pre-registration, add a squared log-step term (or a spline) to the control and require the self-test's false-positive rate to sit at the nominal level.

None blocking as of 2026-10-30.