Developmental checkpoints. Tooling
atlas.py is the analysis side of the atlas: standard
library only, one file, runs on this box against exported tables.
Nothing in it loads a model.
Running atlas.py
python3 atlas.py --selftest
python3 atlas.py TABLE.tsv --threshold 0.05 [--alpha 0.05]just test runs the self-test with every other
--selftest in the repo (about 4 s on this box). The
self-test checks the checkpoint schedule, the probe against its
shuffled-label null on separable and on label-noise data, change-point
recovery on a synthetic kink at step 2000 and no change-point on flat
and on smooth log-linear series, the partial correlation on a tied
series and its false-positive rate on step-only series, and the TSV
round trip and its rejection of bad rows.
The analysis prints one line per (model, measure, layer) series:
above_null_0: the step-0 row'snull_pis below--alpha(question a).partial_r,partial_p: Pearson correlation of value and loss after regressing each on log2(1 + step), with a two-sided permutation p over 999 shuffles (question b). plan.md does not fix the control; linear in log step is the simplest one and matches the change-point axis.cp_step,slope_change,batches: the best single hinge on the log2(1 + step) axis, at least 3 grid points each side, reported only when the slope change per doubling of step exceeds--threshold(question c).batchesis the half-open range [previous grid step, cp_step) of Pile batches; at 1024 sequences per batch the sequence range is the batch range times 1024.
--threshold has no default. It is in the measure's own
units per doubling of step and belongs in the pre-registration; one
value per run, so measures on different scales go in separate runs.
Checkpoint grid
GRID in atlas.py is plan.md's 20 steps: 0, 1, 2, 4, ...,
512, 1000, 2000, 4000, 8000, 16000, 32000, 64000, 100000, 143000.
PYTHIA_STEPS is the published 154 (0, powers of two to 512,
every 1000 to 143000), and the reader rejects any step outside it, so
the dense rerun of direction question Q4 uses the same format.
GPU-side interface
The GPU box writes one TSV per run, tab-separated, one header line,
one row per (model, step, measure, layer).
atlas.write_table writes it; copying atlas.py to the GPU
box and importing it keeps the two sides on one format.
| column | type | meaning |
|---|---|---|
| model | str | Pythia name as on the hub, e.g. pythia-160m,
pythia-160m-deduped |
| step | int | checkpoint step, must be a published Pythia step |
| measure | str | e.g. M1_valence_acc, M1_steer_effect,
M2_jspace_dim, M3_self_other,
M5_residual |
| layer | int | residual-stream layer; -1 for a model-level measure |
| value | float | the measure at this checkpoint |
| null_mean | float | mean of the null draws |
| null_sd | float | standard deviation of the null draws |
| null_p | float | one-sided p of value against the null, (1 + #null >= value) / (1 + n_null) |
| n_null | int | number of null draws |
| loss | float | validation loss of the checkpoint on a fixed held-out Pile sample; identical across rows of one (model, step) |
The null depends on the measure: shuffled labels for a probe accuracy
(the procedure in atlas.probe_null), random directions of
equal norm for a steering effect. The reader rejects a wrong header, a
non-finite number, a step outside the schedule, null_p
outside [0, 1], a duplicate key, and a loss that differs
between rows of one checkpoint.
Still to write, all GPU-side, under the uv lockfile of plan.md step 1:
- Checkpoint loader: stream one Pythia revision
(
step<N>on the hub), run everything for it, delete it. Weights never reach this box (Q2). - Activation capture: residual stream per layer at a fixed token position, for the M1 prompt pairs and the M3 continuations.
- Probe trainer in torch, with the same split, standardisation and
permutation scheme as
atlas.probe_acc/atlas.probe_null. A small exported activation sample run through both gives the cross-check that the two agree. - Steering harness: add the M1 direction at a fixed norm, score continuations with the fixed sentiment classifier, and the random-direction null.
- Loss evaluation on the held-out sample, one number per checkpoint.
- Exporter: rows as above via
atlas.write_table.