01 · The task
Addition on a 97-hour clock
The task could not be simpler to state: given two numbers a
and b between 0 and 96, predict (a + b) mod 97 —
add them, and if the sum passes 96, wrap around, like clock hands passing
midnight. There are exactly 97 × 97 = 9,409 possible questions. The network
trains on a random 40% of them and is tested on the 60% it has
never seen. Memorizing the training set is easy; answering the
unseen questions requires actually learning modular arithmetic.
Try the task yourself
Drag the sliders. The dot walks a steps, then b more, around a 97-position clock.
This widget computes the ground-truth answer to show what the task is. The trained networks answer the held-out test questions with accuracy.
The machine under study
A deliberately small transformer, so that every part can be inspected and intervened on.
Three tokens (a b =) are embedded, flow through two identical blocks — each with attention heads (which move information between positions) and one MLP ( neurons of pointwise computation) — and the final state at = is read out as scores over the 97 possible answers. There is no LayerNorm, which makes the mathematics of attribution exact. Full architecture details.
02 · The phenomenon
Memorize first, understand later
Watch the two curves below. Training accuracy hits 100% almost immediately — by epoch the network answers every question it has seen. But test accuracy — the questions it hasn't seen — stays near zero for thousands more epochs. Then it leaps. This delayed generalization is called grokking, and it happened in all five runs, at different times (epoch ).
Training vs. test accuracy (log-scale epochs)
The “slingshot” collapses
During training, some runs briefly collapse — training accuracy falls from 100% to as low as within a hundred epochs — and then fully recover. By our detection rule (training accuracy dropping below 90% after having reached 99%), such events occur in ; the three deepest collapses (seeds ) fall below 50%, while the others are milder dips. They are marked ● on the chart, recover within about a thousand epochs, and are a known quirk of adaptive optimizers at very low loss — one reason we archive full training histories rather than final numbers alone.
03 · The algorithm
The network invents trigonometry
How would you build wrap-around addition out of smooth,
continuous parts? One elegant answer: put the numbers on a circle. Represent
each number n as a point at angle proportional to n
— then adding numbers becomes rotating, and the wrap-around at 97
happens automatically. A “frequency” k just means the point
laps the circle k times as n counts up.
Numbers as rotations
Each number n becomes a point at angle 2πkn/97. Slide n to count; slide k to change how fast the point laps the circle.
The two coordinates of the point are cos(2πkn/97) and sin(2πkn/97) — a wave pair. A network that stores several such wave pairs can add numbers by rotating, then find which answer lines up best. That is the Fourier multiplication algorithm previously reverse-engineered in one-layer networks by Nanda et al. (2023).
Did our networks learn this? Look inside the learned number embeddings. If the network uses rotations, its embedding table should be dominated by a few frequencies — sharp spikes, not a smear.
What frequencies live in the embeddings?
How strongly does each run support the Fourier story?
Two measurements per run: how concentrated the embedding is in its top 5 frequencies (a flat spectrum would put 10.4% there), and how much of the network's output is explained by the rotation formula cos(ω(a+b−c)) — the R² of the fit.
Four of five runs support the Fourier algorithm strongly. Seed 4 is the honest exception: its embeddings are just as spiky, but the rotation formula explains only of its output — and adding more frequencies doesn't help. Whatever else seed 4 is doing remains uncharacterized, so we report the overall evidence as mixed rather than proven.
04 · Causality
Don't just watch the network — reach in and break it
Seeing wave-like patterns is correlation. To claim a component matters, you intervene. We use three lenses of increasing strength:
Direct attribution asks: what does this component write directly into the answer? (Bookkeeping — exact here, but blind to indirect effects.) Ablation asks: what breaks if I switch the component off, replacing its output with its average? (Necessity.) Activation patching asks: if I copy this component's activity from a correct run into a corrupted one, how much behaviour comes back? (Sufficiency of the information it carries.)
Switch a component off
Choose a run, then press a component to mean-ablate it — every number is a real measurement from the saved artifacts.
Why three different lenses?
They disagree in an instructive way. The layer-1 MLP dominates direct attribution — it writes most of the final answer. But layer-0 components score near zero on direct attribution while ablating them is devastating: they act through layer 1, not directly on the output. Only the causal lenses see that. The component ranking table shows all three lenses side by side, per seed.
05 · The search for a circuit
A negative result, honestly told
Mechanistic interpretability often hopes to find a small circuit — a handful of components that fully implement the behaviour, with the rest prunable. We searched greedily: repeatedly remove whichever component hurts least, and check how much performance the remaining set retains. The result: there is no compact head-level circuit here. At a 95%-faithfulness bar, the minimal set is the entire model in four runs and 9 of 10 components in the fifth. The arithmetic is spread across nearly everything — each attention head carries a share of the wave amplitudes.
The search still tells us something real: at every size from two components upward, the greedy set beats both random same-size sets and sets chosen by sheer activation magnitude (at size one, every selection is at noise level) — importance is graded and structured, just not modular.
How much survives as the model shrinks?
Information flow between components
Edges from path patching: how much performance is destroyed when only that single connection carries corrupted information. Hover any node or edge.
In every run the dominant edge is the layer-1 MLP's direct write into the answer; the next strongest are the layer-1 heads feeding that MLP, with the layer-0 MLP and heads contributing smaller edges spread across many paths — breadth rather than a single channel.
06 · Replication
Five networks, five personalities
Every claim above was measured five times, from five random initializations. The algorithm-level story replicates; the details are idiosyncratic. Each run picks its own handful of frequencies (almost disjoint between runs), groks at its own time, and one run resists the clean Fourier explanation entirely.
Side by side
Highlighted cells mark the extremes of each column. “Neurons kept” is how many of each MLP's 512 neurons are needed to retain 90% of performance — even at neuron level, the computation is not sparse.
07 · What we learned
Findings, sorted by how much to trust them
Methodology & limitations
Five seeds of a 2-layer, 4-head transformer (d_model 128, no LayerNorm) trained full-batch with AdamW (weight decay 1.0) on 40% of all 9,409 modular-addition pairs; a uniform stopping rule (99.9% sustained test accuracy, else a 50,000-epoch cap). Counterfactuals are position-resolved mean ablations; patching metrics are normalized answer-logit differences; the Fourier fit is an exact orthogonal projection. Key limitations: training-duration confound — seeds stopped between epochs under the same rule, so cross-seed differences partly reflect training time (a matched-schedule control config ships with the repository); seed 4's residual structure is unexplained; mean ablation defines the counterfactual; greedy search does not certify minimality; five seeds support descriptive statistics only. The full list, with measured values separated from interpretation, is in SCIENTIFIC_FINDINGS.md.