LayerNorm and the sphereWhy the whole computation lives on a curved surface — and what that turns a transformer into

The residual stream is a running sum, and running sums have a failure mode: they grow. Forty layers of enthusiastic addition would let some tokens' arrows become enormous — and a very long arrow shouts over everything else for reasons of length, not meaning. The fix is one operation, applied relentlessly, and it quietly decides the geometry of everything this library studies.

1. LayerNorm: reproject onto the sphere

Between operations the transformer applies LayerNorm: rescale the arrow back to standard length, keeping only its direction. Since direction is where meaning lives, nothing meaningful is lost — the arrow is reprojected onto the surface of a sphere, and the whole computation effectively lives on that surface.

Drag the point anywhere — far outside the circle or deep inside it. LayerNorm slides it along its own direction back to the circle: length becomes 1, direction is untouched. When the essays say a transformer's states "live on the unit sphere", this operation, applied relentlessly, is the reason. length = —

2. Why a sphere is roomier than it looks

A sphere in two or three dimensions feels cramped — surely forcing every token onto one surface throws information away? Here is the surprise that makes high dimensions different, and it is worth feeling directly. Take two random directions and measure their agreement. On a circle, random pairs agree or oppose all the time — the histogram of dot products is spread wide. Raise the dimension and the histogram collapses toward zero: in 512 dimensions, two random directions are almost always almost exactly perpendicular.

That is called concentration of measure, and it means a high-dimensional sphere is roomy in a precise sense: there is space for enormous numbers of directions that barely interfere with each other. Meanings can coexist without shouting. And when two directions do agree strongly, that agreement is never an accident — which is exactly what makes attention's dot-product scores so informative.

The histogram of agreements (dot products) between thousands of freshly sampled random unit-vector pairs in dimension d. Slide upward: 3 → 16 → 64 → 512. The spread narrows like 1/√d — in a real model's space, random directions are strangers, so genuine agreement stands out like a shout in a silent room. d = 3

3. The payoff: you have just met an oscillator network

Now read the whole module back with this library's eyes. Every token is a unit arrow on a sphere — a phase. Attention is a set of listening channels whose strengths are dot-product agreements — a coupling, choosing its own topology at every step. The residual stream is the shared medium all the coupling acts through — the field. A transformer, described with no metaphor at all, is a network of coupled oscillators.

That identity is the bridge the entire Transformers as Oscillators essay walks across, and with this module in hand you can now walk it — starting with the coupling manifold and the lossless rank-2 projection. When you are ready for the mathematics of motion on the sphere itself — how coupled unit vectors flow, cluster, and carry a token along a path — that is the advanced tier's Lohe dynamics module, and the geometry of thought-shapes is The Physics of Mind §08.

What's real here Everything in this module is a deliberately tiny cartoon — two dimensions standing in for thousands, one head standing in for hundreds — so the moving parts fit in your hands. The measured results built on this machinery live in the Transformers essay with their evidence attached: a derived dead-head threshold τ = 0.96/√d validated at 95–100% precision across five published architectures, attention outputs collapsing onto a handful of effective dimensions, and a rank-2 projection that preserves the model's outputs losslessly. This module teaches the vocabulary; the essay carries the claims, and their limits, with receipts. (The concentration histogram above, though, is not a cartoon — it is the real computation at the stated dimensions, sampled live in your browser.)