Here is a fact that ought to be more famous than it is. A large language model does its thinking in an enormous space: each word being processed carries a running summary — its slot on the residual stream, the shared whiteboard every layer reads and writes — with hundreds or thousands of dimensions. (If those words are new, the Inside-a-transformer module builds them from nothing in ten minutes.) And the model computes in almost none of that space. Measure where the work actually happens, and the joint output of a layer’s attention heads — the listeners that pull information between words — does not fill its 768-or-more dimensions. It concentrates onto a thin surface — a coupling manifold — of effective dimension two to about nineteen. Intelligence, it turns out, has a shape, and the shape is small.
The reason is the one law. Heads are coupled oscillators writing to a shared field — the stream. Coupled oscillators synchronize, and synchronized parts are not independent: most of those hundreds of dimensions are heads agreeing with each other, and only a handful carry genuinely distinct structure. So the question becomes: how many directions is the work really using? Along each independent direction of the heads’ joint output there is a number — a singular value — saying how much of the action lies that way; think of it as that direction’s loudness. Count the directions that are actually loud, and you have counted the dimensions doing work.
Drag the synchronization below — think of it as training proceeding, heads learning to cohere. On the left, twelve head outputs as vectors; watch them fall out of a random scatter into a couple of shared directions. On the right, the loudness of each direction, and the live count, deff, dropping from twelve toward two.
The count you just watched has a standard formula — the participation ratio. You do not need it to follow the chapter; it is here for the reader who wants the machinery:
Click any coloured symbol to see what it means.
This is not lossy approximation; it is where the computation already lives. Measured on a real, published model — GPT-2, at one of its last layers (layer 11) — the effective dimension is about 3.5 inside 768, and projecting the layer’s output onto just its top directions changes the next-token prediction not at all. The other hundreds of dimensions are thermal modes: oscillators agreeing, cancelling at the output. We make that lossless projection tangible in the next chapter.
Read as physics, this surface is where the oscillator network’s dance actually lives — the few shared directions a synchronized system moves in. Which sharpens a puzzle: if the dance is this small, why is the dancer so big? Why do the weights refuse the compression the activations allow? That is the fiber bundle, two chapters on.
The shape itself has more structure than a flat low-dimensional blob — it is a torus, one circular direction per cluster of phase-locked heads, and moving on it is what reasoning is (developed in the paradigm essay, The shape of a thought). Here we stay with the manifold and what it lets us do: project onto it, losslessly.