Tokens as pathsSection 3 of the Lohe module — the payoff: where you end up is meaning, and the path is computation

Now the whole curriculum cashes out. From the transformer primer: after LayerNorm, a token's hidden state is (approximately) a unit vector — a point on a hypersphere. Each layer does two things: attention pulls the token's vector toward the vectors of the tokens it listens to — structurally the Lohe coupling term from section 1, with the agreement-chosen wiring from section 2 — and the MLP stirs it internally; then LayerNorm re-projects to the sphere. Nudge, re-project. Nudge, re-project. A few dozen times.

So a token passing through a transformer traces a path on the sphere — a short trajectory, bent at every step by where its context sits. Where the path ends up is the meaning the model settled on; the bending along the way is the computation.

1. One token, bending

The orange dot is one token; the faint dots are its context (held still here, so the path stays legible). The token moves by agreement-weighted coupling to the context and leaves a trail — its path on the sphere. Press A new prompt arrives: the context re-seeds, and the same token's path bends toward the new mass. The readout tracks the path's statistics: how far the token has traveled versus how far it has gotten — a decisive path goes nearly straight (ratio near 1); a deliberating one wanders first. traveled — · net — · wander ×—

2. Two nearby tokens, parting ways

Here is the part that makes paths worth caring about. Meaning-space is not flat ground — it has watersheds. Two tokens can start almost on top of each other, feel the same context, follow the same rule — and end up in different places, because they started on opposite sides of a divide. A drop of rain an inch to the left of a ridge line reaches a different ocean.

Below, two tokens begin a couple of degrees apart, straddling the watershed between two context clusters. Watch them separate: slowly at first (the divide is a place of indecision — pulls nearly balanced), then decisively, each to its own basin. "Nearby words, different meanings, different destinations" stops being a metaphor and becomes a picture.

Two tokens (orange and blue) seeded a few degrees apart across the watershed between two context clusters, each moving by the same agreement-weighted rule. The readout tracks the angle between them: it starts small and grows as each token commits to its own basin. Divergence is slowest right at the start — near the divide, the pulls nearly cancel, and small differences take time to become different destinies. separation: start — · now —

3. Where this becomes the research

This module is the doorway to the most technical material on the site. The Transformers as Oscillators essay takes the identification seriously — attention's write-back as the coupling term, term by term — and measures its consequences on real models. The Physics of Mind §08 finds that the paths real tokens trace do not wander the whole hypersphere: they live on a surface of a few dimensions inside it. And §15 shows that two models trained separately lay their meaning out on that surface the same way, one rotation apart (primer 07 builds the rotation-group language for that). If you climbed the curriculum from primer 00 to here, you have every concept those chapters use.

What's real here The simulations on this page are illustrative Lohe dynamics on the ordinary 3-dimensional sphere — chosen because you can see it. They are not a transformer, and the watershed pair is deliberately seeded astride a divide so you can watch the parting (real prompts do not come with such courtesies). The identification of attention's write-back with Lohe-type coupling is a claim the Transformers essay argues and measures on real models; the numbers live there, not here. And one honest caveat about dimension: real models live on spheres of hundreds to thousands of dimensions, where two random directions are almost always nearly perpendicular (concentration of measure) — which is precisely what makes the high-dimensional versions of these dynamics tractable at all.