NeurIPS 2026

What Transformer FFNs never see

A feed-forward layer only receives a mixture of value vectors. Two attention patterns can manufacture the exact same mixture, so the layer cannot tell which tokens it used. A short trace of the attention weights makes those patterns separable again.

Playing. Attention is moving along a direction the mixture cannot see.

Six tokens

Value vector and current attention weight.

value vα

What the FFN receives

y = Σ αᵢ vᵢ

Largest change in y: 0.000

Same y, plus a trace

same mixture, different trace

Six tokens, one 8-dimensional mixture. The value vectors are fixed. The slider moves attention along a null direction of V, so every attention score changes and the dominant token switches between T1 and T3. All eight entries of y stay put, because that direction is invisible to the mixture. The panel on the right keeps y and adds which token peaked.
y = Vα α and α′ can satisfy Vα = Vα′ ỹ = y + W r(α)

The collision is structural.

Each head compresses n attention weights into a shorter value mixture. When the sequence is longer than that mixture, different routes are forced to share an output. The feed-forward layer then sees identical inputs and must produce identical outputs. Making the network deeper, wider, or more trained does not put the discarded weights back.

The missing trace is already computed.

The attention weights exist earlier in the same forward pass. Route-Aware FFN keeps a few summaries of them — how peaked the distribution is, and which positions carry the mass — and adds that vector to the mixture before the feed-forward layer. The original mixture stays. The routes no longer look the same.

Four summaries recover the route.

They are computed from the same attention as the figure above. The mixture stays on one vector. The summaries move with the route, and name which tokens carry it.

Tied to the route slider above.

Peak weight

0.000

Entropy

0.000

Lower when attention concentrates on fewer tokens.

Sparsity

0 of 6

Tokens carrying more than an even share of the mass.

Top positions

T1

The three tokens that hold the most attention.

The route comes back as four numbers. Peak weight is the largest attention score. Entropy measures how spread out the scores are. Sparsity counts tokens above an even share. The top three positions say where the mass sits. Slide the route above and these change between T1 and T3, while every entry of y stays put.