Paper questions whether dense self-attention is necessary for strong sequence models, reframing attention as tensor lifting that is expressive but hard to interpret globally.
Proposes Causal Grassmann architecture: reduce token states, map local pairs to Gr(2,r) subspaces, embed via Plücker coordinates, fuse with gated mixing; information flows via subspace deformations across layers/windows instead of explicit L×L weights. Linear in sequence length for fixed rank vs quadratic attention.
Evaluations: Wikitext-2 LM at 13–18M params within 10–15% perplexity of size-matched Transformer; SNLI head on DistilBERT slightly beats Transformer head (val acc 0.8550 vs 0.8545; test 0.8538 vs 0.8511).
Authors aim to de-center attention, not declare it obsolete—arguing geometric evolution on manifolds can match competitive performance with more analyzable structure.