The core correction
The modern stack is plural.
Production model families now use several different efficiency strategies: local or sparse attention, compressed KV representations, recurrent linear attention, selective state-space layers, and genuine Mamba–attention hybrids. Calling all of these “Mamba” erases the most important architectural differences.
At a glance
Four branches of sequence mixing
These mechanisms answer the same question—how should a token use earlier context?—but they store and retrieve that context differently. Mixture-of-Experts routing is a separate FFN/capacity decision and can be combined with any branch.
Recurrent state
Compress the past
LSTMs, Griffin, Mamba, and TTT carry a bounded state forward. Decode memory can remain constant with respect to context length, but information must be compressed. Looped models such as Parcae add a different kind of recurrence by repeating computation over model depth.
Attention
Address the past
Self-attention explicitly compares tokens. Full attention is expressive and parallel during training, while its compute and KV storage motivate local, sparse, compressed, and IO-aware variants.
Linear attention
Update a memory matrix
Linear Transformers, RetNet, DeltaNet, and Gated DeltaNet rewrite attention into recurrent updates. They are related to SSMs, but they are not automatically Mamba.
Hybrid stacks
Assign different jobs
Jamba, Nemotron, Griffin, Taipan, and newer Qwen architectures interleave token mixers so efficient recurrent layers handle most positions while attention protects retrieval and recall.
Complexity without the mythology
Training cost, decode cost, and cache memory are different measurements
“Quadratic Transformer” and “linear-time Mamba” are useful shorthand, but only after the phase of computation and the form of memory are named.
| Token mixer | Training / prefill mixing | Autoregressive state | What is retained |
|---|---|---|---|
| Full attention | O(N²) attention work | O(N) KV cache; O(N) work at the next token | Per-token keys and values |
| Sliding-window attention | O(NW) | O(W) cache for a local layer | The most recent window |
| Compressed attention / MLA | Still attention-dependent | Cache still grows with N, but with a smaller representation | Compressed per-token latent KV information |
| SSM / recurrent linear layer | O(N) algorithmic sequence work | O(1) with respect to N | A fixed-size learned state |
| Hybrid stack | Depends on layer ratio and attention pattern | Recurrent state plus the cache of attention layers | Compressed state and selected explicit token history |
FlashAttention reduces attention’s memory traffic and working-memory requirements; it does not change the exact full-attention operation into an O(N) computation.
The evidence-led timeline
From recurrence to architectural portfolios
Filter the chronology by architectural branch. Dates refer to the first paper or public technical release cited in each entry.
Recurrent baseline
LSTMs dominate production sequence modeling—but attention already exists
LSTM was introduced in 1997, not 2016. By this timeline’s starting point, recurrent encoder–decoder systems commonly used learned alignment mechanisms such as Bahdanau attention, so context did not depend “entirely” on one hidden vector even though recurrence still constrained parallelism.
Causal convolution
WaveNet shows that deep causal convolutions can model sequences
Dilated causal convolutions expand the receptive field without an RNN. Generation remains autoregressive, but training can process positions in parallel and the sequence-layer design space broadens.
Convolution → attention
ConvS2S removes recurrent training; the Transformer removes recurrence entirely
Facebook’s convolutional sequence-to-sequence model arrives in May. In June, Vaswani and colleagues pair multi-head self-attention with position-wise feed-forward networks. Attention mixes information between positions; the FFN transforms each position independently.
Transformer consolidation
GPT and BERT turn the Transformer into the dominant pretraining substrate
The principal change is not a new token mixer. Decoder-only generative pretraining and bidirectional encoder pretraining show that a general Transformer backbone can be adapted across many language tasks.
Segment recurrence
Transformer-XL reuses hidden states across segments
A recurrence mechanism extends usable context beyond a fixed training segment while retaining attention inside the model. “Transformer versus recurrence” is already becoming an incomplete framing.
Sparse, linear & recurrent memory
Longformer, Linear Transformers, and HiPPO attack different bottlenecks
Longformer combines local windows with task-specific global attention. Linear Transformers formulate causal attention as a recurrent state update. HiPPO develops a mathematical framework for compressing continuous history—the foundation later used by structured state-space models.
SSM
S4 makes structured state-space sequence layers practical
S4 combines HiPPO-derived state matrices, structured parameterization, and efficient convolutional computation. It demonstrates strong long-range sequence modeling and creates the most direct research lineage leading to Mamba.
Exact attention
FlashAttention changes the hardware cost without changing the attention result
IO-aware tiling avoids materializing the full attention matrix in high-bandwidth memory. The algorithm remains exact and computationally quadratic, but far more usable on GPUs.
SSM + local mixing
H3 adapts state-space layers to language modeling
The Hungry Hungry Hippos architecture combines two SSMs with multiplicative interactions to improve token comparison and copying behavior—an important bridge between S4 and selective SSMs.
Long convolution & retention
Hyena and RetNet show that “subquadratic” is not one architecture
Hyena uses implicit long convolutions and data-controlled gating. RetNet provides parallel, recurrent, and chunkwise forms of a retention mechanism. Both pursue Transformer-like quality with more efficient long-sequence execution.
Mamba
Input-dependent selection brings content sensitivity into the SSM update
Mamba makes selected SSM parameters functions of the current input and introduces a hardware-aware parallel scan. It delivers linear sequence scaling and constant-size inference state with respect to context length. That state is compact—not zero-cost.
Gated recurrence + attention
Griffin and Jamba make hybridization explicit
Google’s Griffin mixes gated linear recurrences with local attention and becomes the basis of RecurrentGemma. AI21’s Jamba interleaves Transformer and Mamba layers and adds MoE capacity. Neither paper establishes one universal layer ratio.
Mamba-2 + NVIDIA study
Structured State Space Duality connects SSMs and structured attention
Mamba-2 follows from a mathematical connection between selective SSMs and structured masked attention. NVIDIA’s controlled 8B-scale study then finds that pure Mamba-family models remain weaker on some recall-heavy tasks, while a small allocation of attention layers materially improves the hybrid.
Test-Time Training layers
TTT turns the recurrent hidden state into a model that learns at inference time
Researchers from Stanford, UC San Diego, UC Berkeley, and Meta propose TTT-Linear and TTT-MLP. The hidden state is itself a learnable model updated on the current sequence, expanding the design space beyond both attention and Mamba.
Selective attention budget
Taipan spends attention only on tokens selected for long-range interaction
University of Oregon, Auburn, and Adobe researchers combine Mamba-2 with Selective Attention Layers. The result makes “hybrid” a routing decision as well as a fixed layer-interleaving decision.
Gated DeltaNet
Gating and delta updates create another recurrent route to efficient recall
Gated DeltaNet combines adaptive forgetting with targeted memory updates and a parallel training algorithm. It outperforms Mamba-2 in the paper’s evaluations and later becomes central to Qwen’s hybrid designs. It is recurrent linear attention, not Mamba.
DeepSeek MLA
DeepSeek reduces KV-cache size while remaining inside the Transformer framework
DeepSeek-V2 and V3 use Multi-head Latent Attention to compress keys and values into a per-token latent representation. DeepSeek R1 inherits the V3 base architecture. Later V3.2 adds sparse attention—not Mamba.
Nemotron-H → Nano 2 → Nemotron 3
NVIDIA turns the Mamba-2/attention hybrid into a model family
Nemotron-H studies hybrid layer allocation at scale. Nemotron Nano 2 9B uses mostly Mamba-2 and MLP layers with four attention layers. Nemotron 3 Nano extends the pattern with MoE routing: 30B total parameters, roughly 3B active—not an 8B model.
Attention + MoE
gpt-oss-20b demonstrates that efficient open models do not require Mamba
OpenAI’s 21B-parameter, 3.6B-active model remains an attention-based causal MoE model. Optimized attention kernels, expert routing, and low-precision weights address deployment cost without a state-space token mixer.
Gated DeltaNet + attention
Qwen3-Next adopts a recurrent hybrid—but not Mamba
Original Qwen3 models use grouped-query attention. Qwen3-Next begins a new line that interleaves Gated DeltaNet with global attention. The architecture shares Mamba’s goal of bounded recurrent state while using a different memory update.
Recall beyond the recent window
Amazon studies Mamba as a component, not a universal backbone
Amazon researchers use Mamba as a fast drafter for speculative decoding and study Span-Expanded Attention for retrieving distant tokens into a hybrid SSM’s finite attention span. The emphasis is increasingly on assigning the right memory mechanism to each job.
Local + global attention
Gemma 4 is hybrid attention, not a Mamba hybrid
Google interleaves local sliding-window and full global attention and makes the final layer global. The word “hybrid” describes two attention patterns; no selective SSM is involved.
Depth recurrence
Parcae makes stable layer looping a new scaling axis
UC San Diego and Together AI researchers recast a looped Transformer as a dynamical system over the residual stream. By constraining its recurrent injection, Parcae stabilizes repeated passes through shared middle layers. This is recurrence over depth, not Mamba-style recurrence over tokens.
GDN + global/sparse attention
The newest Qwen line strengthens the portfolio model
Qwen3.8-Flash-Next uses Gated DeltaNet with one full-attention layer in every four, then replaces full attention with Qwen Sparse Attention during continued pretraining. The direction is clearly hybrid, but still not Mamba.
What current model families actually use
Mamba adoption is real, but narrower than “hybrid” adoption
This is the practical distinction that gets lost when every subquadratic or local/global architecture is placed in one bucket.
gpt-oss-20b
Attention-based causal MoE.
Gemma 4
Local and global attention.
Nemotron Nano 2 / 3
Mamba-Transformer hybrids; Nemotron 3 adds MoE.
DeepSeek V3 / R1 / V3.2
MLA and later sparse attention.
Qwen3-Next / 3.8
Gated DeltaNet plus attention.
2026 local model comparison
Compare the deployment you will actually run
The table uses one inference envelope: 4-bit weights and an FP8 KV cache, batch size one. It deliberately excludes vendor benchmark scores measured with BF16, FP16, proprietary serving stacks, or undisclosed cache precision.
Compact specialists
Gemma 4 E4B or 12B, Phi-4-reasoning-plus, and Qwen3-VL 8B leave varying amounts of room for FP8 KV and runtime buffers.
The useful local frontier
Gemma 4 26B/31B, Qwen3 30B-A3B, Qwen3-Coder 30B-A3B, gpt-oss-20b, and Nemotron 3 Nano become practical—with context length still managed explicitly.
Open does not mean desktop-sized
Mistral Small 4 and DeepSeek V3.2 can be self-hosted, but their resident weight sets place them outside a 32 GB GPU pool.
| Model | Total / active | Q4 weight floor | Published context | FP8 KV behavior | Best local role | Practical placement |
|---|---|---|---|---|---|---|
| Gemma 4 E4B | 8B stored / 4.5B effective | ≈ 4.0 GB | 128K | Local-window + selected global layers | Fast multimodal assistant; audio, image and text | 1 × 16 GB |
| Gemma 4 12B Unified | 12B / 12B | ≈ 6.0 GB | 256K | Local-window + selected global layers | Single-GPU reasoning and unified multimodality | 1 × 16 GB* |
| Phi-4-reasoning-plus | 14B / 14B | ≈ 7.0 GB | 32K | Full-attention cache grows with context | Math, science and deliberate English reasoning | 1 × 16 GB |
| Qwen3-VL 8B | 8B / 8B | ≈ 4.0 GB | 256K | Long-context attention; visual tokens also consume cache | OCR, UI understanding, visual agents and documents | 1 × 16 GB* |
| gpt-oss-20b | 21B / 3.6B | ≈ 10.5 GB floor; ~16 GB official MXFP4 runtime | 131K | Sliding-window + global attention cache | General reasoning, structured output and tool use | 2 × 16 GB for KV headroom |
| Gemma 4 26B A4B | 25.2B / 3.8B | ≈ 12.6 GB | 256K | Local-window + selected global layers | High-throughput multimodal reasoning | 2 × 16 GB |
| Gemma 4 31B | 30.7B / 30.7B | ≈ 15.4 GB | 256K | Local-window + selected global layers | Dense Gemma-family reasoning when latency is secondary | 2 × 16 GB |
| Qwen3 30B-A3B | 30.5B / 3.3B | ≈ 15.3 GB | 32K native; 131K with YaRN | GQA cache grows with attended context | Multilingual reasoning, chat and tool orchestration | 2 × 16 GB |
| Qwen3-Coder 30B-A3B | 30.5B / 3.3B | ≈ 15.3 GB | 256K native | Repository-scale context makes KV the main variable | Agentic coding, repository exploration and tool calls | 2 × 16 GB* |
| Nemotron 3 Nano | 30B / 3B | ≈ 15.0 GB | 1M | FP8 KV only for attention layers; Mamba layers keep recurrent state | Long-running agents, multi-agent throughput and tool use | 2 × 16 GB minimum* |
| Mistral Small 4 | 119B / 6B | ≈ 59.5 GB | 256K | Hybrid attention; cache depends on served context | Unified reasoning, coding and multimodal server | Server / CPU offload |
| DeepSeek V3.2 | 671B resident class / MoE | ≈ 335.5 GB | Long-context sparse attention | MLA + sparse-attention state; still substantial at scale | Frontier reasoning and agents on clustered infrastructure | Cluster-local |
*Context-constrained. A model can load while its advertised maximum context cannot. FP8 halves KV storage relative to FP16/BF16, but it does not make KV free. Calibrated FP8 scales are preferred; measured tokens-per-second should only be compared on the same engine, GPU, batch size, prompt/decode split and checkpoint format. vLLM FP8 KV guidance
How specialized models change the landscape
The winning local system is becoming a portfolio
A general model no longer has to perform every step. Small models can retrieve, route, inspect images, call tools, classify risk, or handle a domain before a larger reasoner is invoked.
EmbeddingGemma
A compact on-device embedding model can search the local evidence set before generation. Better retrieval often produces more value than spending every request on a larger generator.
Official overviewFunctionGemma
A small function-calling model can turn intent into a constrained tool call. This moves routine control-plane work away from the expensive general reasoner.
Official model cardQwen, Phi, Gemma or Nemotron
The primary model can be selected per request: code agent, compact math reasoner, multimodal analyst, or long-context Mamba hybrid. “Best” becomes conditional on the work state.
Compare local rolesMedGemma
Medical text and image training materially changes the baseline for health workflows. It remains a developer model that requires downstream validation—not an autonomous clinical authority.
Official model cardQwen3-VL or Gemma 4
Vision-language models add OCR, UI understanding, chart reading, video, and document analysis. Visual work no longer needs to be flattened into imperfect text first.
Qwen3-VL model cardShieldGemma + task evaluators
Dedicated safety and verification passes make model outputs inspectable components in a workflow. The architecture now includes routing policy, evidence, and evaluation—not only model weights.
ShieldGemma model cardThe systems implication
Model selection becomes runtime architecture.
Specialization favors explicit routing and measured escalation: send the easy request to the smallest validated model, invoke the domain expert when its evidence applies, and reserve the large reasoner for ambiguity or failed verification. This can improve throughput and privacy while making failures easier to attribute to a specific role.
How to read the decade
The bottleneck keeps moving
2016–2018: remove the sequential training dependency
Convolutions and self-attention win because GPUs can process many positions in parallel. The Transformer’s central achievement is not that it “understands ambiguity mathematically”; it provides a scalable mechanism for content-dependent token interaction.
2019–2022: keep attention, but make context affordable
Segment recurrence, local windows, sparse patterns, linearized attention, and IO-aware kernels each attack a different cost. This is where asymptotic complexity and realized GPU performance visibly diverge.
2023–2024: make recurrent state content-sensitive and hardware-efficient
Mamba is the most influential result in this period, but it arrives inside a broader wave that includes Hyena, RetNet, Griffin, TTT, and delta-rule models. Recall evaluations quickly show why fixed-state compression and explicit token access remain complementary.
2025–2026: architectures become portfolios
Model builders mix layer types, compress KV representations, route tokens through experts, restrict attention windows, and reserve global attention for selected layers or tokens. The design question is no longer “Transformer or Mamba?” It is “which memory mechanism belongs at which layer, for which workload, on which hardware?”
Open only what you need
Corrections to common architecture claims
The main chronology stands on its own. These notes preserve the distinctions that matter when the timeline is used as a technical reference.
01Was contextual disambiguation invented by self-attention?
No. Recurrent networks, bidirectional recurrent networks, and encoder–decoder attention already produced context-dependent representations. Self-attention made direct pairwise token interaction easier to parallelize and scale; it did not create the general concept of contextual representation.
02Does a KV cache make Transformer inference quadratic?
The cache is a memory optimization, not the source of the quadratic training attention matrix. During decode it prevents recomputation of prior keys and values. Cache memory grows linearly with context; each new query attends over a growing prefix, so cumulative full-attention decode work grows quadratically with generated length. The model still computes the new token’s query, key, and value.
03Does Mamba have zero memory overhead?
No. Mamba carries a nonzero recurrent state in every applicable layer. The precise advantage is that the inference state does not grow with the number of preceding tokens. Training activations and implementation workspaces are separate memory costs.
04Are attention layers permanent semantic anchors?
That is a metaphor, not an established mechanism. Hybrid studies support a more modest claim: explicit attention materially helps recall, copying, retrieval, and other tasks that are difficult when the entire past must be compressed into fixed-size state.
05Does “hybrid attention” imply a recurrent layer?
No. Gemma 4 is a clear counterexample: it is called hybrid because it interleaves local sliding-window attention with full global attention. Always identify the actual token mixers instead of inferring them from the word “hybrid.”
Research index
Primary sources by lineage
Papers and official model reports are linked at the point where each claim appears. This index groups the most important sources for follow-up reading.
Linear and learned-state alternatives
Linear Transformers RetNet TTT layers Parcae looped models Gated DeltaNet Qwen3.8-NextHybrid evidence and deployments
Jamba NVIDIA empirical study Nemotron-H Nemotron Nano 2 Nemotron 3 Nano2026 local and specialized models
gpt-oss-20b Gemma 4 Qwen3-Coder 30B-A3B Phi-4-reasoning-plus Mistral Small 4The 2026 conclusion
Architectural hybridization won more clearly than Mamba itself.
Mamba established that a selective recurrent layer could scale and run efficiently on modern accelerators. The broader market response was not wholesale replacement of attention. It was permission to combine specialized memory systems: exact attention where retrieval matters, recurrent state where bounded decode memory matters, sparse or local attention where context can be constrained, and MoE where capacity can be routed independently.
Evidence status: factual dates and architecture descriptions are linked to primary papers or official model documentation. Interpretive claims are identified as synthesis rather than benchmark results.