Mamba (State Space Model)
Mamba: Linear-Time Sequence Modeling with Selective State Spaces · Also known as: Mamba, State space models, Selective state space
Mamba is a sequence model architecture introduced by Gu and Dao in 2023 that achieves linear-time complexity while maintaining strong performance on language modeling tasks. By combining state space models with input-dependent selectivity, Mamba addresses the quadratic complexity of transformers while preserving modeling power.
Read the full method
Sign in with a free account to read this section.
Method map
The neighbourhood of related methods — select a node to explore.
When to use it
Mamba is ideal for extremely long sequence modeling where transformers are infeasible due to memory and compute constraints. It is particularly valuable for streaming applications and real-time inference. Use transformers when you have sufficient compute and need maximum modeling capacity. Mamba excels in tasks like genomics, time series, and long-document understanding.
Strengths & limitations
- Linear computational complexity in sequence length enables efficient processing of arbitrarily long sequences
- Maintains strong performance compared to transformers through input-dependent state selectivity
- Efficient hardware utilization through parallel-scanning and simple operations amenable to acceleration
- Enables new applications like real-time streaming inference with bounded latency
- Architectural novelty means fewer optimized implementations and pre-trained models compared to transformers
- Selectivity mechanism adds complexity compared to simpler SSMs, requiring careful implementation
- Performance on some benchmarks may lag transformer baselines during the transition period
Frequently asked
How does Mamba achieve linear complexity while maintaining transformer-like expressiveness?
Traditional SSMs use fixed state transitions, limiting their ability to selectively attend to important tokens. Mamba introduces input-dependent state transitions: the state matrices depend on the input, enabling selectivity. This allows the model to focus on important information like attention does, while maintaining linear complexity through recurrent state updates rather than all-pairs attention.
What is parallel scanning and why is it important?
Parallel scan is an algorithm that efficiently computes cumulative operations over sequences. For Mamba, it computes the recurrent state updates in parallel across the sequence, avoiding the sequential dependency that would make recurrence slow. Parallel scan has low latency and high throughput, making it ideal for accelerator hardware like GPUs and TPUs.
When is Mamba preferable to transformers?
Mamba excels for long sequences (>10K tokens) where transformer memory becomes prohibitive. It is also preferable for streaming or online processing where latency must be bounded. For short sequences and maximum accuracy on well-studied benchmarks, transformers may still be better. The choice depends on sequence length, latency requirements, and available compute.
Sources
- Gu, A., & Dao, C. (2023). Mamba: Linear-time sequence modeling with selective state spaces. arXiv preprint arXiv:2312.08956. link ↗
How to cite this page
ScholarGate. (2026, June 3). Mamba: Linear-Time Sequence Modeling with Selective State Spaces. ScholarGate. https://scholargate.app/en/deep-learning/mamba
Which method?
Set this method beside its closest kin and read them side by side — the library lays the books on the table; the choice is yours.
- Latent Diffusion ModelsDeep learning↔ compare
- Masked AutoencodersDeep learning↔ compare
- Vision MambaDeep learning↔ compare
- Vision TransformerDeep learning↔ compare