Pyraformer: Pyramidal Attention Transformer for Long-Range Time-Series Forecasting
Pyraformer (Pyramidal Attention for Long-Range Forecasting) · Also known as: Pyramidal Attention Transformer, Pyraformer Transformer, Piramit Dikkat Dönüştürücüsü, Low-Complexity Transformer
Pyraformer is a Transformer-based model for long-range time-series forecasting introduced by Liu et al. at ICLR 2022. Its central innovation is a Pyramidal Attention Module (PAM) that organizes tokens into a multi-resolution hierarchy, enabling the model to capture temporal dependencies across multiple scales while keeping time and memory complexity at O(L log L) rather than the quadratic cost of vanilla self-attention.
Read the full method
Sign in with a free account to read this section.
Method map
The neighbourhood of related methods — select a node to explore.
When to use it
Pyraformer is well-suited for multivariate or univariate long-sequence forecasting tasks (e.g., predicting hundreds to thousands of future steps) where capturing dependencies at multiple temporal granularities matters—electricity load, traffic flow, weather. It assumes the input is a regularly sampled time series. Practitioners should note that, like many Transformer variants, it may underperform simpler linear baselines on certain benchmark datasets; careful ablation against DLinear or PatchTST is advisable before deployment.
Strengths & limitations
- O(L log L) time and memory complexity, making it more scalable than standard Transformers for long sequences
- Explicit multi-resolution hierarchy captures both short-term patterns and long-range dependencies in a single forward pass
- Avoids autoregressive decoding at inference, enabling fast parallel multi-step predictions
- Architecturally principled: the pyramidal structure has a clear inductive bias for time series with multi-scale seasonality
- The CSCM coarsening introduces a fixed hierarchical structure that may not be optimal for all series; irregular or non-stationary patterns can be poorly represented
- Empirical gains over simpler linear models (e.g., DLinear) are not always consistent across datasets, raising questions about whether attention-based inductive biases are always beneficial
- Hyperparameter sensitivity: pyramid depth C and kernel size k require dataset-specific tuning
- Implementation complexity is higher than standard Transformer or convolutional baselines, increasing reproducibility burden
Frequently asked
How does Pyraformer differ from Informer?
Informer reduces attention complexity via a ProbSparse mechanism that selects a subset of query-key pairs based on estimated importance. Pyraformer instead builds an explicit multi-resolution pyramid and restricts attention to local windows within each level plus parent-child links across levels. Both achieve sub-quadratic complexity, but Pyraformer's hierarchy provides a stronger inductive bias for multi-scale temporal patterns.
Is Pyraformer still competitive with more recent models?
Subsequent work (e.g., PatchTST, TimesNet, iTransformer) and even simple linear models (DLinear, NLinear) have shown that Transformer-based forecasters do not uniformly outperform simpler alternatives. Pyraformer remains a valuable architectural reference and may be competitive in specific multi-scale or long-horizon settings, but benchmarking against current baselines on your specific dataset is essential.
What is the role of the CSCM in Pyraformer?
The Coarser-Scale Construction Module (CSCM) is a convolutional sub-network that transforms the original fine-grained sequence into progressively coarser temporal representations, creating the pyramid levels. It effectively learns a data-driven temporal summarization rather than using fixed pooling, allowing the model to adapt the coarsening to the statistical properties of the input series.
Sources
- Liu, S., Yu, H., Liao, C., Li, J., Lin, W., Liu, A. X., & Dustdar, S. (2022). Pyraformer: Low-complexity pyramidal attention for long-range time series modeling and forecasting. ICLR. link ↗
How to cite this page
ScholarGate. (2026, June 2). Pyraformer (Pyramidal Attention for Long-Range Forecasting). ScholarGate. https://scholargate.app/en/deep-learning/pyraformer
Which method?
Set this method beside its closest kin and read them side by side — the library lays the books on the table; the choice is yours.
Compare side by side →