Non-stationary Transformer
Also known as: NS-Transformer, Non-stationary Transformer Network, Stationarization-based Transformer, Durağan-Olmayan Transformer
Non-stationary Transformer is a Transformer-based time-series forecasting architecture introduced by Yong Liu, Haixu Wu, Jianmin Wang, and Mingsheng Long at NeurIPS 2022. It addresses a fundamental tension in applying Transformers to real-world time series: over-stationarization during preprocessing strips out non-stationary signals that carry predictive information, while raw non-stationary inputs cause attention to collapse. The model resolves this through series stationarization paired with a novel de-stationary attention mechanism that restores the original temporal distribution in predictions.
Key highlights
- Explicitly handles non-stationary time series without discarding trend or scale information
- De-stationary attention is lightweight and integrates into any standard Transformer architecture
- Per-sequence normalization generalizes across series with heterogeneous magnitudes
- Demonstrated competitive performance on standard long-range forecasting benchmarks against Transformer-family baselines
Intuition
This section is available to Pro members. Upgrade to Pro
How it works
This section is available to Pro members. Upgrade to Pro
When to use it
Use Non-stationary Transformer when forecasting multivariate or univariate time series that exhibit clear non-stationarity (trends, level shifts, varying variance) over medium to long horizons. It assumes a fixed-length look-back window and requires sufficient training data to learn temporal patterns. It is less suited to very short series or cases where non-stationarity is sparse. Alternatives include Autoformer for longer sequences with strong periodicity, or simple linear models when data volumes are small.
Strengths & limitations
- Explicitly handles non-stationary time series without discarding trend or scale information
- De-stationary attention is lightweight and integrates into any standard Transformer architecture
- Per-sequence normalization generalizes across series with heterogeneous magnitudes
- Demonstrated competitive performance on standard long-range forecasting benchmarks against Transformer-family baselines
- Quadratic attention complexity limits scalability to very long sequences without further approximation
- Performance advantage over simple linear models may be marginal on some univariate benchmarks
- Requires careful hyperparameter tuning (look-back length, number of layers, tau/delta learning rate)
- Like all Transformer models, it demands substantial training data and compute compared to classical statistical methods
Common pitfalls
This section is available to Pro members. Upgrade to Pro
Applications
This section is available to Pro members. Upgrade to Pro
Frequently asked
How is series stationarization different from standard z-score normalization?
Standard z-score normalization typically uses global statistics computed over the entire training set. Series stationarization in Non-stationary Transformer uses per-sequence statistics — mean and standard deviation computed from each individual input window. This means each sample is normalized relative to itself, preserving relative within-window patterns while removing local level and scale, and the removed statistics are stored for later recovery.
What are the tau and delta operators in de-stationary attention?
Tau and delta are learnable rescaling parameters that are conditioned on the original non-stationary statistics of the input sequence. Tau scales the queries and delta shifts the keys inside the scaled dot-product attention. They allow the attention weights to reflect the original temporal distribution rather than the stationarized one, recovering predictive information that would otherwise be lost after normalization.
Can Non-stationary Transformer be applied to univariate series?
Yes. The architecture supports both univariate and multivariate forecasting. In the univariate case a single channel is processed through the stationarization-attention-recovery pipeline. However, some studies suggest that on certain univariate benchmarks simple linear baselines remain competitive, so practitioners should empirically validate the model against lightweight alternatives before deployment.
Sources
- 1.Liu, Y., Wu, H., Wang, J., & Long, M. (2022). Non-stationary transformers: Exploring the stationarity in time series forecasting. NeurIPS.
You have read it. What now?
Cite this page
ScholarGate. (2026, June 2). Non-stationary Transformer. ScholarGate. https://scholargate.app/deep-learning/nonstationary-transformer