Non-stationary Transformer
Non-stationary Transformers for Forecasting · Also known as: NS-Transformer, Non-stationary Transformer Network, Stationarization-based Transformer, Durağan-Olmayan Transformer
Non-stationary Transformer is a Transformer-based time-series forecasting architecture introduced by Yong Liu, Haixu Wu, Jianmin Wang, and Mingsheng Long at NeurIPS 2022. It addresses a fundamental tension in applying Transformers to real-world time series: over-stationarization during preprocessing strips out non-stationary signals that carry predictive information, while raw non-stationary inputs cause attention to collapse. The model resolves this through series stationarization paired with a novel de-stationary attention mechanism that restores the original temporal distribution in predictions.
Read the full method
Sign in with a free account to read this section.
Method map
The neighbourhood of related methods — select a node to explore.
When to use it
Use Non-stationary Transformer when forecasting multivariate or univariate time series that exhibit clear non-stationarity (trends, level shifts, varying variance) over medium to long horizons. It assumes a fixed-length look-back window and requires sufficient training data to learn temporal patterns. It is less suited to very short series or cases where non-stationarity is sparse. Alternatives include Autoformer for longer sequences with strong periodicity, or simple linear models when data volumes are small.
Strengths & limitations
- Explicitly handles non-stationary time series without discarding trend or scale information
- De-stationary attention is lightweight and integrates into any standard Transformer architecture
- Per-sequence normalization generalizes across series with heterogeneous magnitudes
- Demonstrated competitive performance on standard long-range forecasting benchmarks against Transformer-family baselines
- Quadratic attention complexity limits scalability to very long sequences without further approximation
- Performance advantage over simple linear models may be marginal on some univariate benchmarks
- Requires careful hyperparameter tuning (look-back length, number of layers, tau/delta learning rate)
- Like all Transformer models, it demands substantial training data and compute compared to classical statistical methods
Frequently asked
How is series stationarization different from standard z-score normalization?
Standard z-score normalization typically uses global statistics computed over the entire training set. Series stationarization in Non-stationary Transformer uses per-sequence statistics — mean and standard deviation computed from each individual input window. This means each sample is normalized relative to itself, preserving relative within-window patterns while removing local level and scale, and the removed statistics are stored for later recovery.
What are the tau and delta operators in de-stationary attention?
Tau and delta are learnable rescaling parameters that are conditioned on the original non-stationary statistics of the input sequence. Tau scales the queries and delta shifts the keys inside the scaled dot-product attention. They allow the attention weights to reflect the original temporal distribution rather than the stationarized one, recovering predictive information that would otherwise be lost after normalization.
Can Non-stationary Transformer be applied to univariate series?
Yes. The architecture supports both univariate and multivariate forecasting. In the univariate case a single channel is processed through the stationarization-attention-recovery pipeline. However, some studies suggest that on certain univariate benchmarks simple linear baselines remain competitive, so practitioners should empirically validate the model against lightweight alternatives before deployment.
Sources
- Liu, Y., Wu, H., Wang, J., & Long, M. (2022). Non-stationary transformers: Exploring the stationarity in time series forecasting. NeurIPS. link ↗
How to cite this page
ScholarGate. (2026, June 2). Non-stationary Transformers for Forecasting. ScholarGate. https://scholargate.app/en/deep-learning/nonstationary-transformer
Which method?
Set this method beside its closest kin and read them side by side — the library lays the books on the table; the choice is yours.
- Augmented Dickey-Fuller TestEconometrics↔ compare
- AutoformerDeep learning↔ compare
- InformerDeep learning↔ compare