T5 (Text-to-Text Transfer Transformer)
T5: Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer · Also known as: T5, Text-to-Text Transfer Transformer, T5-Small, T5-Base, T5-Large, T5-3B, T5-11B, seq2seq fine-tuning baseline
T5 is a unified sequence-to-sequence deep learning framework introduced by Raffel et al. at Google Brain in 2020, published in the Journal of Machine Learning Research (Vol. 21, No. 140). It reframes every NLP task — classification, translation, summarisation, question answering, and more — as a text-to-text problem: both input and output are always character strings, enabling a single encoder-decoder Transformer to be pre-trained once and fine-tuned across tasks with a consistent interface. T5 introduced span-corruption pre-training and the C4 corpus, and its largest variant (11B parameters) achieved state-of-the-art results across a wide range of NLP benchmarks at the time of publication.
Read the full method
Sign in with a free account to read this section.
Method map
The neighbourhood of related methods — select a node to explore.
When to use it
T5 is appropriate when (1) the task can be expressed as generating a text string from a text string — including classification (labels as words), extraction, translation, summarisation, question answering, and text generation; (2) sufficient labelled data or prompt-based few-shot examples are available for fine-tuning; and (3) computational resources permit loading a pre-trained checkpoint (T5-Small has ~60M parameters; T5-11B requires multi-GPU infrastructure). It is a strong default baseline for sequence-to-sequence NLP tasks, particularly when encoder-only models such as BERT are unsuitable because the output is variable-length text rather than a fixed-size label. T5 assumes textual input and output; it is not designed for purely numeric regression or structured tabular prediction without text encodings.
Strengths & limitations
- Unified text-to-text interface covers virtually every NLP task with one architecture and one training recipe.
- Systematic comparison of pre-training objectives and dataset scales provides a well-documented, reproducible design.
- Span-corruption pre-training captures richer linguistic structure than token-level masking while remaining computationally efficient.
- Relative position biases improve generalisation to input lengths not seen during pre-training.
- Released under Apache 2.0 with checkpoints at five scales (Small, Base, Large, 3B, 11B), enabling flexible deployment trade-offs.
- Widely studied and extended (FLAN-T5, mT5, CodeT5), ensuring long-term community support and documented failure modes.
- Large memory and compute footprint: T5-11B requires tens of gigabytes of GPU memory even for inference.
- Encoder-decoder architecture is slower at inference than decoder-only models for purely generative tasks because the encoder must process the full input before decoding begins.
- The text-to-text framing can be awkward for tasks not naturally expressed as strings (e.g., structured prediction with complex constraints).
- Pre-training on English C4 makes the base model predominantly English; multilingual extension (mT5) requires separate pre-training.
- Fine-tuning on small datasets can lead to forgetting of pre-trained knowledge; careful learning-rate and epoch selection is necessary.
Frequently asked
How is T5 different from BERT?
BERT is an encoder-only model pre-trained with masked token prediction and designed for tasks that require a fixed-size output (classification, token labeling). T5 is an encoder-decoder model that generates arbitrary-length text output, making it suitable for summarisation, translation, and any other task that produces a string rather than a single label. Both use the Transformer attention mechanism, but T5's unified text-to-text framing means its fine-tuning interface is identical across all tasks.
What is span corruption and why is it better than token masking?
Span corruption replaces contiguous groups of tokens (spans) with a single sentinel token and asks the model to reconstruct those spans in the output. Compared to token-level masking as in BERT, span corruption reduces the number of prediction targets per sample, which speeds up pre-training, while still requiring the model to understand longer-range context to fill in a multi-token span. Raffel et al. found empirically that a mean span length of 3 with 15% corruption gave the best results among the objectives they compared.
Which T5 variant should I start with?
T5-Base (~250M parameters) is a practical starting point for most research settings: it fits on a single modern GPU, fine-tunes quickly, and performs well on standard benchmarks. T5-Small (~60M) is appropriate when memory or latency is a hard constraint. Larger variants (Large, 3B, 11B) generally improve results but require multi-GPU setups and longer fine-tuning runs. FLAN-T5 (instruction-tuned) versions are often a better off-the-shelf choice when labelled fine-tuning data are scarce.
Can T5 be used for tasks outside NLP, such as tabular data?
T5 is designed for text input and output. Tabular data can be serialised into text (e.g., column names paired with values as comma-separated strings), and there is research on this approach, but T5 is not optimised for purely numeric or structured data. For tabular prediction tasks, tree-based ensembles or purpose-built tabular deep learning models are generally more appropriate.
Sources
- Raffel, C., Shazeer, N., Roberts, A., Lee, K., Narang, S., Matena, M., Zhou, Y., Li, W., & Liu, P. J. (2020). Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer. Journal of Machine Learning Research, 21(140), 1–67. link ↗
- Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, Ł., & Polosukhin, I. (2017). Attention is All You Need. Advances in Neural Information Processing Systems, 30. link ↗
- Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2019). BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. Proceedings of NAACL-HLT 2019, 4171–4186. link ↗
How to cite this page
ScholarGate. (2026, June 3). T5: Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer. ScholarGate. https://scholargate.app/en/deep-learning/t5
Which method?
Set this method beside its closest kin and read them side by side — the library lays the books on the table; the choice is yours.
- Attention MechanismDeep learning↔ compare
- Transfer LearningMachine learning↔ compare