Semi-supervised Text Summarization
Semi-supervised Text Summarization (Label-efficient Abstractive and Extractive Summarization) · Also known as: semi-supervised summarization, label-efficient summarization, weakly supervised summarization, SSL-based summarization
Semi-supervised text summarization trains summarization models by leveraging large amounts of unlabeled text alongside a small set of human-written reference summaries. By using techniques such as language-model pretraining, pseudo-labeling, and self-training, these methods substantially reduce the annotation burden while maintaining competitive ROUGE scores on benchmark datasets.
Read the full method
Sign in with a free account to read this section.
When to use it
Use semi-supervised summarization when you have abundant unlabeled documents in your domain but only a small budget for human-annotated summaries — a common scenario in specialized domains such as biomedical literature, legal texts, or low-resource languages. It is also appropriate when adapting a general-purpose summarization model (BART, PEGASUS) to a new domain with limited in-domain labeled data. Do not use this approach when you have a large, high-quality labeled dataset — in that case, fully supervised fine-tuning is simpler and equally effective. Avoid it when the unlabeled text distribution is far from the target domain, as pseudo-label noise can degrade performance.
Strengths & limitations
- Dramatically reduces reliance on costly human-annotated reference summaries.
- Exploits large pretrained language models (BART, T5, PEGASUS) as strong starting points, accelerating convergence.
- Flexible: works for both extractive and abstractive summarization paradigms.
- Iterative self-training can progressively close the gap to fully supervised performance.
- Particularly effective for low-resource and domain-specific summarization scenarios.
- Compatible with consistency regularization and data augmentation to further improve robustness.
- Pseudo-label quality depends heavily on the initial pretrained model; poor priors propagate errors through self-training.
- Requires careful confidence filtering or noise-robust training to prevent the model from reinforcing its own mistakes.
- Computationally expensive: pretraining and iterative self-training involve multiple forward and backward passes over large corpora.
- Evaluation remains challenging — ROUGE scores do not fully capture factual faithfulness, and human evaluation is needed for final validation.
- Gains over zero-shot prompting of large models (e.g., GPT-4) are narrowing in English, limiting practical advantage outside low-resource or specialized settings.
Frequently asked
How many labeled examples do I actually need?
Experiments typically show that semi-supervised methods can achieve competitive ROUGE scores with as few as a few hundred to a few thousand labeled pairs, depending on domain proximity to the pretraining corpus. Starting with 500–2000 high-quality labeled summaries is a reasonable baseline.
Which pretrained model should I start from?
PEGASUS and BART are the most widely used starting points for abstractive summarization, as their pretraining objectives are closely aligned with summary generation. T5 is also competitive. For domain-specific text, consider models pretrained on in-domain corpora (e.g., BioBART for biomedical text).
How do I filter noisy pseudo-labels?
Common strategies include keeping only pseudo-summaries whose ROUGE score against extractive baselines exceeds a threshold, applying confidence scores from the model's output probability, or using consistency filtering where only stable predictions across paraphrased inputs are retained.
Is semi-supervised summarization still relevant with GPT-4 available?
For general English summarization, large proprietary models often match or exceed semi-supervised systems without any fine-tuning. However, for specialized domains, low-resource languages, or applications requiring on-premise deployment with small models, semi-supervised fine-tuning remains highly valuable.
How should I evaluate faithfulness beyond ROUGE?
Use factual consistency metrics such as FactCC, QAGS, or DAE alongside ROUGE. BERTScore provides a semantic similarity estimate. For high-stakes applications, human evaluation using faithfulness and coherence rubrics is still the gold standard.
Sources
How to cite this page
ScholarGate. (2026, June 3). Semi-supervised Text Summarization (Label-efficient Abstractive and Extractive Summarization). ScholarGate. https://scholargate.app/en/deep-learning/semi-supervised-text-summarization