方法对比

	基于领域自适应BERT的分类	基于RoBERTa的分类
领域	深度学习	深度学习
方法族	Machine learning	Machine learning
起源年份	2019–2020	2019
提出者	Gururangan et al. (2020); earlier domain-specific instances include Lee et al. (2020) — BioBERT	Liu, Y. et al. (Facebook AI Research / University of Washington)
类型	Domain-adaptive pre-training followed by supervised fine-tuning	Pre-trained transformer fine-tuned for sequence classification
开创性文献	Gururangan, S., Marasovic, A., Swayamdipta, S., Lo, K., Beltagy, I., Downey, D., & Smith, N. A. (2020). Don't Stop Pretraining: Adapt Language Models to Domains and Tasks. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics (ACL 2020), 8342–8360.	Liu, Y., Ott, M., Goyal, N., Du, J., Joshi, M., Chen, D., Levy, O., Lewis, M., Zettlemoyer, L., & Stoyanov, V. (2019). RoBERTa: A Robustly Optimized BERT Pretraining Approach. arXiv preprint arXiv:1907.11692.
别名	DAPT BERT classification, domain-adaptive pre-training, domain-specific BERT fine-tuning, BERT DAPT	RoBERTa classifier, RoBERTa text classification, Robustly Optimized BERT Classification, RoBERTa fine-tuning for classification
相关	6	5
摘要	Domain-adaptive BERT-based classification extends the standard fine-tuning pipeline by first continuing BERT's masked-language-model pre-training on a large corpus of in-domain unlabeled text, then fine-tuning the adapted model on labeled examples for the target classification task. This two-stage approach closes the vocabulary and distributional gap between BERT's general pre-training corpus and specialized domains such as biomedicine, law, finance, or social-media text.	RoBERTa-based Classification applies the RoBERTa pre-trained transformer — trained more robustly than BERT with dynamic masking and larger batches — to text categorisation tasks by adding a lightweight classification head on top of the [CLS] token representation and fine-tuning the entire model on labelled examples. It consistently matches or outperforms BERT on standard NLP benchmarks.