Process / pipeline
BERT Embeddings — Contextual Text Representations
BERT-based text embeddings, introduced by Devlin and colleagues at Google AI in 2019, turn text into context-sensitive dense vectors using a bidirectional Transformer encoder. Because the meaning of a word shifts with its context, BERT produces richer representations than static methods such as Word2Vec or topic models like LDA.
Open in MethodMindSoonVideoSoon
Read the full method
Members only
Sign inSign in with a free account to read this section.
Sources
- Devlin, J., Chang, M.-W., Lee, K. & Toutanova, K. (2019). BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. NAACL-HLT, 4171-4186. DOI: 10.18653/v1/N19-1423 ↗
- Tenney, I., Das, D. & Pavlick, E. (2019). BERT Rediscovers the Classical NLP Pipeline. Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics (ACL), 4593-4601. DOI: 10.18653/v1/P19-1452 ↗
Related methods
Referenced by
Automated Essay ScoringAutomatic Text EvaluationBERTopicCommonsense ReasoningContrastive Learning for NLPCross-lingual Text AnalysisDomain AdaptationFake News DetectionFew-Shot Text ClassificationGender Bias DetectionGloVe EmbeddingsHallucination DetectionHate Speech DetectionIntent DetectionLexical SubstitutionLinguistic Acceptability AssessmentMulti-Document SummarizationMultimodal NLPNMF Topic ModelingParaphrase DetectionRetrieval-Augmented GenerationSemantic SimilaritySentiment AnalysisSocial Media NLPStance DetectionText Coherence ScoringText DeduplicationText InfillingText Regression