Skip to contentScholarGate
LibraryBookshelfDeskReview StudioAssistant
Sign in
Self-supervised Transformer/Evidence
Method evidence record

Self-supervised Transformer

A self-supervised Transformer is a Transformer network pretrained using automatically constructed supervision signals — such as masked token prediction or next-sentence prediction — rather than human-annotated labels. The resulting representations are then fine-tuned or probed on downstream tasks. BERT, GPT, and ViT (Vision Transformer in masked-image modeling mode) are the most widely known instantiations of this paradigm.

Sources recorded, not reviewed

Source record

Citations copied verbatim from the method’s source record. No claim-level verification is inferred from them.

Self-supervised Transformer (Pretraining with Self-generated Supervision)
Taxonomic method record · ml-model / deep-learning
  • Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2019). BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. Proceedings of NAACL-HLT 2019, 4171–4186. · DOI 10.18653/v1/N19-1423
  • Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L., & Polosukhin, I. (2017). Attention Is All You Need. Advances in Neural Information Processing Systems, 30. · URL
Open full method

Curated claims

Claims persisted in the evidence ledger, each with its own assessment.

No curated claims yet

This view does not invent a claim assessment when the ledger has none.

Related methods

Generated from the method graph and shown as machine-suggested relations — no evidence claim is inferred.

Taxonomic bucketBERT-based Classificationmachine-suggested · Relational suggestion, not evidence.Taxonomic bucketFine-Tuned Transformermachine-suggested · Relational suggestion, not evidence.Taxonomic bucketRoBERTa-based Classificationmachine-suggested · Relational suggestion, not evidence.Taxonomic bucketSelf-supervised convolutional neural networkmachine-suggested · Relational suggestion, not evidence.Taxonomic bucketSentence Embeddingsmachine-suggested · Relational suggestion, not evidence.

Evidence status

Sources recorded, not reviewed

Bibliographic sources are present. Claim-level evidence review has not been performed.

Sources

2 recorded citations, copied from the method source record.

Actions

Open method page
ScholarGate

A content-first reference library for research methods — what each one is, how it works, and where it comes from.

Open data (CC-BY)

Explore

  • Library
  • Search the library…
  • Browse by field
  • Fields
  • Journey
  • Compare
  • Which method?

Reference

  • Subjects
  • Atlas
  • Glossary
  • Methodology
  • Philosophy

Your tools

  • Bookshelf
  • Desk
  • Chat

Company

  • About
  • Pricing
  • Contact
  • Suggest a method

Entries are compiled from published sources for reference. Verifying the accuracy and suitability of any information for your own use remains your responsibility.

© 2026 ScholarGate · A research-method reference library
  • Privacy
  • Cookies
  • Terms
  • Delete account