Methoden vergleichen

	Selbstüberwachtes Vision Transformer	Multimodal Vision Transformer
Fachgebiet	Deep Learning	Deep Learning
Familie	Machine learning	Machine learning
Entstehungsjahr	2021–2022	2021
Urheber	Caron et al. (DINO); He et al. (MAE)	Dosovitskiy et al. (ViT); Radford et al. (CLIP multimodal ViT)
Typ	Self-supervised pre-training for vision transformers	Multimodal transformer model
Wegweisende Quelle	Caron, M., Touvron, H., Misra, I., Jegou, H., Mairal, J., Bojanowski, P., & Joulin, A. (2021). Emerging Properties in Self-Supervised Vision Transformers. Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), 9650–9660.	Dosovitskiy, A., Beyer, L., Kolesnikov, A., Weissenborn, D., Zhai, X., Unterthiner, T., Dehghani, M., Minderer, M., Heigold, G., Gelly, S., Uszkoreit, J., & Houlsby, N. (2021). An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale. In International Conference on Learning Representations (ICLR).
Aliasnamen	SSL-ViT, self-supervised ViT, unsupervised ViT pre-training, vision transformer self-supervised pre-training	Multimodal ViT, vision-language transformer, cross-modal vision transformer, multi-modal ViT
Verwandt	4	5
Zusammenfassung	Self-supervised Vision Transformer (SSL-ViT) applies self-supervised pre-training objectives — such as masked patch prediction (MAE) or self-distillation with no labels (DINO) — to the Vision Transformer architecture, enabling powerful visual representations to be learned from large unlabeled image corpora before any task-specific fine-tuning.	Multimodal Vision Transformer (Multimodal ViT) extends the Vision Transformer architecture to jointly process and align representations from multiple modalities — typically images and text — using self-attention and cross-attention mechanisms. By learning shared or aligned embedding spaces across modalities, it enables tasks such as visual question answering, image-text retrieval, visual grounding, and image captioning.