Weakly Supervised Vision Transformer
Weakly Supervised Vision Transformer (WS-ViT) · Also known as: WS-ViT, weakly supervised ViT, weak supervision with vision transformer, ViT with weak labels
Weakly Supervised Vision Transformer (WS-ViT) trains a Vision Transformer on image data that lacks precise pixel-level annotations, instead using cheaper, noisier supervision such as image-level class tags, bounding boxes, or web-scraped text. The global self-attention mechanism of the transformer makes it especially capable of localising objects and learning discriminative features from these incomplete labels.
Read the full method
Sign in with a free account to read this section.
Method map
The neighbourhood of related methods — select a node to explore.
When to use it
Use WS-ViT when you have a large image dataset but obtaining dense per-pixel or even per-instance annotations is prohibitively expensive — for example in medical imaging (pathology slides, radiology), remote sensing, or large-scale web-scraped datasets. It is well-suited to image classification, weakly supervised object localisation, and segmentation seeded from image-level tags. Prefer fully supervised approaches when precise spatial masks are available and the dataset is small, since self-attention models need more data to converge well. Avoid if your images are very low resolution or if model interpretability at the pixel level is a hard regulatory requirement, because attention-based localisation can still be imprecise.
Strengths & limitations
- Dramatically reduces annotation cost by learning from image-level tags or bounding boxes rather than pixel masks.
- Self-attention captures long-range spatial dependencies, enabling better object localisation than CNN-based weak supervision.
- Benefits directly from large pre-trained ViT backbones, making it data-efficient when fine-tuning on small weakly labelled sets.
- Attention rollout and CAM provide interpretable activation maps that support qualitative analysis.
- Applicable across diverse domains including medical imaging, remote sensing, and natural image datasets.
- Vision Transformers are computationally heavy; training from scratch requires substantial GPU resources and large datasets.
- Weak labels introduce noise that can systematically bias the model if not carefully handled with appropriate loss functions.
- Localisation accuracy from image-level supervision alone is still inferior to fully supervised segmentation models.
- Requires careful selection of the weak supervision source; different label types need different loss designs.
- Attention-based localisation can be diffuse and unreliable for small or overlapping objects.
Frequently asked
What makes ViT better than CNN for weakly supervised learning?
Self-attention in ViT allows every image patch to directly interact with every other patch, so the model can localise discriminative regions globally — not just in local neighbourhoods as convolutions do. This global receptive field makes attention-based CAM sharper and more semantically meaningful than gradient-weighted CAM from CNNs, which is advantageous when no pixel-level supervision is available.
What kinds of weak labels can be used?
Common forms include image-level class tags (cheapest), bounding boxes, point annotations, scribbles, and pseudo-labels generated by a teacher network or a stronger model. Each type demands a different loss formulation: image-level tags pair with class activation mapping, bounding boxes with a partial cross-entropy or tightness prior, and pseudo-labels with consistency regularisation or noise-transition modeling.
How much data is needed?
Starting from a large pre-trained ViT (e.g., ViT-B/16 pre-trained on ImageNet-21k or via DINO/CLIP), useful performance can be achieved with a few thousand weakly labelled images. Training from scratch typically requires hundreds of thousands of images because transformers have more parameters and less built-in inductive bias than CNNs.
Is the localisation output reliable enough for clinical use?
Not without additional validation. Attention-based localisation maps should be compared against expert annotations on a held-out set before drawing clinical conclusions. For regulatory or safety-critical contexts, weakly supervised localisation is typically used as a screening or prioritisation tool rather than a definitive spatial diagnosis.
How does multiple instance learning (MIL) relate to WS-ViT?
MIL is a classic weak supervision framework where a bag of instances (patches) receives a single bag-level label. WS-ViT can be viewed as a MIL architecture in which the transformer aggregates patch-level information via attention, replacing hand-crafted bag aggregation functions with learned self-attention pooling, often yielding stronger performance than traditional MIL on image tasks.
Sources
- Dosovitskiy, A., Beyer, L., Kolesnikov, A., Weissenborn, D., Zhai, X., Unterthiner, T., Dehghani, M., Minderer, M., Heigold, G., Gelly, S., Uszkoreit, J., & Houlsby, N. (2021). An image is worth 16x16 words: Transformers for image recognition at scale. In International Conference on Learning Representations (ICLR). link ↗
- Zhou, Z.-H. (2022). A brief introduction to weakly supervised learning. National Science Review, 5(1), 44–53. DOI: 10.1093/nsr/nwx106 ↗
How to cite this page
ScholarGate. (2026, June 3). Weakly Supervised Vision Transformer (WS-ViT). ScholarGate. https://scholargate.app/en/deep-learning/weakly-supervised-vision-transformer
Which method?
Set this method beside its closest kin and read them side by side — the library lays the books on the table; the choice is yours.
- Knowledge DistillationDeep learning↔ compare
- Self-supervised LearningMachine learning↔ compare
- Semi-supervised LearningMachine learning↔ compare
- Vision TransformerDeep learning↔ compare