• The two training stages do different jobs. LoRA fine-tuning with an MLP head teaches the model to emit the label; a second contrastive pass with supervised InfoNCE reshapes the backbone's embeddings so the classes separate.
• Class imbalance gets attacked from three sides at once: oversampling the minority class, augmenting the images, and focal loss so easy examples stop dominating the gradient.
• At inference they blend stage-one token probabilities with stage-two classifier scores using weights tuned on the validation set, rather than trusting either head on its own.
• Binary hate detection reached 0.797 F1 while three-class sentiment reached only 0.518. The same adaptation recipe degrades sharply as the label space gets finer and more subjective.

• LoRA — A cheap way to fine-tune a model by training a small add-on layer instead of changing all of the model's weights.
• multimodal — A model that works with more than text — reading images, audio, or video, and sometimes generating them too.
• fine-tuning — Taking a trained model and training it a bit more on your own examples so it gets better at one specific job.
• embedding — A list of numbers representing a piece of text's meaning, so that similar meanings end up numerically close and can be searched.

Shows a concrete alternative to OCR+translation pipelines for meme/multimodal classification in low-resource scripts by exploiting a VLM's native script understanding, plus a practical two-stage LoRA + contrastive (InfoNCE) recipe for handling class imbalance in shared-task settings.

• articleChinese CLIP: Contrastive Vision-Language Pretraining in ChineseQwen Team
• articleNeoMME: an efficient Multimodal-native and Multilingual Encoderhuggingface.co
• articleGeneralized Visual Language ModelsLilian Weng

Checking sign-in…

Loading comments…