Skill issue: stop deploying vision language models, use them with Skills — Merve Noyan, Hugging Face
Source
AI Engineer
Author
AI Engineer
Date
Key takeaways · AI-distilled
Merve Noyan argues against calling a VLM at runtime for vision tasks: a small task-specific detector can run at around 40 frames per second on weak hardware and still beat the VLM.
Her 'vibe training' flow labels an unlabeled dataset with a 9B open VLM, sends the overlaid boxes to two smaller VLM judges, merges on minimum agreement rather than consensus, and trains an RF-DETR detector, for about $3-4 end to end.
Minimum agreement mattered because one judge rejected far more than the other; requiring consensus would have left too few training examples. She also says the judge prompts still need human approval.
She warns that even strong coding agents make domain mistakes, such as horizontally flipping traffic signs or color-jittering traffic lights during augmentation, until explicitly told not to.
She also flags licensing: teams often deploy a popular detector without noticing its copyleft license, which is why her toolkit hands coding agents Apache 2.0 models.
Terms in this piece · Glossary
AI agent — An AI system that doesn't just answer once but works toward a goal in a loop — taking actions, reading the results, and deciding what to do next.
inference — Running a trained model to get answers — the phase where AI is actually used, as opposed to trained.
Why it matters
Gives a reproducible, cheap ($3-4) recipe for turning a coding AI agentAn AI system that doesn't just answer once but works toward a goal in a loop — taking actions, reading the results, and deciding what to do next.Full definition → plus open VLMs into a trained, real-time object detector, plus concrete pitfalls like agent-introduced label bias for anyone automating dataset labeling.
videoFrom Ingestion to Agents: How AI Teams Build on Document Intelligence — Adit Abraham, Reducto
videoYou’re Not Thinking Big Enough: Rebuilding Food Systems with AI Agents — Cody Menefee, Firecrawl
videoFrom VLM/VLA's to Embodied Agents — Armen Aghajanyan, Perceptron AI
videoFrom Scratch to SOTA: Training a 3B State-Space Vision Model — Krishna Prasad Srinivasan, Sarvam
