Ai2 used spoken descriptions to collect richer training material for Molmo, its family of models that answer questions about images. For the PixMo caption dataset, annotators described images aloud for 60 to 90 seconds, including objects’ positions and relationships. Ai2 reports that this produced more detail in less time than written descriptions. The recordings were transcribed and refined using language models. Its release article explains the process.
This is the data-collection mechanism discussed in the paper club: people supplied the visual observations, reducing dependence on captions produced by proprietary image-understanding models.
Terms in this piece · Glossary
distillation — Training a small, cheap model to imitate a big one's outputs, keeping much of the capability at a fraction of the cost.
