Tarsier

Why it's ranked here

The research package combines model weights, inference code, evaluation tooling, a public benchmark, and interactive demos. Its reported results cover captioning, question answering, grounding, and hallucination tests. The case is tempered by acknowledged instruction-following weakness and visible setup friction in the supplied code.

What's good

The architecture is easy to reason about: independently encoded frames feed a language model through a projection layer. Uniform sampling includes the first and last frames. The repository also supports resumable, chunked batch inference and maps several captioning and question-answering benchmarks to appropriate evaluation methods.

Tradeoffs

The released Tarsier2 model received post-training only on video caption tasks, and its authors warn that it can struggle to follow instructions. Multiple-choice evaluation requires careful prompt formatting. The dependency list also omits several packages imported directly by the supplied source, increasing setup work.

How to use it well

Use Tarsier for local video-captioning research, visual question-answering experiments, benchmark runs, and model comparisons. Start with the interactive demo, then move repeatable workloads into chunked batch inference. Treat prompts as part of the experiment. It does not replace a dependable general instruction-following assistant.

Observed

License

Apache License 2.0 headers appear throughout the supplied source.

Primary language

Python

Install surface

Pinned Python dependencies are listed in requirements.txt.

Interfaces

Python library code, command-line workflows, and a Gradio web interface.

Accepted media

JPEG, PNG, TIFF, GIF, MP4, AVI, WebM, MOV, MKV, and WMV inputs.

Model distribution

Pretrained model weights are linked through Hugging Face.