Broken on Arrival: Silently Defective LLM Artifacts in Public Model Registries and How to Catch Them

A downloaded model file can look normal and still fail at the work it was published for. A study of compressed model distributions identified five defective files among 305 eligible files from the Ollama library. Each failed all 164 follow-up coding tasks on both tested hardware backends, while independent conversions of the same models performed better, and for two confirmed defects the output's surface statistics sat inside the healthy range.

The comparison matters because poor performance can also come from compressing a small model too aggressively or from a particular execution setup. The study used independent conversions and multiple backends to distinguish those possibilities. It supports testing the actual downloaded file on the intended system: the original model's reputation and plausible-looking output do not establish that its distributed conversion works.

“Developers increasingly run large language models locally by pulling quantized GGUF artifacts from public registries, yet nothing in the distribution pipeline functionally tests these conversions before they reach users.”

“The official library carries five silently defective artifacts, a batch of four Qwen2.5-Coder-3B conversions and one phi3.5-mini conversion, that solve zero of 164 tasks and zero of the smoke suite on both backends while independent conversions of the same models work: 1.6% of official artifacts, 2 of 29 model-and-size conversion groups.”

“Two confirmed defects produce output whose surface statistics sit inside the healthy range, invisible to any low-noise heuristic short of execution.”

“We release the audit dataset, the quantcheck acceptance-testing tool, and disclosure reports for every confirmed defect ( this https URL ), and argue that model registries need the acceptance gate that package registries already run.”