← All IntelClip / OtherNew architectures make quantization implementation harder
From Compression at the Edge — Chris Alexiuk, NVIDIA · ≈33:36
“So it's always a challenge um both from like the implementation perspective but then actually you know running into the quantization bit and making sure it still works correctly.”
“Um so we sometimes leave it in higher precision um as a default just because we actually want people to have a better experience uh versus you know quantizing it down”
What’s in it
- Explains why Ollama sometimes ships smaller models at higher precision
- Reveals how Ollama tests multiple quantization levels before picking a default
- Shows the hidden work behind supporting new model variants at launch
Clip transcript
up with this? >> Yeah. Um there's kind of two facets to it I would say. Uh the first is actually just the model implementation itself. U there's you know been times where a model lab would come to us early and we're kind of working with them to implement the model uh beforehand and we do this except you know they come and sometimes there's like five different variations of a model and you know we need to have them all working. So the implementation is one side of it. Um and I'm sure other people also have to put a lot of work in it. But the other kind of other side is you kind of have to run through the quantization bit and seeing you know kind of which one works best. And atama we kind of do a UX thing of like giving a default uh model quantization for most things uh for most models. Um, and a big part of that is actually, you know, us spending the time one quantizing it, but then seeing if it actually works well with like different harnesses and, you know, is it actually usable after and so we find that sometimes when you have a very small parameter size model, um, you don't get like great quantization after that. Um so we sometimes leave it in higher precision um as a default just because we actually want people to have a better experience uh versus you know quantizing it down quantiz yeah yeah quantizing it down to too little of a precision um and not having the model actually work well. So it's always a challenge um both from like the implementation perspective but then actually you know running into the quantization bit and making sure it still works correctly.
Comments
Sign in to comment.
Loading comments…