today than it was when it was everything was just long all the way down. >> Yeah. Like in the olden days you know every model was dense the transformer plus plus. So it's called transformer plus architecture. You know it's an old transformer. Okay. Plus RMS lay plus some extra tricks. That's called transformer plus. And then now it's like oh my it's like transformer plus+ plus and then plus this thing plus that thing minus this thing minus that thing a different activation function linear attention here sliding window window window attention you know how many layers are sliding window how many layers are global oh let's delete global do something else blah blah blah blah blah um you know everyone likes to do their own thing and you know they like to compare you know like okay this one does better for long context you know this one does worse for long context or something like this so there there's always like reasons why they like to change the architecture Um you know some folks even change some of the you know layer norm epsilons like you know change 1 eus 5 to 1 eus 6 okay which one's better and so on. So they do a lot of abilations you know they do a lot of testing you know this one seems to be better than this one. Um and yes it has complicated compression and quantization dramatically. Um you know you have your old huristics okay this works well for for this model but then when you go to the world oh you can quantize thee layers to like one bit and it doesn't break. Um but then you know when you go to the linear attention world you cannot quantize the linear attention layers. So if you quantize the linear we found that if you quantize the linear attention layers okay it looks like it's doing good but then when you do long context benchmarks you know when you actually use the model in real production it becomes gibberish. Um and so like there are some layers you cannot quantize with these new architectures. Some layers you can you know quantize very low to like one bit you know you can even delete some layers if you like. Um and so it's like these new architectures complicate the process. Um but to be honest very happy with this because we need more different architectures. We don't want everyone to be like thinking the same way and you know open source has been you know the open model era has been like there's so many different architectures and it's very good to have like you know a variety of different opinions and architectures. Yeah. I mean