ggml 1.0.0 removes deprecated API and the legacy DIAG_MASK ops and bumps the RPC protocol, so code built on it may need updates. It also adds FP8 support and faster CUDA, Metal, Vulkan and SYCL paths for local inferenceRunning a trained model to get answers — the phase where AI is actually used, as opposed to trained.Full definition →.
Terms in this piece · Glossary
quantization — Shrinking a model by storing its numbers less precisely — like rounding — so it runs faster and fits on smaller hardware, at a small quality cost.
mixture-of-experts — A model built from many specialist sub-networks where only a few activate per token, giving big-model capability at small-model running cost.
inference — Running a trained model to get answers — the phase where AI is actually used, as opposed to trained.
