Local inferenceRunning a trained model to get answers — the phase where AI is actually used, as opposed to trained.Full definition → now gets realtime audio in and out through the server plus several new model architectures, so engineers running llama.cpp can build voice and omni workflows without a separate stack.
Terms in this piece · Glossary
streaming — Sending a model's response token by token as it is generated, so the reader sees text immediately instead of waiting for the whole answer.
mixture-of-experts — A model built from many specialist sub-networks where only a few activate per token, giving big-model capability at small-model running cost.
inference — Running a trained model to get answers — the phase where AI is actually used, as opposed to trained.
