GridCore – a scheduler that lets several local LLMs share one GPU
- Source
- blokhin.us
- Author
- w512
- Date

Context
GridCore is an open-source scheduler that decides which local gets the GPU when several apps want it at once. In the author's write-up, separate model servers on one card either split its memory evenly until it runs out, or serve requests in arrival order, so a long indexing job can stall a chat reply.
GridCore sits in front of the models and accepts requests through an OpenAI-compatible interface. Apps tag requests as interactive, background or batch, and interactive work always goes first. Large background requests are cut into chunks of 32 inputs so the GPU can switch to chat between chunks, though a chat reply already in progress is never interrupted. Models are evicted by reload cost rather than simple recency.
The measurements are the author's, from one machine, an RTX 4060 Ti with 16 GB. In a five-minute mixed test with four models loaded he reports no errors and a median chat time that went from 1.9 to 3.4 seconds. He lists multi-GPU support and interrupting a chat mid-generation as things left out.
- LLM — A large language model — the neural network behind tools like Claude and ChatGPT, trained on huge amounts of text to predict what comes next.
- embedding — A list of numbers representing a piece of text's meaning, so that similar meanings end up numerically close and can be searched.
- AI agent — An AI system that doesn't just answer once but works toward a goal in a loop — taking actions, reading the results, and deciding what to do next.
Checking sign-in…
Loading comments…


