Vibeleaderboard
← All Intel
Intel / repo

LLM-as-a-Verifier Plugin for DeepSeek Harness

Source
github.com
Author
gazebushka
Date

Context

The llm-verifier plugin for DeepSeek turns one request into several candidate solutions, then asks a model to choose among them. Its default workflow runs three independent attempts and compares their results against criteria covering the specification, correctness and errors. Repeated pairwise judgments are combined into a ranking.

That extra selection step has a substantial cost: the documented default uses 36 grading calls in addition to the three attempts, with retries potentially adding more. The resulting score expresses the judging model's assessment. It can help select a promising candidate, but remains a model judgment rather than proof that the selected output is correct.

Terms in this piece · Glossary
  • LLM — A large language model — the neural network behind tools like Claude and ChatGPT, trained on huge amounts of text to predict what comes next.
  • agent harness — The scaffolding around a model that turns it into a working agent — the loop, the tools it can call, and the rules for when to stop.
  • AI agent — An AI system that doesn't just answer once but works toward a goal in a loop — taking actions, reading the results, and deciding what to do next.
Recommended reads
Comments

Checking sign-in…

Loading comments…