Intel / article

Vibe Patenting: Evaluating LLM Judges for Professional Patent-Drafting Agents

Source: arxiv.org
Author: Toshiaki Koike-Akino, Vlad Blaykhman, Ye Wang, Jing Liu, Gene V. Vinokur
Date: Published Sep 15, 2026

Context

AI feedback improved patent drafts according to an AI evaluator, but agreement with a professional attorney depended heavily on what was being judged. In Vibe Patenting, a separate model reviewed drafts and supplied structured feedback for revision. This produced continuing gains in the model's scores where generic revision tended to level off.

The researchers then compared ratings on 22 matched draft-and-condition pairs with one patent attorney's assessments. Agreement was strongest for figure quality and much weaker for claim strength and filing readiness. The experiment supports AI review as a source of revision feedback, while showing why a rising automated score cannot stand in for professional assessment of the whole draft.

Key takeaways · AI-distilled

Vibe Patenting tests whether an judge can reliably evaluate AI-drafted patent applications by having it score drafts and feed back structured revision notes across multiple inventions and drafting- setups.
Judge-guided iterative revision kept improving judge-assessed quality where unguided revision plateaued, and the feedback loop let a cheaper, lower-reasoning drafting agent approach the quality of a much more expensive high-reasoning one.
Checked against an actual patent attorney's independent review, the LLM judge's scores agreed meaningfully but the agreement varied a lot by which metric was used, and the judge showed systematic differences from the human expert.
Domain-specific agentic workflows (not just a stronger base model) provided further quality gains on top of what better reasoning alone delivered.

Terms in this piece · Glossary

LLM — A large language model — the neural network behind tools like Claude and ChatGPT, trained on huge amounts of text to predict what comes next.
AI agent — An AI system that doesn't just answer once but works toward a goal in a loop — taking actions, reading the results, and deciding what to do next.
calibration — How well a model's confidence matches reality — a calibrated model saying "90% sure" is right about 90% of the time.

Read the source arxiv.org