promptcheckup

About

A Python tool that checks whether a prompt change made an AI agent better. It runs the old and new prompt on the same cases and reports better, worse or not sure yet with a 95% confidence interval. Running both prompts on the same inputs cancels out shifts such as model updates or changing traffic, which would otherwise make week-to-week comparisons misleading. The gate command returns exit codes so a pull request that makes the agent worse can fail CI.