NerfWatch()
nerfwatch.lol- Category
- Developer Tools
- Rank
- No. 2757Tools index
Previous survey · No. 2701 ·
- Type
- APP
- Use case
- Model & Agent Evaluation
- Date
About
NerfWatch() runs the same benchmark test against several AI models (such as GPT, Grok, Muse, and Claude) every day and compares each model only against its own first week of scores, aiming to surface silent quality degradation ("nerfing") over time. It pairs the benchmark numbers with community votes on whether a model feels degraded, and offers an email alert when a model's verdict changes.
At a glance
NerfWatch runs automatically graded prompts on a schedule and compares a model with its own starting baseline. It is worth checking for a signal to investigate when a model feels different.
Good fit
People monitoring a model over time who want to separate test results from community impressions.
Consider something else when
Use your own regression suite when the question is whether a model still handles your specific workflow.
Start here
Pick one model and inspect its baseline and run count. If the score moves, compare the failing task types and repeat a few examples from your own work.
Cost and requirements
The project publishes a public dashboard and source for running your own tests. Self-run tests use model API access and may incur charges.
Limits to know
A change on these synthetic questions does not establish a broad capability change or explain its cause. Small run counts and community votes are separate evidence.
Sources checked 2026-11-05. Source-based assessment.
Why it made the leaderboard
Runs the same benchmark against major models every day and pairs it with community votes on whether a model 'feels' nerfed, giving practitioners an ongoing, independent signal on silent model quality regressions.
Tags
Tech Stack
Comments (0)
No comments yet
Editorially curated, with community endorsements as a secondary signal. Corrections welcome.