Drift Watch
A grade tells you how your agent did on our tasks. It can't see your live traffic, and no judge sits on every real answer. But two numbers can be watched on live traffic without judging anything, and a jump in either is nearly always something real.
Thresholds
Change these and press Check again. On quiet tasks a small move can be just a couple of requests, so keep the first one at a few dozen.
The two checks
How often the agent says it can't help or doesn't know. A rise on a task it used to answer usually means a tool broke, a source went away or the model got more cautious. A fall can mean you fixed something, or that it has started guessing. And zero refusals on questions about data is its own warning, because some of those questions should get "I don't know".
It barely moves for months, so a change of 30% or more is worth a look. Up and down mean different things. Longer is usually a tool returning more, a prompt edit or a model swap. Shorter is flagged louder, because it can be the model quietly cutting answers off.
With timestamps we compare the last 7 days with everything before. Without them we compare the last third of your rows with the rest. We only compare a task once each side has at least 30 answers, because on a quiet task a 10 point move can be two requests. You can change all three thresholds on the page, or send them as options to the endpoint. Refusals are spotted from the wording near the start of each answer. On the 33 graded answers to our "unknowable revenue" question it caught 25 of the 27 honest answers and none of the 6 made-up ones. If you already know which answers were refusals, send refused and we use that instead.
What to send
One row per answer with a task (or type, category, route) and the answer (or output, response). Add time if you have it. JSON lines, a JSON array or CSV all work. Don't want to share the text? Send length and refused instead of the answer.
From your code
The same check as a free endpoint, so you can run it from a nightly job and alert yourself when the status is alert:
POST https://agent-report-card.vercel.app/api/drift/
{"rows": [{"time": "2026-11-18T09:00:00Z", "task": "price-lookup", "answer": "The price is $42."}, ...],
"options": {"minPerWindow": 30, "refusalPoints": 0.1, "lengthChange": 0.3}} (options are optional)
→ {"status": "alert", "tasks": [{"task": "price-lookup",
"baseline": {"n": 168, "refusalRate": 0.04, "avgLength": 28},
"recent": {"n": 42, "refusalRate": 0.62, "avgLength": 28},
"flags": [{"kind": "refusals-up", "message": "Refusals rose from 4% to 62%. ..."}]}]}Up to 20,000 rows a request. We don't keep anything you send.
Then get graded
Drift Watch tells you something changed. A regrade tells you whether the change made the agent better or worse, with independent judges scoring the answers.