Where I got stuck
I built this on Vercel. The parts I expected to be hard were the parts that gave the least trouble: isolated execution, durable runs, keeping credentials out of the sandbox. These are the places I lost time anyway. Two of them turned out to be my own fault and say so. I added no external observability tooling, and the one thing I did reach for when the platform couldn’t tell me something was my own Postgres, which is what Vercel’s own guidance recommends anyway.
Nothing tells you a route has gone quiet
This is the one I’d most like fixed and I ran into it twice. The first time was small. A submit button did nothing, so I went to the runtime logs to find the failing request and there wasn’t one. No request at all, which turned out to be the diagnosis, though it only worked because I already knew what should have been sitting there.
The second time I didn’t catch at all until very late on. I was reading production logs for an unrelated reason when I noticed that the part of this product that does the actual work, the bit that clones a repository and runs its documented setup in a sandbox and times it, had never once been invoked on the deployment. Every report on this site was computed on my laptop, against the same database the deployed app reads from. The only evidence was a single request path that quietly wasn’t happening. What did I reach for? Nothing existed to reach for. I went and ran one by hand. It worked first time and produced 90 workflow invocations that were all fine, so nothing was broken and nothing ever needed fixing. I had just never checked.
Alerts only fire above a baseline. The usage rule wants volume four standard deviations above a 24-hour average and the error rule wants a rate unusually high against its recent one, so neither has anything to say about a number that went to nothing. There’s a guide called Monitor uptime for AI-native apps with Vercel Alerts and it can’t detect a route that stopped being called. I know the obvious objection is that this is what uptime monitoring is for, but a prober only tells you a route still answers, not that anyone is reaching it.
A warning fired, and pointed at the wrong thing
The cause of that first absence was mine. I skipped step two of a four-step setup. BotID wants redirects configured through withBotId() in the Next config on top of the client-side protection and the server-side checkBotId() I’d already added, and without the redirects the challenge script 404s, the client never gets a token, and the patched fetch gives up before it opens a connection. Hence a network error in the browser and nothing on the server.
There was a warning, and I want to give it credit before I complain about it. checkBotId()noticed something was wrong and logged Possible misconfiguration of Vercel BotId. But what it told me to do was ensure my client-side protection was properly configured, and it printed the { path, method } entry to add to it. My client-side protection was already correct. The missing piece was the config wrapper, which the warning never mentions. So a message that was right about there being a misconfiguration sent me to inspect the one part of the setup that was fine, and I discounted it and went back to the browser.
It fires on a request arriving without the header, which can mean a missing rewrite, a missing client entry, or an actual bot, and it names only the second. That is a hard thing to get right, and I don’t think the answer is a longer message.
The model kept returning nothing, and nothing was wrong
Three audits in a row came back with no score. Not an error, just a clean valid empty answer. I assumed the prompt was bad and rewrote it, then assumed the model was bad and switched models, then finally started printing the raw response, which was {} every single time and perfectly schema-valid. The schema was a Zod partialRecord, and compiled to JSON Schema that becomes an object with propertyNames and no properties, so what I had actually described was a thing with no named keys for anyone to fill in. The model was doing exactly what I asked.
The SDK isn’t hiding anything. NoObjectGeneratedError carries a finish reason, its cause is a validation error with the Zod issues still attached, and both were sitting there while I chased the schema instead. What tripped me up was the message string, which reads response did not match schema whether the response genuinely didn’t match or was simply cut short, so the obvious next move of rewriting the schema is the wrong move about half the time.
properties, since that is a shape nothing can ever fill and you can tell before the request leaves. Putting the finish reason in the message would fix the rest.Three sources, three answers, one property
I built a polling loop around getCommand(id).exitCode and it told me every command was still running, forever. So I went to find out what the property was supposed to do, and got a different answer everywhere I looked. The SDK reference says it starts as null for detached commands and is populated after you await wait(). The REST specification says a plain GET returns the command data along with the exit code if the command did finish, which is the opposite. The TypeScript type is number | null and stays that way regardless, because getCommand() never narrows to the finished variant, and the property carries no comment, so the editor settles nothing.
What I observed matched the SDK reference and not the REST spec. Reading them in the other order would have cost me the afternoon in the other direction. While I was in there I hit the neighbouring one, which is that reattaching to a command replays its logs from the beginning: there is no offset, cursor or resume parameter anywhere in the SDK or the REST API, and nothing in the docs mentions it, so every reattaching consumer has to build its own dedupe. My first attempt at streaming output cheerfully duplicated everything.
null is a bug rather than a documentation gap.Nothing tells you the model answered differently this time
The hardest question in this whole project was why the same repository at the same commit scored differently on consecutive runs. I went looking for the failure and there wasn’t one. Every run succeeded, on time, with nothing unusual anywhere. The variance was in the content, because a model reading the same documentation proposed a different set of commands each time, so every run was measuring something slightly different. Three runs of one repository at one commit gave identical gate results, scores of 75, 73 and 71, and time-to-first-value readings of 87, 49 and 50 seconds.
Half of what I want already exists, which I found late. AI Gateway takes tags you supply and reports cost and tokens grouped by them, and that was the part I had assumed was missing. What it can’t tell me is that the answers differed, and Vercel’s own guidance on building evaluation dashboards says as much, that the Gateway doesn’t retain prompt and response content or join runs across time and that this work belongs in your own database. That is a defensible default. It is also exactly why nothing on the platform can see this kind of failure.
So the fix I shipped isn’t a measurement, it’s an avoidance: once a repository produces a clean run its journey is pinned and re-runs repeat that plan rather than drawing a new one. That makes two runs comparable by making them the same run, which is useful and is not the same as being able to see the drift. Detecting it, rather than routing around it, would mean keeping the plans and diffing them, and that is work I haven’t done. I don’t think the answer is for the Gateway to start retaining everyone’s content by default. I do think an application that returns 200 every time while quietly doing something different is a failure mode the platform has no vocabulary for yet, and I would rather be told I’m wrong about that than carry on believing it.