How this is scored

Nine areas

Overall DX is round(Σ weight × score / 100). An area that does not apply to a repository keeps its row, its weight goes to 0, and the remaining weights renormalise to 100. Applicability comes from positive signals only: a repository cannot make an area not apply by deleting a file. If the applicable weight falls under 60 of 100, the verdict is UNVERIFIED and no number is shown.

Evidence labels and ceilings

Every number carries the label of the evidence behind it, and the label sets how high the number can go. Observed means we executed it in a sandbox from a clean clone and can reach 100. CI-observed and Reviewed — a passing CI run on the audited commit, or review of the code and its generated output — cap at 84. Estimated — prose only, because the sandbox was skipped, timed out, or the planner had to guess — caps at 69. An estimate never proves a verdict.

Bands and verdict

The verdict is orthogonal to the band and computed in code, never by a model. FAIL: any hard gate fired. PASS WITH DEBT: no gate, but P2 or P3 findings remain. PASS: no gate, no P2. UNVERIFIED: the sandbox and CI evidence were both unavailable for a hard gate, or the applicable weight was under 60. A score of 85 or more displays as World-class candidate only when every gate passes, every applicable area is at least 75, no P0 or P1 is open, and the magic path was Observed or CI-observed inside 15 minutes; otherwise it displays as Good with a note.

Hard gates

A gate is a defect serious enough that no average can wash it out. One fired gate makes the verdict FAIL, whatever the number says — and the number is never clamped, because clamping would make it lie in the other direction. A gate fires only on an observed failure or an observed elapsed time beyond the standard's limit. We execute for at most 10 minutes per audit; the standard allows 15 for the magic path and 10 for local development. Running out of our budget is UNVERIFIED for the affected gates and a P2 finding, never FAIL.

Private repositories

Sign in with GitHub asks for your profile only. Auditing a private repository needs the OAuth repo scope, which GitHub grants for all your private repositories at once — there is no per-repository option for OAuth apps. We say so before you decide, use the token in one step to download a tarball, encrypt it at rest, never log it, and never let it into the sandbox. Per-repository access via a GitHub App is on the roadmap.

How often we produce a number

We watch eleven public repositories. A repository is on the list if it is public, widely used, and audits inside our execution budget; nothing is taken off it for scoring badly, so the FAILs stay. One has been removed under that rule — vercel/ai is 368 MB and its install has never finished inside the wall on any engine, so it only ever produced the budget finding twice over. It is excluded for being unauditable by us, not for its number: it never had one.

Of those we do score, a number is produced for roughly half. The rest return UNVERIFIED, and it is worth being exact about why: none of them is marked unverified for anything the repository did wrong. Every one is a limit of ours.

Some run a test suite for longer than the run allocated it — openai/openai-node at 180 seconds, sindresorhus/execa at 203. Some need a tool the sandbox did not provide: stripe/stripe-node documents just, and one run of vercel/serve invoked a bare binary that was not on the path. One run of vercel/workflow was given npx workflow web as its first-value step — a development server, which never exits and so was killed.

A missing tool or an exhausted budget is our failure to observe, not the repository's failure to work, so it reads unverified and fires no gate. That is the rule the whole product rests on, and six of twelve is what it costs.

The largest single cause of variation between runs is the planner. It proposes a journey for each audit, and a different journey measures a different thing: vercel/serve scored 69 on one run and returned no number on the next because the plan invoked serve rather than npx serve. Three runs of vercel/swr at one commit produced identical gate results, scores of 75, 73 and 71, and time-to-first-value readings of 87, 49 and 50 seconds. A repository's journey is now pinned once it produces a clean run, so a re-run repeats that plan instead of drawing a new one and measures the same thing twice. That removes the variance; it does not measure it, and this page will not claim more stability than the pinned path gives.

The calibration set

Five repositories are the calibration set: vercel/swr, vercel/ai, vercel/workflow, better-auth/better-auth and sindresorhus/execa. They are audited by hand today; the nightly job that watches them for drift is not built yet, so treat the numbers below as a baseline rather than a running series.

The first baseline, on engine 0.1.0: swr scored 75 and passed with debt; workflow scored 40 and failed, because its build exits 1 after a clean install; better-auth scored 34 and failed; execa and ai are both unverified, because our own budget cut them off before the run finished. All four verdicts appear in five repositories.

The stability standard is that the same repository and commit audited twice lands within 3 points with identical gate results. Three consecutive runs of swr scored 75, 73 and 71, with identical gates every time. The gate half holds; the number is one point outside on the widest pair, and the variation is in the planner rather than the scorer. We rank no one, and there is no leaderboard. We tried one, ordered by time to first value on the argument that a stopwatch is steadier than a composite; the same three runs timed 87, 49 and 50 seconds, because the planner proposes a different journey each time. The metric picked for being stable was the noisier of the two, so the board shows readings in a fixed order and sorts nothing.

The scores we do publish are on the boards: a watched set of public repositories, each with the evidence label that produced it, in a fixed order that nothing sorts by score.