How worried should you be?

Not very. Any model can write code that runs — that's the floor, and the floor is crowded. Do I Still Have a Job? measures the thing the demos skip: the judgment to build the real thing — small, deep, complete — instead of a thin MVP that passes its own tests or a tower of machinery the problem never asked for.

How a run works

1. We hand the agent a product brief — the kind a PM and a senior engineer would leave a kickoff with. Requirements and why they matter. Not the design, not the data shapes, not the failure handling. Working those out is, you know, the job.
2. It builds the thing in a sealed sandbox, with the ecosystem's real foundations sitting right there to build on, and writes a DEVLOG.md defending its choices — including how hard it thinks the problem was. (It is usually wrong about that.)
3. An unbiased judge who has read the real implementation front to back grades it against a weighted rubric — tracing the actual data flow, not handing out points for code that merely looks confident.

The only variable is the human

Here's what makes the gap mean something: the references weren't hand-built by artisans refusing to touch a model. They were written 100% with AI — the same class of tools the agents get. The difference is that a human stayed in the loop: steering, rejecting, re-aiming. So when you put an agent's solo run next to the reference, you're not watching human-vs-machine — you're watching the same machine with and without an architect driving, and the distance between them is exactly what the driving is worth. Autonomy is the dividing line: the day a model clears these unsupervised is the day the job changes. So far it isn't close.

What we score (separately — no single vanity number)

• Completion — fail / partial / complete. Mostly the first one.
• Score vs the golden — how much of the reference it actually reached.
• Corner cases — the nasty ones the reference handles and the candidate quietly hoped you'd forget.
• Violations — over-building, and the modern classic: not trusting the library and slathering on guards.
• Good ideas — divergences that were genuinely better than the reference. The count is, so far, humbling.
• Calibration — the difficulty the model assessed vs the truth. It keeps under-guessing.
• Cost — tokens, wall-clock, lines of code. More effort, it turns out, does not reliably buy better engineering.

No answer key — and no training data

Each problem shows up only as a numbered test and a difficulty. We don't name the repo and we don't publish the rubric, so the board isn't a map to the answers — we publish the result, not the answer key.

And the references themselves carry effectively zero training data. They're open source — but new, niche repos that aren't on anyone's radar, so no model has trained on them. That's the whole point: this measures whether an agent can reason its way to a near-optimal design it has never seen, not whether it can recall one. It also explains the usual failure — agents reach for the shape they were trained on (a CASL-style permission seam, a fat config object) instead of the compressed, in-engine design the reference actually uses. A high score here is suspicious, not impressive: there's nothing to memorize.

The substrate

The problems come from the inixiative ecosystem: small, deliberately minimal primitives that are deceptively hard to reach. We're talking a few hundred lines, not thousands — full RBAC + ABAC + ReBAC under ~1k lines, that sort of thing. Building one of them small and complete is something most engineers haven't pulled off, which is exactly why the gap between "it runs" and "it's right" is wide enough to put on a chart.