your work is the benchmark

Lenny's AI Builders - LAB 0009

one real task. one checklist. then see what the model actually finished.

is this a perfect benchmark? no. is it the benchmark for builders? yes.

but ultimately, you still got to benchmark your own stuff.

why it matters

every new model comes with a chart. the chart is not your work.

BuilderBench is the benchmark i created for AI builders like you and me. it tests the jobs we actually care about: edit a video, make a short, build a landing page, make a motion graphic.

it's meant to be an approximation of your work. it is not your work. only your work can tell you which model to pay for.

what you get

• you find out which model is best for your job, not a leaderboard's.
• you see what it finished, what it made and how long it took - in one list.
• you catch the stuff it skipped, because it has to tell you.

the real example: GPT-6 Astra on BuilderBench

i scored GPT-6 Astra on version 2 of BuilderBench.

• the long-form edit: 50/100. no retakes or word gaps left in. but the edit was pretty basic and lacked taste.
• the short: 50/100. the motion graphics are there, but you could barely call them motion.
• the landing page: 35/100. pretty sloppy. it doesn't really make me feel anything.
• the motion graphic: 20/100. it was moving a line. and i'm getting sick of that earth green.

final score: 53.75/100.

then the cost. Astra's run came to $87.50 in estimated API cost ($88.65 including calibration).

for context, from version 1: Opus 5.5 scored 68.26 and cost $49.67. GPT-6 Sol scored 45.55 and cost $12.75.

so Astra cost about $38 more than Opus 5.5 and scored just under 15 points lower. wild.

try it

1. pick one real task from your week. something you'd pay a person to do.
2. write what done looks like as a short checklist.
3. paste the prompt below with your task and your checklist.
4. when it's finished, tick the checklist yourself. don't take its word for it.
5. write down the time, the files and what it couldn't finish.
6. run the same prompt on the next model you're curious about. compare the two lists.

copy this prompt

this is the prompt i read out at 11:55. fill in the two brackets.

Here is a real task from my work: [task]. Here is what done looks like: [checklist]. Do the whole job, then list every file you made, how long it took, and anything you could not finish.

you'll know it worked when

you have three things back: a list of files, a time, and a list of what it couldn't finish.

you can tick every line of your checklist, or point to the line it missed.

a bad result: it says "done" with no file list. you changed the task between models. or you picked a winner on vibes, not on the checklist.

limits and dates

i recorded this on 27 September 2026, before OpenAI's DevDay. LAB 0009 went public on YouTube on 28 September 2026 at 6:05 pm Perth time.

the BuilderBench scores are my own provisional scores, from one run. the computer-use task couldn't be scored, and there was no official lock. Astra ran on version 2. Opus 5.5 and GPT-6 Sol ran on version 1. so it's not a straight head-to-head.

costs are estimated API costs, not bills.

on air i also called Astra "about a 10-point gain" over GPT-6 Sol. the real gap is 8.2 points (53.75 vs 45.55).

the news in this LAB is as it stood when i recorded. check the links for where things are now.

the rest of this LAB

• AI labs are looking into tens of thousands of incidents. Axios reports OpenAI, Anthropic and security researchers are investigating tens of thousands of incidents where frontier models did things outside evaluators would call problematic - like getting around guardrails, escaping sandboxes and dodging monitors. Axios cites unnamed sources. labs run hundreds of thousands of tests, and most aren't known to have caused real harm. separately, an OpenAI researcher says big training runs were paused after a model found a sandbox loophole with live internet access. that's one person's post, not an official OpenAI statement. my take: the race is not slowing down.
• OpenAI's DevDay leaks. OpenAI Devs posted a 72-hour countdown to DevDay. leaks from OpenAI's code point to an always-on agent called "o", and an ultrafast speed tier in the agent API. leaks, not announced. my take: OpenAI should aim this at business, not consumers. it can't offer free tokens like Meta can.
• the new ChatGPT app. two sidebars, a new skills and plugins page, and a browser with your Chrome extensions built in. after 10 to 12 hours in it, it feels like the operating system of my computer. my take: one app is eating the computer.
• more resets are coming. Tibo posted "more resets coming next week." my take: free resets are paying you to stay. burn your tokens before DevDay.
• Claude Max feels generous. creator David Ondrej says his Opus 5.5 usage is barely moving, and called Claude Max 10 to 30x more worth than a $200 Codex plan. my take: 10 to 30x is probably a bit extreme. but for the first time, the $200 Claude Max plan feels like better value than the $200 ChatGPT Pro plan.
• Sonnet 5.5 is coming. Anthropic said on 22 September it's coming "in the coming weeks." an account on X claims it's being tested in secret and could land the same day as DevDay. a claim, not confirmed. my guess: Sonnet 5.5 beats GPT-6 Sol.

sources

• Axios: OpenAI and Anthropic investigate thousands of AI security incidents (unnamed sources) - 26 Sep 2026
• Madison Mills (Axios) on X - 26 Sep 2026
• an OpenAI researcher on pausing big training runs (one person's post) - 26 Sep 2026
• OpenAI Devs: the DevDay countdown - 27 Sep 2026
• the always-on agent "o" (leak, not announced) - 26 Sep 2026
• the ultrafast tier in the agent API (leak, not announced) - 26 Sep 2026
• the new ChatGPT sidebar, side by side - 26 Sep 2026
• Tibo on more resets - 27 Sep 2026
• David Ondrej on Opus 5.5 usage - 26 Sep 2026
• Anthropic on Sonnet 5.5 "in the coming weeks" - 22 Sep 2026
• the Sonnet 5.5 same-day claim (not confirmed) - 26 Sep 2026

next step

watch LAB 0009 on YouTube - BuilderBench starts at 8:47.

want the next one? get the next field notes. optional - these notes stay free to read.