price the job, not the token

from the episode: i gave Sonnet 5.5 4 hours of my real work. it scored 48.98 | LAB 0010

Lenny's AI Builders - LAB 0010

the token price is the sticker. the finished job is the bill.

check the cost per task. that is the guiding star metric.

cheap sticker, but a big bill.

the takeaway is price the job.

why it matters

Sonnet 5.5 costs $2 in and $10 out per million tokens. that's the same price as GPT-6 Sol.

but the price per token is only the sticker. Artificial Analysis says Sonnet 5.5 uses the most tokens they've ever measured - about $7.60 per task.

so a model with a cheap sticker can still run up a big bill. and a cheap model that does worse work isn't cheap. the only way to know is to run your own job.

what you get

you see what one finished job really costs, not the price per token.
you see which model actually finishes your job, and what it skipped.
you stop paying Opus prices for Sonnet jobs.

the real example: Sonnet 5.5 on BuilderBench

BuilderBench is the benchmark for AI builders. it scores real jobs, weighted into one score out of 100. each run is four hours on extra high effort.

i guessed Sonnet 5.5 would get something like 65, or maybe 63.

the results are in, and they're not what i expected: 48.98.

85 for debugging and fixing a skill.
60 for three title and thumbnail pairs.
45 for a Threadify landing page. i really enjoyed how clear the workflow section was. it was just a tad too long.
36 for writing.

LAB 0010 frame at 11:52: how it stacks up - GPT-6 Sol 45.55, Sonnet 5.5 48.98, GPT-6 Astra 53.75, Opus 5.5 68.26 — source: LAB 0010 episode · 11:52 - Sonnet 5.5 next to the other three models

then the price of one run:

GPT-6 Sol: 45.55, $12.75
Sonnet 5.5: 48.98, $17.59
Opus 5.5: 68.26, $49.67
GPT-6 Astra: 53.75, $87.50

that puts Opus 5.5 in a league of its own. among these four, Anthropic has the best offerings right now.

LAB 0010 frame at 12:13: cost vs score for the four runs — source: LAB 0010 episode · 12:13 - what one run cost against what it scored

try it

pick one real task you'd pay a person to do.
write what done looks like as a short checklist.
paste the prompt below with your task and your checklist.
run the same prompt on each model you're choosing between.
for each run, write down the cost, the time and what it couldn't finish.
compare the cost of each finished job, not the price per token.

copy this prompt

this is the prompt i read out at 12:31. fill in the two brackets.

Lenny’s AI Builders / LAB 0010

Here is a real task from my work: [task]. Here is what done looks like: [checklist]. Do the whole job. At the end, list every file you made, how long it took, and anything you could not finish.

LAB illustration: "price the job" - four agents, each holding its own checklist — source: LAB illustration made for these notes with ImageGen · not a screenshot from the episode

you'll know it worked when

you have four things for each model: a file list, a time, a list of what it couldn't finish, and a cost.

you can tick every line of your checklist, or point to the line it missed. and you can say which model you'd pay for this job, and why.

a bad result: it says "done" with no file list. you compared token prices instead of run costs. you changed the task between models. or you treated one run as the answer for every job.

limits and dates

i recorded this on the night of 29 September 2026 (Perth time), about an hour before OpenAI's DevDay keynote. nothing announced at DevDay is in this episode. LAB 0010 went public on YouTube on 30 September 2026 at 6:47 pm Perth time.

the BuilderBench scores are my own scores, still provisional. two jobs couldn't be rated. Sonnet 5.5 ran on BuilderBench v3, GPT-6 Astra on v2, and Opus 5.5 and GPT-6 Sol on v1. so it's not a straight head-to-head.

run costs are recorded usage, not bills.

the $200 plan maths are estimates, not OpenAI's numbers. Sonnet 5.5's benchmark numbers are the ones Anthropic chose to show.

on air i said the Dots leak says "without your approval". the leaked text says "with your approval".

the rest of this LAB

Anthropic's IPO filing. Reuters saw the filing. Anthropic made nearly $4.6 billion last year, and $11.5 billion in just April to June this year. it lost more than $8 billion last year. it has promised to spend $518 billion on computers and cloud, and wants to be worth more than $2 trillion. the founders keep 50.1% of the votes. the filing warns its AI could "resist shutdown, conceal or manipulate information" and "act in ways resembling blackmail". my take: the bill is huge.
OpenAI cancels GPT-6.1 Astra's October release. the Wall Street Journal broke the news: internal tests showed more deception. OpenAI's head of safety systems said it "didn't quite meet the bar". my take: an expensive miss.
OpenAI's Dots agents. leaked text, posted by TestingCatalog, points to agents you can text, call, Slack and email, that can buy things with your approval. a leak before DevDay, not an announcement. my take: most agents end up on cron jobs, eating tokens and doing fuck all. i'm guilty of that.
the $200 Pro plan. Tibo says it will "net out at half the dollar in API spend" compared to the old one. Theo measured Claude's $200 plan at about $9,000 of Opus a month. another creator estimated the old Pro plan at about $4,900, so half is about $2,450. estimates. my take: same price, half the work.
Sonnet 5.5. same token price as GPT-6 Sol. 70.6% on Terminal-Bench 4.0, to Opus 5.5's 66.4%, on Anthropic's chosen charts. Artificial Analysis measured about $7.60 per task. my take: cheap sticker, big bill.
Anthropic's guide. use Sonnet 5.5 for clear, everyday and high-volume jobs, and Opus 5.5 for the hardest, longest ones. start at medium effort. my take: most jobs don't need Opus. the less prompting you do, use Opus. the more you plan up front, use Sonnet.

sources

next step

watch LAB 0010 on YouTube - BuilderBench starts at 8:54.

want the next one? get the next field notes. optional - these notes stay free to read.