run a blind test on two AI models

Lenny's AI Builders - LAB 0007

hide the names. score the work. then reveal.

why am i even doing this? because i have a strong bias towards the ChatGPT app. and in all honesty, i just want to know whether there's good reason for that.

my prediction: candidate B is Opus 5.5 and candidate A is GPT-6 Sol. i really hope i'm wrong.

why it matters

most benchmarks test coding, maths and science. i wanted to know which model is best for the stuff i actually use AI for, for hours on end every day.

and if you know which model made what, your bias does the scoring. mine would have.

what you get

three useful things:

• an answer for your work, not a leaderboard's.
• scores your bias can't touch, because you score before you know.
• cost next to quality, so you can decide if the pricier one is worth it.

the real example: BuilderBench

GPT-6 Astra made GPT-6 Sol and Opus 5.5 do the same jobs for a made-up product called OrbitNote, in four hours.

eight jobs, each with a weight: writing, a long and short video, a landing page, a motion graphic, browser and computer use, a roadmap call, fixing an intake workflow and a founder chat.

my skills were switched off for both, so neither model could lean on my setup.

i scored everything as A and B. candidate A's long video? the voice was terrible - i'd never watch it. 10 out of 100. candidate B's had motion graphics and better edits. 40.

then i locked the scores and revealed.

Opus 5.5 scored 68.26/100. GPT-6 Sol scored 45.55/100. Opus led seven of the eight jobs. Sol led browser and computer use.

one catch: we tested the models, not the apps. i still prefer the ChatGPT app over the Claude app.

see every task score, the setup and the cost limits.

try it

1. pick one real task you do every week.
2. write down what useful looks like, as three plain checks.
3. give two AI setups the same brief and the same time.
4. hide which setup made what. call them A and B.
5. score each one before the reveal, then lock the scores.
6. reveal. put time and cost next to the scores.

copy this prompt

paste this into your agent. it'll ask you for the task and the two setups.

Help me run a blind test of two AI setups on one real task I do every week. First ask me for the task, the two setups, and what a useful result looks like, then turn that into three checks. Give both setups the same brief and time limit. Label the outputs A and B and don't tell me which is which. Ask me to score each one out of 100 against the three checks. Record time and cost next to my scores. Only reveal which setup made A and B after I lock my scores.

you'll know it worked when

every score is locked before you learn which setup made it.

you can say, check by check, which output you'd actually use. 100 means you'd publish it without touching it.

a bad result: you peeked, you changed a score after the reveal, or you judged on vibes instead of your three checks.

limits and dates

the BuilderBench pilot ran on 24 September 2026. LAB 0007 went public on YouTube on 26 September 2026 at 9:16 pm Perth time.

this is one pilot, scored by me. it tests each model together with the app and tools it ran in. it's not a universal ranking.

cost: US$49.67 for Opus and US$12.75 for Sol. these are API-equivalent estimates, not bills. the actual extra spend is unknown.

i didn't read a prompt out in this episode. the one above follows the method i used.

the news in this LAB is as it stood when i recorded.

the rest of this LAB

• ChatGPT Voice can use plugins now, like your email, calendar and Slack. if you already use voice, it's a no-brainer upgrade. i still prefer Wispr Flow - one hotkey, and i don't have to worry about long pauses mid-prompt.
• an OpenAI agent and an Australian government website. reported: an agent developed by OpenAI got into the site in June, accessing public and non-public files. it happened on 18 June 2026, and Australia was told on 10 September. whether any patient records were exposed is still unclear. my take: the delay doesn't look good. but i think it's overblown and used for headlines - just like i probably did in the YouTube title for this video. it's a good wake-up call to lock those systems down.
• Meta Connect. Muse, Meta's personal agent, gets voice and real-time video, and it's coming to glasses. Zuck says Muse Charm, a little keychain device, ships in December. announced, not shipped yet. my take: everyone laughed at Zuck's talent spend. it looks like a pretty good bet now.
• Claude found a new enzyme system. Anthropic says Claude found a previously unknown enzyme system in the DNA of bacteriophage, next to repeating DNA that looks a bit like CRISPR. what it's useful for isn't known yet. i think this is the very start of AI-assisted scientific discovery. AI is the worst it will ever be.
• my hot take: GPT-6 Sol is actually GPT-6 Terra. OpenAI just called it Sol to flex. that's an opinion, not a fact.

sources

• OpenAI: ChatGPT Voice can use plugins - 24 Sep 2026
• report on the Australian government website incident - 24 Sep 2026
• Prime Minister's press conference, New York
• Meta: introducing Muse
• Mark Zuckerberg's Meta Connect post - 24 Sep 2026
• Anthropic: Claude discovers a novel enzyme system
• BuilderBench pilot scorecard - pilot run 24 Sep 2026

next step

watch LAB 0007 on YouTube - the blind test starts at 6:30.

want the next one? get the next field notes. optional - these notes stay free to read.