Writing
How to Pick an AI Model: A 15-Minute Bake-Off
updated 2026-08-12
Every model launch comes with a leaderboard victory lap. Fine. I still have to ship something by Friday.
Benchmarks and day-to-day usefulness measure different things. A model can win the exam and still feel exhausting in a real session. It over-apologizes. It stops early. It asks you to confirm a one-line fix. That friction is a real cost. It never shows up in ARC scores.
This page is a how-to. Use it when a new model drops and the chart says you should switch.
What I optimize for
Score the model on your work, not on the internet's favorite exam.
- Does it stay out of my way? If I fight the personality more than the problem, it is not an upgrade.
- Is it better than what I already have? Most people are not freely shopping across labs. Ask "is this better than today's default?" not "did it beat the internet this week?"
- Can I turn the dial down? Effort and thinking settings are not linear. I have seen peaks below max. I have seen cheaper settings that barely lose quality. Try mid before max. Keep whichever wins on your task.
- Is the harness cleaner than last time? Sometimes the win is not the weights. It is a shorter system prompt. Cut one instruction. Re-run the same task. Keep the trim if quality holds.
A 15-minute bake-off
When a new model drops, do this instead of reading the chart.
- Pick one real task you already know well. A pull-request review. A refactor. A draft rewrite.
- Run it on your current default and the contender. Same prompt. Same context.
- Score only what you feel: time to useful output, how often you had to steer, and whether you would happily do it again tomorrow.
- Keep the winner as default for a week. Revisit if the work changes.
If a model solves the task with a strategy nobody else tried, that gets my attention more than another #1 badge. Leaderboards tell you who won the exam. A weird new approach tells you the model might actually be exploring.
Keep a tiny scorecard. Three numbers. No essay.
| Signal | What you count | Winner |
|---|---|---|
| Time to useful output | Minutes until you would paste, commit, or send | Lower is better |
| Steering | Times you had to correct course | Lower is better |
| Would I use this tomorrow? | Yes or no | Must be yes |
If the contender wins the chart and loses this card, keep your default.
What the charts miss
Public coding benches (SWE-Bench, Terminal-Bench, and friends) are useful as a first filter. They are not the job.
They do not measure:
- How often the model asks you to confirm a tiny change
- How often it stops before the task is done
- How much of the session you spend fighting tone
- Whether a cheaper effort setting is good enough
- Whether a bloated system prompt is the real bottleneck
A model can look like it beat the field on paper and still lose a bake-off on those points. I have seen that gap in public reviews of frontier launches. The pattern is stable: benchmark leadership and practical usefulness are different measurements.
What I am not doing
I am not pretending capital-markets headlines or lab drama help me pick a model for a ticket. Fun to watch. Wrong input for the decision.
If multi-model routers keep winning, "which model should I use" content has a shelf life anyway. The durable skill is knowing what good looks like in your workflow — speed, reliability, tone, cost — and swapping tools when those signals change.
Benchmarks are not useless. They are just not the job. The job is useful output without wanting to throw the laptop.
FAQ
How do I pick an AI model for real work?
Run a 15-minute bake-off. Take one task you already know. Run it on your current default and the new model with the same prompt. Keep the one that gets you to useful output faster with less steering.
Are AI benchmarks useless?
No. They are a first filter. They tell you who won a public exam. They do not tell you how the model feels on your ticket, your repo, or your writing.
Should I always use the highest reasoning or effort setting?
No. Try a mid setting first. Effort is not linear. A cheaper setting often holds quality on a task you already understand.
What if the new model wins the leaderboard?
Still run the bake-off. Leaderboard wins can hide a personality you will not want to live with. If you would not happily use it tomorrow, it is not your default.
How often should I switch models?
When the work changes, or when a bake-off beats your current default for a week. Do not switch because a launch thread told you to.
Sources
- AI Daily Brief podcast, "Where Should Claude Opus 5 Fit In Your Model Rotation." The episode is where I first wrote down the bake-off rule. The "benchmarks vs real use" split is the reusable claim. I do not copy the episode's recap of lab news.
- Public model-launch reviews that keep showing the same gap: a model can win a bench and still feel neurotic in a working session. Treat those reviews as signal, not as a shopping list.