Open this card when a tool arrives promising to match the best models. It might be an orchestrator, a layer that splits your task across models from different vendors behind one API, or simply a new model with an impressive benchmark score. The card does not tell you whether the tool is good. It tells you whether it beats what you use today, on your work and with your money.
The blocks follow the order of a test: tasks first, then grading, measurement, interpretation traps, integration, and finally the decision.
1. Write down exactly what you are testing
The vendor's claim is a starting point, not a result. Before you run anything, copy it word for word and separate what the vendor says from what you need to know.
- [ ] Write the vendor's claim in one sentence and note which benchmarks it was measured on.
- [ ] Label it plainly as a claim until you have confirmed it on your own tasks.
- [ ] Establish whether this is a new model or a layer that manages other models; it changes what you should expect.
- [ ] Pick a baseline: the model or tool you use for the same work today.
- [ ] Write down what result would make you switch; without that, any result can be read as encouraging afterwards.
A test without a written switching threshold ends in an impression, not a decision.
2. Build a task set with objective results
A one-to-ten scale invites taste. Pass/fail tasks leave less room for interpretation and give you a comparison that holds up in front of a skeptic.
- [ ] Collect tasks from your own work, not from the vendor's materials.
- [ ] For each task, write a pass condition that can be checked without asking you.
- [ ] Spread the set across types: puzzles, trap questions, work against a spec, heavier algorithms, or their equivalents in your field.
- [ ] Add a few easy tasks your current tool finishes in seconds; they expose the overhead of coordination.
- [ ] Split the set into waves and grade them in turn, so you see early whether the test is heading somewhere useful.
Easy tasks matter as much as hard ones. They are where the cost of a management layer shows first.
3. Hand grading to someone outside the comparison
When one of the systems being compared writes the tasks or grades the results, the test stops being fair. The grader has to be independent of both sides.
- [ ] Choose a grader that is neither of the systems being compared.
- [ ] Ideally, let the same grader write the tasks, so neither side sets its own exam.
- [ ] Give both sides identical prompts and identical inputs.
- [ ] Hide from the grader which output came from which system.
- [ ] Review a few grades by hand to check the grader sticks to the pass conditions.
How fair a test is depends more on who grades it than on how many tasks it has.
4. Measure three things, not one
A quality-only comparison nearly always looks harmless, because a tool that draws on the best models rarely does worse. The real question is what the same result costs.
- [ ] Record the result of every task: pass, fail, tie.
- [ ] Measure the time from sending the prompt to getting the answer, per task and in total.
- [ ] Record cost under pay-as-you-go billing rather than a subscription, so the numbers compare.
- [ ] Work out the cost per passed task for both sides.
- [ ] Flag the tasks with the biggest time gap and check whether they are the easy ones.
A quality tie at a higher cost and a longer wait is a loss, not a tie.
5. Check the interpretation traps
You can misread a result even when the measurement is sound. The most common trap with orchestrators: your baseline sits inside the pool of models the orchestrator picks from.
- [ ] Check whether the model you compare against is one of the models the orchestrator uses.
- [ ] If it is, note that next to the result; a tie then largely means a model compared with itself.
- [ ] Write down what your set does not cover, such as long refactors of a large codebase or team work.
- [ ] Don't carry the result over to kinds of work you did not test.
Every result has limits. Write them next to the numbers before someone reads the numbers without them.
6. Count the integration cost and the limits
The announcement shows quality. It does not show how much work it takes to connect the tool to your agent harness, the program in which a model reads files and runs tools, or how fast the limits run out.
- [ ] Check whether connecting it is a simple swap of endpoint and key or a separate setup.
- [ ] Run a few ordinary tasks in your agent harness: with your skills, research and files.
- [ ] Watch for unusual behaviour, such as a context counter that does not grow over a long conversation.
- [ ] Check how fast the subscription limits fill up at your normal pace of work.
- [ ] Compare that with the limits of the tool you use today.
A tool you cannot connect easily or use for a full working day loses, whatever the test says.
7. Make the decision
The decision rule is short, and it is worth writing down before the test, not after.
- [ ] Same quality, longer wait, higher cost → stay with what you use.
- [ ] Higher quality on the tasks that matter to you → work out whether the gain is worth the extra time and cost.
- [ ] Consider orchestration if you do heavy software engineering and want one vendor's model to plan and another vendor's model to review.
- [ ] Consider it also if many people on your team work on one codebase and already run two tools side by side.
- [ ] For knowledge work, such as writing, research or analysis, require a clear quality advantage before you pay more.
- [ ] Record the result and the test date; in a few months you will rerun it on the same set.
You pay for coordination in time and money. If you cannot see its effect on your tasks, don't pay.
After the test
You are left with a task set you can reuse for every future announcement. That is the most valuable part of the work, because the next test will take a fraction of the time.
The principle I take from this: don't lock yourself into one provider, and don't hand the choice of model to a layer whose decisions you cannot see either. Look for the cheapest model that keeps the quality of each task, and check it on your own data.