Keep this open in a second window while you decide. The first three blocks pick the tool, the rest decide whether you can walk away from it, and the order matters in both halves.
Where a detail below belongs to one product it is OpenAI's Codex desktop app. The questions belong to the task and will outlive whichever app you use.
1. The task, before you pick anything
You cannot choose a tool for a task you have not written down. Do this part on paper, with no app open, because the tool you already like will answer for you otherwise.
The task: ______ How often: ______ What one manual run costs me: ______
What it produces: ______ Where the output has to land: ______
- [ ] The task is one you actually did more than twice last month, not one you expect to start.
- [ ] You can write the whole flow as numbered steps a colleague could follow unaided.
- [ ] The output is a specific object: a named file, a row in a system, a message to a person.
- [ ] The destination is named. A folder path or a system, not "wherever it usually goes".
A task you cannot write as steps is not ready to automate. It is ready to be written down.
2. The three questions, in order
Ask these top to bottom and stop the moment one of them answers yes. They work as a ladder: every rung below the one that answered costs you more money and more variance for the same result.
Does the tool have an API?
- [ ] Open the tool's documentation and look for an API or integrations page before concluding there isn't one.
- [ ] Check your automation app's plugin list. Many APIs arrive there already wired up.
- [ ] If you found one, stop here and build against it. It is the fastest and most consistent route.
If there is no API, are your clicks identical every time?
- [ ] Run the task by hand once and write down every click, in order.
- [ ] Run it again on a different day and compare the two lists.
- [ ] Mark every step where the two runs differed. Zero differences means you are on this rung.
- [ ] Ask a coding assistant for a script that repeats exactly those steps. You get it back in minutes, even if you have never written one.
- [ ] Stop here if the script completes. It costs almost nothing to run and it cannot surprise you.
Only if the screen has to be read and judged do you go lower.
- [ ] Name the step that needs judgement in one sentence: the layout shifts, a dialog appears sometimes, the next step depends on what came back.
- [ ] If you cannot name that step, go back up. You are choosing the interesting option, not the right one.
- [ ] Write the flow out in plain language, naming the page, the button, the file, and the destination.
- [ ] Budget for it being slower and less repeatable than either answer above.
Most tasks stop at the first question. The mistake I see most often is starting at the last one, because the last one looks like the future.
3. The exception the ladder does not cover
Cheapest and most reliable do not always point at the same rung, so treat the ladder as the order you test in rather than a law. A browser agent can beat an API that exists and works, for reasons worth checking before you commit.
What the API charges per run: ______ What an agent run costs on top of what I already pay: ______
- [ ] Check what the API bills per call, and whether that lands on top of a subscription that already covers the agent.
- [ ] Check whether the API produces the formatting you need, not just the text. Awkward formatting is a common reason teams give up on one.
- [ ] Check whether it can attach the media, the file, or the image the task requires.
- [ ] Count the time spent fixing a mangled API result as part of the API's cost.
- [ ] If the evidence points lower, take it and write down why, or somebody undoes it in three months on principle.
The ladder tells you what to test first. It does not tell you what to end up with.
4. Credentials, and why a chat log is a document
A conversation is a document. It is stored, synced, scrolled back through, and every so often pasted into a ticket by someone in a hurry. A password that has been in one is a password you no longer control, and deleting the message later does not change that.
- [ ] Scroll back through the conversation and confirm no password, key, or one-time code was typed into it.
- [ ] Import your logins into the agent's password store instead. One row per login: name, URL, username, password, and a notes field you can leave empty.
- [ ] Confirm the agent fills the login form from that store and the password never appears in the conversation.
- [ ] Give the agent its own working account for each service that allows one, rather than your personal login.
- [ ] Rotate any credential that has been in a chat, instead of deciding it is probably fine.
- [ ] Know which single entry you would pull to cut the agent off from one service.
The password store is the rule here, including on the day you are in a hurry.
5. Sessions, and the two hard stops
Signing in once is what makes any of this usable: you log in by hand inside the agent's browser and the next run starts already inside the account. That holds for most of the everyday web, and it fails deliberately on the platforms you would most like it to work on.
- [ ] Sign in by hand once, then start a fresh run and confirm it opens already logged in.
- [ ] Test again a day later. A session that survives an afternoon may not survive a night.
- [ ] For each account in the flow, note which of the two it is: a session that persists, or a platform that ends it on purpose.
- [ ] Where a platform ends sessions on purpose, plan for a human to sign in and hand over.
- [ ] Mark any step that asks for a code from your phone as a hard stop, and design the flow to finish before it or begin after it.
- [ ] Do not route the second factor to the machine the agent controls.
A second factor works precisely because a machine cannot complete it alone. An automation that gets around it has deleted the thing you were paying for.
6. Watch it work before you look away
One successful run proves nothing. There is no recorded sequence underneath, only a decision made fresh at every step, so the run that matters is the one where the page is not what it was yesterday.
- [ ] Run it headed and watch the whole run, not just the final screen.
- [ ] Run it again on real data, more times than feels necessary, before you leave the room.
- [ ] Note every point where it hesitated or backtracked, and add a sentence to the instruction that removes the doubt.
- [ ] Force an awkward run: a slow load, a banner in the way, a missing item, a moved button.
- [ ] Confirm what it does when it cannot finish. Stopping and saying so is the behavior you want; clicking something plausible is the one that costs you.
- [ ] Open the output and check it. A report that the file was downloaded is not evidence that the file is right.
You are not checking that it can do the task. It already did it once.
7. Instructions strict enough to leave alone
The agent decides from a picture of a screen, so every sentence you leave loose is a decision you have handed to a guess. Tightening the wording is the cheapest reliability work you have, and the step most people skip once the demo works.
- [ ] The instruction names the exact page, the exact button, the exact file, and the exact destination folder.
- [ ] Read it back cold and circle every noun a stranger could read two ways. Replace each one.
- [ ] It says what not to touch, not only what to do.
- [ ] It says what to do when something is missing: stop and report, never improvise.
Raise the bar when the run touches: money ______ documents ______ anything that leaves the company ______
- [ ] For anything in those three, decide the number of watched runs in advance instead of settling for "a few". The higher the stakes, the longer I would watch.
- [ ] Anything irreversible (a payment, a send, a deletion) stays a step you press yourself.
Strict instructions are what turn a demo into something you can walk away from.
The first unattended week
Come back to these after the first run that happened while you were not watching.
- [ ] Compare the output of the first unattended run against the last one you watched, line by line.
- [ ] Put the flow on a schedule only after it has survived a run nobody supervised.
- [ ] Save the working flow as a named skill so the second automation costs a fraction of the first.
- [ ] Count the payoff honestly. A single run saves minutes, not hours. What you stop paying is the interruption of leaving your own work and finding your way back to it.
- [ ] Re-ask the first question every few months. A tool with no API today may ship one, and moving back up a rung is a win.