The Five-Part Automation Test

Origin

Sparked by a hand-drawn note: a 5-part test for what should become an AI agent — repeated trigger, stable inputs, clear tools, measurable finish line (automation-ready), plus judgment required in the middle (agent, not just automation). I hadn't written this down before. But I'd been doing most of it already, unnamed, across a few weeks of building real automation scripts. This is that practice, written down.

The 5-part test, stated plainly

A task is automation-ready if it has all four:

1. A repeated trigger — it happens on a schedule or a recurring event, not once.
2. Stable inputs — the shape of what comes in doesn't change unpredictably.
3. Clear tools — there's an unambiguous way to act (an API call, a file write, a query).
4. A measurable finish line — you can check, objectively, whether it worked.

If a fifth thing is also true — real judgment sits in the middle of the task (should this actually go out, is this the right call, does this need a human's read) — the task doesn't stay "Automation." It becomes something that needs to be an agent with a check in it, not silent, unattended automation, and not full autonomy either.

Where this differs from how automation is popularly discussed

Most public automation content — the viral screenshots, the "I automated my entire workflow" threads — is:

• Demo-and-hype-first. You see the win. You almost never see what didn't ship, or why.
• Tool-first, not judgment-first. The conversation is about which tool did it, not about how anyone decided what needed a human.
• Build = ship. Getting the code working and putting it into someone's actual workflow are treated as the same moment.
• Survivorship-biased. Nobody posts "I built this and it sat unused for two months."

My own practice, once I actually looked at it, was closer to an engineering discipline than a demo reel — mechanisms for judgment, honest tracking of what never got used, caution calibrated to actual risk rather than to how impressive something looks. That gap is the actual content here.

The core thesis

A judgment step that's named but has no forcing mechanism attached to it doesn't happen. Not "might not happen" — in my own experience, twice, it simply didn't. Writing "a human should review this before it goes out" in a spec is not the same as building something that makes that review actually occur. The second one needs a deadline, an owner, or a timer. The first one is a hope.

Four principles, with real evidence

1. Calibrate caution to blast radius, not novelty

A brand-new script I built got armed — turned fully live, unattended — the same day it was finished. A different script, also fully built and fully tested, sat deliberately un-armed for a week. The difference wasn't how new or interesting either one was. It was what happens if each one is wrong: the first only reads data and reports back to me privately — if it's wrong, I notice and shrug. The second can send a real message to a real person on my behalf, unattended — if it's wrong, someone outside my own head sees it. Match the caution to who gets hurt if it's wrong, not to how impressive the build is.

2. Build ≠ armed

Finishing the code and turning the system on are two different decisions, and I now treat them that way on purpose. A script can pass every test, handle every edge case, degrade gracefully on every failure — and still sit un-launched, because "does this work" and "should this be live, unattended, right now" are different questions with different owners. Almost all popular automation content collapses these into one moment. I don't anymore.

3. A review step needs a timer or an owner, not a hope

Two real examples, from my own work:

• I built a pipeline that generated dozens of genuinely good, personalized outreach messages — real, usable output. The plan was "review before sending." Nobody attached a deadline or an owner to that review. It never happened. The messages are still sitting there, unsent, months later.
• A separate orchestration script I built had one line in its own checklist: "present to execs before any auto-send." That line sat unchecked for six straight weeks. The whole pipeline just went quiet, waiting on a step that had no mechanism forcing it to occur.

Contrast that with a different script I built later: instead of "review before sending" as a hope, it drafts the message, pings me directly with a clear "send in 12 minutes unless you say no," and only sends if I don't respond. Same underlying judgment call — is this okay to send — but one version is a mechanism and the other is a wish. Only the mechanism actually works.

4. Fail-closed is a house standard, not a per-feature choice

The rule "if anything is uncertain or breaks, do nothing rather than guess" shows up in two very different places in my own scripts, which is exactly why I trust it: once as protection against sending the wrong thing to someone (an error means no send, full stop, fall back to a normal manual process), and once, completely separately, as protection against corrupting my own data (a tracking file that a script updates every day correctly chose to leave the file untouched, rather than overwrite it with a bad or partial result, the one time an underlying API call failed). Same principle, unrelated failure types. That's what makes it a real design standard instead of a one-off patch.

What this means if you're building your own automation

Don't stop at naming the judgment step. Everyone gets that far — "a human should check this" is in nearly every spec ever written. Go one step further and ask: what specifically forces that check to happen? A deadline with teeth? A timer? A named owner who's actually accountable? If the honest answer is "I'm counting on someone remembering," that step will not happen — not because people are careless, but because a hope isn't a mechanism. Build the mechanism, not the hope.

And publish your real numbers, not just your wins. Out of the automation scripts I've built recently, a meaningful share never ran for real at all — built, tested, and simply never used in production. That's not a failure worth hiding. It's the actual, honest state of the practice, and it's more useful to anyone learning from this than another highlight reel.