Case Study · Agentic Operations
I don't just build agents.
I operate them.
A fleet of Claude agents works on my systems every night while I sleep. This is the operating system of rules, ledgers, verifiers, and kill switches that makes that safe — and the receipts that prove it runs.
Live figures as of July 2026, from the system's own ledgers.
The problem nobody demos
Every AI portfolio in 2026 has an agent demo. Almost none answers the question that decides whether agents create value or incidents: what happens when nobody is watching?
An agent with shell access, working unattended at 3 a.m., will eventually face a choice the demo never showed: commit a 1,400-line "cleanup," delete the failing test that looks outdated, invent the env-var name it can't find. Intelligence doesn't prevent any of these. Operations does.
So I built — and run, daily — an Agentic OS: the permanent layer of files, checks, ledgers, and loops that lets frontier models work alone on my production systems and be trusted to. The models supply the intelligence. The OS supplies the honesty.
The architecture
Four load-timing tiers — the top-level view of an eight-layer operating model. One placement rule: every piece of policy lives at the cheapest layer that still gets it applied.
Everything unattended reports upward into a single 2-minute morning read: what ran, what passed, what's blocked on exactly one question.
Three design moves that do the real work
1 · The maker is never the grader
Agents verify each other's work — but never their own, and never on trust. Verifier seats run as separate zero-tool processes: they see the diff and the test output, never the builder's narration, and hold no tools, so they cannot be talked into actions. Where a verifier could once be socially engineered, it is now physically incapable.
Best practice — fresh-context verification (Anthropic research): the grader sees artifacts — diffs, test output, logs — never the maker's narration. Measured to catch far more real issues than same-context self-critique.
2 · Autonomy is earned per category, from a ledger
No agent "earns trust" globally. Each category of work climbs a ladder on logged evidence — and demotion is automatic and loud.
Best practice — the Auto Mode safety model: graduated trust with human approval on consequential actions; no unattended path to production.
3 · Done is a state, not an event
Finished work never closes — it converts into a standing goal: a falsifiable predicate (exit codes, never grep) re-verified in CI every day, forever. Each predicate is proven able to fail — its covering test was introduced by the fix itself, so pre-fix code fails it by construction. The sentinel only detects; fixes go through normal review. The first silent regression it catches pays for the whole layer.
Best practice — verification over vibes: "done" is a checkable predicate, re-verified continuously — not a self-certified event.
The receipt that proves the routing works
Minimum effective intelligence: route each task to the cheapest model that can verifiably do it — and escalate where it counts.
Four Claude tiers are in service, each with a job: frontier judgment on exception, a default top tier for planning and review, a mid tier for well-specified implementation, a fast tier for anything describable in advance. The policy question was whether the exception tier ever earns its cost. So I ran a controlled benchmark — identical adversarial review task, identical read-only checkout, a correctness-critical concurrent subsystem — frontier tier versus the default top tier:
Default top tier: Reviewed the concurrency path and
certified it correct — an explicit "no race here" verdict.
Frontier tier: Found a genuine ordering race on the same
path — a real defect the cheaper review had signed off as safe.
Confirmed afterward by direct code read, not taken on the
model's word.
Outcome: the finding was fixed, and the frontier tier is now
reserved by written policy for exactly this class of work:
deep adversarial review of correctness-critical code, where a
missed defect is expensive and a second, sharper pass pays
for itself. That is the whole cost-governance philosophy in one decision: the expensive model is a scalpel, the routing table is the price tag, and every escalation has to earn its keep in findings the cheaper tier provably missed — proven here by a benchmark, not assumed.
Best practice — Building Effective Agents: match the model to the task; add complexity, and cost, only when it pays.
An engineering culture you can read
The system's most underrated artifact is its failure catalog — 21 process lessons, each logged the day it was earned, each priced, each ending in a rule whose compliance is visible in a receipt:
L-013 · Inspect untracked files before ANY delete/--force
Cost: permanently deleted a file the agent didn't create.
SYMPTOM Cleanup ran `git worktree remove --force`, asserting
"only the gitignored symlink is untracked." That was
an assumption. The status output showed a second file.
CAUSE Asserting instead of reading; --force suppressed git's
own refusal — the safety net for exactly this case.
RULE Before any destructive command: read the untracked
list, narrate it in the receipt. Not yours → stop. Rules graduate to standing policy only after repeated recurrence with objective evidence — and even then as proposed diffs a human applies. Once a month, one rule gets deleted on purpose: if nothing breaks, it was overhead.
What this demonstrates
| Competency | Evidence in this system |
|---|---|
| Systems thinking | An 8-layer operating model where every mechanism names the failure it prevents — and the check that would catch that failure silently returning. |
| Security judgment | Prompt-injection treated as a standing threat: readers of untrusted content are quarantined from actors with write power; rule files are structurally un-editable by the agents they govern. |
| Evaluation rigor | Model-tier decisions made by benchmark, not brand loyalty — n=1 caveats stated, variance floors respected, verdicts confirmed by direct code read before acting. |
| Cost governance | Tiered routing with an escalation budget: the top model appears exactly twice per loop — deciding what to build, and deciding whether it was built. |
| Operational honesty | Receipts over trust: "done" requires verification output; failures escalate with exactly one question attached; nothing is self-certified — including by me. |
Grounded in the published best practice
Not just principled — aligned, mechanism by mechanism, with Anthropic's own engineering guidance.
Every layer and move above maps to current Anthropic publications: Building Effective Agents (match the model to the task; add complexity only when it pays), Effective Harnesses for Long-Running Agents (bounded work per cycle, session-start verification, never edit tests to pass), Effective Context Engineering (a lean always-on layer, just-in-time retrieval, state on disk), and the Auto Mode safety architecture (a reasoning-blind classifier as an automated approver, graduated trust). Mapped component by component to that guidance, the system aligns in the top decile.
Two mechanisms go beyond what is publicly documented: the maker-is-never-the-grader verifier seats — zero-tool processes that cannot be socially engineered — and a memory anti-poisoning gate that treats every remembered fact as untrusted until it earns provenance. The gaps the review surfaced were surface-area hygiene, not architecture — the mark of a system that is sound, not merely elaborate.
The longer version
This system is also the source of Agentic Ops, the operations mini-course I author inside CareerForge, and the operating discipline behind PACCA (healthcare prior-authorization agents), CareerForge, and CurricMesh — all live in production. Two patents, 35+ years, HP and Microsoft before this. I'm happy to walk any part of it — with the ledgers open.