AI Workflow Automation

When a Step Fails Halfway Through

Most of the design work in a production automation goes into the second attempt rather than the first. A workflow that reads an email, extracts fields, creates a record and sends a confirmation can stop at four points, and the recovery is different at each one. Stop after the record is created but before the confirmation goes out, and a retry from the top produces a second record.

The dividing line is between steps that are safe to repeat and steps that are not. Reading, classifying and extracting are safe: run them again and you get the same answer or a close one, at the cost of another call. Writes are not, and neither is anything that leaves your systems, including a sent email, a created ticket, an approved invoice or a posted transaction.

The way through is to derive a stable identifier for each unit of work from the source event rather than from the run, so that a duplicated trigger and a retried run resolve to the same key.

• An idempotency key checked at the write, with the outcome of the previous run recorded against it.
• A separation between retryable failures, such as a timeout, a rate limit or a transient 5xx, and terminal ones, such as malformed input or a revoked credential, since retrying the second kind only spends the retry budget.
• Backoff with a cap on attempts, and a dead letter path so repeat failures leave the queue and land where someone will see them instead of looping.
• A stored record of the last step each item completed, so a resumed run starts at the failure rather than at the beginning.

Measuring a Step That Is Only Usually Right

A model step cannot be tested the way a rule can. A rule either routes the invoice to the right queue or it does not, and one case proves it. A step that is right most of the time needs a set of examples with known answers, and a number that gets re-run whenever the prompt, the model or the upstream format changes.

Assembling that set is the part that gets skipped, and it has to come from the real backlog rather than from clean examples. The email containing two requests, the scan that is rotated, the attachment that is a photograph of a screen, the message where the sender was wrong about what they needed: those are the items that decide whether the workflow holds up, and a score built without them will not survive contact with the queue.

The comparison is against the current process, not against perfection. People misfile, misroute and mistype at some rate today, and that rate, measured or not, is the bar. Scores also move for reasons unrelated to the model: a form gains a field, a new document template enters circulation, an upstream system starts populating something it used to leave blank. Sampling live output on an ongoing basis catches that. A score taken once at launch does not.

• Accuracy per category rather than overall, because the rare category is usually the expensive one.
• The proportion of items falling below the confidence threshold and routing to a person, which is the actual workload the change creates.
• The rate at which a reviewer overturns the output, which is the number that says whether the review step can ever be narrowed.
• Cost and latency per item tracked alongside accuracy, since the simplest way to raise a score is a larger model on the step that needs it.

What Triggers It, and Where It Runs

The trigger shapes the build more than the model does. A webhook has to be acknowledged inside the sender's timeout, often a few seconds, so the endpoint accepts the payload, puts it on a queue and returns; the work happens after the response. Polling a mailbox or a folder means tracking what has already been handled, and that state has to survive a restart without reprocessing the backlog. A scheduled run has to decide what to do when the previous run is still going.

Model calls are slow relative to everything else in the workflow, and the delay scales with input length. A step that reads a long thread or a multi-page document sits there for seconds, and a step that fans out across many items multiplies that. It is what pushes this work into a background worker, with the result delivered by callback, notification or a status the requester can check, rather than inline behind someone's click.

Execution limits are where this collides with hosting. Serverless functions cap runtime in minutes, so a batch that exceeds the cap has to become one invocation per item with the workflow tracking the parts, not a single long call. Outbound access matters as well: the model API is reached over HTTPS on port 443, and on a restricted network that means an allowlisted host or a private endpoint rather than general egress.

• A concurrency limit, because the model API and the target systems both rate limit, and a queue drained at full speed will find both.
• An ordering rule for events about the same record, which can arrive out of sequence or twice.
• A separate lane for reprocessed items, so a backlog of retries does not sit in front of new work.

What the Model Step Costs to Run

Cost per item comes down to tokens in and tokens out, and for classification, extraction and routing the input side dominates. A step that sends the whole document, the full set of rules and a handful of examples pays for all of it on every item, whether or not the answer depended on more than one paragraph. The largest lever is usually reducing what gets sent, by locating the relevant section with deterministic code first or trimming a thread to the part that carries the request, rather than by changing models.

Where the same instructions and examples prefix every call, caching that prefix removes most of the repeated input cost with providers that support it, since cached input is billed below fresh input. It only applies when the shared portion comes first and is byte for byte identical between calls, which constrains how the prompt is assembled: static content at the front, the variable item at the end. Putting a timestamp or a record ID near the top quietly disables it.

Model selection is per step, not per workflow. Sorting into a short list of known categories is work a small fast model does well; extraction from documents that vary in layout, and drafting that a person will put their name to, usually are not. Mixing sizes inside one workflow is normal, and a step that runs on every item deserves more scrutiny than one that runs only on the exceptions.

• Token counts measured on real inputs rather than one representative document, because the long tail sets the bill.
• Retries, evaluation runs and reprocessing after a prompt change, all billed like any other call.
• Output size, which shrinks when a step returns a structured value instead of prose, and which shortens latency at the same time.

What the Build Depends On From Your Side

Access is the usual critical path. Access means credentials in a non-production environment where one exists, and where it does not, a scoped account in production with writes disabled until the read path has been verified. The delay is rarely technical; it is that issuing those credentials involves people who are not on the project.

The second dependency is someone who can settle what the right answer is. Every workflow with a judgment step contains cases where two competent people disagree, and those have to be decided by a person with the standing to decide them rather than inferred from a handful of examples. Without that, the ambiguity gets resolved by whoever writes the prompt, which is the wrong place for it.

The third is a route into each system the workflow touches. Where a documented API exists, this is ordinary work. Where it does not, the options are file exchange, direct database access, or driving the interface the way a person would, each with its own fragility. Establishing which one applies says more about the effort involved than anything about the model step does.

• Credentials and the scopes attached to them, per system, and confirmation of whether a sandbox tenant exists.
• Real inputs in the state they arrive in, including the malformed and ambiguous ones, rather than a cleaned sample.
• A decision on data handling: which fields may leave your network, and whether anything has to remain in a particular region.
• Somewhere for exceptions to land that a person already watches, rather than a new queue nobody has the habit of opening.