The Latency Tax in Agentic Workflows

Twelve sequential model calls at 800 milliseconds each spend 9.6 seconds just waiting. Cut each call to 200 milliseconds and the same path is 2.4 seconds. That arithmetic is not a benchmark. It is why a modest per-call difference becomes the bill when an agent stacks steps.

An agentic workflow's latency is not one response time. It is the critical path through model calls, tools, network hops, validation, retries, memory, and human approvals. Make one faster by shortening or removing the slowest sequential boundaries without weakening task completion.

For this revision we compared the workflow patterns described by Anthropic, OpenAI, Google Cloud, LangGraph, and several primary research papers. We reduced them to one trace schema and one decision rule: optimize the whole successful run, not the fastest isolated API call.

We did not benchmark a production agent for this article. Numeric examples below are labeled arithmetic, not model measurements.

Discretion is not a synonym for more model calls

The same label covers systems with very different levels of autonomy. A fixed pipeline might always classify a request, fetch a record, draft a response, and ask for approval. A more autonomous agent might choose its own tools, revise its plan after every observation, delegate subtasks, and decide when it has enough evidence to stop.

Anthropic’s architecture guide draws a useful line between workflows, where models and tools follow predefined code paths, and agents, where the model directs its own process and tool use. OpenAI’s agent guide defines the core agent as a model with tools and instructions, then puts the model inside a run loop with exit conditions and guardrails.

Autonomy changes how latency should be measured:

Table 1 Download CSV

System shape	Who chooses the next step?	Latency behavior	Main control
Deterministic workflow	Application code	Stable path with known branches	Timeouts and service budgets
Agentic workflow	Code and model	Partly predictable, with model-selected branches	Step limits, validation, and traces
Autonomous agent	Model within broad rules	Variable path length and tool use	Permissions, stopping rules, and escalation
Multi-agent system	Manager, router, or peer agents	Added handoffs; possible parallel work	Delegation boundaries and shared state

ReAct made the loop concrete by interleaving reasoning with actions against external tools or environments. That design lets an agent update a plan from new evidence, but every observation also creates another boundary. A call to search is useful because it changes what the model knows. It is expensive in wall time because the next decision waits for search to finish.

This is why “agentic” should not become a synonym for “many LLM calls.” The model needs discretion only where a deterministic rule cannot make the decision reliably. A fixed parser should parse a timestamp. A permission check should remain code. A database transaction should not be delegated to prose.

The first latency optimization is often removing a model call that never needed to exist.

The critical path is the latency tax

An agent trace is a directed graph. Nodes are model generations, tools, validators, handoffs, persistence operations, and human checkpoints. Edges are dependencies. If node B needs the output of node A, B cannot start first. The longest dependency chain is the critical path, and it sets the minimum end-to-end time even when other work runs in parallel.

A useful latency budget separates six contributors:

Table 2 Download CSV

Component	What it includes	What usually changes it
Queue and network	Provider queueing, connection setup, regional distance	Capacity, connection reuse, deployment region
Prompt processing	Reading instructions, history, retrieved context, tool definitions	Context size, caching, prompt design
Generation	Time to first token and output decoding	Model, reasoning effort, output length
Tool execution	Search, database, browser, code, or business API time	Tool design, parallelism, external service health
Orchestration	Serialization, state writes, handoffs, guardrails	Runtime design, persistence, validation placement
Recovery	Retries, repair calls, fallback models, human review	Error rate, timeout policy, output contracts

Hold the graph fixed and the same twelve-call toy path is a 7.2-second gap before any tool even runs. This is not a benchmark, and a real trace also contains tools and network time. It shows why swapping models looks like a latency project when the waits are stacked.

Now change the graph instead of the model. If four independent research calls each take two seconds, running them sequentially costs eight seconds before synthesis. Running them concurrently makes the branch roughly as slow as the slowest call, plus dispatch and gather overhead. The model is unchanged. The critical path is shorter.

Parallelism has a hard boundary: two tasks can overlap only if neither needs the other’s result. Search queries for four independent markets can fan out. “Read the search result, decide whether it is credible, then formulate a follow-up query” cannot. Pretending a dependency is independent usually moves the wait into rework.

LLMCompiler studied this distinction directly. Its planner identifies function calls that can execute in parallel, then dispatches them through a task-fetching and execution layer. Reported gains do not transfer automatically to every agent, but the mechanism does: expose the dependency graph before trying to optimize it.

Latency also changes shape across the distribution. A workflow that feels fast at the median can become unusable at the 95th percentile when one slow tool triggers a retry and a fallback. Record successful, failed, and escalated paths separately. Combining them into one average hides the path users complain about.

Six patterns, and each one costs something

Architecture diagrams often show the happy path. Production latency lives in the arrows, branch conditions, and return loops. These six patterns cover most agentic workflow designs without requiring a new agent framework for each one.

Prompt chaining

Prompt chaining sends one model output into the next fixed step. It works when the task decomposes cleanly: extract facts, draft an answer, then check the draft. The path is easy to inspect, but every stage is sequential. Add a chain step only when it raises acceptance enough to repay another model call.

The main failure is decorative decomposition. Splitting one well-specified generation into five roles can increase latency, tokens, and opportunities for information loss. If no intermediate output has an independent acceptance test, the chain may be ceremony.

Routing

A router classifies the input and sends it to a specialized prompt, tool, model, or agent. Routing can reduce latency when most requests are simple and only a minority need an expensive path. It can also add latency to every request if the router is another large model call.

Use deterministic routing for stable signals such as account tier, language, file type, or an explicit user choice. Use a model when the boundary depends on meaning. Then measure both route accuracy and the cost of misrouting. A fast router that sends hard tasks to a weak path creates retries downstream.

Parallel fan-out and gather

Parallel workflows dispatch independent subtasks together and synthesize their results. Google Cloud’s agent design-pattern guide notes the central tradeoff: concurrency can reduce elapsed time, but it raises resource use, token cost, and gather complexity.

The gather step deserves its own budget. Four fast workers can still produce a slow workflow if the synthesizer receives redundant context, resolves conflicts, or asks workers to repeat missing work.

Orchestrator and workers

An orchestrator dynamically decomposes a task, delegates pieces, and combines the results. This fits coding, research, and document tasks where the required subtasks are not known before the input arrives. It is more flexible than fixed fan-out and less predictable in path length.

Cap delegation depth. Record how many workers were requested, started, completed, cancelled, and retried. An orchestrator that delegates the same question three ways may be buying diversity. One that delegates because the first prompt was vague is paying a tax for unclear instructions.

Evaluator and optimizer

An evaluator scores or critiques an output, then an optimizer revises it. The loop can improve quality when the evaluation criteria are explicit and revision is genuinely useful. It can also run forever because “make it better” has no stopping rule.

Set an acceptance threshold, a maximum number of revisions, and an exit for disagreement. The evaluator should return structured reasons tied to the rubric. Otherwise the optimizer receives another essay and has to infer what failed.

Tool-use loop

A tool-use agent selects an action, waits for the tool, reads the result, and repeats. It is the most general pattern and the easiest one to leave unbounded. The exit condition may be a final answer, a structured result, a maximum turn count, an error, or a human handoff.

Every tool needs an observable contract: arguments, timeout, side-effect class, retry safety, output schema, and permission rule. Tool descriptions affect model selection; tool implementation determines whether the selected action returns quickly enough to be useful.

These patterns combine. A customer-support workflow may route the issue, retrieve policy and order data in parallel, run a tool loop to resolve the case, then use an evaluator before issuing a refund. The composition is valid. It also creates several places where latency and failure can multiply.

Four workflows make the wait visible

Examples are useful only when the call boundaries are visible. “An agent handles support” says nothing about the number of decisions, tools, or approvals involved.

Customer-support resolution

A support agent identifies the issue, retrieves the customer record and relevant policy, chooses an allowed action, drafts the response, and escalates sensitive cases. Record lookup and policy retrieval may run in parallel. Refund approval cannot begin until the requested action and amount are known.

The fast path is a known issue with complete data and a reversible action. The slow path includes missing identity, policy conflict, repeated tool errors, or a high-risk side effect. The workflow should branch early instead of discovering the risk after drafting a final response.

Research with source verification

A research agent decomposes a question, runs independent searches, reads sources, extracts claims, checks conflicts, and synthesizes an answer. Search can fan out. Verification remains partly sequential because a follow-up query depends on what the first source did not establish.

The latency mistake is sending every source through the full conversation history. Each worker needs the research question, its assigned subproblem, the evidence rules, and an output schema. The synthesizer needs the checked claims and citations, not every intermediate thought.

Document intake and exception handling

An intake workflow classifies a document, extracts fields, validates required values, looks up matching records, and routes exceptions. Most of the path can be deterministic once extraction finishes. A model is useful when layout, wording, or document type varies; code should handle schema validation and exact business rules.

Batch independent pages when the model and document allow it. Stop early on an unreadable file rather than paying for extraction, lookup, and judge calls that cannot succeed.

Coding and repository work

A coding agent reads the issue, inspects the repository, plans a change, edits files, runs tests, interprets failures, and revises. Parallel workers can inspect independent areas or run separate checks, but edits to the same files and tests against shared state need coordination.

The tool latency often outweighs generation: dependency installation, builds, browser tests, and remote CI are real work. A faster model will not fix a 12-minute test suite. The workflow needs selective checks during iteration and the full suite at the release gate.

Model routing is a quality decision first

Using one model for every node is simple. It can also waste time and money when short, checkable steps do not need the same reasoning depth as planning or synthesis. A two-model design keeps ambiguous work on a primary path and moves narrow tasks to a faster path behind validation.

Good candidates for the fast path have three properties:

1. the output is narrow enough to validate;
2. the cost of a wrong answer is bounded;
3. a fallback can recover without an irreversible side effect.

Tool routing, field extraction, bounded scoring, and query rewriting often fit. Open-ended planning, ambiguous exception handling, long synthesis, and consequential approvals often do not.

Celeris-1 is one candidate for those short, structured nodes. Its model page collects the current architecture, context, API, and verification details. Treat it as a model to test against the node’s labeled cases, not as a default replacement for the primary path.

Research on model cascades and routing supports the concept but not a universal configuration. FrugalGPT studied cascades that accept an earlier model result when it is sufficient and escalate otherwise. RouteLLM learned to choose between stronger and weaker models from preference data. Both frame routing as a quality-and-resource decision, not a permanent belief that small models handle “easy” work.

Production routing needs a labeled set from the target node. For a tool router, label the correct tool and arguments. For extraction, score fields rather than prose similarity. For a judge, compare against human decisions and inspect disagreement by category. For query rewriting, evaluate retrieval results, because a fluent query can still omit the entity that mattered.

Do not treat the router’s confidence as proof. Calibrate it against observed correctness, and define an abstention region that sends uncertain cases to the primary model. The fallback rate belongs in the latency budget. If half the requests take the fast path and then fall back, the workflow pays for both models on those cases.

The tool-use model comparison can form a candidate list. The LLM speed page can identify models worth timing. Neither page selects the model for a node; the node’s labeled cases do.

A timer does not name the wait

An end-to-end timer says the workflow is slow. A trace shows why. The OpenAI Agents SDK tracing model is a useful neutral vocabulary even if the application uses another runtime: a trace represents one workflow run, spans represent timed operations, and parent-child relationships preserve the execution tree.

Capture enough information to rebuild the critical path without logging secrets:

Table 3 Download CSV

Trace field	Why it matters	Privacy-safe value
trace_id and parent_id	Reconstructs dependencies and parallel branches	Random identifiers
Node and operation type	Separates model, tool, guardrail, handoff, and state time	Stable internal name
Start and end time	Calculates duration and critical path	UTC timestamps
Model and settings	Explains changes in reasoning and output length	Model ID, effort, token limits
Input and output size	Connects context growth to latency	Token or byte counts
Validation result	Shows whether speed produced usable output	Pass, fail, reason code
Retry and fallback	Exposes recovery cost	Attempt number and destination
Side-effect class	Distinguishes reads from risky writes	Read, reversible write, irreversible
Outcome	Keeps speed tied to task completion	Success, partial, failed, escalated

Inputs and outputs may contain personal, proprietary, or regulated data. Hashing a user ID is not enough if the prompt body remains in the trace. Store sizes, schema results, and reason codes by default; sample content only under a documented access and retention policy.

Then compute four views:

• Node distribution: p50, p95, and failure rate for each model and tool node.
• Critical-path contribution: time each node adds to the longest dependency chain.
• Path frequency: how often each branch, retry, and fallback occurs.
• Outcome slice: latency for successful, failed, partial, and escalated runs.

Throughput belongs in a separate chart. A service can complete many concurrent requests while one user-facing workflow still waits on a long sequential chain. Tokens per second also misses time to first token, tool time, and non-streaming structured responses. Match the metric to the interface.

State persistence adds another boundary but prevents worse recovery. LangGraph’s persistence documentation describes checkpoints at graph steps for interruption, human review, and fault-tolerant resumption. A checkpoint may add storage time to each step; without one, a failed long-running workflow may repeat every earlier call.

Measure both.

Build the smallest loop that can finish

Start with the smallest loop that can complete the task. OpenAI’s guide recommends growing from a single agent before adding multi-agent orchestration. That advice also protects latency: every handoff introduces another prompt, another context boundary, and another failure mode.

Use this build order:

1. Name one outcome. “Resolve an eligible return or escalate with a reason” is testable. “Help the customer” is not.
2. Define state. Store the facts, tool results, pending action, validation status, retry count, and completion reason. Do not rely on conversation text as the only database.
3. Expose narrow tools. Give each tool typed arguments, explicit permissions, a timeout, and an idempotency rule.
4. Set the loop boundary. Specify what counts as final output, when to stop, and the maximum turns, tool calls, elapsed time, and spend.
5. Validate before side effects. Check schemas, permissions, business rules, and required evidence before a write.
6. Trace the first version. Record the fields above before optimizing models or adding agents.
7. Build an evaluation set. Include normal, sparse, adversarial, high-risk, timeout, and partial-failure cases.
8. Change one boundary. Try parallelism, a faster model, a shorter context, a deterministic rule, or a cached result one node at a time.
9. Replay the full set. Compare completion, correctness, p95 latency, fallback rate, tool errors, and human escalations.
10. Roll out by risk. Start with read-only or reversible actions, then expand only when traces show why failures are contained.

The evaluation must score the final state, not only the last message. The τ-bench paper evaluates tool-using agents by comparing the database state at the end of a conversation with the annotated goal state. That principle transfers well: a polite refund message is not success if the order was never updated, and a correct database update is not acceptable if the agent violated the policy that governed it.

Test consistency across repeated runs. Agents may choose different tools or arguments from the same input. One successful demonstration says the path exists; repeated trials show whether the workflow can depend on it.

Set timeouts at the node and run levels. A tool timeout should return a typed failure the agent knows how to handle. A run timeout should stop new side effects, preserve the last valid checkpoint, and return an honest status. “Still working” is not a completion state.

Human intervention is part of the architecture, not evidence that the agent failed. Escalate when a retry limit is reached, required evidence is missing, permissions are insufficient, or the action is too consequential for automated approval. The useful metric is not zero escalations. It is appropriate escalation with enough context for a person to continue.

Faster can fail the task

Lower latency can reduce quality when it removes the reasoning, context, or verification the task actually needs. Parallel workers can duplicate effort or disagree. A fast router can misclassify the request. A shorter prompt can omit policy. A stricter timeout can abandon a slow but valid tool call. A cached result can be stale.

Cost can move in the opposite direction. Fan-out reduces wall time by buying concurrent work. Evaluator loops spend extra tokens to raise confidence. Speculative execution starts branches that may be cancelled. The right objective is a service-level target constrained by correctness and cost, not minimum milliseconds at any price.

High-stakes steps need a different threshold. Financial, legal, medical, security, access-control, and irreversible actions require stronger evidence, explicit authorization, and often human review. A short output does not make the decision simple.

And some workflows should not be agentic. If the path is stable, the inputs are structured, and the rules are complete, ordinary software will be faster, cheaper, and easier to test. Use a model where interpretation is the problem. Keep the rest as code.

The next useful number is not tokens per second on an isolated call. It is p95 on the successful path after you have removed every model turn that code could have taken.