Deterministic first, model second: how I design AI agents with LangGraph

Most agent systems I see fail the same way: someone hands the LLM the steering wheel and then spends months writing prompts begging it to drive safely. The model decides what to do next, which tools to call, when it is done — and the whole system inherits the model's variance. Demo looks great. Production is a coin flip per request.

My rule after building intake systems, forecasting tools and guardrails around this: the control flow is deterministic code, and the model only fills bounded slots inside it. LangGraph is useful precisely because it makes this separation explicit — but only if you use it that way.

The graph is the product

In a LangGraph system, the graph — nodes, edges, state transitions — is where the business logic lives. That part should be written the way you would write any backend: typed state, explicit transitions, exhaustive branches. If you can draw your workflow on a whiteboard, the graph should match the whiteboard, and a colleague should be able to predict the path a given input takes without running a model.

The LLM appears at specific nodes with a narrow job description: extract structure from messy text, draft a narration of numbers the engine already computed, classify into an enum you defined. Every model output passes through a schema (Zod, Pydantic, whatever your side of the stack uses) before it touches state. If validation fails, that is a normal branch in the graph — retry, degrade, or route to a human — not an exception.

What never goes through the model

• Routing and escalation decisions where being wrong is expensive. In Medbay, safety escalation is plain code: it costs zero tokens, runs in constant time and gives the same answer twice.
• Arithmetic and aggregation. In Konstellation and AtlasOS, deterministic engines produce every number; the model is only allowed to narrate them, and the narration is validated against the source values.
• Anything you need to replay. Reproducibility dies the moment a model output feeds another model's input without a frozen, validated artifact between them.

Why bother — the economics

This is not just an elegance argument; it is a cost structure argument. A model call on every decision means your unit cost scales with conversation length and your p99 latency is hostage to a third-party API. Deterministic control flow means model calls per request are a small constant you chose, cost per unit of work is predictable enough to put in a spreadsheet, and most requests touch the expensive path exactly once or not at all.

It also changes testing. A deterministic graph is testable with ordinary unit tests over state transitions. The model slots are testable with a small eval set per slot — dramatically cheaper than end-to-end evals of a free-roaming agent, because the blast radius of each slot is bounded by its schema.

The uncomfortable admission

Designing this way is slower at the start. Letting the model improvise gets you a demo in a day. Drawing the graph forces you to actually understand the workflow first. But that work was always owed. The improvising agent just deferred it to production, with interest.