System Design in the AI Age: What Actually Works

The Optional Feature That Bottlenecked the Endpoint

We shipped an AI feature on a Thursday. By Monday the p99 latency (the latency of the slowest one percent of requests) on one endpoint had gone from 400 milliseconds to eleven seconds, and nobody had touched a line of business logic.

The model was a latest one, the code too was tested thoroughly but the boundary around it was a single HTTP call in the synchronous path: a 30-second timeout, two retries, no circuit breaker, sitting between the auth check and the database read. Every user traversed this path, including those who never interacted with the AI feature. This anti-pattern is one of the most common mistakes in modern AI system design..

The feature was optional by design, a suggestion layered on a flow that already worked and that is what made the failure more galling. An optional component had taken the whole endpoint hostage because we treated it as optional in intent but mandatory in the request path. We did not see it until a situation arose where I read the trace.

That day we had confused the code with the system. We did not see the difference between the code and the system. We treated the model as if it were just another piece of code, when in reality it was fundamentally different class of dependency. This is the pattern I keep seeing in the AI age. Teams sprinkle LLM calls across every component, treat them as magic and forget the first law of distributed computing: network calls are expensive and they will fail. AI is not the exception to that law. AI is its loudest example.

In our initial setup, the LLM call had a 30-second timeout, two retries, no circuit breaker, no fallback, and it sat between the auth check and the database read. When the provider got slow, the retries stacked, and the endpoint inherited eleven seconds of tail latency that had nothing to do with what the user asked for. The fix was not a better model. It was a better boundary. We moved the call out of the synchronous path, gave it a 2.5-second timeout, one retry, a circuit breaker that opened after three failures, and a rule-based fallback on the critical path. Same model, same feature, p99 back to 420 milliseconds.

The model is not the system. The boundary around the model is the system.

This is not a small-team failure. It is the same confusion we just had, cut in the other direction: we under-built the boundary, there are teams who over-build the system. I have seen senior teams jump straight into distributed systems when a simpler approach would serve them better, then wonder why every change takes three sprints. I have seen teams adopt a vector database, a graph database and a time-series store in the same quarter because each one sounded impressive in a demo. AI amplifies every one of these mistakes, because it lets you add complexity at a speed nobody has accounted for.

The Mental Model: Two Halves, One Boundary

Every modern AI system is two different kinds of software bolted together, and most system failures occur when treating them as one.

• The deterministic half is the harness: orchestration, state, business logic, databases, retries, audit. Code you can test, version and replay. When it breaks, it breaks loudly, and you can reproduce it.
• The non-deterministic half is the model and the agents built on it. Output is probabilistic, no two runs are identical, and failure is semantic instead of loud: the agent returns a clean, confident, wrong answer. You cannot unit-test your way out of it, and you cannot debug it by rerunning the same test.

The design rule everything below follows: the non-deterministic half never touches anything important without the deterministic half in between. The model suggests. The harness decides, validates, enforces and records. Every control layer in this article is that boundary aimed at one specific risk.

One rule governs how the deterministic half should grow. Gall's Law: a complex system that works is invariably found to have evolved from a simple system that worked. Start as a single, well-structured application, and split only when users press on real boundaries, because a premature split is a distributed monolith, the worst of both worlds. In the AI age this is load-bearing: the model layer can become obsolete in six months, so the best code you can write is code that is easy to change later.

The boundary in practice looks like this:

The control layers, at a glance: jump straight to any of them.

• 1. The Harness: Process and Reliability Controls
• 2. Optimization and Cost Control
• 3. Quality Control: Testing the Non-Deterministic
• 4. Audit, Versioning, Observability and Editing the Output

1. The Harness: Process and Reliability Controls

The boring rules work. Treat every AI call like the external dependency it is:

• Strict timeouts. 2.5 seconds, not 30.
• Retry budget, not retries. One retry on timeout, never on a 429 or a 4xx.
• Fast circuit breaker. Opens early, stays open long enough for the provider to recover.
• Fallback on every critical path. Even if the fallback is dumber.
• Per-user rate limits. The circuit breaker protects you from the provider; rate limits protect you from the user. Inference is expensive enough that a single runaway script can drain thousands of dollars, so token quotas per user or tenant belong in the harness.

A rule-based answer that is right eighty percent of the time beats a graceful failure that serves nobody. None of this is new: we did it for payment gateways and SMS providers. People skip the patterns for the LLM because it feels different. But abandoning these patterns is exactly when AI stops acting like magic and starts acting like a liability.

Keep inference separate from business logic. Treat the model as a suggestion engine: it recommends, your rules decide. That one separation buys three things at once. The validation layer catches nonsense output. You swap providers or models by changing one layer instead of rewriting the application. And your core logic stays testable without depending on a model behaving.

The same instinct extends to agents. When an agent wants to execute an irreversible or regulated action, the harness enforces a trust boundary: state machine interrupts suspend the run, a cryptographically signed approval token puts the exact payload in front of the right human, and dangerous tools run sandboxed so the model cannot bypass the orchestrator. You cannot prompt your way to safety. You engineer it.

2. Optimization and Cost Control

Cost and latency are architecture decisions, not accounting line items. Every token has a price and a time cost. Four controls do most of the work:

• Model routing. Route by task: a small model for classification, a larger one for generation, and a strict rule for when a call is expensive enough to matter. Routing cut our inference cost by roughly two thirds on the same workload.
• Semantic caching. A semantically identical question asked minutes ago is answered from cache before the model ever sees the request. A lightweight vector store, or even Redis with stored embeddings, is enough to hold it, because a traditional exact string match cannot catch the same intent phrased differently.
• Context compression. Compress the payload between the application and the provider: cluster repeated log patterns, keep function signatures, collapse JSON bodies. This cuts token usage by 60 to 95 percent while preserving the signals the model needs.
• Version pinning. Pin model and prompt versions together, so changing either is a reviewed, reversible release. Never let a silent provider update change your behaviour or your bill.

If you are not measuring cost per request next to latency per request, you are designing blind.

The instrument you watch shifts with the system:

3. Quality Control: Testing the Non-Deterministic

Traditional unit tests cannot cover a non-deterministic model, so the approach must shift. The controls that work:

• Contract tests verify the response shape even when the content varies.
• Evaluation sets, curated examples where you know what good looks like, run on every model or prompt change, with scores gated like a test suite gates a merge. Build them from production traces: an input that caused a support ticket belongs in the set.
• Shadow testing runs a new model beside the old one and compares before switching.
• LLM-as-a-judge grades the messy parts: a stronger, constrained model checks facts, tone and hallucination against the source context. Frameworks like Braintrust, LangSmith or open-source evaluators ship most of this harness, so you are choosing an evaluation platform, not building one.
• Human review loops for anything that touches money or people's records. Not a compromise, the design.

For agents, the unit of behaviour is the trace, not the completion. Curate golden trajectories, the exact path a perfect execution should take, score them with deterministic evaluators on every commit, and measure Pass@K instead of pass or fail: run the same prompt ten times, and nine clean traversals is a 90 percent Pass@10.

You are not testing whether the model is correct. You are testing whether the system handles the model gracefully in every scenario.

4. Audit, Versioning, Observability and Editing the Output

Two properties separate a demo from a deployment: you can prove what happened, and you can take it back.

• Audit. Every run is recorded: which prompt, which model version, which provider, how many tokens, how long, and who approved what. Trace IDs span the model call. For regulated industries the audit becomes immutable and hash-chained, so an auditor can verify nothing was altered after the fact.
• Versioning and observability. Model and prompt versions are released together and pinned. The evaluation set is versioned like code. And because agents are non-deterministic, observability is the debugging tool: trace the state handed between agents, not just the text, so a silent failure can be replayed and fixed.
• Editing, not regenerating. Treat the output as a tree of independent blocks, not a flat string. The user asks to punch up the conclusion, and only the conclusion block goes back to the model and swaps in place: faster, a fraction of the cost, and the rest of the document stays untouched. Regenerating a whole report because one paragraph changed is the most expensive UX mistake in generative AI.
• Security. If user data leaves your boundary to reach a third-party model, that is a design decision with legal weight. Build for data residency: bring the model to the data. When data must leave, a scrubbing layer inside the harness strips personally identifiable information (PII) from the payload first, so the provider only ever sees what it needs. Treat prompt injection as an input validation problem, because that is what it is.

The Future: The Harness Is the System

Something underneath all of this is changing. If code becomes cheap to produce, the system is no longer the code. The system is the harness: what plans the work, executes it, verifies it, deploys it, maintains it and, increasingly, retires it. The earlier rules do not disappear. They move up a level: the timeout, the fallback and the verification that used to wrap a model call now wrap the AI that writes the code. Teams that treat AI as a faster compiler build yesterday's applications faster. Teams that focus on building the harness can deploy applications that were previously uneconomical to create.

Test coverage is where this becomes concrete. For twenty years, 80 percent coverage was the practical ceiling, because maintaining the last 20 percent by hand cost more than the bugs it caught. AI removes the maintenance cost: it writes the edge-case tests, the property tests and the contract tests, and regeneration keeps them honest. Near 100 percent coverage of behaviour becomes a design target instead of a fantasy, and the harness verifies the tests too, with mutation testing and evaluation sets gating a code change the way they gate a model change.

Once the harness can build and retire applications on demand, the architecture inverts. Business teams have always needed many short-lived applications, a triage tool for a supply chain crisis, a compliance report generator for one quarter, a campaign dashboard for one launch, and they never got built because the fixed cost exceeded the value of an application that lives for three months. They got built anyway, as spreadsheets and shadow systems, each with its own copy of the data. This creates fragmented systems by default rather than by design.

The future of system design is a thinner, disposable application layer over a permanent, governed core.

The model is a commodity that gets cheaper every quarter. The data is not: it is permissioned, protected and lives in systems you already paid for. This is exactly why at Kappal Software we advocate for bring the model to the data. No data copies, no drift, no decommissioning problem, because the data never left home.

Going deeper. Each control layer here deserves its own treatment, and we are writing them up as separate posts: the planner/executor architecture of multi-step agents, human approval and trust boundaries, debugging silent failures in multi-agent systems, testing LLMs and agents, and editing generative output block by block. This post is the map. Those are the territories.