I see that people use runtime AI Agents just for the sake of using agents. This can result in unnecessary complexity, bad performance and cost. I share a simple rule to guide when to use runtime AI Agents.
Definitions
Before we look at the problem, I want to define the two kinds of agents that we use during software development life cycle (SDLC).
Build time agents These are agents like Claude Code and Codex. These agents accept prompts from the user and generate progams. These AI-generated programs need not have any dependency on the AI agents at runtime.
Runtime agents These agents are used at the runtime of the program. Once the system receives an input these agents are invoked to process the input. The responsiveness of the system and the cost is directly dependent on how fast the agent can generate output tokens (output tokens/second) and how expensive the agent is ($/output token). Also the agent that generated the code need not necessarliy be the same agent used at runtime. In fact, most of the time they both will be different.
These days build time AI Agents are ubiquitous. I hardly see anyone who does not benefit from the power of the build time AI agents. But using runtime AI Agents is a choice, often done wrong. There are cases when using AI Agents at runtime will only lead to unnecessary slowness and cost. I wanted to give a very simple rule of thumb to decide if you need an AI Agent at runtime or not.
Before stating the rule, let us understand the concept of semantic equivalents. Semantic equivalents mean that the same idea expressed in multiple ways: same meaning, different forms. Like "The program gave an error", "I got an error from the program", "Error was returned by the program" all could mean that the user encountered an error while using the program.
The Rule
The rule is very simple. Consider the inputs to the system at runtime.
If the inputs are from a finite set of values known during the build time, you don't need to use AI Agents at runtime. If the inputs can have semantic equivalents (like plain English text entered by a human), then you may benefit from using AI Agents at runtime.
If your inputs can be enumerated ahead of time, then you should not introduce any runtime AI agent dependency, even if the enumerated input set can be large. If the set of values in your inputs change over time (new values are added), your program can still be written such a way that it indicates clearly that a new input has been encountered. By simply monitoring for the new inputs, you can keep updating your program.
Let us look at a few examples to get clarity.
Examples
Example 1: An alert handling system
The list of alerts that your system can raise are finite. The playbook to handle these alerts can be easily generated based on a set of rules. So you definitely don't need runtime AI Agent.
Example 2: An issue ticketing system
If the issue tickets are submitted by humans in natural language, then you definitely need a runtime AI Agent. The AI Agent can understand the human input and convert that into an execution plan and provide an answer.
Example 3: A chatbot in a website answering questions from users
You definitely need a runtime AI Agent for this case, as the inputs from users will be in natural language.
Optimizing for performance and cost
Encountering unknown inputs every once in a while is acceptable. In that case, you will review the logs for unknown input and you can update code to handle that case. But what if you want the runtime AI agent to handle unknown input cases as well?
You can keep the AI-generated program and the runtime AI agent in tandem as given below.
This will make your solution more robust even when an unknown input is encountered. You will be periodically reviewing how many times an unknown input has been encountered that the runtime AI Agent had to be invoked. Your goal is to keep the number of times the runtime AI agent is invoked to be zero. As an when a new input is identified, you will update your AI-generated program to handle that input. The feedback loop is the part that will make this solution most effective. But this model comes with increased complexity.
Updated: 05/28/2026