The word “agent” gets thrown around a lot right now. If you string two API calls together, someone is going to call it an autonomous AI agent. But if you’ve actually tried to build a system that you can run in production, and get real work done without constant hand holding, you know this is not going to cut it.

Building production ready agentic workflows requires a specific architecture. Over a lot of customer engagements Aviato have found it easiest to think about this stack in four distinct layers plus the underlying plumbing that keeps it all from exploding in production.

Here’s a practical look at how the modern agentic stack is actually built, and what you need to productionise AI Systems:

Layer 1: Large Language Models (LLM’s)

At the absolute bottom of the stack sits your foundation model. This is where you’re dealing with the raw mechanics: pinging APIs, handling tokenization, tweaking inference parameters, and prompt engineering.

You give it instructions, and it responds. On its own, it doesn’t care about your long term objectives, it forgets what happened five minutes ago, and it definitely can’t orchestrate a complex, multi step workflow. To get that, you have to move up the stack.

Layer 2: Agents

This is where we take a reactive model and actually turn it into an agent. We’re wrapping the LLM in code that gives it persistence, structure, and a goal.

Instead of just answering a question, a Layer 2 agent can actually pursue an objective. To make that happen, you have to bolt on a few things:

• Memory and state management: So it doesn’t lose the plot halfway through a task.
• Tools: Structured function calling so it can actually do things like hit an external API or query a database.
• Context: This is where your RAG (Retrieval-Augmented Generation) pipelines live.
• Reasoning: The logic required to look at a big task, break it down into steps, and adapt if step two completely fails.

Unlike a standalone model, a Layer 2 agent acts, looks at the intermediate result of that action, and adapts its next move based on what just happened.

Layer 3: Multi-Agent Systems

Eventually, you’re going to give a single agent a task that’s simply too big. The context window is exhausted, it loses focus, and the whole thing falls apart.

That’s when you need to bring in a multi-agent system. Instead of writing one prompt to rule them all, you build a distributed team of specialist sub agents. This layer handles the collaboration between them, including:

• Inter-agent communication protocols so they can talk to each other (e.g. A2A)
• Intelligent routing to make sure the coding task goes to the coding agent, not the research agent.
• Shared state coordination so they aren’t overwriting each other’s work.
• Parallel workflows so multiple agents can grind at the same time.

By splitting up the work, the whole system becomes drastically faster, more robust, and way less prone to hallucinating under pressure.

Layer 4: Agentic Ecosystems

When you have a bunch of specialized agents running around asynchronously, things turn into chaos fast. Without structured orchestration, a multi-agent setup is just a cool local demo. With it, you get a scalable, reliable system that can actually survive real-world constraints.

For a reliable production system you need:

• Evaluation Tooling to help understand if a modified system prompt or new version of your LLM will improve or degrade your agents (Vertex AI Evaluation Engine is a good fit).
• A mechanism to escalate to a human
• Monitoring & Logging, to provide transparency into what the agent is doing (Google Cloud Logging)
• Orchestration to manage the flow of execution and keep the system coherent at scale (Vertex AI Agent Engine, with ADK)
• Guardrails and sandboxing to prevent undesirable actions
• Error Handling, Retries, and loop prevention

These are not sexy, but are how you ensure accountability, mitigate failure modes, and actually preserve trust in the automated decisions your software is making.

Aviato have run a number of PoC’s our current cost is 6 weeks and 80k AUD, to prove an agentic system can meet your needs. Moving these to production requires a team (or Aviato SRE’s) to manage them, and a lot of additional thought.