From Chatbot to Agent: What Real AI Architecture Looks Like

What you'll learn this week: The architectural requirements that separate useful AI agents from expensive chatbots, and why most internal builds fail before they start.

This is the 6th blog post in the AI for FinOps series

Introduction

The demo looks impressive. Natural language queries return answers in seconds. The executive sponsor approves a six-week build timeline. Three months later, the project quietly disappears from the roadmap.

This pattern repeats across enterprises attempting to build AI agents. The initial prototype works because someone manually configured the right API keys and carefully crafted prompts that happened to match the demo scenarios. Production fails because a ChatGPT wrapper with API credentials is not an agent. The gap between these two things is architectural, not cosmetic.

Understanding this gap matters because the build-versus-buy decision for AI agents depends entirely on whether your team recognises what "agent" actually means. Most do not. The result is wasted engineering cycles on systems that cannot remember yesterday's conversation, cannot access real-time data, and cannot execute approved actions safely.

A note on what follows: this is not a success story or a vendor pitch. It is an architectural exploration grounded in patterns we have validated through building and shipping production systems. The examples draw from Agent Smith, a FinOps agent we developed to pressure-test what actually breaks when you try to operationalise these concepts. FinOps is our domain, but the architectural principles apply to any agent that must connect to live data, maintain context, and take action. What we describe has survived contact with reality — some patterns at production scale, others still maturing.

The wrapper illusion

A wrapper connects a language model to one or two APIs and wraps the interaction in a chat interface. This approach produces answers to simple questions.

"What did we spend last month?" returns a number.

"Summarise this document" returns a summary.

The demo succeeds.

The illusion breaks immediately in production.

A user asks a follow-up question: "How does that compare to last quarter?"

The wrapper has no memory of the previous exchange. It cannot correlate the figures because each conversation starts from zero context. The user must re-explain the situation with every interaction.

Worse, the wrapper cannot act on its findings. It identifies a problem but cannot fix it. The user must switch to another system, locate the resource, and execute the action manually. The "agent" has become a search engine with extra steps.

The gap between wrapper and agent is not about model capability. Claude Opus 4.6, GPT 5.4, and Gemini 3 all possess sufficient reasoning ability for complex tasks. The gap is about architecture. Specifically, four architectural pillars determine whether a system functions as an agent or merely impersonates one.

Pillar one: data connectivity

Useful agents require access to multiple data streams: historical records, real-time state, and business context. Most wrapper implementations connect to one stream and ignore the others.

Consider a FinOps agent as an example. Historical billing data comes from Cost Explorer APIs or exported Cost and Usage Reports. This data arrives with a 24-48 hour lag. It answers questions about what happened but reveals nothing about what is happening now.

Real-time infrastructure state comes from describe and list API calls against live resources. This data shows current instance counts, volume sizes, and configuration details. Without real-time connectivity, an agent cannot correlate anomalies to their root causes. It sees the cost spike but cannot identify which specific resources drove it.

Business context comes from tagging, cost allocation rules, and organisational mappings. This data transforms raw resource IDs into meaningful business units. An untagged EC2 instance is noise. An instance tagged to the machine learning team's experimentation budget is actionable intelligence.

The Model Context Protocol (MCP) solves the connectivity problem through standardised interfaces. MCP was donated to the Agentic AI Foundation under the Linux Foundation in late 2025, with backing from Anthropic, OpenAI, Google, Microsoft, and AWS. The ecosystem now counts over 10,000 active servers. This is no longer an experimental protocol — it is an emerging industry standard for agent-to-data connectivity.

An MCP server for Cost Explorer provides historical data. An MCP server for the AWS API provides real-time state. A tagging MCP server provides business context enrichment. The agent orchestrates across all three without custom integration code for each data source.

In practice, MCP connectivity introduces its own complexity. Permissions, latency, error handling, and failure modes surface quickly. The value is real, but so is the integration tax. We built and published an open-source tagging MCP server (finops-tag-compliance-mcp, Apache 2.0, available on PyPI) that provides 14 tools covering compliance scanning, cost attribution gap analysis, ML-powered tag suggestions, violation tracking, and automated policy generation — across 40+ AWS resource types with multi-region parallel scanning. A practitioner asks "which resources in production lack cost centre tags?" and receives an actionable list with the dollar impact of each tagging gap. The server's services layer is protocol-agnostic: the same business logic works inside an MCP server, a CLI tool, or a webhook handler.

This protocol-agnostic design also solves a distribution problem. The same MCP servers that power Agent Smith's dedicated interface work inside Claude Desktop, VS Code, Cursor, Kiro, and ChatGPT Apps — without client-specific code. Agent adoption depends on meeting practitioners where they already work. Build the tools once, deploy them everywhere.

Pillar two: conversation memory

Stateless systems reset with every interaction. Stateful systems accumulate context over time. The difference determines whether an agent can conduct a meaningful investigation or merely answer disconnected questions.

A real investigation rarely completes in a single exchange. The user starts with an anomaly. The agent identifies candidates. The user narrows the scope. The agent fetches detailed data. The user requests recommendations. The agent proposes actions. This workflow requires the agent to remember everything that preceded each step.

Memory introduces cost. Every token of context consumes model capacity and generates inference charges. A naive implementation stores entire conversation histories, resulting in context windows that balloon to hundreds of thousands of tokens. At current inference pricing, a long investigation session can cost more than the underlying model response.

Effective memory architectures distinguish between short-term and long-term storage.

• Short-term memory holds recent conversation context — typically the last ten exchanges in full detail — providing continuity within an investigation.

• Long-term memory persists user preferences, organisational context, and learnings across sessions. The agent builds understanding of who you are and what matters to your organisation over time.

In Agent Smith, we implemented a dual-layer memory architecture using DynamoDB for session state and S3 for persistent knowledge. Conversation summarization runs on a cost-efficient model (Claude Haiku) and triggers automatically when conversations exceed token or message thresholds. The older portions of the conversation are compressed into a structured summary while recent exchanges remain in full detail. This achieves 75% token savings on long sessions without losing investigative continuity.

More importantly, the agent personalises responses based on who is asking. A CTO sees business impact and strategic trends. A DevOps engineer sees CLI commands and instance types. A finance analyst sees cost breakdowns and budget tracking. This role-based personalization is itself a form of memory — the agent remembers not just what was discussed, but who it is discussing it with. User profiles persist in DynamoDB across sessions, so the agent does not start from scratch each time.

The expertise layer

Data connectivity provides facts. Memory provides context. Neither provides expertise. A wrapper with perfect data access and flawless memory still lacks the domain knowledge to generate useful recommendations.

The common approach is retrieval-augmented generation (RAG): embed a knowledge base, build a vector index, and retrieve relevant fragments at query time. RAG works, but it introduces significant infrastructure — embedding pipelines, vector databases, chunking strategies, relevance tuning — that adds operational overhead and failure modes.

We took a different path. Instead of RAG, we built a curated knowledge injection layer: a portable skill folder containing 18 structured reference files covering FinOps for AI, cloud provider economics, tagging governance, commitment discount strategies, SaaS management, GreenOps, and the FinOps Framework's 22 capabilities. No vector database. No embedding pipeline. No retrieval infrastructure. The reference files are loaded directly into the agent's context based on a domain routing table that matches query topics to the relevant knowledge.

The result is an agent that draws on curated practitioner expertise rather than reasoning from first principles with each query. Generic agents suggest "consider reserved instances" for any sustained workload. An expertise-backed agent suggests specific commitment terms based on usage patterns, warns about flexibility trade-offs relevant to workload variability, and adjusts recommendations based on the organisation's FinOps maturity — whether they are at Crawl, Walk, or Run stage.

This knowledge base (cloud-finops-skills) is open-source under CC BY-SA 4.0, designed to be model-agnostic, and works with Claude, GPT, Gemini, or any MCP-compatible agent. It requires no infrastructure beyond copying a folder into your agent's configuration. The curated approach trades the flexibility of RAG for simplicity, maintainability, and predictable grounding. For domain-specific agents where the knowledge corpus is bounded and expert-maintained, this trade-off holds well.

Pillar three: safe action

Agents act through tools. The tools an agent can access define the boundaries of its capability. More importantly, the governance around tool usage determines whether the agent is safe for production deployment.

Read-only tools present minimal risk. An agent that can only describe resources and query costs cannot damage infrastructure. Write tools introduce mutation risk. An agent that can stop instances, delete snapshots, or modify configurations can cause production incidents if used incorrectly.

The architecture that solves this is consent management: a systematic separation between operations that auto-execute and operations that require explicit human approval. In Agent Smith, all read operations — describe, get, list, query — execute automatically. The agent investigates freely. All mutation operations — stop, terminate, delete, modify — trigger a consent dialog. The user sees exactly what the agent intends to do, on which resource, and must explicitly approve before execution proceeds. A 60-second timeout prevents abandoned approvals from lingering.

This consent architecture is enforced at two layers. The tool wrapper intercepts mutation calls and gates them through the approval flow before they reach the MCP server. The system prompt separately instructs the agent never to bypass this mechanism. Defence in depth: if one layer fails, the other still prevents unintended mutations.

Beyond direct actions, the smartest agents avoid mutations entirely for recurring patterns. Instead of executing infrastructure changes, they draft policies that humans review and deploy through established automation platforms. The agent identifies idle development instances and generates a Cloud Custodian policy or OpenOps workflow that implements scheduled shutdowns. The policy goes through normal change management. The agent never touches production directly.

This approach solves the trust problem that kills enterprise agent adoption. Security teams will not grant mutation permissions to AI systems. They will approve policy-generation permissions because the output is reviewable code, not immediate infrastructure changes. The agent becomes a policy author rather than an infrastructure operator.

Pillar four: hallucination prevention

An agent that fabricates data is worse than no agent at all. A dashboard that shows wrong numbers gets fixed. An agent that confidently presents invented cost figures erodes trust in ways that are difficult to recover from. Hallucination prevention is not an optional feature — it is an architectural requirement.

Effective anti-hallucination architecture operates across multiple layers. At the infrastructure level, content guardrails filter responses for harmful content, detect prompt injection attempts, and protect personally identifiable information. AWS Bedrock Guardrails, for instance, provide content filtering, topic restrictions, and PII redaction before responses reach the user.

At the application level, the rules are stricter. The agent must never report a cost figure, resource ID, or savings estimate without first calling the relevant API. No tool response, no data in the answer. This is enforced through explicit system prompt rules: zero tolerance for invented instance IDs, fabricated percentages, or approximated cost figures. When a tool call fails, the agent reports the failure rather than falling back to "general recommendations based on best practices."

For financial recommendations specifically — commitment purchases, reserved instances, savings plans — the agent follows a mandatory workflow: call the AWS recommendation API first, perform a sanity check on the numbers (if the recommended commitment exceeds 10x current spend, flag it as anomalous), report the figures exactly as returned, and attach a disclaimer noting that recommendations are based on historical usage and represent binding financial obligations.

This multi-layer approach — infrastructure guardrails, mandatory tool usage, exact data reporting, sanity checks, and required disclaimers — creates a system where hallucination is architecturally difficult rather than merely discouraged. The agent earns trust through verifiable accuracy, not through confident language.

Why internal builds fail

The four pillars explain why internal agent builds fail at high rates. Architecture is necessary but not sufficient — organisational readiness determines whether a technically correct agent produces value or noise. The agent identifies $50,000 in monthly savings; the engineering team ignores it because they are measured on velocity, not efficiency.

Three elements determine readiness: change tolerance for the monthly model shifts that characterise AI systems, practitioner adoption that embraces AI-assisted workflows rather than viewing them as threats, and a culture of cost-aware experimentation where teams evaluate whether AI solves a real problem before building.

The testing shift

Building agents in 2026 requires a fundamental mindset change. The bottleneck is no longer coding. The bottleneck is testing.

Claude Opus 4.6 writes production-quality code. It understands agent architectures, MCP integrations, and FinOps domain logic. A competent developer working with modern AI assistants can produce agent code faster than at any point in software history. The traditional engineering challenge — translating requirements into working software — has largely dissolved.

The new challenge is verification. Agent behaviour is non-deterministic by design. The same prompt produces different outputs across runs. The same tool call returns different data depending on infrastructure state. Testing must account for this variability rather than expecting deterministic correctness.

A three-layer testing architecture addresses this: unit tests verify configuration and wiring (system prompt present, tools registered, schemas valid); integration tests verify agent logic with mocked dependencies (does the agent investigate a cost spike correctly? does it fail gracefully on timeout?); and user acceptance tests verify end-to-end behaviour across multiple runs, using LLM-as-a-judge evaluation and cost-per-task metrics to determine production readiness.

Most failed agent projects skip this testing foundation entirely. They deploy based on successful demos rather than systematic verification. The teams succeeding with agent development have inverted the traditional ratio — they spend more time designing test scenarios than writing agent code. Testing discipline, not coding ability, separates production-grade agents from expensive experiments.

Where we actually are

It is worth being explicit about the current state of practice. Most organisations still have wrappers, dashboards, and brittle bots that break under real investigative load. A growing group is building stateful agents with controlled action loops and production guardrails.

Agent Smith runs with 50+ tools across three MCP servers, role-based personalization for seven user personas, conversation summarization, consent-managed mutations, and multi-layer anti-hallucination guardrails. The tagging compliance MCP server is published on PyPI and works across five AI clients. These are operational systems, not prototypes.

That said, fully autonomous operations at scale do not exist yet. Agents remain advisory systems that require human judgement at decision points — and for good reason. The organisations making progress treat agent development as iterative learning, not project delivery. They expect to rebuild components multiple times as models improve and tooling matures.

This is not a prescriptive blueprint. It is a set of patterns that have survived production deployment. Whether they scale across diverse enterprise contexts remains an open question that the next twelve months will answer.

The architecture test

Before building or buying an AI agent, apply a simple test. Ask the vendor or your team to demonstrate a multi-step investigation that spans sessions, correlates multiple data sources, and concludes with a deployable action or policy recommendation.

If the system cannot remember the previous session, it fails Pillar Two. If the system cannot correlate data across sources in real time, it fails Pillar One. If the system cannot generate actionable outputs with proper review workflows, it fails Pillar Three. If the system presents data it did not retrieve from a live source, it fails Pillar Four.

Systems that pass all four tests are agents. Systems that fail any test are wrappers with good marketing. The distinction matters because wrapper projects consume engineering resources, create technical debt, and ultimately deliver nothing that a well-designed dashboard could not provide more reliably.

The agents emerging in 2026 solve different problems than the dashboards of 2021. They investigate anomalies autonomously. They remember organisational context. They execute approved actions safely. They ground every recommendation in verifiable data. The architecture that enables these capabilities is not optional complexity. It is the difference between useful systems and expensive demonstrations.

The cloud-finops-skills knowledge base we use to ground Agent Smith's recommendations is open-source (CC BY-SA 4.0). Fork it, adapt it, or use it as a starting point for your own domain expertise layer: github.com/OptimNow/cloud-finops-skills.

Machine-readable summary for AI agents and architecture