The Agent That Learns: How One Week of Memory, Reflection, and MCP Turned a Stateless API Into Something That Actually Remembers
I spent Week 6 making every agent decision observable. Trace trees, retrieval scores, latency breakdowns — all flowing into a self-hosted LangFuse stack. It was beautiful. But by Sunday I had a nagging feeling I couldn't shake: observability tells me what happened. It doesn't make the agent better at its job.
My agent was starting fresh every single conversation. No memory of the last user. No knowledge of what worked before. No awareness that it was about to make a mistake.
Week 7 started with a simple premise: an agent that doesn't remember isn't an agent. It's just a fancy API endpoint.
1. Engineering Impact 40%
Memory, Precedent, Reflection, and a Protocol
I shipped four features in parallel, each solving a specific gap I'd felt during Weeks 5 and 6.
Feature 1: Session Memory — The Agent Remembers the Conversation
The symptom was obvious during testing. A user would say: "I forgot my password." The agent would classify it as password_reset. Then the user would say: "Actually, never mind — my colleague helped me. Now I have a billing question." The agent would start over, treating the new message as if no history existed.
I built app/memory.py — a lightweight SessionMemory dataclass that tracks conversation turns per session_id:
def classifier_node(state: AgentState) -> dict: result = classify(state["message"]) session = get_session(state.get("session_id") or "default") session.add_turn("user", state["message"]) session.current_intent = result.intent session.confidence = result.confidence context_history = session.to_context_block()
The to_context_block() method formats the last 5 turns into a prompt-ready context block injected into every subsequent request:
## Conversation History
1. User: I forgot my password and can't log in
2. Agent: Let me help with a password reset...
3. User: Actually, never mind — my colleague helped me. Now I have a billing question.
This was the single highest-leverage change in the entire Week 7 cycle. Before session memory, the agent misclassified follow-ups 40–50% of the time — it had no idea the user was the same person. After session memory, follow-up accuracy jumped significantly. The classifier didn't get better; it just got context.
Feature 2: The Precedent Store — The Agent Learns From Its Mistakes
Session memory keeps one conversation alive. But what about patterns across conversations? If 5 users all ask about password resets in the same confused way, shouldn't the agent get faster at recognizing that pattern?
I built app/precedent_store.py — a PostgreSQL + pgvector-backed memory that stores every triage decision with its semantic embedding:
precedents = find_precedent(symptoms=state["message"], top_k=2) if precedents: # "Previous similar case → password_reset (User had same issue last week)"
Each precedent stores:
- The pattern hash (SHA-256 of the query text) — exact match detection
- The semantic embedding — similar query detection via cosine similarity
- The human correction — if a human overrode the classification, that override becomes the new truth
I seeded the precedent store with 6 test cases and a deliberate miss. Then I sent similar queries through the agent. Does the agent use precedent to improve, or does precedent just add noise?
Precedent helps most on borderline cases. When confidence is high (>85%), the agent doesn't need precedent — it's already sure. When confidence is low (<60%), precedent is the difference between a correct escalation and an incorrect one. The system effectively bootstraps its own training data: every correct classification becomes a reference for the next similar query.
Feature 3: LLM-as-Judge Reflection — The Agent Reviews Itself Before Acting
This was the hardest feature to get right — and the one I'm most proud of.
The idea: before the agent executes a tool call, a cheaper model reviews the classification. If it spots an error or a safety issue, it overrides the intent before tool_runner_node fires.
The app/reflection.py module sends the user query + agent classification to a second model — the same "cheap-classifier" model from the LiteLLM proxy:
You receive: - The customer's query - The agent's classification (intent) - Confidence score You evaluate: 1. Accuracy — Is the intent correct? 2. Escalation need — Does this need a human? 3. Policy compliance — Does the handling align with protocol?
I ran 20 test queries through the pipeline with reflection on. Then I ran the same 20 with reflection off. I expected small improvements.
Reflection caught 3 misclassifications that would have triggered the wrong tool. In one case, the user wrote "I can't access my account" — the classifier said technical_support. Reflection flagged it: "The user can't access their account due to a forgotten password, not a system outage. This should be password_reset." It was right.
Reflection also had a false positive rate. About 10% of correct classifications were flagged unnecessarily — the reflection model was too cautious, second-guessing classifications that were fine. I tuned the prompt to only flag when confidence dropped below 0.7, but the behavior persisted. The lesson: reflection is a safety net, not a primary classifier. Use it to catch outliers, not to gate every decision.
Feature 4: MCP Server — One Protocol to Rule Them All
The Model Context Protocol (MCP) server wraps the agent's existing DB tools (faq_lookup, ticket_lookup) as standard MCP tools:
@mcp.tool() def faq_search(intent: str, user_message: str) -> str: result: ToolResult = faq_lookup(intent, user_message) return result.data
Why MCP instead of direct function calls? Because I wanted the agent's tools to be callable from anywhere — not just from the LangGraph pipeline. An IDE plugin could call faq_search. Claude Desktop could call it. A future multi-agent system could call it. MCP standardizes the interface.
The client side (app/mcp_client.py) uses asyncio.to_thread() to run the MCP subprocess — the same trick that avoids the 20-second SentenceTransformer cold-load on every request:
@asynccontextmanager async def mcp_tool_context(): server_params = StdioServerParameters( command="python", args=["-m", "app.mcp_server"] ) async with stdio_client(server_params) as (read, write): async with ClientSession(read, write) as session: await session.initialize() # yield call_tool — ready to use
MCP added complexity — subprocess management, async context managers, protocol handshakes — for a feature that doesn't change the agent's behavior yet. But it changes what the agent can become. Without MCP, every new tool integration requires modifying the agent's source code. With MCP, adding a tool means writing one @mcp.tool() function.
Engineering Decisions — Why I Chose What I Chose
Beyond shipping features, this week was about making architectural choices that would define the agent for months. Here are the five decisions I'm most confident about — and why.
1. In-memory dict over Redis for session memory
_sessions: dict[str, SessionMemory]) · Rejected: RedisSessions are ephemeral by nature — they last the length of a conversation, not the lifetime of the system. A Redis dependency adds infrastructure complexity, network latency on every turn, and a failure domain that can crash the agent. An in-memory dict is simpler, faster, and — critically — sessions survive as long as the API server is alive, which is long enough. The trade-off: restarting the server loses active sessions. I accepted this because sessions are short-lived and restarts during a conversation are rare. If production data shows otherwise, I'll migrate to Redis — but not before.
2. pgvector over a dedicated vector database (Pinecone, Qdrant)
The precedent store needs ACID guarantees — when a human correction is recorded, it must persist atomically. Dedicated vector DBs are optimized for throughput, not transactional integrity. pgvector runs on the same PostgreSQL instance as the FAQ articles and support tickets, so precedent lookups can join with other data in a single query. No separate infrastructure to deploy, monitor, or secure. The concern: pgvector's ANN search degrades at very large scale (millions+ vectors). At 6 seeded precedents growing at ~100/day, I have about 27 years before that's a problem.
3. MCP stdio transport over HTTP transport
stdio_client subprocess · Rejected: HTTP SSE transportMCP supports two transport modes: stdio (subprocess with stdin/stdout) and HTTP (SSE). Stdio is simpler — no network setup, no authentication, no port conflicts. The subprocess inherits the parent's Python environment, so all dependencies are available. The risk: spawning a subprocess per connection has overhead (~200ms). I mitigated this by keeping the MCP client alive as an async context manager within the request lifecycle. HTTP would be necessary if the tools ran on a different machine, but in my architecture, everything is local.
4. One reflection model over two-pass classification
The obvious approach is to classify twice with different models and compare results. I tried this in Week 2 and abandoned it: cost doubles, latency doubles, and both models usually agree. Reflection doesn't re-classify — it reviews. The prompt is shorter, the output is structured JSON, and the model doesn't need context about the tool ecosystem. The result: half the cost for equal or better accuracy. The key insight is that reviewing is fundamentally cheaper than classifying — it's a binary check with structured output, not an open-ended decision.
5. Fail-silent over fail-closed for every integration
try/except pass · Rejected: Fail on integration errorEvery Week 7 feature follows the same pattern: if it fails, the agent proceeds without it. Reflection fails? No revision. Precedent store offline? No precedent context. MCP server crashes? Falls back to direct tool calls. Session memory exhausted? Starts fresh. This is the opposite of bank-grade security (fail-closed), but it's the right trade-off for an agent that needs to be available. A crash caused by a broken integration is worse than a slightly less informed response. I made this call deliberately: the governance layer in Post 6b is where audit-grade guarantees live. The agent layer is where availability matters more.
2. Technical Leadership & Growth 30%
What I Learned From Watching My Agent Fail
The experiment that changed my mind
When I first built session memory, I was proud. My agent remembers now. Then I tested it.
I wrote to the agent as "Yahya": "I need billing help." It classified billing. I said: "How do I update my payment method?" It correctly followed up.
Then I sent a message from a new session with the exact same wording. The agent had no idea it had answered this question before. It classified billing again — correctly — but with no awareness that the same question was asked yesterday.
That's when I realized: session memory isn't enough. You need precedent memory too.
The experiment forced me to distinguish between two fundamentally different kinds of memory:
| Memory Type | Lifetime | Storage | Purpose |
|---|---|---|---|
| Session | Minutes to hours | In-memory dict | Track conversation flow |
| Precedent | Days to forever | PostgreSQL + pgvector | Learn from past decisions |
An agent with only session memory is polite but forgetful. An agent with only precedent memory is knowledgeable but socially awkward. It needs both.
Why I chose self-review over two-pass classification
The obvious alternative to LLM-as-Judge reflection is a two-pass classifier: classify twice with different models, compare results. I tried this in Week 2 and abandoned it — the cost doubles, the latency doubles, and the two classifications usually agree, wasting the second call.
Reflection is different. It doesn't re-classify. It reviews. The reflection prompt is shorter, the output is structured JSON, and the model doesn't need context about the tool ecosystem — it just needs to say "this looks wrong" or "this looks fine."
| Approach | Tokens | Est. Cost | Accuracy |
|---|---|---|---|
| Two-pass classification | ~800 (2 × 400) | ~$0.00030 | Same classifier twice |
| Reflection (this week) | ~500 (400 + 100) | ~$0.00015 | Classifier + reviewer |
Half the cost, same or better accuracy.
The fail-silent pattern, applied again
Every new feature follows the fail-silent pattern established in Week 4:
- Reflection fails? → Agent proceeds without revision
- Precedent store offline? → Agent works without precedent context
- MCP server crashes? → Agent falls back to direct tool calls
- Session memory exhausted? → Agent starts fresh
Not a single Week 7 feature can crash the agent. This wasn't accidental — I deliberately wrapped every integration in try/except blocks:
try: precedents = find_precedent(symptoms=state["message"], top_k=2) except Exception: pass # Precedent is a bonus, not a requirement
This is production engineering. Every optional feature must prove it's worth keeping by being invisible when missing.
3. Saudi Vision 2030 & Knowledge Economy 20%
Context That Compounds
Why memory matters for government services
A citizen calling a Saudi government helpline shouldn't have to repeat their ID number five times. They should be recognized from context, routed to the right service, and treated as a known identity.
Session memory makes this possible. A single API client can maintain session_id across an entire interaction, and the agent carries context from step 1 to step N.
Why precedent matters for compliance
When the Saudi Data & AI Authority (SDAIA) asks: "Has this agent handled this type of request before, and what was the outcome?" — the precedent store answers with deterministic evidence. Every similar case, every human override, every semantic match.
The audit hash chain (coming in Post 6b) proves the precedent store wasn't tampered with. The precedent store proves the agent learned from past cases. Together, they form a closed loop: the agent acts, the agent learns, the agent's learning is auditable.
Why MCP matters for the ecosystem
Model Context Protocol is now the connectivity standard — 97M+ SDK downloads, adopted by Anthropic, OpenAI, Google, Microsoft, and AWS. By building my tool layer on MCP, I'm not just solving a local problem. I'm aligning with an infrastructure standard that every GCC enterprise will use.
When a Saudi bank or telecom wants to connect my triage agent to their CRM system, they won't need a custom integration. They'll just point MCP at their API and go.
4. What's Next 10%
The Roadmap Ahead
These four features gave the agent what every knowledge worker needs: context, memory, self-awareness, and standard tools. But each feature opens a question the next post answers.
Immediate: Post 6b — ProofLayer: Governance as Infrastructure
Session memory and precedent store produce data. But how do you prove that data is authentic? The audit hash chain wraps every decision in a tamper-evident SHA-256 ledger. The context graph (pl_nodes/pl_edges) captures causal relationships between decisions, policies, and context. The ISO 42001 compliance engine reads both to produce audit evidence. Post 6b is the governance layer that makes this week's work defensible — not just functional.
Next: Post 6c — Arabic First: A Bilingual Agent for the Kingdom
The MCP server and reflection pipeline work for English. But the Kingdom speaks Arabic. Post 6c covers the full bilingual agent architecture — 20 mirrored files, Arabic-specific guard classifiers, code-switching handling, and the surprising discovery that the reflection model doesn't need Arabic prompts to reason about Arabic content. The MCP server becomes the bridge between both language pipelines and a shared tool ecosystem.
Beyond: Multi-Agent Orchestration and Zero-LLM Routing
With MCP standardizing tool access, the next architectural step is specialist sub-agents — a Billing Agent, a Technical Support Agent, an Escalation Agent — each with its own MCP toolset, coordinated by a Router Agent. And with precedent store confidence data, I can build PRISM-style routing: simple intents skip the LLM entirely, going straight to a direct FAQ lookup. The observability data from Week 6 proves which intents are safe to route without an LLM call. That proof is the foundation of the next phase.
Week 6 made the agent observable. Week 7a made it learnable. Week 7b will make it governable. And Week 7c will make it bilingual. The infrastructure compounds.
.jpg)
0 Comments
Post a Comment