Self-hosting LangFuse v3, instrumenting every retrieval call, and shipping a real-time observability stack on a single machine
In my last post, I promised that Week 6 would be about observability. Not as an afterthought — as a prerequisite. You can't improve what you can't measure, and the Week 5 agent was flying blind.
The classifier returned an intent. The tool runner found a FAQ. The responder formatted a message. But between those nodes — inside the embeddings, the vector search, the database connection — there was a black box. When a query returned a low-similarity match, I didn't know why. When it took 2 seconds instead of 200ms, I couldn't trace the bottleneck.
This week, I changed that. By the end, every /triage request produces a full waterfall trace from HTTP handler down to the pgvector cosine distance operator, with retrieval quality scores attached to every span. And I'm running the entire observability stack on my local machine.
The Architecture: What I Built
The full stack now spans five services, each with a narrow responsibility:
Browser -> http://localhost/
|
v
nginx (:80)
|
v
FastAPI (:8000)
POST /triage GET /health
|
v
+------------------------+
| LangGraph Agent |
| classifier -> tool |
| -> responder |
+-----------+------------+
|
v
+------------------------+
| PostgreSQL + pgvector | FAQ search, ticket creation
+-----------+------------+
|
v
+------------------------+
| LangFuse v3 Stack |
| (6 containers) |
| web -> MinIO -> |
| worker -> ClickHouse |
+------------------------+The critical difference from Week 5: every node in the LangGraph now publishes traces, spans, and scores to a self-hosted LangFuse instance. The agent doesn't just execute — it reports.
Part 1: Self-Hosting LangFuse v3
Why Not the Cloud?
LangFuse offers a generous cloud tier. For a demo, that's fine. But for a project that aims to demonstrate production readiness, self-hosting tells a different story: I can deploy the entire observability stack alongside the application, on my own infrastructure, with no external dependencies.
For the Saudi market, this matters. Under PDPL, many government and regulated-industry AI systems cannot send tracing data to third-party cloud services. Self-hosted observability isn't a nice-to-have — it's a compliance requirement.
The Docker Stack
LangFuse v3 ships as six containers, each with a distinct role:
Container | Role | External Port
-----------------+-------------------------------+--------------
langfuse-web | UI + API | :3100
langfuse-worker | Async event processor | (internal)
postgres | Transactional data | (internal)
clickhouse | OLAP observability store | (internal)
redis | Cache + job queue | (internal)
minio | S3-compatible event buffer | :9090 (API)
| | :9091 (console)The write path is event-driven:
app -> langfuse-web -> MinIO (raw JSON events)
-> langfuse-worker (async)
-> ClickHouse (queryable)Every trace, span, and score your agent emits lands in MinIO first as a raw JSON object. The worker picks it up asynchronously and inserts it into ClickHouse. This means a burst of traffic won't overwhelm the database — the S3 buffer absorbs spikes, and the worker drains at its own pace.
ReplicatedMergeTree table engine, which requires ZooKeeper or an embedded ClickHouse Keeper. Without it, the migration fails silently and no traces appear. I added a clickhouse-config.xml with a single-node Keeper configuration — 48 lines of XML that solved two hours of debugging.
Part 2: The Trace Hierarchy
Here's what a single /triage request looks like in the LangFuse trace view today:
triage [@observe on POST /triage]
|
+-- classifier-node [@observe on agent_graph.py]
| +-- classify [@observe(as_type="span")]
| +-- LiteLLM gen [auto-traced via litellm callbacks]
|
+-- tool-runner-node [@observe on agent_graph.py]
| +-- faq_lookup [@observe on tools.py]
| +-- trace-embedding [@observe on metrics.py]
| | +-- embed [@observe on embeddings.py]
| +-- trace-vector-search [@observe on metrics.py]
| +-- Scores:
| retrieval_hit_rate = 1.0 (found)
| retrieval_mrr = 1.0 (rank 1)
| latency_ms_embedding = 45.2
| latency_ms_vector_search = 12.8
|
+-- responder-node [@observe on agent_graph.py]Four levels of nesting from HTTP request to model inference. Every span has a name, a duration, and — where relevant — a score.
The deepest span, embed, measures the actual sentence-transformers model inference time. This is where you discover that the first request after a cold start takes 15 seconds (model loading), while subsequent requests take 45ms (cache hit). Without this span, you'd be debugging a ghost.
@observe(name="embed") decorator on a 3-line function is the most valuable span in the entire trace. It tells you whether performance problems are in the model, the network, or the database — before you start guessing.
Part 3: Retrieval Metrics
Traces tell you what happened. Scores tell you how well it happened. I attached five scores to each retrieval call:
| Score | What It Measures | Source |
|---|---|---|
retrieval_hit_rate | Did the vector search return a result? | find_faq() return value |
retrieval_mrr | Mean Reciprocal Rank of first match | 1.0 if hit, 0.0 if miss |
latency_ms_embedding | Time to generate embedding vector | trace_latency() context manager |
latency_ms_vector_search | Time to run pgvector cosine query | trace_latency() context manager |
ticket_creation_success | Was the escalation ticket inserted? | DB write success/failure |
These scores aggregate across all requests in the LangFuse UI. Over time, you can trend hit rate per intent, detect embedding latency regressions, and compare retrieval quality before and after a model change.
The Offline Eval Harness
I also built a script that runs 7 known-answer queries through the real retrieval pipeline and logs aggregate metrics:
- 6 queries that should hit FAQ articles (password reset, billing, tech support)
- 1 query that should miss ("lorem ipsum" — no semantic overlap)
$ python scripts/eval_retrieval.py
Running 7 eval cases...
[1/7] v HIT (expected HIT) sim=0.832 | I forgot my password and can't log in
[2/7] v HIT (expected HIT) sim=0.791 | How do I reset my password?
[3/7] v HIT (expected HIT) sim=0.845 | My invoice looks wrong
[4/7] v HIT (expected HIT) sim=0.768 | How do I update my billing information?
[5/7] v HIT (expected HIT) sim=0.811 | The app keeps crashing on startup
[6/7] v HIT (expected HIT) sim=0.739 | I can't connect to the service
[7/7] v MISS (expected MISS) sim= -- | Lorem ipsum dolor sit amet
Aggregate hit_rate=0.857 MRR=0.857 (6/7 hits)
Scores flushed to Langfuse.The eval scores (eval_hit_rate, eval_mrr) are logged to LangFuse as well, on their own eval-retrieval-run trace. This gives me a repeatable, versioned benchmark I can run after every retrieval system change.
Part 4: The Chat UI
Observability isn't just for debugging — it's for demonstration. I built a single-file HTML chat interface (860 lines, Linear-inspired dark theme) that lets you:
- Send messages and see the agent's full response inline
- View intent, confidence, latency, and hit/miss chips on every response
- Expand raw JSON for any response
- Manage multiple sessions (persisted in localStorage)
- Hit example prompts with one click
GET / and also proxied through an nginx container on port 80 — the same pattern you'd use in production. No CORS issues, no separate ports, no "works on my machine."
Engineering Decisions
Every week produces architectural trade-offs worth documenting. Here are the four that mattered most for Week 6.
1. Self-Hosted vs. Cloud LangFuse
Decision: Run the full LangFuse v3 stack locally via Docker Compose instead of LangFuse Cloud.
Why: The cloud tier is free and zero-setup. But self-hosting proves the observability stack can travel with the application — no external API keys, no data leaving the network, no dependency on third-party uptime. For the Saudi market, this is a PDPL compliance requirement: many government AI systems cannot send tracing data outside their infrastructure boundary.
Cost: ~2GB RAM and 4 CPU cores for the full 6-container stack. The setup cost was about 2 hours — mostly debugging ClickHouse Keeper and S3 env var names.
2. Event-Driven Write Path
Decision: Accept LangFuse v3's default architecture: app -> langfuse-web -> MinIO S3 -> langfuse-worker -> ClickHouse, rather than writing directly to ClickHouse.
Why: Direct writes would be simpler and remove two moving parts. But the event-driven path means a burst of 100 traces in 1 second doesn't lock a ClickHouse table — MinIO absorbs the spike, the worker drains at its own pace. For a local dev machine, this is arguably over-engineered, but building with the production architecture from day one prevents painful migrations later.
3. Decorator-Based Tracing vs. LangChain Callbacks
Decision: Use @observe() decorators on individual functions rather than LangFuse's LangChain CallbackHandler.
Why: The agent doesn't use LangChain's abstractions — it uses raw litellm.completion() and custom Python functions. Adding LangChain just for its callback system would introduce a heavy dependency for a thin integration layer. The @observe() decorator gives per-function spans with zero architectural coupling. The trade-off: each function must be explicitly decorated (4 decorators across the graph), whereas a CallbackHandler would auto-capture everything.
Result: 4 decorators maintain full trace visibility without a single LangChain import in the agent code.
4. Named Spans vs. Anonymous Observations
Decision: Every @observe() decorator uses an explicit name= parameter matching the trace hierarchy ("triage", "faq_lookup", "trace-embedding", "embed").
Why: LangFuse auto-generates span names from function names. But once you have multiple nodes calling the same underlying function, auto-naming produces ambiguous traces. Explicit naming creates a consistent, readable trace tree that maps directly to the architecture diagram — anyone reading the trace can reconstruct the code flow without opening a single file.
Three Bugs That Took the Longest
Every engineering week has its debugging stories. These three consumed the most time:
1. ClickHouse Keeper
LangFuse v3 uses ReplicatedMergeTree for ClickHouse tables. On a single-node deployment, this requires an embedded Keeper (replaces ZooKeeper). Without it, migrations fail silently. The fix: a 48-line XML config file mounted into the ClickHouse container defining a single-node Keeper with RAFT consensus on localhost.
2. Environment Variable Ordering
The langfuse Python SDK reads LANGFUSE_BASE_URL, LANGFUSE_PUBLIC_KEY, and LANGFUSE_SECRET_KEY at import time. If your import langfuse runs before load_dotenv(), the SDK picks up the previous session's cloud URL instead of localhost:3100. The fix: call load_dotenv() at the top of classifier.py, before any from langfuse import ... statement.
3. LangFuse v4 API Changes
The original code used get_current_trace().score() (v3 API). LangFuse v4 replaced this with get_client().score_current_span(). Silent failure — no error, no scores appearing in the UI. The fix: grep for all scoring calls and update to the new API signature.
Honoring the Roadmap: What I Promised vs. What I Built
Last week I committed to "LangFuse traces on every query_faq() and query_similar_tickets() call. Hit rate, MRR, latency." That's what shipped. But I also added two things I didn't promise: a full chat UI and an nginx reverse proxy. Both were driven by the need to test and demonstrate the observability stack — you can't verify trace quality without a way to trigger traces.
| Pattern | Status | What Happened |
|---|---|---|
| Retrieval Observability | DELIVERED | Every faq_lookup() call produces a 4-level trace waterfall with 5 retrieval scores attached. Hit rate, MRR, and latency per operation are logged to LangFuse in real time. |
| Self-Hosted LangFuse | DELIVERED | 6-container Docker stack running locally on port 3100 with ClickHouse, MinIO, Redis. Event-driven write path verified end-to-end. |
| Offline Eval Harness | DELIVERED | scripts/eval_retrieval.py — 7 known-answer queries, aggregate hit rate/MRR scores logged to LangFuse as a repeatable benchmark. |
| Chat UI | DELIVERED | 860-line single-file HTML with session management, raw JSON viewer, example prompts. Served at GET / from FastAPI and proxied through nginx on port 80. |
| Memory | NEXT WEEK | The roadmap called for Memory + Reflection in Week 7. Observability was the prerequisite — now it's ready, the trace data will tell us whether the memory layer actually improves retrieval quality. |
| Reflection | NEXT WEEK | Self-critique node planned for Week 7, now with the measurement infrastructure in place to validate its impact. |
The Revised Roadmap
| Week | Focus | Deliverable |
|---|---|---|
| 5 DONE | LangGraph + pgvector RAG | Stateful agent, dual retrieval, tool dispatch |
| 6 DONE | Observability | Self-hosted LangFuse, full trace hierarchy, retrieval quality scores, chat UI |
| 7 | Memory + Reflection | Long-term memory via pgvector, self-critique node, reflection quality scoring |
| 8 | Planner + Router Agent | Task decomposition, multi-step execution, specialist routing |
| 9+ | Multi-Agent Supervisor | Sub-agent graphs, least-agency tool scoping, supervisor topology |
Why This Matters for Saudi Arabia AI Future
1. Observability as a Compliance Prerequisite
Under Saudi PDPL, AI systems processing personal data must demonstrate traceability. A self-hosted LangFuse instance behind your own nginx proxy, on your own infrastructure, with full audit trails of every classification decision — that's not just good engineering, it's a compliance artifact. Few teams in the region are shipping with this level of instrumentation.
2. The Cost of Blind Agents
The GCC market is seeing a surge in AI agent deployments — customer service bots, HR assistants, government service portals. Most of them are flying blind. A 2024 survey of regional AI implementations found that fewer than 15% had any production observability beyond basic request logging. This week's work demonstrates that full-stack observability is achievable on a single machine with open-source tooling. The barrier is not cost or complexity — it's knowing what to build.
3. Measured Improvement vs. Aspirational Roadmaps
One of Vision 2030's AI goals is "data-driven decision making across government services." That applies to the systems themselves, not just the dashboards they power. An agent that reports its own retrieval quality scores, surfaces its own latency bottlenecks, and generates its own eval benchmarks is a system that can improve itself methodically. That's the kind of infrastructure Saudi AI needs — not demos, but instrumented systems capable of measured iteration.
What's Next: Memory + Reflection
With observability in place, Week 7 targets two interconnected patterns:
Long-term memory. The pgvector store already holds FAQ embeddings and ticket embeddings. I'm adding a memory_entries table that stores conversation summaries per session_id. On future requests, the agent retrieves relevant past memories via embedding similarity — letting it remember user context across sessions.
Reflection. A 4th LangGraph node after the responder that critiques the agent's own output for hallucinations, incomplete answers, or missing context. The reflection quality score itself becomes a LangFuse score, creating a feedback loop: the system that measures its own output quality.
The data from this week's observability stack will tell me if these changes actually improve retrieval quality. That's the point.
The best AI system isn't the one with the most patterns. It's the one you can see into, measure, and improve with confidence. This week gave the Triage Agent that visibility.
.jpg)
0 Comments
Post a Comment