Building a Self-Improving Agent Stack with Langfuse, RAG, and Human Feedback

Most agent teams ship observability, retrieval, and human review as separate projects. One team owns the trace viewer. Another owns the vector index. A third owns the review queue, usually a spreadsheet. The agent still gets better, but only because engineers manually read logs on Friday afternoons and tweak prompts on Monday. That is not a learning system. It is a slow, lossy manual pipeline wearing a dashboard.

A self-improving agent stack wires these three functions into one closed loop: every production run is traced, every trace is scorable, every score feeds a curation queue, and every promoted artifact (prompt, retrieval index, tool schema) is versioned and gated by evaluation before it ever touches live traffic. The reference stack below assumes the whole thing runs inside a client VPC or air-gapped enclave, which is the setting that rules out most SaaS evaluation vendors and forces honest choices about where state lives.

How do you build that loop without a data-exfiltration surface, and without a flywheel that quietly degrades the model it is supposed to improve?

The Closed Loop in One Picture

Four things happen on every request. The agent emits a trace covering the LLM call, every retrieval, every tool invocation, and the final action. A reviewer or an automated judge attaches scores to that trace. Curated traces land in a versioned dataset. New prompts, retrievers, or policies are evaluated against that dataset and promoted through a gate before serving traffic.

The non-negotiable property is that each stage writes to the next through an append-only, queryable store. Nothing lives in a notebook. Nothing lives in a reviewer's head. If an answer regressed last Tuesday, you can name the prompt version, the corpus version, the retrieved chunks, the judge score, and the reviewer who labeled the related golden example. This is the same discipline our team describes in AI agent architecture work with regulated buyers, and it is what distinguishes a self-improving agent from an agent that merely logs well.

Self-Hosting Langfuse Inside the VPC

Langfuse captures traces for both LLM and non-LLM calls, including retrieval, embeddings, and tool invocations, with native SDKs for Python and JavaScript and over 100 framework integrations. It is built on OpenTelemetry, which matters when you want agent traces in the same observability backplane as the rest of the platform.

For regulated deployments the relevant fact is that Langfuse can be self-hosted as an MIT-licensed open source project, with a hardened edition that adds FIPS-compatible deployment, air-gapped operation, and SSO enforcement. A production topology typically runs stateless web and worker tiers, Postgres as the control plane, ClickHouse for the trace data plane, Redis for queueing, and object storage for raw events. AWS's partner writeup notes tens of thousands of events per minute with 50-100ms prompt fetch latency on this architecture, which is the useful range to size against when agents call the prompt API on every turn.

Two deployment constraints come up repeatedly. First, ClickHouse growth is linear with event volume, so a retention policy belongs in the first week of operation, not the first incident. Second, the prompt store becomes a hard dependency of the agent runtime; it needs the same uptime budget as the model gateway itself.

Versioning the Retrieval Corpus Like Code

The retrieval corpus is the second half of the agent's behavior, and in most incidents it is the half that moved. A corpus refresh that improves recall on one query class can degrade precision on another, which is why mature teams version the corpus with a hash, a date, and a chunk count and run a regression eval of 200 to 500 queries before promoting a new index.

A workable schema records, per chunk: source document ID, source revision, extractor version, chunker config hash, embedding model ID and dimension, ingestion timestamp, and a corpus-version tag. Promotion of a new corpus is a label flip against that tag, not an in-place overwrite. The old index stays live until the new one clears gates, and rollback is a second label flip. Teams coming from pipeline tooling will recognize this as the pattern described in our canary deployments piece, applied to retrieval rather than to code.

Human Feedback as First-Class Trace Signal

Automated scoring is necessary and insufficient. The RAGAS framework for RAG evaluation covers four core metrics — faithfulness, answer relevancy, context precision, and context recall — using LLM-as-a-judge scoring to approximate human judgment. That gets you cheap, continuous coverage. It does not get you ground truth. Recent work on LLM-as-a-judge self-inconsistency shows judges assign different scores to the same output across runs, which makes judge-only gates brittle at the margins where they matter most.

Human labels fill the gap, but only if the schema is tight enough to aggregate. A workable reviewer record carries: trace ID, agent version, prompt version, corpus version, a correctness verdict (correct, partially correct, wrong, refused-correctly, refused-incorrectly), a severity tier, a failure category drawn from a fixed vocabulary (retrieval miss, grounding violation, tool-arg error, policy breach, hallucinated citation), free-text rationale, and the reviewer's own ID for inter-rater analysis. Severity should map to the same action tiers used in approval workflows, so the loop that catches a wrong wire transfer in review is the same loop that would have blocked it at execution.

Promotion Gates and Rollback Discipline

The gate is the only thing standing between a promising offline score and a production regression. A defensible gate holds three conditions simultaneously: an absolute floor on each core metric, a noninferiority margin against the incumbent version, and a hard-fail list for safety categories (grounded-citation rate, policy-violation count, tool-arg validity). A candidate that improves mean faithfulness but drops grounded-citation rate below floor does not promote; the decision is INCONCLUSIVE until a reviewer sweeps the regressions.

Three artifacts need independent version lanes: the agent prompt, the retrieval corpus, and the tool registry. Each should be promotable and reversible on its own, which is the discipline covered in agent versioning. Mixing them into one deployable guarantees that the smallest change blocks the biggest fix.

Failure Modes to Design Against

A closed loop that learns is also a closed loop that can poison itself. The failure modes worth hard-coding against are the ones where feedback quality degrades silently.

Reviewer drift is first. Two labelers trained a month apart will diverge on edge cases unless the rubric is version-controlled and inter-rater agreement is sampled weekly. Judge contamination is second: if the same model family evaluates its own outputs, scores inflate. Use a different family, or anchor the judge against a human-labeled holdout each week. Reward hacking is third. The survey on open problems in RLHF documents deployed models that leaked private information, hallucinated, spread biases, and became sycophantic, all after feedback-driven fine-tuning. Treat any training signal derived from the loop as a candidate for the same safety evaluations you run on third-party models.

Three structural controls help. Keep a frozen holdout the loop never sees. Keep an append-only lineage from any shipped model back to the exact traces, labels, and corpus version that produced it. And keep a kill switch that reverts prompt, corpus, and tool registry to a last-known-good tuple in one operation. These are the same hygiene points covered in the first 90 days of an agent deployment, and they are what the loop fails without.

What This Buys You

A stack built this way does not promise a self-tuning agent. It promises something more useful: every regression is attributable, every improvement is defensible against the previous version, and every reviewer hour compounds into a dataset the next prompt is measured against. The precedent for the mechanism is strong. OpenAI's InstructGPT work showed that a 1.3B parameter model fine-tuned with human feedback was preferred over the 175B GPT-3, with gains in truthfulness and reductions in toxic output, which is the clearest existing evidence that disciplined human signal beats raw scale.

The loop is not a product you buy. It is a schema, a few gates, and the operational will to version three things that most teams version zero of. Done inside the VPC, with Langfuse as the trace spine and reviewers as the ground-truth source, it is the shortest path from a demo-grade agent to one that keeps earning its production slot.