Add blog post on chatbot eval setup

content/blog/2026-07-16-steal-our-eval-setup.mdx

@@ -0,0 +1,203 @@
+---
+title: "Yes, you can steal our eval setup"
+date: 2026/07/16
+description: "Evals need to be custom to your app. At least some of them. A good chunk you can, and should be copying. Here is our setup for our docs chatbot."
+tag: engineering
+author: Annabell
+highlight: true
+---
+
+import { BlogHeader } from "@/components/blog/BlogHeader";
+
+<BlogHeader
+  title="Yes, you can steal our eval setup"
+  description="Evals need to be custom to your app. At least some of them. A good chunk you can, and should be copying. Here is our setup for our docs chatbot."
+  date="July 16, 2026"
+  authors={["annabellschafer"]}
+/>
+
+"Build custom evals," they say. Yes, but "custom" doesn't mean "from scratch." For a huge chunk of eval design, especially for something as well-trodden as a chatbot, teams are solving a problem thousands of teams have already solved. Here's our setup for our docs chatbot, in full. Ready to copy.
+
+<Callout type="info">
+  Explore the [Langfuse example project](/docs/demo), the public Langfuse
+  project across cloud regions [EU](https://cloud.langfuse.com/project/clkpwwm0m000gmm094odg11gi), [Japan](https://jp.cloud.langfuse.com/project/cmmnj9e9n0006ad06nyz28646), and [US](https://us.cloud.langfuse.com/project/cmeic4yr500qmad0772bw0tol),
+  and the [open-source chatbot
+  code](https://github.com/langfuse/langfuse-docs/tree/main/components/qaChatbot).
+</Callout>
+
+<div className="not-prose my-6 overflow-hidden rounded border border-line-structure bg-surface-1">
+  <div className="border-b border-line-structure px-4 py-3">
+    <p className="font-mono text-[11px] font-medium uppercase text-text-tertiary">
+      Chatbot setup snapshot
+    </p>
+  </div>
+  <dl className="grid sm:grid-cols-2">
+    <div className="border-b border-line-structure px-4 py-3 sm:border-r">
+      <dt className="font-sans text-[12px] font-[430] text-text-tertiary">
+        Runtime
+      </dt>
+      <dd className="mt-1 font-sans text-[14px] font-medium text-text-primary">
+        <a
+          href="https://vercel.com/docs/ai-sdk"
+          className="underline decoration-line-structure underline-offset-2 transition-colors hover:text-text-secondary hover:decoration-text-tertiary"
+        >
+          Vercel AI SDK v7
+        </a>
+      </dd>
+    </div>
+    <div className="border-b border-line-structure px-4 py-3">
+      <dt className="font-sans text-[12px] font-[430] text-text-tertiary">
+        Generation
+      </dt>
+      <dd className="mt-1 font-sans text-[14px] font-medium text-text-primary">
+        OpenAI models
+      </dd>
+    </div>
+    <div className="border-b border-line-structure px-4 py-3 sm:border-b-0 sm:border-r">
+      <dt className="font-sans text-[12px] font-[430] text-text-tertiary">
+        Retrieval
+      </dt>
+      <dd className="mt-1 font-sans text-[14px] font-medium text-text-primary">
+        Inkeep-powered RAG
+      </dd>
+    </div>
+    <div className="px-4 py-3">
+      <dt className="font-sans text-[12px] font-[430] text-text-tertiary">
+        Tool access
+      </dt>
+      <dd className="mt-1 font-sans text-[14px] font-medium text-text-primary">
+        Langfuse docs MCP
+      </dd>
+    </div>
+  </dl>
+</div>
+
+The structure below follows the [AI engineering loop](/academy/ai-engineering-loop): trace production behavior, monitor the right signals, turn reviewed examples into datasets, run experiments, and feed what we learn back into the chatbot.
+
+## Trace every step
+
+Before anything else, you need to see what the chatbot is actually doing, not just what it says at the end. Every retrieval call, every tool use, the full input and output at each step. Tracing is the foundation for everything else: no traces means no monitoring, no clear issue investigation, and no foundation for good datasets.
+
+<Frame fullWidth>
+  ![Trace detail view showing user ID, session ID, tool use, scores, input, and output](/images/blog/2026-07-16-steal-our-eval-setup/chatbot-trace-detail.png)
+</Frame>
+
+For chatbots, make sure each trace also carries the surrounding context you will need later. We pass a `userId` so we can see whether the same person comes back with the same topic over time, and a `sessionId` so related turns are grouped into one [session](/docs/observability/features/sessions). A good default is one trace per user message and one session per conversation. That keeps individual traces easy to inspect while still letting us replay the full thread and compare behavior across returning users in the [Users view](/docs/observability/features/users).
+
+<Frame fullWidth>
+  ![User detail view showing a focused list of chatbot traces for one returning user](/images/blog/2026-07-16-steal-our-eval-setup/user-focused-traces.png)
+</Frame>
+
+What we look for is the user's question, the retrieval step showing which docs got pulled in, the tool calls, and the generation step with the final answer. If something is wrong with the answer, the trace tells us whether the problem was bad retrieval or bad generation on top of good retrieval. Those are two very different fixes.
+
+## Monitor what is going on in prod
+
+Once production tracing is in place, we check for two things:
+
+- Which chatbot interactions are interesting enough to review?
+- How do quality and usage behave over time?
+
+<Frame fullWidth>
+  ![Running evaluator scores for the docs chatbot including disagreement, relevance, cursing, and user intent](/images/blog/2026-07-16-steal-our-eval-setup/running-evaluator-scores.png)
+</Frame>
+
+### What counts as interesting?
+
+We could review every conversation by hand. We decided to evaluate where we need to look and call that monitoring. A few things reliably mark a chatbot conversation as worth a closer look:
+
+- The user pushes back or disagrees with the answer.
+- The question is out of scope for what the bot can actually help with.
+- The user is cursing or broadly complaining about the system.
+- The user is clearly frustrated.
+
+The first three run as [LLM-as-a-judge evaluators](/docs/evaluation/evaluation-methods/llm-as-a-judge) in Langfuse and target observations that contain the full message history. You can start from the managed evaluator library in Langfuse or copy the prompts from our related post on [high-signal production monitoring](/blog/2026-04-01-llm-as-a-judge-production-monitoring). User frustration is tracked with a [code-based all-caps detector](/docs/evaluation/evaluation-methods/code-evaluators), also running in Langfuse, scoped to observations where the user input comes first.
+
+This is what turns "review everything" into "review the right things."
+
+### How do we define quality?
+
+On top of the event detectors, a lightweight always-on quality check helps:
+
+- Is our output relevant to what the user is asking?
+- What are the rough intent categories of our users over time?
+
+This is not a fully fledged quality evaluation system, but it gives us a signal for quality, usage patterns, and drift over time, which tells us when to take action.
+
+Both relevance and user intent are set up as LLM-as-a-judge evaluators: one numeric, one categorical. The user intent categories are application-specific. For our use case, we use `conceptual-question`, `implementation-question`, `self-hosting`, `pricing-and-comparison`, `ui-feedback`, and `irrelevant-to-langfuse`.
+
+We then set up a [production monitoring dashboard](/docs/metrics/features/custom-dashboards) that shows average scores, user intent classifications, total score counts, and cost over time. On top of that, we have [monitors that notify us](/docs/metrics/features/monitors) in Slack when certain thresholds are reached, such as increased cost, increased time to first token, or a high number of users disagreeing with our chatbot.
+
+<Frame fullWidth>
+  ![Production score monitoring dashboard for the docs chatbot](/images/blog/2026-07-16-steal-our-eval-setup/score-monitoring-dashboard.png)
+</Frame>
+
+<Frame fullWidth>
+  ![Monitor configuration for high user disagreement with warning and alert thresholds](/images/blog/2026-07-16-steal-our-eval-setup/monitor-configuration.png)
+</Frame>
+
+## Annotation queues, and why we still look ourselves
+
+Our traces end up in two [annotation queues](/docs/evaluation/evaluation-methods/annotation-queues) worth working through: one with flagged conversations, and one with general samples of everything, reviewed on a regular cadence regardless of whether anything was flagged.
+
+Your coding agent could review those queues for you. But especially in the early setup, most of the value, we would put it somewhere around 80%, comes from a human actually looking at the data (learn more why in our article [AI is eating the AI engineering loop](/blog/2026-06-09-ai-is-eating-ai-engineering)). Because lightweight monitoring already points us at the relevant stuff, that look is a lot more efficient than it sounds.
+
+For each conversation you review, attach two things: a pass/fail call and a short freeform note on what actually went wrong. Use open coding, not a fixed category list yet. After a first pass, group failure modes and label all failed conversations with their respective category. For each category, decide what to do about it: ignore it, monitor it further, or fix it. The full guide on how to do this systematically is in our [error analysis guide](/academy/monitoring/error-analysis).
+
+<Frame fullWidth>
+  ![Annotation queue item with pass/fail assessment and open coding fields](/images/blog/2026-07-16-steal-our-eval-setup/annotation-queue-item.png)
+</Frame>
+
+That comment is what matters most, more than the score. It is what helps to craft an expected output when this example becomes a dataset item.
+
+## From annotations to a dataset
+
+Once we annotated enough real conversations and noted what we think the right output should have been, we built a proper [dataset](/docs/evaluation/experiments/datasets). A dataset item needs an input and can have an expected output, or none, depending on what you are testing.
+
+<Frame fullWidth>
+  ![General QA chatbot dataset with production and synthetic examples](/images/blog/2026-07-16-steal-our-eval-setup/general-qa-chatbot-dataset.png)
+</Frame>
+
+Two things matter most when building it:
+
+- The input distribution should reflect what people actually ask, not what you imagine they ask. Mix random sampling with your annotated examples.
+- You need to know what you are evaluating before you build the dataset, not after.
+
+For chatbots, three dimensions have held up well as a starting point and are close to universal across different bots:
+
+- **Correctness against an expected answer**, checked with an LLM judge.
+- **Keyword overlap**, especially useful for making sure product names and specific terms actually show up.
+- **Behavior match**, whether the bot did the right thing given the question: answer directly, ask a follow-up, defer because it is out of scope, or call the right tool.
+
+<Frame fullWidth>
+  ![Expected output on a chatbot dataset item with sample reply, keyword overlap, and behavior label](/images/blog/2026-07-16-steal-our-eval-setup/dataset-expected-output.png)
+</Frame>
+
+Correctness and behavior match are set up as [LLM-as-a-judge evaluators](/docs/evaluation/evaluation-methods/llm-as-a-judge) in the app. They are scoped to experiment runs on this specific dataset. Keyword overlap runs as part of the [experiment runner](/docs/evaluation/experiments/experiments-via-sdk) ([script](https://github.com/langfuse/langfuse-docs/blob/main/components/qaChatbot/experiments/run_general_qa_chatbot_experiment.ts)) and is a deterministic match, so a code-based eval.
+
+We use this dataset with its evaluators to run [experiments](/docs/evaluation/experiments/experiments-via-sdk) and understand the quality of our system. We frequently run experiments on newly released models, after changes to the docs, and whenever we add new edge cases to the dataset.
+
+We use the [dataset item metadata](/docs/evaluation/experiments/data-model#datasetitem-object) field to categorize items by where they come from, what the item is for, and which input language we are using. This lets us filter, slice, and dice score data, and makes it easier to keep the dataset balanced over time across those dimensions.
+
+That dataset doubles as our regression suite. We run it as part of [CI](/docs/evaluation/experiments/experiments-ci-cd), so a prompt or model change gets checked against real failure cases before it ships, not after.
+
+## The failure mode that is actually ours
+
+CI catches the general stuff. But every system has failure modes that are specific to it, not to chatbots in general.
+
+For us, it is staying grounded in the actual docs instead of training data. LLMs know old versions of SDKs, are trained on old integration pages, and multiple versions exist across the ecosystem, so a fluent, confident, wrong answer is an easy failure to produce and an easy one to miss.
+
+We built a separate dataset just for this. It is reference-free: inputs only, no expected output, because the correct answer changes every time the SDK does. What we evaluate instead is whether the answer is actually grounded in the retrieved documentation rather than in what the model already "knows."
+
+<Frame fullWidth>
+  ![SDK integration grounding dataset item with input and metadata](/images/blog/2026-07-16-steal-our-eval-setup/sdk-grounding-dataset-item.png)
+</Frame>
+
+Your version will be different. Maybe it is making sure a specific tool gets called when it should. Maybe it is knowing when to hand off to a human instead of guessing. The mechanism is the same either way: find the failure mode through your own error analysis, then build a dataset around that specific thing. Do not wait for a generic evaluator to catch it, because it will not.
+
+## It is a loop, not a milestone
+
+Keep monitoring production, keep doing error analysis, keep pulling interesting traces into your datasets. That is what turns this from a one-time setup into a system that actually gets better over time.
+
+The more mature the setup gets, the more of this you can hand off to a coding agent: triaging queues, drafting dataset items, flagging candidates for review. Early on, though, look yourself. That is where most of the value is.
+
+The full setup is live in our [public Langfuse project](https://cloud.langfuse.com/project/clkpwwm0m000gmm094odg11gi), including the traces, scores, dashboards, datasets, and experiment setup described above.

scripts/update-github-stars.js

@@ -6,6 +6,14 @@ const path = require("path");
 const GITHUB_REPO_API_URL = "https://api.github.com/repos/langfuse/langfuse";
 const STARS_FILE_PATH = path.join(__dirname, "..", "src", "github-stars.ts");
 
+function getExistingStarCount() {
+  if (!fs.existsSync(STARS_FILE_PATH)) return null;
+
+  const existingContent = fs.readFileSync(STARS_FILE_PATH, "utf8");
+  const match = existingContent.match(/GITHUB_STARS\s*=\s*(\d+)/);
+  return match ? Number(match[1]) : null;
+}
+
 async function updateGitHubStars() {
   try {
     console.log("Fetching GitHub stars...");
@@ -50,7 +58,16 @@ async function updateGitHubStars() {
     console.log(`✅ Updated ${STARS_FILE_PATH} with ${starCount} stars`);
   } catch (error) {
     console.error("❌ Error updating GitHub stars:", error.message);
-    process.exit(1); // Fail the build
+    const existingStarCount = getExistingStarCount();
+
+    if (existingStarCount !== null) {
+      console.warn(
+        `Using existing GitHub stars value (${existingStarCount}) from ${STARS_FILE_PATH}.`,
+      );
+      return;
+    }
+
+    process.exit(1); // Fail only if no checked-in fallback exists.
   }
 }