docs(evals): update trace-level evaluator upgrade guide

[marliessophie]

Summary

• explain why trace-level evaluators need to be upgraded for the v4 data model
• document the SDK-first and UI-recommended Langfuse Assistant / coding assistant flow
• clarify the score-cardinality change from one score per trace to one score per matching observation
• move configuration classifications and the before/after mapping example into an optional collapsible reference

Verification

• pnpm run format:check
• node scripts/check-h1-headings.js
• git diff --check

Full build and link checks were not run for this focused documentation-only change.

Greptile Summary

This PR updates the trace-level evaluator upgrade documentation to reflect the v4 data model and the deprecation timeline (end of October 2026). The "Traces (Legacy)" UI tab is removed from the main LLM-as-a-Judge setup guide and replaced with a deprecation callout; the migration FAQ is substantially rewritten around the new agentic (Langfuse Assistant / coding assistant) upgrade flow.

• llm-as-a-judge.mdx: Removes the "Traces (Legacy)" configuration tab and adds a deprecation warning callout with a deep link to the v4 rollout timeline; the matching explicit anchor is added to both this heading and the v4.mdx rollout-timeline heading.
• llm-as-a-judge-migration.mdx: Full rewrite — restructures the guide into three steps (SDK upgrade → Langfuse UI upgrade → review/validate), documents the in-app vs. coding-assistant paths, clarifies the score-cardinality change (one score per matching observation instead of per trace), and moves the before/after config example into a collapsible technical reference.
• v4.mdx: Adds a trace-level evaluator deprecation entry to the rollout timeline table and an explicit #rollout-timeline anchor so incoming deep links from the callout resolve correctly.

Confidence Score: 4/5

Documentation-only change that is safe to merge; one method-name discrepancy between the migration guide and v4.mdx should be resolved before the page goes live to avoid confusing users attempting the transition.

The rewrite is clear and the cross-file anchor links are consistent. The one concrete problem is the mismatch between the deprecated method names used in the new migration guide (set_current_trace_io() / setActiveTraceIO()) and those referenced in the existing v4.mdx (set_trace_io). Users following both pages simultaneously during an active upgrade will encounter conflicting names, and calling the wrong one will fail silently or raise a runtime error. Everything else — flow, callout placement, score-cardinality explanation, checklist — reads correctly.

content/faq/all/llm-as-a-judge-migration.mdx (line 46) and content/docs/v4.mdx (line 143) need to be reconciled on the canonical deprecated method names for both the Python and JS/TS SDKs.

Flowchart

%%{init: {'theme': 'neutral'}}%%
flowchart TD
    A[User has trace-level evaluators] --> B[Step 1: Upgrade SDK or ingestion path]
    B --> B1[Python SDK v4.7.0+]
    B --> B2[JS/TS SDK v5.4.0+]
    B --> B3[OTel: x-langfuse-ingestion-version: 4]
    B1 & B2 & B3 --> C{Keep legacy evaluator running during transition?}
    C -->|Yes — trace IO now empty| D[Use deprecated set_current_trace_io or setActiveTraceIO temporarily]
    C -->|No legacy trace IO used| E[Step 2: Open Evaluation screen in Langfuse UI]
    D --> E
    E --> F{Recommended upgrade path}
    F -->|Minimal config changes| G[In-app Langfuse Assistant upgrade]
    F -->|Broader instrumentation changes| H[Coding assistant upgrade]
    G & H --> I[Step 3: Review proposed config changes]
    I --> J[Create observation-level evaluators]
    J --> K{Score cardinality change?}
    K -->|Multiple observations match filter| L[Narrow filters using name/type/Is Root Observation]
    K -->|Single observation matches| M[Validate new evaluator results]
    L --> M
    M --> N{Keep legacy evaluator active temporarily?}
    N -->|Yes| O[Compare scores in parallel, then deactivate]
    N -->|No| P[Deactivate legacy evaluator immediately]
    O & P --> Q[Upgrade complete]

Loading

%%{init: {'theme': 'base', 'themeVariables': {"darkMode": true, "background": "#0d1117", "primaryColor": "#21262d", "primaryTextColor": "#e6edf3", "primaryBorderColor": "#8b949e", "lineColor": "#8b949e", "textColor": "#e6edf3", "edgeLabelBackground": "#161b22", "actorBkg": "#21262d", "actorBorder": "#8b949e", "actorTextColor": "#e6edf3", "actorLineColor": "#8b949e", "signalColor": "#8b949e", "signalTextColor": "#e6edf3", "noteBkgColor": "#373320", "noteBorderColor": "#d4a72c", "noteTextColor": "#f0e6c0", "labelBoxBkgColor": "#21262d", "labelBoxBorderColor": "#8b949e", "labelTextColor": "#e6edf3", "loopTextColor": "#e6edf3", "activationBkgColor": "#30363d", "activationBorderColor": "#8b949e"}}}%%
flowchart TD
    A[User has trace-level evaluators] --> B[Step 1: Upgrade SDK or ingestion path]
    B --> B1[Python SDK v4.7.0+]
    B --> B2[JS/TS SDK v5.4.0+]
    B --> B3[OTel: x-langfuse-ingestion-version: 4]
    B1 & B2 & B3 --> C{Keep legacy evaluator running during transition?}
    C -->|Yes — trace IO now empty| D[Use deprecated set_current_trace_io or setActiveTraceIO temporarily]
    C -->|No legacy trace IO used| E[Step 2: Open Evaluation screen in Langfuse UI]
    D --> E
    E --> F{Recommended upgrade path}
    F -->|Minimal config changes| G[In-app Langfuse Assistant upgrade]
    F -->|Broader instrumentation changes| H[Coding assistant upgrade]
    G & H --> I[Step 3: Review proposed config changes]
    I --> J[Create observation-level evaluators]
    J --> K{Score cardinality change?}
    K -->|Multiple observations match filter| L[Narrow filters using name/type/Is Root Observation]
    K -->|Single observation matches| M[Validate new evaluator results]
    L --> M
    M --> N{Keep legacy evaluator active temporarily?}
    N -->|Yes| O[Compare scores in parallel, then deactivate]
    N -->|No| P[Deactivate legacy evaluator immediately]
    O & P --> Q[Upgrade complete]

Loading

Prompt To Fix All With AI

Fix the following 1 code review issue. Work through them one at a time, proposing concise fixes.

---

### Issue 1 of 1
content/faq/all/llm-as-a-judge-migration.mdx:46-47
**Method name mismatch with v4.mdx**

The warning callout names the Python deprecated method as `set_current_trace_io()` and the JS/TS one as `setActiveTraceIO()`, but `content/docs/v4.mdx` (line 143) refers to the same methods generically as `set_trace_io`. The old removed tab in `llm-as-a-judge.mdx` also used `set_trace_io()` / `setTraceIO()`. Users reading both pages during their upgrade will see three different names for what should be the same API — whichever set of names is wrong will cause an `AttributeError` / `TypeError` at runtime. Please confirm the canonical names across the SDK and align all references.

_{Reviews (1): Last reviewed commit: "chore: push" | Re-trigger Greptile}

Greptile also left 1 inline comment on this PR.

[vercel]

The latest updates on your projects. Learn more about Vercel for GitHub.

Project	Deployment	Actions	Updated (UTC)
langfuse-docs	Error		Jul 19, 2026 3:08pm

[github-actions]

@claude review

[greptile-apps]

Method name mismatch with v4.mdx

The warning callout names the Python deprecated method as set_current_trace_io() and the JS/TS one as setActiveTraceIO(), but content/docs/v4.mdx (line 143) refers to the same methods generically as set_trace_io. The old removed tab in llm-as-a-judge.mdx also used set_trace_io() / setTraceIO(). Users reading both pages during their upgrade will see three different names for what should be the same API — whichever set of names is wrong will cause an AttributeError / TypeError at runtime. Please confirm the canonical names across the SDK and align all references.

Prompt To Fix With AI

This is a comment left during a code review.
Path: content/faq/all/llm-as-a-judge-migration.mdx
Line: 46-47

Comment:
**Method name mismatch with v4.mdx**

The warning callout names the Python deprecated method as `set_current_trace_io()` and the JS/TS one as `setActiveTraceIO()`, but `content/docs/v4.mdx` (line 143) refers to the same methods generically as `set_trace_io`. The old removed tab in `llm-as-a-judge.mdx` also used `set_trace_io()` / `setTraceIO()`. Users reading both pages during their upgrade will see three different names for what should be the same API — whichever set of names is wrong will cause an `AttributeError` / `TypeError` at runtime. Please confirm the canonical names across the SDK and align all references.

How can I resolve this? If you propose a fix, please make it concise.

[claude]

This PR is mostly a well-scoped documentation update, but it does not build: the new AvailabilityBanner on the FAQ page passes selfHosted: "v4", which isn't a valid availability id and throws at render time (the Vercel deployment already shows an Error status consistent with this). Please fix that value before merging; deferring for human review.

Extended reasoning...

Overview

The PR rewrites the trace-level → observation-level evaluator upgrade guide across three docs pages (llm-as-a-judge.mdx, v4.mdx, and the FAQ migration guide), adds an anchor to an existing heading, and restructures content into collapsible reference sections. All changes are content/documentation only — no application logic, auth, or data-handling code is touched.

Security risks

None. This is a documentation-only change with no new code paths, inputs, or data handling.

Level of scrutiny

This would normally warrant light scrutiny as a docs-only change, but the PR introduces a genuine build-breaking bug: content/faq/all/llm-as-a-judge-migration.mdx passes selfHosted: "v4" to <AvailabilityBanner />, an id not present in the component's fixed availabilities list (ee, team-add-on, full", "private-beta", "public-beta", "not-available"). This causes plan.availabilityto beundefinedand the render to throwCannot read properties of undefined (reading 'label'), which will fail static generation for this page during pnpm build`. This is already corroborated by the timeline: the Vercel deployment comment on this PR shows deployment status "Error". The PR description itself notes the full build/link checks were not run, which is consistent with this being missed.

Other factors

This is a real, build-breaking defect (already flagged as an inline finding), not a stylistic nit, so I'm deferring rather than approving despite the otherwise low-risk nature of the content changes.

[claude]

🔴 The FAQ page passes selfHosted: "v4" to <AvailabilityBanner />, but "v4" isn't one of the ids the component recognizes (ee, team-add-on, full, private-beta, public-beta, not-available). This throws a TypeError: Cannot read properties of undefined (reading 'label') when the page is statically rendered, breaking the build for this page. Change selfHosted: "v4" to selfHosted: "full" (this feature is available on self-hosted per the guide's own "OSS and self-hosting" section).

Extended reasoning...

content/faq/all/llm-as-a-judge-migration.mdx (lines 9-18) adds an <AvailabilityBanner /> with:

<AvailabilityBanner
  availability={{
    hobby: "full",
    core: "full",
    pro: "full",
    enterprise: "full",
    selfHosted: "v4",
  }}
/>

components/Availability.tsx defines a fixed availabilities array (lines 13-25) containing only these ids: ee, team-add-on, full, private-beta, public-beta, not-available. "v4" is not among them.

The exact failure path:

1. AvailabilityBanner builds availablePlans via plans.map(plan => ({ ...plan, availability: availabilities.find(a => a.id === props.availability[plan.id]) })) (lines 33-40).
2. For the selfHosted plan, props.availability["selfHosted"] is "v4".
3. availabilities.find(a => a.id === "v4") matches nothing and returns undefined.
4. The render then does <span>{plan.availability.label}</span> (line 54) with no null/optional-chaining guard.
5. Accessing .label on undefined throws TypeError: Cannot read properties of undefined (reading 'label').

Why TypeScript doesn't catch it: the prop type is Record<(typeof plans)[number]["id"], (typeof availabilities)[number]["id"]> (lines 28-31), but availabilities is explicitly annotated as { id: string; label?: string }[] (lines 9-12). That annotation widens id to plain string rather than inferring a literal union from the array contents, so any string — including "v4" — type-checks fine even though it doesn't match any actual entry.

Why this matters for this PR specifically: AvailabilityBanner is a globally registered MDX component (wired in mdx-components.tsx), so it renders unconditionally wherever it's used — there's no lazy/conditional path that would avoid triggering the crash. Since FAQ pages under content/faq/ are statically rendered (SSG) via Fumadocs/MDX, this throws during pnpm build for /faq/all/llm-as-a-judge-migration, not just in some edge-case runtime path. The PR description explicitly states "Full build and link checks were not run for this focused documentation-only change," which is exactly why this wasn't caught before opening the PR. Every other existing usage of AvailabilityBanner in the repo uses a valid id (ee, full, not-available, public-beta); this is the only occurrence of "v4" anywhere.

Fix: replace selfHosted: "v4" with a valid id. Given the page's own "OSS and self-hosting" section states "This preview is currently available on Langfuse Cloud. We are working on the migration path for OSS deployments," the intended value is most likely "full" (already available) or "private-beta"/"not-available" depending on the precise self-hosted rollout status the author wants to communicate — but any of the six valid ids resolves the crash. As a durability improvement (optional, separate from this PR), the availabilities array's id field could be typed as a literal union (e.g. via as const) so TypeScript would catch invalid ids like "v4" at compile time instead of only failing at render.