From 3e1f267fd60f980b66c5048c44780b991021e285 Mon Sep 17 00:00:00 2001 From: Marlies Mayerhofer <74332854+marliessophie@users.noreply.github.com> Date: Fri, 17 Jul 2026 01:14:47 +0200 Subject: [PATCH 01/13] docs(evals): update trace-level evaluator upgrade guide --- content/faq/all/llm-as-a-judge-migration.mdx | 221 ++++++------------- 1 file changed, 73 insertions(+), 148 deletions(-) diff --git a/content/faq/all/llm-as-a-judge-migration.mdx b/content/faq/all/llm-as-a-judge-migration.mdx index 7c8e1acd23..a42afdb227 100644 --- a/content/faq/all/llm-as-a-judge-migration.mdx +++ b/content/faq/all/llm-as-a-judge-migration.mdx @@ -1,194 +1,119 @@ --- -title: How do I upgrade my trace-level evaluators to observation-level evaluators? +title: How do I upgrade trace-level evaluators to observation-level evaluators? tags: [evaluation] --- -# Evaluator Upgrade Guide +# Upgrade trace-level evaluators -Langfuse has introduced running evaluators on observations as the recommended approach for live data LLM-as-a-Judge evaluations. This guide helps you upgrade existing live data evaluators to the new system. +Trace-level evaluators use Langfuse's legacy trace-centric data model and are being deprecated as part of the [Langfuse v4 upgrade](/docs/v4). Existing evaluators continue to run while you upgrade. Based on existing configurations, the overwhelming majority can be upgraded by targeting the observation that already contains the required input, output, and context. -**Setting up new evaluators?** Use the [LLM-as-a-Judge setup guide](/docs/evaluation/evaluation-methods/llm-as-a-judge#set-up-step-by-step) instead. Start with observation-level evaluators directly for the best experience. This guide is only for upgrading existing trace-level evaluators. +**Setting up a new evaluator?** Start with an observation-level evaluator in the [LLM-as-a-Judge setup guide](/docs/evaluation/evaluation-methods/llm-as-a-judge#set-up-step-by-step). This page is only for upgrading existing trace-level evaluators. -## Why Upgrade to Observation-Level Evaluators? +## Why this upgrade is needed -Observation-level evaluators offer: +- **Real-time execution:** The legacy engine waits for and assembles a full trace before evaluating it. Observation-level evaluators run as observations are ingested. +- **Scalable by design:** Evaluating one observation avoids loading the full trace and scales with larger traces and higher evaluation volume. +- **Required for future capabilities:** Any future multi-span support will also use the observation-centric data model. Upgrading now is the foundation for it. -- **Dramatically faster execution**: Evaluations complete in seconds instead of minutes. Eliminates evaluation delays and backlog issues. Optimized architecture processes thousands of evaluations per minute. -- **Operation-level precision**: Evaluate specific operations (final LLM responses, retrieval steps) not entire workflows. Reduces evaluation volume and cost. -- **Compositional evaluation**: Run different evaluators on different operations simultaneously -- **Improved reliability**: More predictable behavior, better error handling and retry logic -- **Future-proof architecture**: Built for upcoming features and capabilities +## How to upgrade -Trace-level evaluators will continue to work. This is an upgrade path, not a deprecation. + +**First, upgrade your SDK or ingestion path. Do this before changing evaluator configurations.** -## When to Upgrade +Observation-level evaluators run in real time with: -### ✅ Upgrade Now If: +- **Python SDK:** v4.7.0+ ([upgrade guide](/docs/observability/sdk/upgrade-path/python-v3-to-v4)) +- **JS/TS SDK:** v5.4.0+ ([upgrade guide](/docs/observability/sdk/upgrade-path/js-v4-to-v5)) +- **Direct OpenTelemetry ingestion:** `x-langfuse-ingestion-version: 4` ([setup guide](/integrations/native/opentelemetry)) -- **You're experiencing evaluation delays, backlogs, or slow evaluation processing** (observation-level evaluations complete in seconds) -- You're using or planning to use OTel-based SDKs (Python v3+ or JS/TS v4+) -- You're setting up new evaluators +After upgrading to the [v4 data model](/docs/v4), input, output, and context are no longer available as separate trace fields. The equivalent values must be available on an observation. -### ⏸️ Stay with Trace-Level For Now If: +{/* TODO: Link the in-product evaluator upgrade flow directly to the SDK upgrade guides in this section. */} -- You're on legacy SDKs (Python v2 or JS/TS v3) and can't upgrade yet -- Your trace evaluators work perfectly for your use case -- You need to evaluate aggregate workflow data spanning multiple operations - -## Prerequisites - -### SDK Requirements for Observation-Level Evaluators - -| Requirement | Python | JS/TS | -| ----------------------- | ---------------------------------------------------------------------- | ----------------------------------------------------------------- | -| **Minimum SDK version** | v3+ (OTel-based) | v4+ (OTel-based) | -| **Upgrade guide** | [Python v2 → v3](/docs/observability/sdk/upgrade-path/python-v2-to-v3) | [JS/TS v3 → v4](/docs/observability/sdk/upgrade-path/js-v3-to-v4) | - -### Zero-Downtime Migration Strategy - -You can set up observation-level evaluators before upgrading your SDK version. This allows you to: - -1. Deploy new evaluators in parallel with existing trace-level evaluators -2. Upgrade your SDK when ready without evaluation downtime - -This is not required but offers the smoothest upgrade path for production systems. - -## Upgrade Process - -Langfuse provides a built-in wizard to upgrade your evaluators. The wizard helps translate your trace-level configuration to observation-level. - -### Step 1: Access the Upgrade Wizard - -1. Navigate to your project → Evaluation → LLM-as-a-Judge -2. Find evaluators marked "Legacy" -3. Click on the evaluator row you want to upgrade -4. Click "Upgrade" in the top right corner - -### Step 2: Review and Configure - -The wizard shows your configuration side-by-side: - -- **Left side (read-only)**: Your current trace-level configuration -- **Right side (editable)**: Observation-level configuration - -**Key configuration steps**: + -1. **Filters**: Add filters to ensure you target the same trace-level attributes as before. We recommend adding new observation-level filters to target a specific observation within the trace. - - Trace-level filters (e.g., `traceName`, `userId`, `tags`) - - Observation type (e.g., `generation`, `span`, `event`) - - Observation name (e.g., `chat-completion`) +### 1. Start the upgrade in Langfuse -2. **Variable Mapping**: Map variables from observation fields - - Change from `trace.input` → `observation.input` - - Change from `trace.output` → `observation.output` - - Use `observation.metadata` for custom fields +Open **Evaluation > LLM-as-a-Judge**, select the legacy evaluator, and start the upgrade. Langfuse reviews its filters and variable mappings, then recommends the appropriate assistant. -3. **Choose upgrade action**: - - **Keep both active**: Test new evaluator alongside old one - - **Mark old as inactive**: Old evaluator stops, new one takes over (recommended initially) - - **Delete old evaluator**: Permanently remove legacy evaluator +### 2. Follow the recommended assistant path -### Step 3: Verify Execution +- **Langfuse Assistant:** Recommended when the evaluator can be upgraded by selecting an existing observation and translating its configuration. This is the fastest path and does not require manual UI setup. +- **Coding assistant:** Recommended when the SDK or instrumentation must change first. Langfuse provides the relevant context and prompt to use in your coding editor. -After upgrading: +Some upgrades use both: update the instrumentation with a coding assistant, then return to Langfuse Assistant to select the correct observation and create the new evaluator. -1. **Check execution logs** - - Go to Evaluator Table → find new evaluator → click "Logs" - - Verify evaluations are running +### 3. Review and create the evaluator -2. **Compare results** (if running both) - - Use [score analytics](/docs/evaluation/scores/score-analytics) to compare scores - - Ensure evaluation logic consistency - - Note: Scores from observation-level evaluators appear on the specific observation in the trace tree, not on the trace itself. Depending on your filter criteria, multiple observations may match the criteria and result in multiple scores per trace. +Review the proposed target observation, filters, and variable mappings. Create the observation-level evaluator, but keep the legacy evaluator available until you validate the results. -## Upgrade Example: Simple Trace Evaluator +## How scores change -Most trace evaluators have input/output that map directly to a specific observation's input/output. The upgrade targets this observation directly. +A trace-level evaluator produces one score per matching trace. An observation-level evaluator produces one score per matching observation. -**Scenario**: You have a trace-level evaluator that evaluates a chatbot's response. The trace has a span observation named "user-workflow" containing the same input/output as the trace. +If three observations in one trace match the filters, the upgraded evaluator produces three scores. This is expected. To keep one score per trace, narrow the filters to one observation using its name, type, or root-observation status. - ![Upgrade example](/images/docs/evaluator-upgrade-example.png) + ![Before and after the upgrade: the evaluator score moves from the trace to the matched observation](/images/docs/evaluator-score-output-example.png) -**Before (Trace-level)**: - -``` -Target: Traces -Filter: name = "user-workflow" - AND environment not in ["sdk-experiment"] -Variables: - - query: trace.input.key - - generation: trace.output -``` - -**After (Observation-level)**: +## Validate the upgrade -``` -Target: Observations -Filter: traceName = "user-workflow" - AND environment not in ["sdk-experiment"] - AND type = "span" -Variables: - - query: observation.input.key - - generation: observation.output -``` +1. Keep the legacy and observation-level evaluators active briefly. +2. Confirm that the new evaluator receives the expected input, output, and context. +3. Check that the number of matched observations and scores per trace is intentional. +4. Deactivate the legacy evaluator once the new results are validated. -**Key Changes**: +If you need to revert, deactivate the new evaluator and reactivate the existing legacy evaluator. Historical evaluation results remain accessible. -1. **Added observation-level filter**: Identify the specific observation within the trace tree - - `observation.type = "span"` targets the specific span observation +## How existing configurations are grouped -2. **Changed variable sources**: Variables now come from observation instead of trace - - `trace.input.key` → `observation.input.key` - - `trace.output` → `observation.output` +Langfuse uses the current variable mappings to recommend an upgrade path. You do not need to classify the evaluator yourself. -3. **Result**: Evaluator now runs on each matching observation individually, providing: - - Dramatically faster execution (completes in seconds, not minutes—no need to load entire trace) - - More precise targeting (only evaluates the specific operation) - - Better cost control (can filter to specific operation types) +
+View configuration cases and a before/after example -**Important**: The resulting score is now **attached to the observation**, not the trace. This means: +| Current variable mapping | How Langfuse handles it | +| -------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | +| **`Observation` for every variable, all from one observation** | Configuration-only upgrade. Langfuse Assistant targets the same span or generation and translates the filters and variable mappings. | +| **`Trace` for every variable** | Langfuse Assistant inspects representative traces and helps select the observation holding the equivalent input, output, and metadata. For end-to-end evaluations, this is often a root observation. | +| **A mix of `Trace` and one `Observation`** | Target that observation and make the trace fields available there via [`propagate_attributes()`](/docs/observability/sdk/instrumentation#add-attributes) or an instrumentation update. | +| **More than one `Observation`, with or without `Trace`** | Update instrumentation first: write the required values to a root or dedicated evaluation observation, then target it directly. Any future multi-span support will also use the observation-centric model. | -- Scores appear on the specific observation in the trace tree -- You can have multiple scores per trace (one per evaluated observation) - -Observation-level evaluators can only map variables from the matched observation. They do not automatically read sibling or child observations from the same trace. If the judge needs an end-to-end request and response, target a root observation that records those values, and write any additional required context onto that root observation. - - - ![Score output example](/images/docs/evaluator-score-output-example.png) - +### Before and after example -## SDK Version-Specific Guidance +This example upgrades a trace-level evaluator that reads trace input and output to target the span containing the same values. -### For Users on Legacy SDKs (Python v2, JS/TS v3) +**Before: trace-level evaluator** -**Option 1: Upgrade to OTel-based SDK** (Recommended) - -1. Upgrade to Python SDK v3+ or JS/TS SDK v4+ (see [Prerequisites](#prerequisites)) -2. Update your instrumentation code using the migration guides -3. Upgrade evaluators using the wizard - -**Option 2: Continue with Trace-Level Evaluators** - -- No changes needed—trace-level evaluators will continue to work -- Upgrade when ready - -### For Users on OTel-based SDKs (Python v3+, JS/TS v4+) - -If you have existing trace-level evaluators, we recommend upgrading to observation-level evaluators using the wizard to get the full benefits of the new architecture. - -## Rollback Plan +```text +Target: Trace +Filters: + environment not in ["sdk-experiment"] + name = "user-workflow" +Variable mapping: + query = trace.input.key + generation = trace.output +``` -If you need to revert after upgrading: +**After: observation-level evaluator** + +```text +Target: Observation +Filters: + environment not in ["sdk-experiment"] + traceName = "user-workflow" + type = "SPAN" +Variable mapping: + query = observation.input.key + generation = observation.output +``` -1. **If you kept both evaluators**: Mark the new one as inactive, reactivate the old one -2. **If you deleted the old evaluator**: Create a new trace-level evaluator with the old configuration -3. **Data is preserved**: All historical evaluation results remain accessible +
-## Getting Help +## Getting help - **Troubleshooting**: [Why is my observation-level evaluator not executing?](/faq/all/observation-eval-not-executing) - **Documentation**: [LLM-as-a-Judge step-by-step set up guide](/docs/evaluation/evaluation-methods/llm-as-a-judge) @@ -197,4 +122,4 @@ If you need to revert after upgrading: --- -_Last updated: February 10, 2026_ +_Last updated: July 17, 2026_ From 3191c7d672ffe76fb404082ac344600ae7a136ca Mon Sep 17 00:00:00 2001 From: Marlies Mayerhofer <74332854+marliessophie@users.noreply.github.com> Date: Fri, 17 Jul 2026 01:25:56 +0200 Subject: [PATCH 02/13] docs(evals): address upgrade guide feedback --- content/faq/all/llm-as-a-judge-migration.mdx | 37 ++++++++++---------- 1 file changed, 19 insertions(+), 18 deletions(-) diff --git a/content/faq/all/llm-as-a-judge-migration.mdx b/content/faq/all/llm-as-a-judge-migration.mdx index a42afdb227..bf298083bb 100644 --- a/content/faq/all/llm-as-a-judge-migration.mdx +++ b/content/faq/all/llm-as-a-judge-migration.mdx @@ -36,20 +36,20 @@ After upgrading to the [v4 data model](/docs/v4), input, output, and context are ### 1. Start the upgrade in Langfuse -Open **Evaluation > LLM-as-a-Judge**, select the legacy evaluator, and start the upgrade. Langfuse reviews its filters and variable mappings, then recommends the appropriate assistant. +Open **Evaluation > LLM-as-a-Judge**. You will be greeted with the upgrade screen, which walks you through all evaluators that need to be upgraded and recommends the appropriate assistant for each one. ### 2. Follow the recommended assistant path - **Langfuse Assistant:** Recommended when the evaluator can be upgraded by selecting an existing observation and translating its configuration. This is the fastest path and does not require manual UI setup. - **Coding assistant:** Recommended when the SDK or instrumentation must change first. Langfuse provides the relevant context and prompt to use in your coding editor. -Some upgrades use both: update the instrumentation with a coding assistant, then return to Langfuse Assistant to select the correct observation and create the new evaluator. +The upgrade screen and recommended assistant guide you through the full process. You do not need to recreate evaluator configurations manually in the UI. ### 3. Review and create the evaluator -Review the proposed target observation, filters, and variable mappings. Create the observation-level evaluator, but keep the legacy evaluator available until you validate the results. +Review the proposed target observation, filters, and variable mappings. When creating the observation-level evaluator, you can keep the legacy evaluator active to compare results or deactivate it immediately. -## How scores change +## How resulting evaluation scores might look different A trace-level evaluator produces one score per matching trace. An observation-level evaluator produces one score per matching observation. @@ -61,26 +61,27 @@ If three observations in one trace match the filters, the upgraded evaluator pro ## Validate the upgrade -1. Keep the legacy and observation-level evaluators active briefly. -2. Confirm that the new evaluator receives the expected input, output, and context. -3. Check that the number of matched observations and scores per trace is intentional. -4. Deactivate the legacy evaluator once the new results are validated. +1. Confirm that the new evaluator receives the expected input, output, and context. +2. Check that the number of matched observations and scores per trace is intentional. +3. If you kept the legacy evaluator active, compare the results and deactivate it when ready. If you need to revert, deactivate the new evaluator and reactivate the existing legacy evaluator. Historical evaluation results remain accessible. -## How existing configurations are grouped +## Optional: How configurations might change -Langfuse uses the current variable mappings to recommend an upgrade path. You do not need to classify the evaluator yourself. +The upgrade screen and recommended assistant guide you through every change. You do not need to classify or update configurations manually. Expand this section only if you want to understand the technical details.
-View configuration cases and a before/after example - -| Current variable mapping | How Langfuse handles it | -| -------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -| **`Observation` for every variable, all from one observation** | Configuration-only upgrade. Langfuse Assistant targets the same span or generation and translates the filters and variable mappings. | -| **`Trace` for every variable** | Langfuse Assistant inspects representative traces and helps select the observation holding the equivalent input, output, and metadata. For end-to-end evaluations, this is often a root observation. | -| **A mix of `Trace` and one `Observation`** | Target that observation and make the trace fields available there via [`propagate_attributes()`](/docs/observability/sdk/instrumentation#add-attributes) or an instrumentation update. | -| **More than one `Observation`, with or without `Trace`** | Update instrumentation first: write the required values to a root or dedicated evaluation observation, then target it directly. Any future multi-span support will also use the observation-centric model. | +View technical configuration cases and a before/after example + +Langfuse groups existing configurations by their variable mappings and recommends an assistant based on the required change: + +| Current variable mapping | Typical configuration change | +| -------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | +| **`Observation` for every variable, all from one observation** | Target the same span or generation and translate the filters and variable mappings. | +| **`Trace` for every variable** | Select the observation holding the equivalent input, output, and metadata. For end-to-end evaluations, this is often a root observation. | +| **A mix of `Trace` and one `Observation`** | Target that observation and make the trace fields available there via [`propagate_attributes()`](/docs/observability/sdk/instrumentation#add-attributes) or an instrumentation update. | +| **More than one `Observation`, with or without `Trace`** | Write the required values to a root or dedicated evaluation observation, then target it directly. Any future multi-span support will also use the observation-centric model. | ### Before and after example From 9ce60fedd88356275bb078246ea80f869c3b747b Mon Sep 17 00:00:00 2001 From: Marlies Mayerhofer <74332854+marliessophie@users.noreply.github.com> Date: Sun, 19 Jul 2026 10:30:34 +0200 Subject: [PATCH 03/13] docs(evals): restructure evaluator upgrade guide --- content/faq/all/llm-as-a-judge-migration.mdx | 77 +++++++++++++------- 1 file changed, 49 insertions(+), 28 deletions(-) diff --git a/content/faq/all/llm-as-a-judge-migration.mdx b/content/faq/all/llm-as-a-judge-migration.mdx index bf298083bb..c5a72a0878 100644 --- a/content/faq/all/llm-as-a-judge-migration.mdx +++ b/content/faq/all/llm-as-a-judge-migration.mdx @@ -3,26 +3,27 @@ title: How do I upgrade trace-level evaluators to observation-level evaluators? tags: [evaluation] --- +import { Steps, Step } from "@/components/docs"; + # Upgrade trace-level evaluators -Trace-level evaluators use Langfuse's legacy trace-centric data model and are being deprecated as part of the [Langfuse v4 upgrade](/docs/v4). Existing evaluators continue to run while you upgrade. Based on existing configurations, the overwhelming majority can be upgraded by targeting the observation that already contains the required input, output, and context. +Trace-level evaluators use Langfuse's legacy trace-centric data model and are being deprecated as part of the [Langfuse v4 upgrade](/docs/v4). Existing evaluators continue to run while you upgrade. + +Observation-level evaluators run in real time and scale with larger traces and higher evaluation volume. Any future multi-span support will also use the observation-centric data model, so upgrading is the foundation for future evaluation capabilities. **Setting up a new evaluator?** Start with an observation-level evaluator in the [LLM-as-a-Judge setup guide](/docs/evaluation/evaluation-methods/llm-as-a-judge#set-up-step-by-step). This page is only for upgrading existing trace-level evaluators. -## Why this upgrade is needed +## Upgrade your evaluator -- **Real-time execution:** The legacy engine waits for and assembles a full trace before evaluating it. Observation-level evaluators run as observations are ingested. -- **Scalable by design:** Evaluating one observation avoids loading the full trace and scales with larger traces and higher evaluation volume. -- **Required for future capabilities:** Any future multi-span support will also use the observation-centric data model. Upgrading now is the foundation for it. + -## How to upgrade + - -**First, upgrade your SDK or ingestion path. Do this before changing evaluator configurations.** +### Before you start -Observation-level evaluators run in real time with: +First, upgrade your SDK or ingestion path. Observation-level evaluators run in real time with: - **Python SDK:** v4.7.0+ ([upgrade guide](/docs/observability/sdk/upgrade-path/python-v3-to-v4)) - **JS/TS SDK:** v5.4.0+ ([upgrade guide](/docs/observability/sdk/upgrade-path/js-v4-to-v5)) @@ -32,44 +33,54 @@ After upgrading to the [v4 data model](/docs/v4), input, output, and context are {/* TODO: Link the in-product evaluator upgrade flow directly to the SDK upgrade guides in this section. */} - + + + + +### Upgrade in Langfuse -### 1. Start the upgrade in Langfuse +Open **Evaluation > LLM-as-a-Judge**. The upgrade screen walks you through all evaluators that need to be upgraded and recommends one of two paths for each evaluator: -Open **Evaluation > LLM-as-a-Judge**. You will be greeted with the upgrade screen, which walks you through all evaluators that need to be upgraded and recommends the appropriate assistant for each one. +- **Langfuse Assistant:** Selects an existing observation and translates the evaluator configuration in the app. +- **Coding assistant:** Uses the provided evaluator context and prompt to make any required SDK or instrumentation changes in your coding editor. -### 2. Follow the recommended assistant path +The agentic flow makes the upgrade conversational. You can inspect the proposed target observation, filters, and variable mappings, ask why a change is needed, and refine the configuration before creating the evaluator. You do not need to translate or recreate the configuration manually. -- **Langfuse Assistant:** Recommended when the evaluator can be upgraded by selecting an existing observation and translating its configuration. This is the fastest path and does not require manual UI setup. -- **Coding assistant:** Recommended when the SDK or instrumentation must change first. Langfuse provides the relevant context and prompt to use in your coding editor. + -The upgrade screen and recommended assistant guide you through the full process. You do not need to recreate evaluator configurations manually in the UI. + -### 3. Review and create the evaluator +### Review, create, and validate -Review the proposed target observation, filters, and variable mappings. When creating the observation-level evaluator, you can keep the legacy evaluator active to compare results or deactivate it immediately. +**How the configuration may change** -## How resulting evaluation scores might look different +Review the target observation, filters, and variable mappings proposed by the assistant. Depending on the existing evaluator, the configuration may target the same observation as before, replace trace fields with values from one observation, or require an instrumentation change so all required values are available together. The assistant guides you through the appropriate change; the [technical reference](#technical-reference) below explains the cases in detail. -A trace-level evaluator produces one score per matching trace. An observation-level evaluator produces one score per matching observation. +**How resulting scores may differ** -If three observations in one trace match the filters, the upgraded evaluator produces three scores. This is expected. To keep one score per trace, narrow the filters to one observation using its name, type, or root-observation status. +Before creating the evaluator, confirm how the resulting scores may change: + +- A trace-level evaluator produces one score per matching trace. +- An observation-level evaluator produces one score per matching observation. +- If three observations in one trace match the filters, the upgraded evaluator produces three scores. To keep one score per trace, narrow the filters to one observation using its name, type, or root-observation status. ![Before and after the upgrade: the evaluator score moves from the trace to the matched observation](/images/docs/evaluator-score-output-example.png) -## Validate the upgrade +**Create and validate** -1. Confirm that the new evaluator receives the expected input, output, and context. -2. Check that the number of matched observations and scores per trace is intentional. -3. If you kept the legacy evaluator active, compare the results and deactivate it when ready. +Create the observation-level evaluator, then confirm that it receives the expected input, output, and context and produces the intended number of scores. You can keep the legacy evaluator active temporarily to compare results, or deactivate it immediately. If you need to revert, deactivate the new evaluator and reactivate the existing legacy evaluator. Historical evaluation results remain accessible. -## Optional: How configurations might change + + + + +## Technical reference [#technical-reference] -The upgrade screen and recommended assistant guide you through every change. You do not need to classify or update configurations manually. Expand this section only if you want to understand the technical details. +The recommended assistant guides you through every change. You do not need to classify or update configurations manually. Expand this section only if you want to understand how an existing configuration may be translated.
View technical configuration cases and a before/after example @@ -114,6 +125,16 @@ Variable mapping:
+## Upgrade checklist + +- [ ] Upgrade the SDK or OpenTelemetry ingestion path. +- [ ] Open the evaluator upgrade screen in **Evaluation > LLM-as-a-Judge**. +- [ ] Follow the recommended Langfuse Assistant or coding assistant path. +- [ ] Review the target observation, filters, and variable mappings. +- [ ] Confirm the expected number and placement of scores. +- [ ] Decide whether to keep the legacy evaluator active for comparison. +- [ ] Validate the evaluator results and deactivate the legacy evaluator when ready. + ## Getting help - **Troubleshooting**: [Why is my observation-level evaluator not executing?](/faq/all/observation-eval-not-executing) @@ -123,4 +144,4 @@ Variable mapping: --- -_Last updated: July 17, 2026_ +_Last updated: July 19, 2026_ From 5fecbac26b695e00669decbd71bb7d343389f47b Mon Sep 17 00:00:00 2001 From: Marlies Mayerhofer <74332854+marliessophie@users.noreply.github.com> Date: Sun, 19 Jul 2026 10:41:02 +0200 Subject: [PATCH 04/13] docs(evals): clarify guided upgrade flow --- content/faq/all/llm-as-a-judge-migration.mdx | 32 +++++++++++--------- 1 file changed, 17 insertions(+), 15 deletions(-) diff --git a/content/faq/all/llm-as-a-judge-migration.mdx b/content/faq/all/llm-as-a-judge-migration.mdx index c5a72a0878..5847035925 100644 --- a/content/faq/all/llm-as-a-judge-migration.mdx +++ b/content/faq/all/llm-as-a-judge-migration.mdx @@ -29,7 +29,7 @@ First, upgrade your SDK or ingestion path. Observation-level evaluators run in r - **JS/TS SDK:** v5.4.0+ ([upgrade guide](/docs/observability/sdk/upgrade-path/js-v4-to-v5)) - **Direct OpenTelemetry ingestion:** `x-langfuse-ingestion-version: 4` ([setup guide](/integrations/native/opentelemetry)) -After upgrading to the [v4 data model](/docs/v4), input, output, and context are no longer available as separate trace fields. The equivalent values must be available on an observation. +In the [v4 data model](/docs/v4), a trace is the logical group of observations that share a trace ID, rather than a separate record. Input and output belong to observations. In most cases, an existing root or workflow observation already carries the same input and output that was previously written to the trace. The evaluator upgrade targets that observation. If no single observation contains all required values, the coding assistant helps update your instrumentation. {/* TODO: Link the in-product evaluator upgrade flow directly to the SDK upgrade guides in this section. */} @@ -39,12 +39,12 @@ After upgrading to the [v4 data model](/docs/v4), input, output, and context are ### Upgrade in Langfuse -Open **Evaluation > LLM-as-a-Judge**. The upgrade screen walks you through all evaluators that need to be upgraded and recommends one of two paths for each evaluator: +Open **Evaluation > LLM-as-a-Judge**. The upgrade screen shows the recommended path for your project: -- **Langfuse Assistant:** Selects an existing observation and translates the evaluator configuration in the app. -- **Coding assistant:** Uses the provided evaluator context and prompt to make any required SDK or instrumentation changes in your coding editor. +- **In-app upgrade:** Recommended when only minimal configuration changes need to be reviewed and accepted. Langfuse Assistant guides you through these changes in the app. +- **Coding assistant:** Recommended when the upgrade requires broader configuration or instrumentation changes. The UI provides the skill and project context to use in your coding editor. -The agentic flow makes the upgrade conversational. You can inspect the proposed target observation, filters, and variable mappings, ask why a change is needed, and refine the configuration before creating the evaluator. You do not need to translate or recreate the configuration manually. +In both paths, the agentic flow makes the upgrade conversational: you can inspect the proposed changes, ask questions, and refine them before creating the new evaluators. You do not need to translate or recreate configurations manually. @@ -52,13 +52,21 @@ The agentic flow makes the upgrade conversational. You can inspect the proposed ### Review, create, and validate -**How the configuration may change** +The recommended agent guides you through reviewing the proposed changes, creating the observation-level evaluators, and validating the result against your project data. You can ask questions and refine the target observation, filters, variable mappings, and expected score behavior until the upgrade looks correct. -Review the target observation, filters, and variable mappings proposed by the assistant. Depending on the existing evaluator, the configuration may target the same observation as before, replace trace fields with values from one observation, or require an instrumentation change so all required values are available together. The assistant guides you through the appropriate change; the [technical reference](#technical-reference) below explains the cases in detail. +When ready, create the observation-level evaluators. You can keep the legacy evaluators active temporarily to compare results, or deactivate them immediately. Confirm that the new evaluators receive the expected input, output, and context and produce the intended number of scores. + +If you need to revert, deactivate the new evaluators and reactivate the existing legacy evaluators. Historical evaluation results remain accessible. + +The agent explains the relevant changes during the upgrade. The following sections summarize what may differ if you want to understand the underlying behavior. + +**How configurations may change** + +Depending on the existing evaluator, the proposed configuration may target the same observation as before, replace trace fields with values from one observation, or require an instrumentation change so all required values are available together. The [technical reference](#technical-reference) below explains the cases in detail. **How resulting scores may differ** -Before creating the evaluator, confirm how the resulting scores may change: +Score placement and cardinality may change: - A trace-level evaluator produces one score per matching trace. - An observation-level evaluator produces one score per matching observation. @@ -68,12 +76,6 @@ Before creating the evaluator, confirm how the resulting scores may change: ![Before and after the upgrade: the evaluator score moves from the trace to the matched observation](/images/docs/evaluator-score-output-example.png) -**Create and validate** - -Create the observation-level evaluator, then confirm that it receives the expected input, output, and context and produces the intended number of scores. You can keep the legacy evaluator active temporarily to compare results, or deactivate it immediately. - -If you need to revert, deactivate the new evaluator and reactivate the existing legacy evaluator. Historical evaluation results remain accessible. - @@ -85,7 +87,7 @@ The recommended assistant guides you through every change. You do not need to cl
View technical configuration cases and a before/after example -Langfuse groups existing configurations by their variable mappings and recommends an assistant based on the required change: +Existing evaluator configurations typically fall into these cases: | Current variable mapping | Typical configuration change | | -------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | From 4d68aaf4aa81609ea875024fdf6513a8cc871e57 Mon Sep 17 00:00:00 2001 From: Marlies Mayerhofer <74332854+marliessophie@users.noreply.github.com> Date: Sun, 19 Jul 2026 16:12:08 +0200 Subject: [PATCH 05/13] docs(evals): add transition bridge and availability notes to upgrade guide - Document deprecated set_trace_io/setActiveTraceIO bridge for keeping legacy trace-level evaluators running during the transition - Add AvailabilityBanner and self-hosted scoping with GitHub Discussion link - Add frontmatter description, Fast Preview terminology, Is Root Observation filter name, Langfuse Assistant link, remove stale last-updated footer - Add explicit anchor to the linked LLM-as-a-Judge setup heading Co-Authored-By: Claude Fable 5 --- .../evaluation-methods/llm-as-a-judge.mdx | 2 +- content/faq/all/llm-as-a-judge-migration.mdx | 33 ++++++++++++------- 2 files changed, 22 insertions(+), 13 deletions(-) diff --git a/content/docs/evaluation/evaluation-methods/llm-as-a-judge.mdx b/content/docs/evaluation/evaluation-methods/llm-as-a-judge.mdx index d2be42b1f9..d1a4245b20 100644 --- a/content/docs/evaluation/evaluation-methods/llm-as-a-judge.mdx +++ b/content/docs/evaluation/evaluation-methods/llm-as-a-judge.mdx @@ -207,7 +207,7 @@ Each experiment run generates traces that are automatically scored by your selec -## Set up step-by-step +## Set up step-by-step [#set-up-step-by-step] diff --git a/content/faq/all/llm-as-a-judge-migration.mdx b/content/faq/all/llm-as-a-judge-migration.mdx index 5847035925..22e16e7ccc 100644 --- a/content/faq/all/llm-as-a-judge-migration.mdx +++ b/content/faq/all/llm-as-a-judge-migration.mdx @@ -1,5 +1,6 @@ --- title: How do I upgrade trace-level evaluators to observation-level evaluators? +description: Step-by-step guide for upgrading legacy trace-level LLM-as-a-Judge evaluators to observation-level evaluators as part of Langfuse Fast Preview (v4). tags: [evaluation] --- @@ -7,7 +8,17 @@ import { Steps, Step } from "@/components/docs"; # Upgrade trace-level evaluators -Trace-level evaluators use Langfuse's legacy trace-centric data model and are being deprecated as part of the [Langfuse v4 upgrade](/docs/v4). Existing evaluators continue to run while you upgrade. + + +Trace-level evaluators use Langfuse's legacy trace-centric data model and are being deprecated as part of [Langfuse Fast Preview (v4)](/docs/v4). Existing evaluators continue to run while you upgrade. Fast Preview is currently available on Langfuse Cloud; for self-hosted support updates, follow the [GitHub Discussion](https://github.com/orgs/langfuse/discussions/12518). Observation-level evaluators run in real time and scale with larger traces and higher evaluation volume. Any future multi-span support will also use the observation-centric data model, so upgrading is the foundation for future evaluation capabilities. @@ -31,7 +42,9 @@ First, upgrade your SDK or ingestion path. Observation-level evaluators run in r In the [v4 data model](/docs/v4), a trace is the logical group of observations that share a trace ID, rather than a separate record. Input and output belong to observations. In most cases, an existing root or workflow observation already carries the same input and output that was previously written to the trace. The evaluator upgrade targets that observation. If no single observation contains all required values, the coding assistant helps update your instrumentation. -{/* TODO: Link the in-product evaluator upgrade flow directly to the SDK upgrade guides in this section. */} + +**Keep legacy evaluators running during the transition.** After the SDK upgrade, trace input and output are no longer written automatically. If your existing trace-level evaluators read trace input or output, keep supplying these values with the deprecated `set_current_trace_io()` (Python) / `setActiveTraceIO()` (JS/TS) methods until the new observation-level evaluators are validated, then remove the calls. See [Why are the input and output of a trace empty?](/faq/all/empty-trace-input-and-output) for details. + @@ -39,9 +52,9 @@ In the [v4 data model](/docs/v4), a trace is the logical group of observations t ### Upgrade in Langfuse -Open **Evaluation > LLM-as-a-Judge**. The upgrade screen shows the recommended path for your project: +Click on the **Evaluation** navigation item. The upgrade screen shows the recommended path for your project: -- **In-app upgrade:** Recommended when only minimal configuration changes need to be reviewed and accepted. Langfuse Assistant guides you through these changes in the app. +- **In-app upgrade:** Recommended when only minimal configuration changes need to be reviewed and accepted. The [Langfuse Assistant](/docs/langfuse-assistant) guides you through these changes in the app. - **Coding assistant:** Recommended when the upgrade requires broader configuration or instrumentation changes. The UI provides the skill and project context to use in your coding editor. In both paths, the agentic flow makes the upgrade conversational: you can inspect the proposed changes, ask questions, and refine them before creating the new evaluators. You do not need to translate or recreate configurations manually. @@ -70,7 +83,7 @@ Score placement and cardinality may change: - A trace-level evaluator produces one score per matching trace. - An observation-level evaluator produces one score per matching observation. -- If three observations in one trace match the filters, the upgraded evaluator produces three scores. To keep one score per trace, narrow the filters to one observation using its name, type, or root-observation status. +- If three observations in one trace match the filters, the upgraded evaluator produces three scores. To keep one score per trace, narrow the filters to one observation using its name, type, or the **Is Root Observation** filter. ![Before and after the upgrade: the evaluator score moves from the trace to the matched observation](/images/docs/evaluator-score-output-example.png) @@ -130,10 +143,9 @@ Variable mapping: ## Upgrade checklist - [ ] Upgrade the SDK or OpenTelemetry ingestion path. -- [ ] Open the evaluator upgrade screen in **Evaluation > LLM-as-a-Judge**. +- [ ] Open the evaluator upgrade screen by clicking on the **Evaluation** navigation item. - [ ] Follow the recommended Langfuse Assistant or coding assistant path. -- [ ] Review the target observation, filters, and variable mappings. -- [ ] Confirm the expected number and placement of scores. +- [ ] Review the newly created observation-level evaluators. Confirm the expected number and placement of scores. - [ ] Decide whether to keep the legacy evaluator active for comparison. - [ ] Validate the evaluator results and deactivate the legacy evaluator when ready. @@ -141,9 +153,6 @@ Variable mapping: - **Troubleshooting**: [Why is my observation-level evaluator not executing?](/faq/all/observation-eval-not-executing) - **Documentation**: [LLM-as-a-Judge step-by-step set up guide](/docs/evaluation/evaluation-methods/llm-as-a-judge) +- **Questions on Fast Preview**: Ask in the dedicated [GitHub Discussion](https://github.com/orgs/langfuse/discussions/12518) - **GitHub**: Report issues at [github.com/langfuse/langfuse](https://github.com/langfuse/langfuse) - **Support**: Contact support@langfuse.com for enterprise customers - ---- - -_Last updated: July 19, 2026_ From f1415f8b7238f0d33e866bb506897f35aca16f96 Mon Sep 17 00:00:00 2001 From: Marlies Mayerhofer <74332854+marliessophie@users.noreply.github.com> Date: Sun, 19 Jul 2026 16:15:32 +0200 Subject: [PATCH 06/13] docs(evals): remove trace-level evaluator setup, add deprecation timeline - Remove the legacy "Traces (Legacy)" tab from the LLM-as-a-Judge setup guide; observation-level is now the only live-data path - Add deprecation note: legacy trace-level evaluators sunset by end of October 2026, multi-span evals build on the observation-centric model - Add the deprecation date to the Fast Preview rollout timeline on /docs/v4 with an explicit #rollout-timeline anchor Co-Authored-By: Claude Fable 5 --- .../evaluation-methods/llm-as-a-judge.mdx | 34 ++----------------- content/docs/v4.mdx | 4 ++- 2 files changed, 5 insertions(+), 33 deletions(-) diff --git a/content/docs/evaluation/evaluation-methods/llm-as-a-judge.mdx b/content/docs/evaluation/evaluation-methods/llm-as-a-judge.mdx index d1a4245b20..3c77a1dfb1 100644 --- a/content/docs/evaluation/evaluation-methods/llm-as-a-judge.mdx +++ b/content/docs/evaluation/evaluation-methods/llm-as-a-judge.mdx @@ -263,9 +263,6 @@ With your evaluator and model selected, configure which data to run the evaluati - - - **Configuration Steps** 1. Select "Live Observations" as your evaluation target @@ -279,37 +276,10 @@ With your evaluator and model selected, configure which data to run the evaluati - [JS/TS v3 → v4 migration guide](/docs/observability/sdk/upgrade-path/js-v3-to-v4) - **When filtering by trace attributes**: To filter observations by trace-level attributes (`userId`, `sessionId`, `version`, `tags`, `metadata`, `traceName`), use [`propagate_attributes()`](/docs/observability/sdk/instrumentation#add-attributes-to-observations) in your instrumentation code. Without this, trace attributes will not be available on observations. If you do set up trace-level attribute filtering and are not propagating attributes to observations, your observations will not be matched by the evaluator. - - - - - -**Performance consideration**: We recommend using Observation-level evaluators for production monitoring. They complete in seconds (vs minutes for trace-level), eliminating evaluation delays and backlogs. They also offer better precision and cost efficiency. See [upgrade guide](/faq/all/llm-as-a-judge-migration). - - -**Configuration Steps** - -1. Select "Live Traces" as your evaluation target -2. Filter traces by name, tags, userId, and other trace-level attributes -3. Choose whether to run on new traces only or include existing traces (backfilling) -4. Configure sampling percentage (e.g., 5%) to manage evaluation costs and throughput -5. Preview matched traces from the last 24 hours to validate your filter configuration - - - ![Production tracing data](/images/docs/evaluator-trace-filter.png) - - -**Requirements** - -- **OTel-based SDKs**: If you're using Python v3+ or JS/TS v4+, trace input/output is derived from the root observation by default. To explicitly set trace input/output for these evaluators, use `set_trace_io()` (Python) or `setTraceIO()` (JS/TS). See the [Python v3 → v4](/docs/observability/sdk/upgrade-path/python-v3-to-v4) and [JS/TS v4 → v5](/docs/observability/sdk/upgrade-path/js-v4-to-v5) migration guides. - - -We recommend migrating to [observation-level evaluators](/faq/all/llm-as-a-judge-migration) instead of using `set_trace_io()` / `setTraceIO()`. Once migrated, you can remove these calls from your codebase entirely. + +**Deprecation of trace-level evaluators:** The legacy trace-level evaluator feature is built on the old trace-centric data model and will be deprecated by the end of October 2026 — see the [Fast Preview rollout timeline](/docs/v4#rollout-timeline). Multi-span evaluations will build on the new observation-centric data model. To move existing trace-level evaluators, follow the [upgrade guide](/faq/all/llm-as-a-judge-migration). - - - diff --git a/content/docs/v4.mdx b/content/docs/v4.mdx index f6a29b183d..1dce7047df 100644 --- a/content/docs/v4.mdx +++ b/content/docs/v4.mdx @@ -54,7 +54,7 @@ As agentic applications grow more complex, a single trace can contain hundreds o /> -## Preview access and rollout timeline +## Preview access and rollout timeline [#rollout-timeline] | Organization creation date (Langfuse Cloud) | Fast Preview default | "Preview" toggle visibility | | ---------------------------------------------- | -------------------- | -------------------------------- | @@ -73,6 +73,8 @@ The toggle remains reversible for organizations created before the cutoff date. **Export source cutoff (2026-05-20):** Cloud projects created on or after this date are locked to **Enriched observations (recommended)** as the export source for blob storage, PostHog, and Mixpanel integrations. Legacy export sources are hidden from the UI for these projects. Projects created before this date retain full choice across all export sources. +**Trace-level evaluator deprecation (end of October 2026):** Legacy trace-level LLM-as-a-Judge evaluators are built on the old trace-centric data model and will be deprecated by the end of October 2026. Upcoming multi-span evaluations will build on the observation-centric data model. Follow the [evaluator upgrade guide](/faq/all/llm-as-a-judge-migration) to move existing evaluators to the observation level. + Data from SDKs `langfuse-js` < `5.0.0`, `langfuse-python` < `4.0.0`, or direct OpenTelemetry exporters that do not send `x-langfuse-ingestion-version: 4` can be delayed by up to 10 minutes in the new UI. Upgrade to the latest SDKs or set that header on your OTEL span exporter to see new data in real time. From f2c89e207cf8cc317d6d3a83ef6e0a4161d3620f Mon Sep 17 00:00:00 2001 From: Marlies Mayerhofer <74332854+marliessophie@users.noreply.github.com> Date: Sun, 19 Jul 2026 16:24:57 +0200 Subject: [PATCH 07/13] chore: push --- content/faq/all/llm-as-a-judge-migration.mdx | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/content/faq/all/llm-as-a-judge-migration.mdx b/content/faq/all/llm-as-a-judge-migration.mdx index 22e16e7ccc..853e6c2d1b 100644 --- a/content/faq/all/llm-as-a-judge-migration.mdx +++ b/content/faq/all/llm-as-a-judge-migration.mdx @@ -14,7 +14,7 @@ import { Steps, Step } from "@/components/docs"; core: "full", pro: "full", enterprise: "full", - selfHosted: "not-available", + selfHosted: "v4", }} /> From 2feaed8dba14d0655bba0f2e1ebcc1136c9f9347 Mon Sep 17 00:00:00 2001 From: Marlies Mayerhofer <74332854+marliessophie@users.noreply.github.com> Date: Sun, 19 Jul 2026 17:06:10 +0200 Subject: [PATCH 08/13] docs(evals): update deprecation notes for trace I/O methods - Clarify the usage of deprecated trace I/O methods in the v4 documentation, specifying the exact method names for both Python and JS/TS. - Enhance the migration guide to ensure users are aware of the transition from trace-level to observation-level evaluators and the necessary steps to maintain functionality during the upgrade. Co-Authored-By: Claude Fable 5 --- content/docs/v4.mdx | 2 +- content/faq/all/llm-as-a-judge-migration.mdx | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/content/docs/v4.mdx b/content/docs/v4.mdx index 1dce7047df..987a3eaf25 100644 --- a/content/docs/v4.mdx +++ b/content/docs/v4.mdx @@ -140,7 +140,7 @@ This preview is powered by a simplified observation-centric data model. Trace-le | ---------------------- | ----------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | **Storage** | Separate mutable Traces and Observations tables | Single immutable Observations table | | **Context attributes** | Stored on the trace, joined at query time | Propagated to every observation on SDK side. | -| **Trace input/output** | Set directly on the trace | Removed, use observation IO instead. (For legacy LLM-as-a-judge evaluators that depend on trace input and output, users can use the deprecated `set_trace_io` methods in the SDKs.) | +| **Trace input/output** | Set directly on the trace | Removed, use observation IO instead. (For legacy LLM-as-a-judge evaluators that depend on trace input and output, users can use the deprecated trace I/O methods: `set_current_trace_io()` / `span.set_trace_io()` in Python, `setActiveTraceIO()` / `span.setTraceIO()` in JS/TS.) | | **Mutability** | Traces and observations are mutable | Observations are immutable (written once) | ## OSS and self-hosting diff --git a/content/faq/all/llm-as-a-judge-migration.mdx b/content/faq/all/llm-as-a-judge-migration.mdx index 853e6c2d1b..44a8c6af58 100644 --- a/content/faq/all/llm-as-a-judge-migration.mdx +++ b/content/faq/all/llm-as-a-judge-migration.mdx @@ -43,7 +43,7 @@ First, upgrade your SDK or ingestion path. Observation-level evaluators run in r In the [v4 data model](/docs/v4), a trace is the logical group of observations that share a trace ID, rather than a separate record. Input and output belong to observations. In most cases, an existing root or workflow observation already carries the same input and output that was previously written to the trace. The evaluator upgrade targets that observation. If no single observation contains all required values, the coding assistant helps update your instrumentation. -**Keep legacy evaluators running during the transition.** After the SDK upgrade, trace input and output are no longer written automatically. If your existing trace-level evaluators read trace input or output, keep supplying these values with the deprecated `set_current_trace_io()` (Python) / `setActiveTraceIO()` (JS/TS) methods until the new observation-level evaluators are validated, then remove the calls. See [Why are the input and output of a trace empty?](/faq/all/empty-trace-input-and-output) for details. +**Keep legacy evaluators running during the transition.** After the SDK upgrade, trace input and output are no longer written automatically. If your existing trace-level evaluators read trace input or output, keep supplying these values with the deprecated trace I/O methods — `set_current_trace_io()` or `span.set_trace_io()` in Python, `setActiveTraceIO()` or `span.setTraceIO()` in JS/TS — until the new observation-level evaluators are validated, then remove the calls. See [Why are the input and output of a trace empty?](/faq/all/empty-trace-input-and-output) for details. From 672892a479cde8051ff8cb60f409d06016fe2854 Mon Sep 17 00:00:00 2001 From: Max Deichmann Date: Tue, 21 Jul 2026 20:47:51 +0200 Subject: [PATCH 09/13] docs: address review feedback on trace-level evaluator deprecation - Replace the hard "end of October 2026" date with "deadline will be announced" in the v4.mdx timeline entry and the llm-as-a-judge callout, matching the v4 launch messaging. - Rename "Fast Preview" to "Langfuse v4" and "legacy" to "deprecated" / "existing trace-level" in the new prose (FAQ description, intro, callout, rollback text, checklist, Getting help). - Remove em dashes from the PR-added callouts. - Fix the broken instrumentation anchor (#add-attributes-to-observations -> #add-attributes). - Run Prettier on v4.mdx (fixes the failing format CI check). The preceding merge of origin/main fixes the FAQ page crashing on AvailabilityBanner's selfHosted "v4" value, which only exists on newer main. Co-Authored-By: Claude Opus 4.8 --- .../evaluation-methods/llm-as-a-judge.mdx | 4 ++-- content/docs/v4.mdx | 12 ++++++------ content/faq/all/llm-as-a-judge-migration.mdx | 16 ++++++++-------- 3 files changed, 16 insertions(+), 16 deletions(-) diff --git a/content/docs/evaluation/evaluation-methods/llm-as-a-judge.mdx b/content/docs/evaluation/evaluation-methods/llm-as-a-judge.mdx index ba8c81ca31..37ad29ea5e 100644 --- a/content/docs/evaluation/evaluation-methods/llm-as-a-judge.mdx +++ b/content/docs/evaluation/evaluation-methods/llm-as-a-judge.mdx @@ -274,10 +274,10 @@ With your evaluator and model selected, configure which data to run the evaluati - **SDK version**: Python v3+ (OTel-based) or JS/TS v4+ (OTel-based) - [Python v2 → v3 migration guide](/docs/observability/sdk/upgrade-path/python-v2-to-v3) - [JS/TS v3 → v4 migration guide](/docs/observability/sdk/upgrade-path/js-v3-to-v4) -- **When filtering by trace attributes**: To filter observations by trace-level attributes (`userId`, `sessionId`, `version`, `tags`, `metadata`, `traceName`), use [`propagate_attributes()`](/docs/observability/sdk/instrumentation#add-attributes-to-observations) in your instrumentation code. Without this, trace attributes will not be available on observations. If you do set up trace-level attribute filtering and are not propagating attributes to observations, your observations will not be matched by the evaluator. +- **When filtering by trace attributes**: To filter observations by trace-level attributes (`userId`, `sessionId`, `version`, `tags`, `metadata`, `traceName`), use [`propagate_attributes()`](/docs/observability/sdk/instrumentation#add-attributes) in your instrumentation code. Without this, trace attributes will not be available on observations. If you do set up trace-level attribute filtering and are not propagating attributes to observations, your observations will not be matched by the evaluator. -**Deprecation of trace-level evaluators:** The legacy trace-level evaluator feature is built on the old trace-centric data model and will be deprecated by the end of October 2026 — see the [Fast Preview rollout timeline](/docs/v4#rollout-timeline). Multi-span evaluations will build on the new observation-centric data model. To move existing trace-level evaluators, follow the [upgrade guide](/faq/all/llm-as-a-judge-migration). +**Deprecation of trace-level evaluators:** Trace-level evaluators are built on the old trace-centric data model and are deprecated; the removal deadline will be announced (see the [Langfuse v4 rollout timeline](/docs/v4#rollout-timeline)). Multi-span evaluations will build on the new observation-centric data model. To move existing trace-level evaluators, follow the [upgrade guide](/faq/all/llm-as-a-judge-migration). diff --git a/content/docs/v4.mdx b/content/docs/v4.mdx index 987a3eaf25..f907fbc7b7 100644 --- a/content/docs/v4.mdx +++ b/content/docs/v4.mdx @@ -73,7 +73,7 @@ The toggle remains reversible for organizations created before the cutoff date. **Export source cutoff (2026-05-20):** Cloud projects created on or after this date are locked to **Enriched observations (recommended)** as the export source for blob storage, PostHog, and Mixpanel integrations. Legacy export sources are hidden from the UI for these projects. Projects created before this date retain full choice across all export sources. -**Trace-level evaluator deprecation (end of October 2026):** Legacy trace-level LLM-as-a-Judge evaluators are built on the old trace-centric data model and will be deprecated by the end of October 2026. Upcoming multi-span evaluations will build on the observation-centric data model. Follow the [evaluator upgrade guide](/faq/all/llm-as-a-judge-migration) to move existing evaluators to the observation level. +**Trace-level evaluator deprecation (deadline will be announced):** Trace-level LLM-as-a-Judge evaluators are built on the old trace-centric data model and are deprecated; the removal deadline will be announced. Upcoming multi-span evaluations will build on the observation-centric data model. Follow the [evaluator upgrade guide](/faq/all/llm-as-a-judge-migration) to move existing evaluators to the observation level. Data from SDKs `langfuse-js` < `5.0.0`, `langfuse-python` < `4.0.0`, or direct OpenTelemetry exporters that do not send `x-langfuse-ingestion-version: 4` can be delayed by up to 10 minutes in the new UI. Upgrade to the latest SDKs or set that header on your OTEL span exporter to see new data in real time. @@ -136,12 +136,12 @@ This preview is powered by a simplified observation-centric data model. Trace-le ### What changes -| | Classic model | Observation-centric model | -| ---------------------- | ----------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -| **Storage** | Separate mutable Traces and Observations tables | Single immutable Observations table | -| **Context attributes** | Stored on the trace, joined at query time | Propagated to every observation on SDK side. | +| | Classic model | Observation-centric model | +| ---------------------- | ----------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | +| **Storage** | Separate mutable Traces and Observations tables | Single immutable Observations table | +| **Context attributes** | Stored on the trace, joined at query time | Propagated to every observation on SDK side. | | **Trace input/output** | Set directly on the trace | Removed, use observation IO instead. (For legacy LLM-as-a-judge evaluators that depend on trace input and output, users can use the deprecated trace I/O methods: `set_current_trace_io()` / `span.set_trace_io()` in Python, `setActiveTraceIO()` / `span.setTraceIO()` in JS/TS.) | -| **Mutability** | Traces and observations are mutable | Observations are immutable (written once) | +| **Mutability** | Traces and observations are mutable | Observations are immutable (written once) | ## OSS and self-hosting diff --git a/content/faq/all/llm-as-a-judge-migration.mdx b/content/faq/all/llm-as-a-judge-migration.mdx index 44a8c6af58..c97953dc85 100644 --- a/content/faq/all/llm-as-a-judge-migration.mdx +++ b/content/faq/all/llm-as-a-judge-migration.mdx @@ -1,6 +1,6 @@ --- title: How do I upgrade trace-level evaluators to observation-level evaluators? -description: Step-by-step guide for upgrading legacy trace-level LLM-as-a-Judge evaluators to observation-level evaluators as part of Langfuse Fast Preview (v4). +description: Step-by-step guide for upgrading deprecated trace-level LLM-as-a-Judge evaluators to observation-level evaluators as part of Langfuse v4. tags: [evaluation] --- @@ -18,7 +18,7 @@ import { Steps, Step } from "@/components/docs"; }} /> -Trace-level evaluators use Langfuse's legacy trace-centric data model and are being deprecated as part of [Langfuse Fast Preview (v4)](/docs/v4). Existing evaluators continue to run while you upgrade. Fast Preview is currently available on Langfuse Cloud; for self-hosted support updates, follow the [GitHub Discussion](https://github.com/orgs/langfuse/discussions/12518). +Trace-level evaluators use Langfuse's old trace-centric data model and are deprecated as part of [Langfuse v4](/docs/v4). Existing evaluators continue to run while you upgrade. Langfuse v4 is currently available on Langfuse Cloud; for self-hosted rollout updates, follow the [GitHub Discussion](https://github.com/orgs/langfuse/discussions/12518). Observation-level evaluators run in real time and scale with larger traces and higher evaluation volume. Any future multi-span support will also use the observation-centric data model, so upgrading is the foundation for future evaluation capabilities. @@ -43,7 +43,7 @@ First, upgrade your SDK or ingestion path. Observation-level evaluators run in r In the [v4 data model](/docs/v4), a trace is the logical group of observations that share a trace ID, rather than a separate record. Input and output belong to observations. In most cases, an existing root or workflow observation already carries the same input and output that was previously written to the trace. The evaluator upgrade targets that observation. If no single observation contains all required values, the coding assistant helps update your instrumentation. -**Keep legacy evaluators running during the transition.** After the SDK upgrade, trace input and output are no longer written automatically. If your existing trace-level evaluators read trace input or output, keep supplying these values with the deprecated trace I/O methods — `set_current_trace_io()` or `span.set_trace_io()` in Python, `setActiveTraceIO()` or `span.setTraceIO()` in JS/TS — until the new observation-level evaluators are validated, then remove the calls. See [Why are the input and output of a trace empty?](/faq/all/empty-trace-input-and-output) for details. +**Keep existing trace-level evaluators running during the transition.** After the SDK upgrade, trace input and output are no longer written automatically. If your existing trace-level evaluators read trace input or output, keep supplying these values with the deprecated trace I/O methods (`set_current_trace_io()` or `span.set_trace_io()` in Python, `setActiveTraceIO()` or `span.setTraceIO()` in JS/TS) until the new observation-level evaluators are validated, then remove the calls. See [Why are the input and output of a trace empty?](/faq/all/empty-trace-input-and-output) for details. @@ -67,9 +67,9 @@ In both paths, the agentic flow makes the upgrade conversational: you can inspec The recommended agent guides you through reviewing the proposed changes, creating the observation-level evaluators, and validating the result against your project data. You can ask questions and refine the target observation, filters, variable mappings, and expected score behavior until the upgrade looks correct. -When ready, create the observation-level evaluators. You can keep the legacy evaluators active temporarily to compare results, or deactivate them immediately. Confirm that the new evaluators receive the expected input, output, and context and produce the intended number of scores. +When ready, create the observation-level evaluators. You can keep the existing trace-level evaluators active temporarily to compare results, or deactivate them immediately. Confirm that the new evaluators receive the expected input, output, and context and produce the intended number of scores. -If you need to revert, deactivate the new evaluators and reactivate the existing legacy evaluators. Historical evaluation results remain accessible. +If you need to revert, deactivate the new evaluators and reactivate the existing trace-level evaluators. Historical evaluation results remain accessible. The agent explains the relevant changes during the upgrade. The following sections summarize what may differ if you want to understand the underlying behavior. @@ -146,13 +146,13 @@ Variable mapping: - [ ] Open the evaluator upgrade screen by clicking on the **Evaluation** navigation item. - [ ] Follow the recommended Langfuse Assistant or coding assistant path. - [ ] Review the newly created observation-level evaluators. Confirm the expected number and placement of scores. -- [ ] Decide whether to keep the legacy evaluator active for comparison. -- [ ] Validate the evaluator results and deactivate the legacy evaluator when ready. +- [ ] Decide whether to keep the trace-level evaluator active for comparison. +- [ ] Validate the evaluator results and deactivate the trace-level evaluator when ready. ## Getting help - **Troubleshooting**: [Why is my observation-level evaluator not executing?](/faq/all/observation-eval-not-executing) - **Documentation**: [LLM-as-a-Judge step-by-step set up guide](/docs/evaluation/evaluation-methods/llm-as-a-judge) -- **Questions on Fast Preview**: Ask in the dedicated [GitHub Discussion](https://github.com/orgs/langfuse/discussions/12518) +- **Questions on Langfuse v4**: Ask in the dedicated [GitHub Discussion](https://github.com/orgs/langfuse/discussions/12518) - **GitHub**: Report issues at [github.com/langfuse/langfuse](https://github.com/langfuse/langfuse) - **Support**: Contact support@langfuse.com for enterprise customers From c51ccc00f70c9a3b43897bdddc7ff5785bb683b6 Mon Sep 17 00:00:00 2001 From: Marlies Mayerhofer <74332854+marliessophie@users.noreply.github.com> Date: Wed, 22 Jul 2026 10:13:21 +0200 Subject: [PATCH 10/13] docs: remove legacy trace-level evaluator content and update migration guidance --- .../evaluation-methods/llm-as-a-judge.mdx | 41 +------------------ content/faq/all/llm-as-a-judge-migration.mdx | 16 ++++---- 2 files changed, 9 insertions(+), 48 deletions(-) diff --git a/content/docs/evaluation/evaluation-methods/llm-as-a-judge.mdx b/content/docs/evaluation/evaluation-methods/llm-as-a-judge.mdx index 37ad29ea5e..5f694d4333 100644 --- a/content/docs/evaluation/evaluation-methods/llm-as-a-judge.mdx +++ b/content/docs/evaluation/evaluation-methods/llm-as-a-judge.mdx @@ -76,13 +76,6 @@ import { Card, CardContent, CardDescription, CardHeader, CardTitle } from "@/com - - -
Traces (Legacy)
-
Complete workflow executions
- - - @@ -121,9 +114,6 @@ import { Card, CardContent, CardDescription, CardHeader, CardTitle } from "@/com Evaluate live production traffic to monitor your LLM application performance in real-time. - - - Run evaluators on individual observations within your traces—such as LLM calls, retrieval operations, embedding generations, or tool calls. #### Context available to observation-level evaluators [#observation-evaluator-context] @@ -149,35 +139,6 @@ At ingest time, each observation is evaluated against your filter criteria. Matc - Monitor toxicity scores on all customer-facing LLM generations - Track retrieval relevance for RAG systems by targeting document retrieval observations - - - - -Run evaluators on complete traces, evaluating entire workflow executions from start to finish. - - -Trace-level LLM-as-a-Judge evaluators are legacy. Migrate to observation-level evaluators and put the data the judge needs on the target observation. - - - -**Consider targeting Observations instead**: Observation-level evaluators complete in seconds (vs minutes for trace-level), eliminating evaluation delays. They also offer better precision for production monitoring. See [upgrade guide](/faq/all/llm-as-a-judge-migration). - - -**Why target Traces** - -- Your evaluation requires full context spanning multiple operations -- You're on legacy SDK versions (Python v2 or JS/TS v3) and cannot upgrade - -**Data Flow** - -At ingest time, each trace is evaluated against your filter criteria. Matching traces are added to an evaluation queue and processed asynchronously. Scores are attached to the trace itself, resulting in one score per trace per evaluator. - -**Example Use Cases** - -- Score the accuracy of a multi-step agent workflow, if and only if evaluator needs full context spanning multiple operations (e.g., retrieval → reranking → generation → citation) - - - @@ -277,7 +238,7 @@ With your evaluator and model selected, configure which data to run the evaluati - **When filtering by trace attributes**: To filter observations by trace-level attributes (`userId`, `sessionId`, `version`, `tags`, `metadata`, `traceName`), use [`propagate_attributes()`](/docs/observability/sdk/instrumentation#add-attributes) in your instrumentation code. Without this, trace attributes will not be available on observations. If you do set up trace-level attribute filtering and are not propagating attributes to observations, your observations will not be matched by the evaluator. -**Deprecation of trace-level evaluators:** Trace-level evaluators are built on the old trace-centric data model and are deprecated; the removal deadline will be announced (see the [Langfuse v4 rollout timeline](/docs/v4#rollout-timeline)). Multi-span evaluations will build on the new observation-centric data model. To move existing trace-level evaluators, follow the [upgrade guide](/faq/all/llm-as-a-judge-migration). +**Deprecation of trace-level evaluators:** Trace-level evaluators are built on the old trace-centric data model and are deprecated; the removal deadline will be announced (see the [Langfuse v4 rollout timeline](/docs/v4#rollout-timeline)). Existing trace-level evaluators will continue to run until the deprecation timeline. After the deprecation timeline, the evaluators will stop producing results, and go stable. Multi-span evaluations will build on the new observation-centric data model. To move existing trace-level evaluators, follow the [upgrade guide](/faq/all/llm-as-a-judge-migration). diff --git a/content/faq/all/llm-as-a-judge-migration.mdx b/content/faq/all/llm-as-a-judge-migration.mdx index c97953dc85..2edd98a306 100644 --- a/content/faq/all/llm-as-a-judge-migration.mdx +++ b/content/faq/all/llm-as-a-judge-migration.mdx @@ -75,7 +75,7 @@ The agent explains the relevant changes during the upgrade. The following sectio **How configurations may change** -Depending on the existing evaluator, the proposed configuration may target the same observation as before, replace trace fields with values from one observation, or require an instrumentation change so all required values are available together. The [technical reference](#technical-reference) below explains the cases in detail. +Depending on the existing evaluator, the proposed configuration may target the same observation as before, replace trace fields with values from one observation, or require an instrumentation change so all required values are available together. The [common configuration scenarios](#common-configuration-scenarios) below explains the cases in detail. **How resulting scores may differ** @@ -93,7 +93,7 @@ Score placement and cardinality may change: -## Technical reference [#technical-reference] +## Common Configuration Scenarios [#common-configuration-scenarios] The recommended assistant guides you through every change. You do not need to classify or update configurations manually. Expand this section only if you want to understand how an existing configuration may be translated. @@ -142,12 +142,12 @@ Variable mapping: ## Upgrade checklist -- [ ] Upgrade the SDK or OpenTelemetry ingestion path. -- [ ] Open the evaluator upgrade screen by clicking on the **Evaluation** navigation item. -- [ ] Follow the recommended Langfuse Assistant or coding assistant path. -- [ ] Review the newly created observation-level evaluators. Confirm the expected number and placement of scores. -- [ ] Decide whether to keep the trace-level evaluator active for comparison. -- [ ] Validate the evaluator results and deactivate the trace-level evaluator when ready. +1. Upgrade the SDK or OpenTelemetry ingestion path. +2. Open the evaluator upgrade screen by clicking on the **Evaluation** navigation item. +3. Follow the recommended Langfuse Assistant or coding assistant path. +4. Review the newly created observation-level evaluators. Confirm the expected number and placement of scores. +5. Decide whether to keep the trace-level evaluator active for comparison. +6. Validate the evaluator results and deactivate the trace-level evaluator when ready. ## Getting help From 21541832215b4dd086f2e86123dae6244c5c41dd Mon Sep 17 00:00:00 2001 From: Marlies Mayerhofer <74332854+marliessophie@users.noreply.github.com> Date: Wed, 22 Jul 2026 11:14:11 +0200 Subject: [PATCH 11/13] docs(observability): deprecate trace input and output --- .../faq/all/empty-trace-input-and-output.mdx | 381 ++++++------------ 1 file changed, 115 insertions(+), 266 deletions(-) diff --git a/content/faq/all/empty-trace-input-and-output.mdx b/content/faq/all/empty-trace-input-and-output.mdx index a710e6d3c1..19bf72a8d4 100644 --- a/content/faq/all/empty-trace-input-and-output.mdx +++ b/content/faq/all/empty-trace-input-and-output.mdx @@ -1,130 +1,41 @@ --- -title: Why are the input and output of a trace empty? -description: This article explains why the input and output of a trace might be empty and how to fix it. +title: Why are trace input and output empty? +description: Trace-level input and output are deprecated in Langfuse v4. Store input and output on observations instead. tags: [observability, observability-get-started] --- import { Details, Summary } from "@/components/Details"; -# Why are the input and output of my trace empty? +# Why are trace input and output empty? -Having input and output on your traces affects several important features: +Trace-level input and output are deprecated as part of [Langfuse v4](/docs/v4). In the observation-centric data model, a trace ID groups related observations; the trace is no longer a separate record with its own input and output. -- **Browsing traces**: The traces list shows input/output previews, making it easier to find what you're looking for -- **Evaluations**: LLM-as-a-judge evaluators use trace input/output to assess quality. Empty fields mean evaluations won't work -- **Search**: You can search traces by their input/output content, but only if it's populated +For new instrumentation, write: -## How trace input/output works +- The overall request and response to the root or workflow observation. +- Step-specific input and output to the observation representing that operation. -Before jumping into solutions, this section helps to understand how Langfuse populates trace input/output. +In most applications, the root observation already carries the same input and output that was previously written to the trace. -### Traces and observations - -In Langfuse, a **trace** represents a complete request or operation. Inside each trace, you have **observations**. - -``` -Trace: "User asks about weather" -├── Observation: Parse user intent -├── Observation: Call weather API -└── Observation: Generate response (LLM call) -``` - -Both traces and observations can have their own input/output fields. They serve different purposes: - -| | Trace input/output | Observation input/output | -| ---------------------- | ------------------------------------------------- | ---------------------------- | -| **What it represents** | The overall request and response | Each individual step | -| **Where you see it** | Traces list, evaluations | Inside the trace detail view | -| **How it's set** | Inherited from root observation OR set explicitly | Set on each observation | - -### The "root observation" rule - -By default, trace input/output is copied from the observation at the top level (called the "root observation"). - -This means: - -- If your root observation has input/output → the trace will too -- If your root observation has no input/output → your trace will be empty (unless you set it explicitly, see [below](#solution-b)) - -This is why your trace might show empty fields even though you can see data in the individual observations inside it. - -## Troubleshooting - -The most common reasons for empty trace input/output are: - -### 1. You're using a short-lived application (scripts, serverless, notebooks) - -**Symptoms**: Traces sometimes appear with missing data, or don't appear at all. - -[Langfuse sends data in the background to keep your application fast](/docs/observability/data-model#background-processing). If your script or serverless function exits before the data is sent, it gets lost. - -**Solution**: Call `flush()` before your application exits. - - - - -```python -from langfuse import get_client - -langfuse = get_client() - -# Your code here... - -# Before your script ends: -langfuse.flush() -``` - -If using the `@observe()` decorator: - -```python -from langfuse import observe, get_client - -@observe() -def main(): - # Your code here... - pass - -main() -get_client().flush() -``` - - - - -```typescript -import { NodeSDK } from "@opentelemetry/sdk-node"; -import { LangfuseSpanProcessor } from "@langfuse/otel"; - -const langfuseSpanProcessor = new LangfuseSpanProcessor(); - -const sdk = new NodeSDK({ - spanProcessors: [langfuseSpanProcessor], -}); -sdk.start(); - -async function main() { - // Your code here... -} - -// Ensure data is sent before exit -main().finally(() => langfuseSpanProcessor.forceFlush()); -``` - - - + +An empty trace input or output does not necessarily indicate missing data. Check the observations in the trace. If the relevant observation contains the expected input and output, the data was ingested correctly. + ---- +## Use observation input and output -### 2. You haven't set input/output on your root span +Observation input and output support the same core workflows without relying on the deprecated trace fields: -**Symptoms**: Trace input/output is always empty, but observations inside the trace have data. +| Workflow | Recommended approach | +| ---------------------- | ------------------------------------------------------------------------------------------------------------------------------------- | +| Browse and search data | Filter or search observations directly. For the overall request and response, use the root observation. | +| Run evaluations | Create an observation-level evaluator that targets the observation containing the required input, output, and context. | +| Record a summary | Store the summary on a root or dedicated workflow observation instead of writing separate input and output values to the trace level. | -If you're manually creating spans, you need to either: +If you still use trace-level LLM-as-a-Judge evaluators, follow the [evaluator upgrade guide](/faq/all/llm-as-a-judge-migration). -- Set input/output on your root span, OR -- Explicitly set input/output on the trace itself +### Set input and output on the root observation -**Solution A**: Set input/output on your root span +Use a root observation to represent the overall application or agent invocation: @@ -136,16 +47,14 @@ langfuse = get_client() with langfuse.start_as_current_observation( as_type="span", - name="my-pipeline" + name="my-pipeline", ) as root_span: user_input = "What's the weather like?" result = process_request(user_input) - # Set input/output on the root span - # This will automatically populate the trace root_span.update( input={"query": user_input}, - output={"response": result} + output={"response": result}, ) ``` @@ -159,8 +68,6 @@ await startActiveObservation("my-pipeline", async (rootSpan) => { const userInput = "What's the weather like?"; const result = await processRequest(userInput); - // Set input/output on the root span - // This will automatically populate the trace rootSpan.update({ input: { query: userInput }, output: { response: result }, @@ -171,11 +78,13 @@ await startActiveObservation("my-pipeline", async (rootSpan) => { - +## If observation input or output is missing + +### Flush short-lived applications -**Solution B**: Set input/output directly on the trace +Langfuse [sends data in the background](/docs/observability/data-model#background-processing). Scripts, serverless functions, and notebooks can exit before all observations are exported. -Sometimes the trace input/output should be different from any observation (e.g., you want a clean summary). You can set it explicitly: +Call `flush()` or `forceFlush()` before the application exits: @@ -185,89 +94,38 @@ from langfuse import get_client langfuse = get_client() -with langfuse.start_as_current_observation( - as_type="span", - name="my-pipeline" -) as root_span: - user_input = "What's the weather like?" - result = process_request(user_input) +# Your code here... - # Set trace input/output explicitly (deprecated — only for backward compat with legacy trace-level LLM-as-a-judge evaluators) - root_span.set_trace_io( - input={"user_question": user_input}, - output={"answer": result} - ) +langfuse.flush() ``` ```typescript -import { startActiveObservation, setActiveTraceIO } from "@langfuse/tracing"; - -await startActiveObservation("my-pipeline", async (rootSpan) => { - const userInput = "What's the weather like?"; - const result = await processRequest(userInput); - - // Set trace input/output explicitly (deprecated — only for backward compat with legacy trace-level LLM-as-a-judge evaluators) - setActiveTraceIO({ - input: { user_question: userInput }, - output: { answer: result }, - }); -}); -``` - - - - ---- - -### 3. Your root span is being filtered out - -**Symptoms**: Trace input/output is empty when using `shouldExportSpan` to filter spans. - -If you're using `shouldExportSpan` in your `LangfuseSpanProcessor` to filter spans, and your root span gets filtered out, the trace input/output will be empty. This happens because trace input/output is [derived from the root observation by default](#the-root-observation-rule). - -**Solution**: If you want the root span to stay filtered out, use `setTraceIO()` to manually set trace input/output from within a child span that is not filtered out: - - -`setTraceIO()` and `set_trace_io()` are deprecated. They are only needed if you still rely on trace-level [LLM-as-a-judge](/docs/evaluation/evaluation-methods/llm-as-a-judge) evaluators that read trace input/output. Once you have [migrated to observation-level evaluators](/faq/all/llm-as-a-judge-migration), these calls have no effect and can be removed from your codebase. - +import { NodeSDK } from "@opentelemetry/sdk-node"; +import { LangfuseSpanProcessor } from "@langfuse/otel"; -```typescript -span.setTraceIO({ - input: yourInputData, - output: yourOutputData, +const langfuseSpanProcessor = new LangfuseSpanProcessor(); +const sdk = new NodeSDK({ + spanProcessors: [langfuseSpanProcessor], }); -``` - -See the [advanced features docs](/docs/observability/sdk/advanced-features#filtering-by-instrumentation-scope) for more details on `shouldExportSpan`. ---- - -### 4. You're using the `@observe()` decorator but input/output capture is disabled - -**Symptoms**: Decorated functions don't show input/output, even though they return values. - -The `@observe()` decorator [automatically captures function arguments as input and return values as output](/docs/observability/sdk/instrumentation#observe-wrapper). But this can be disabled. +sdk.start(); -**Check if capture is disabled**: +async function main() { + // Your code here... +} -```bash -# Check your environment variables -echo $LANGFUSE_OBSERVE_DECORATOR_IO_CAPTURE_ENABLED +main().finally(() => langfuseSpanProcessor.forceFlush()); ``` -If this is set to `false`, input/output won't be captured. - -**Solution**: Enable capture + + -```bash -# In your environment -export LANGFUSE_OBSERVE_DECORATOR_IO_CAPTURE_ENABLED=true -``` +### Check decorator input/output capture -Or enable it per-function: +The Python [`@observe()` decorator](/docs/observability/sdk/instrumentation#observe-wrapper) captures function arguments and return values by default. If `LANGFUSE_OBSERVE_DECORATOR_IO_CAPTURE_ENABLED=false`, enable capture globally or for the relevant function: ```python from langfuse import observe @@ -277,31 +135,22 @@ def my_function(data): return process(data) ``` ---- - -### 5. You're using an OpenTelemetry-based integration +### Ensure the relevant observation is exported -**Symptoms**: Traces from OTEL integrations (OpenLLMetry, Logfire, etc.) show empty input/output. +If `shouldExportSpan` filters out your root observation, its input and output are not sent to Langfuse. Keep the root observation, or store the required values on another exported observation and target that observation in downstream workflows. -Different OpenTelemetry providers use different attribute names for input/output. Langfuse looks for specific attributes and may not find them if your provider uses different names. +See the [span filtering documentation](/docs/observability/sdk/advanced-features#filtering-by-instrumentation-scope) for details. -**Solution**: Set the attributes Langfuse expects +### Map OpenTelemetry input and output -Langfuse maps these OTEL span attributes to observation input/output (checked in this order): +OpenTelemetry integrations use different attribute names. Langfuse maps these span attributes to observation input and output, in priority order: -| Observation field | OTEL attributes (in priority order) | +| Observation field | OpenTelemetry attributes | | ----------------- | ---------------------------------------------------------------------------------------- | | `input` | `langfuse.observation.input`, `gen_ai.prompt`, `input.value`, `mlflow.spanInputs` | | `output` | `langfuse.observation.output`, `gen_ai.completion`, `output.value`, `mlflow.spanOutputs` | -For trace-level input/output, Langfuse looks for: - -- `langfuse.trace.input` / `langfuse.trace.output`, OR -- The root span's observation input/output (using the attributes above) - -See the [complete property mapping reference](/integrations/native/opentelemetry#property-mapping) for all supported attributes. - -**Example**: Manually set the attributes Langfuse recognizes: +See the [OpenTelemetry property mapping](/integrations/native/opentelemetry#property-mapping) for the complete reference. @@ -313,13 +162,14 @@ import json tracer = trace.get_tracer(__name__) with tracer.start_as_current_span("my-operation") as span: - # Set attributes that Langfuse recognizes - span.set_attribute("input.value", str(input_data)) - span.set_attribute("output.value", str(output_data)) - - # Or use the langfuse namespace for guaranteed mapping - span.set_attribute("langfuse.observation.input", json.dumps(input_data)) - span.set_attribute("langfuse.observation.output", json.dumps(output_data)) + span.set_attribute( + "langfuse.observation.input", + json.dumps(input_data), + ) + span.set_attribute( + "langfuse.observation.output", + json.dumps(output_data), + ) ``` @@ -331,13 +181,14 @@ import { trace } from "@opentelemetry/api"; const tracer = trace.getTracer("my-service"); tracer.startActiveSpan("my-operation", (span) => { - // Set attributes that Langfuse recognizes - span.setAttribute("input.value", JSON.stringify(inputData)); - span.setAttribute("output.value", JSON.stringify(outputData)); - - // Or use the langfuse namespace for guaranteed mapping - span.setAttribute("langfuse.observation.input", JSON.stringify(inputData)); - span.setAttribute("langfuse.observation.output", JSON.stringify(outputData)); + span.setAttribute( + "langfuse.observation.input", + JSON.stringify(inputData), + ); + span.setAttribute( + "langfuse.observation.output", + JSON.stringify(outputData), + ); span.end(); }); @@ -347,83 +198,81 @@ tracer.startActiveSpan("my-operation", (span) => {
-**Which attributes does my OTEL provider use?** - -Different providers use different semantic conventions. Here's how to find out what your provider sends: +**Which attributes does my OpenTelemetry provider use?** -1. **Enable debug logging** in your OTEL exporter to see the raw span attributes -2. **Check the trace in Langfuse**: Open a trace, click on an observation, and look at the "Metadata" tab to see which attributes were received -3. **Consult your provider's documentation** for their semantic conventions +1. Enable debug logging in your OpenTelemetry exporter to inspect the raw span attributes. +2. Open the observation in Langfuse and check its metadata. +3. Review the provider's semantic conventions. -Common providers and their conventions: +Common conventions include: -- **OpenLLMetry**: Uses `gen_ai.prompt` and `gen_ai.completion` -- **OpenInference**: Uses `input.value` and `output.value` -- **MLflow**: Uses `mlflow.spanInputs` and `mlflow.spanOutputs` -- **Pydantic Logfire**: Uses custom attributes (Langfuse has specific support since PR #5841) +- **OpenLLMetry:** `gen_ai.prompt` and `gen_ai.completion` +- **OpenInference:** `input.value` and `output.value` +- **MLflow:** `mlflow.spanInputs` and `mlflow.spanOutputs` -If your provider uses different attribute names, you have two options: - -1. Manually set the attributes Langfuse expects (as shown above) -2. Open a [GitHub issue](https://github.com/langfuse/langfuse/issues) requesting support for your provider's conventions +If your provider uses different names, map them to the Langfuse observation attributes shown above.
-Many OTEL-specific issues have been fixed in recent Langfuse versions. If you're self-hosting, make sure you're on the latest version. - ---- - -## Still having issues? +## Temporary compatibility for legacy evaluators -If none of the above solutions work: + +The trace I/O methods are deprecated. Use them only while an existing trace-level evaluator still reads trace input or output. Do not add them to new instrumentation. + -1. **Enable debug logging** to see what's being sent: +After upgrading your SDK or ingestion path, trace input and output are no longer written automatically. Until you have [upgraded the evaluator](/faq/all/llm-as-a-judge-migration), you can continue supplying the legacy fields explicitly: ```python -from langfuse import Langfuse +from langfuse import get_client -langfuse = Langfuse(debug=True) -``` +langfuse = get_client() +user_input = "What's the weather like?" - - +with langfuse.start_as_current_observation( + as_type="span", + name="my-pipeline", +) as root_span: + result = process_request(user_input) -```bash -# In your environment -export LANGFUSE_DEBUG="true" + root_span.set_trace_io( + input={"query": user_input}, + output={"response": result}, + ) ``` - - -2. **Check your SDK version** and update if needed: - - -```bash -pip install --upgrade langfuse -``` +```typescript +import { + setActiveTraceIO, + startActiveObservation, +} from "@langfuse/tracing"; - - +await startActiveObservation("my-pipeline", async () => { + const userInput = "What's the weather like?"; + const result = await processRequest(userInput); -```bash -npm update @langfuse/tracing @langfuse/otel @langfuse/client + setActiveTraceIO({ + input: { query: userInput }, + output: { response: result }, + }); +}); ``` -3. **Look at the trace structure** in the Langfuse dashboard: - - Open a trace and look at the observation tree - - Is there a single root observation? - - Do the individual observations have input/output? +Once the observation-level evaluator is validated, remove these calls. + +## Still having issues? + +1. Update to the latest Langfuse SDK version. +2. Open the trace and verify that the expected observation was exported. +3. Check that the observation itself has input and output. +4. For short-lived applications, confirm that the client flushes before exit. -4. **Ask for help**: Open a [GitHub discussion](https://github.com/langfuse/langfuse/discussions) with: - - Your SDK version - - A code snippet showing how you're creating traces - - A screenshot of the trace structure in the dashboard +If the observation data is still missing, open a [GitHub discussion](https://github.com/langfuse/langfuse/discussions) with your SDK version, instrumentation snippet, and a screenshot of the observation tree. From daa7ff9b9df09101871b939c6de5642f8af87e01 Mon Sep 17 00:00:00 2001 From: Marlies Mayerhofer <74332854+marliessophie@users.noreply.github.com> Date: Wed, 22 Jul 2026 11:52:22 +0200 Subject: [PATCH 12/13] docs(evals): decouple guide from v4 release docs --- .../evaluation-methods/llm-as-a-judge.mdx | 3 +-- content/docs/v4.mdx | 16 +++++++--------- content/faq/all/llm-as-a-judge-migration.mdx | 2 +- 3 files changed, 9 insertions(+), 12 deletions(-) diff --git a/content/docs/evaluation/evaluation-methods/llm-as-a-judge.mdx b/content/docs/evaluation/evaluation-methods/llm-as-a-judge.mdx index 5f694d4333..d2fcc03d9f 100644 --- a/content/docs/evaluation/evaluation-methods/llm-as-a-judge.mdx +++ b/content/docs/evaluation/evaluation-methods/llm-as-a-judge.mdx @@ -139,7 +139,6 @@ At ingest time, each observation is evaluated against your filter criteria. Matc - Monitor toxicity scores on all customer-facing LLM generations - Track retrieval relevance for RAG systems by targeting document retrieval observations - @@ -238,7 +237,7 @@ With your evaluator and model selected, configure which data to run the evaluati - **When filtering by trace attributes**: To filter observations by trace-level attributes (`userId`, `sessionId`, `version`, `tags`, `metadata`, `traceName`), use [`propagate_attributes()`](/docs/observability/sdk/instrumentation#add-attributes) in your instrumentation code. Without this, trace attributes will not be available on observations. If you do set up trace-level attribute filtering and are not propagating attributes to observations, your observations will not be matched by the evaluator. -**Deprecation of trace-level evaluators:** Trace-level evaluators are built on the old trace-centric data model and are deprecated; the removal deadline will be announced (see the [Langfuse v4 rollout timeline](/docs/v4#rollout-timeline)). Existing trace-level evaluators will continue to run until the deprecation timeline. After the deprecation timeline, the evaluators will stop producing results, and go stable. Multi-span evaluations will build on the new observation-centric data model. To move existing trace-level evaluators, follow the [upgrade guide](/faq/all/llm-as-a-judge-migration). +**Deprecation of trace-level evaluators:** Trace-level evaluators are built on the old trace-centric data model and are deprecated; the removal deadline will be announced as part of [Langfuse v4](/docs/v4). Existing trace-level evaluators will continue to run until the deprecation timeline. After the deprecation timeline, the evaluators will stop producing results, and go stable. Multi-span evaluations will build on the new observation-centric data model. To move existing trace-level evaluators, follow the [upgrade guide](/faq/all/llm-as-a-judge-migration). diff --git a/content/docs/v4.mdx b/content/docs/v4.mdx index f907fbc7b7..f6a29b183d 100644 --- a/content/docs/v4.mdx +++ b/content/docs/v4.mdx @@ -54,7 +54,7 @@ As agentic applications grow more complex, a single trace can contain hundreds o /> -## Preview access and rollout timeline [#rollout-timeline] +## Preview access and rollout timeline | Organization creation date (Langfuse Cloud) | Fast Preview default | "Preview" toggle visibility | | ---------------------------------------------- | -------------------- | -------------------------------- | @@ -73,8 +73,6 @@ The toggle remains reversible for organizations created before the cutoff date. **Export source cutoff (2026-05-20):** Cloud projects created on or after this date are locked to **Enriched observations (recommended)** as the export source for blob storage, PostHog, and Mixpanel integrations. Legacy export sources are hidden from the UI for these projects. Projects created before this date retain full choice across all export sources. -**Trace-level evaluator deprecation (deadline will be announced):** Trace-level LLM-as-a-Judge evaluators are built on the old trace-centric data model and are deprecated; the removal deadline will be announced. Upcoming multi-span evaluations will build on the observation-centric data model. Follow the [evaluator upgrade guide](/faq/all/llm-as-a-judge-migration) to move existing evaluators to the observation level. - Data from SDKs `langfuse-js` < `5.0.0`, `langfuse-python` < `4.0.0`, or direct OpenTelemetry exporters that do not send `x-langfuse-ingestion-version: 4` can be delayed by up to 10 minutes in the new UI. Upgrade to the latest SDKs or set that header on your OTEL span exporter to see new data in real time. @@ -136,12 +134,12 @@ This preview is powered by a simplified observation-centric data model. Trace-le ### What changes -| | Classic model | Observation-centric model | -| ---------------------- | ----------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -| **Storage** | Separate mutable Traces and Observations tables | Single immutable Observations table | -| **Context attributes** | Stored on the trace, joined at query time | Propagated to every observation on SDK side. | -| **Trace input/output** | Set directly on the trace | Removed, use observation IO instead. (For legacy LLM-as-a-judge evaluators that depend on trace input and output, users can use the deprecated trace I/O methods: `set_current_trace_io()` / `span.set_trace_io()` in Python, `setActiveTraceIO()` / `span.setTraceIO()` in JS/TS.) | -| **Mutability** | Traces and observations are mutable | Observations are immutable (written once) | +| | Classic model | Observation-centric model | +| ---------------------- | ----------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | +| **Storage** | Separate mutable Traces and Observations tables | Single immutable Observations table | +| **Context attributes** | Stored on the trace, joined at query time | Propagated to every observation on SDK side. | +| **Trace input/output** | Set directly on the trace | Removed, use observation IO instead. (For legacy LLM-as-a-judge evaluators that depend on trace input and output, users can use the deprecated `set_trace_io` methods in the SDKs.) | +| **Mutability** | Traces and observations are mutable | Observations are immutable (written once) | ## OSS and self-hosting diff --git a/content/faq/all/llm-as-a-judge-migration.mdx b/content/faq/all/llm-as-a-judge-migration.mdx index 2edd98a306..cc6756d3ed 100644 --- a/content/faq/all/llm-as-a-judge-migration.mdx +++ b/content/faq/all/llm-as-a-judge-migration.mdx @@ -14,7 +14,7 @@ import { Steps, Step } from "@/components/docs"; core: "full", pro: "full", enterprise: "full", - selfHosted: "v4", + selfHosted: "not-available", }} /> From 63bd5a9685b148531a582c6a6b5da5cd93ab8978 Mon Sep 17 00:00:00 2001 From: Marlies Mayerhofer <74332854+marliessophie@users.noreply.github.com> Date: Wed, 22 Jul 2026 12:09:15 +0200 Subject: [PATCH 13/13] docs(evals): separate trace input output guide --- .../faq/all/empty-trace-input-and-output.mdx | 381 ++++++++++++------ 1 file changed, 266 insertions(+), 115 deletions(-) diff --git a/content/faq/all/empty-trace-input-and-output.mdx b/content/faq/all/empty-trace-input-and-output.mdx index 19bf72a8d4..a710e6d3c1 100644 --- a/content/faq/all/empty-trace-input-and-output.mdx +++ b/content/faq/all/empty-trace-input-and-output.mdx @@ -1,41 +1,130 @@ --- -title: Why are trace input and output empty? -description: Trace-level input and output are deprecated in Langfuse v4. Store input and output on observations instead. +title: Why are the input and output of a trace empty? +description: This article explains why the input and output of a trace might be empty and how to fix it. tags: [observability, observability-get-started] --- import { Details, Summary } from "@/components/Details"; -# Why are trace input and output empty? +# Why are the input and output of my trace empty? -Trace-level input and output are deprecated as part of [Langfuse v4](/docs/v4). In the observation-centric data model, a trace ID groups related observations; the trace is no longer a separate record with its own input and output. +Having input and output on your traces affects several important features: -For new instrumentation, write: +- **Browsing traces**: The traces list shows input/output previews, making it easier to find what you're looking for +- **Evaluations**: LLM-as-a-judge evaluators use trace input/output to assess quality. Empty fields mean evaluations won't work +- **Search**: You can search traces by their input/output content, but only if it's populated -- The overall request and response to the root or workflow observation. -- Step-specific input and output to the observation representing that operation. +## How trace input/output works -In most applications, the root observation already carries the same input and output that was previously written to the trace. +Before jumping into solutions, this section helps to understand how Langfuse populates trace input/output. - -An empty trace input or output does not necessarily indicate missing data. Check the observations in the trace. If the relevant observation contains the expected input and output, the data was ingested correctly. - +### Traces and observations + +In Langfuse, a **trace** represents a complete request or operation. Inside each trace, you have **observations**. + +``` +Trace: "User asks about weather" +├── Observation: Parse user intent +├── Observation: Call weather API +└── Observation: Generate response (LLM call) +``` + +Both traces and observations can have their own input/output fields. They serve different purposes: + +| | Trace input/output | Observation input/output | +| ---------------------- | ------------------------------------------------- | ---------------------------- | +| **What it represents** | The overall request and response | Each individual step | +| **Where you see it** | Traces list, evaluations | Inside the trace detail view | +| **How it's set** | Inherited from root observation OR set explicitly | Set on each observation | + +### The "root observation" rule + +By default, trace input/output is copied from the observation at the top level (called the "root observation"). + +This means: + +- If your root observation has input/output → the trace will too +- If your root observation has no input/output → your trace will be empty (unless you set it explicitly, see [below](#solution-b)) -## Use observation input and output +This is why your trace might show empty fields even though you can see data in the individual observations inside it. -Observation input and output support the same core workflows without relying on the deprecated trace fields: +## Troubleshooting -| Workflow | Recommended approach | -| ---------------------- | ------------------------------------------------------------------------------------------------------------------------------------- | -| Browse and search data | Filter or search observations directly. For the overall request and response, use the root observation. | -| Run evaluations | Create an observation-level evaluator that targets the observation containing the required input, output, and context. | -| Record a summary | Store the summary on a root or dedicated workflow observation instead of writing separate input and output values to the trace level. | +The most common reasons for empty trace input/output are: -If you still use trace-level LLM-as-a-Judge evaluators, follow the [evaluator upgrade guide](/faq/all/llm-as-a-judge-migration). +### 1. You're using a short-lived application (scripts, serverless, notebooks) -### Set input and output on the root observation +**Symptoms**: Traces sometimes appear with missing data, or don't appear at all. -Use a root observation to represent the overall application or agent invocation: +[Langfuse sends data in the background to keep your application fast](/docs/observability/data-model#background-processing). If your script or serverless function exits before the data is sent, it gets lost. + +**Solution**: Call `flush()` before your application exits. + + + + +```python +from langfuse import get_client + +langfuse = get_client() + +# Your code here... + +# Before your script ends: +langfuse.flush() +``` + +If using the `@observe()` decorator: + +```python +from langfuse import observe, get_client + +@observe() +def main(): + # Your code here... + pass + +main() +get_client().flush() +``` + + + + +```typescript +import { NodeSDK } from "@opentelemetry/sdk-node"; +import { LangfuseSpanProcessor } from "@langfuse/otel"; + +const langfuseSpanProcessor = new LangfuseSpanProcessor(); + +const sdk = new NodeSDK({ + spanProcessors: [langfuseSpanProcessor], +}); +sdk.start(); + +async function main() { + // Your code here... +} + +// Ensure data is sent before exit +main().finally(() => langfuseSpanProcessor.forceFlush()); +``` + + + + +--- + +### 2. You haven't set input/output on your root span + +**Symptoms**: Trace input/output is always empty, but observations inside the trace have data. + +If you're manually creating spans, you need to either: + +- Set input/output on your root span, OR +- Explicitly set input/output on the trace itself + +**Solution A**: Set input/output on your root span @@ -47,14 +136,16 @@ langfuse = get_client() with langfuse.start_as_current_observation( as_type="span", - name="my-pipeline", + name="my-pipeline" ) as root_span: user_input = "What's the weather like?" result = process_request(user_input) + # Set input/output on the root span + # This will automatically populate the trace root_span.update( input={"query": user_input}, - output={"response": result}, + output={"response": result} ) ``` @@ -68,6 +159,8 @@ await startActiveObservation("my-pipeline", async (rootSpan) => { const userInput = "What's the weather like?"; const result = await processRequest(userInput); + // Set input/output on the root span + // This will automatically populate the trace rootSpan.update({ input: { query: userInput }, output: { response: result }, @@ -78,13 +171,11 @@ await startActiveObservation("my-pipeline", async (rootSpan) => { -## If observation input or output is missing - -### Flush short-lived applications + -Langfuse [sends data in the background](/docs/observability/data-model#background-processing). Scripts, serverless functions, and notebooks can exit before all observations are exported. +**Solution B**: Set input/output directly on the trace -Call `flush()` or `forceFlush()` before the application exits: +Sometimes the trace input/output should be different from any observation (e.g., you want a clean summary). You can set it explicitly: @@ -94,38 +185,89 @@ from langfuse import get_client langfuse = get_client() -# Your code here... +with langfuse.start_as_current_observation( + as_type="span", + name="my-pipeline" +) as root_span: + user_input = "What's the weather like?" + result = process_request(user_input) -langfuse.flush() + # Set trace input/output explicitly (deprecated — only for backward compat with legacy trace-level LLM-as-a-judge evaluators) + root_span.set_trace_io( + input={"user_question": user_input}, + output={"answer": result} + ) ``` ```typescript -import { NodeSDK } from "@opentelemetry/sdk-node"; -import { LangfuseSpanProcessor } from "@langfuse/otel"; +import { startActiveObservation, setActiveTraceIO } from "@langfuse/tracing"; -const langfuseSpanProcessor = new LangfuseSpanProcessor(); -const sdk = new NodeSDK({ - spanProcessors: [langfuseSpanProcessor], +await startActiveObservation("my-pipeline", async (rootSpan) => { + const userInput = "What's the weather like?"; + const result = await processRequest(userInput); + + // Set trace input/output explicitly (deprecated — only for backward compat with legacy trace-level LLM-as-a-judge evaluators) + setActiveTraceIO({ + input: { user_question: userInput }, + output: { answer: result }, + }); }); +``` -sdk.start(); + + -async function main() { - // Your code here... -} +--- -main().finally(() => langfuseSpanProcessor.forceFlush()); +### 3. Your root span is being filtered out + +**Symptoms**: Trace input/output is empty when using `shouldExportSpan` to filter spans. + +If you're using `shouldExportSpan` in your `LangfuseSpanProcessor` to filter spans, and your root span gets filtered out, the trace input/output will be empty. This happens because trace input/output is [derived from the root observation by default](#the-root-observation-rule). + +**Solution**: If you want the root span to stay filtered out, use `setTraceIO()` to manually set trace input/output from within a child span that is not filtered out: + + +`setTraceIO()` and `set_trace_io()` are deprecated. They are only needed if you still rely on trace-level [LLM-as-a-judge](/docs/evaluation/evaluation-methods/llm-as-a-judge) evaluators that read trace input/output. Once you have [migrated to observation-level evaluators](/faq/all/llm-as-a-judge-migration), these calls have no effect and can be removed from your codebase. + + +```typescript +span.setTraceIO({ + input: yourInputData, + output: yourOutputData, +}); ``` - - +See the [advanced features docs](/docs/observability/sdk/advanced-features#filtering-by-instrumentation-scope) for more details on `shouldExportSpan`. + +--- + +### 4. You're using the `@observe()` decorator but input/output capture is disabled + +**Symptoms**: Decorated functions don't show input/output, even though they return values. + +The `@observe()` decorator [automatically captures function arguments as input and return values as output](/docs/observability/sdk/instrumentation#observe-wrapper). But this can be disabled. + +**Check if capture is disabled**: + +```bash +# Check your environment variables +echo $LANGFUSE_OBSERVE_DECORATOR_IO_CAPTURE_ENABLED +``` -### Check decorator input/output capture +If this is set to `false`, input/output won't be captured. -The Python [`@observe()` decorator](/docs/observability/sdk/instrumentation#observe-wrapper) captures function arguments and return values by default. If `LANGFUSE_OBSERVE_DECORATOR_IO_CAPTURE_ENABLED=false`, enable capture globally or for the relevant function: +**Solution**: Enable capture + +```bash +# In your environment +export LANGFUSE_OBSERVE_DECORATOR_IO_CAPTURE_ENABLED=true +``` + +Or enable it per-function: ```python from langfuse import observe @@ -135,22 +277,31 @@ def my_function(data): return process(data) ``` -### Ensure the relevant observation is exported +--- + +### 5. You're using an OpenTelemetry-based integration -If `shouldExportSpan` filters out your root observation, its input and output are not sent to Langfuse. Keep the root observation, or store the required values on another exported observation and target that observation in downstream workflows. +**Symptoms**: Traces from OTEL integrations (OpenLLMetry, Logfire, etc.) show empty input/output. -See the [span filtering documentation](/docs/observability/sdk/advanced-features#filtering-by-instrumentation-scope) for details. +Different OpenTelemetry providers use different attribute names for input/output. Langfuse looks for specific attributes and may not find them if your provider uses different names. -### Map OpenTelemetry input and output +**Solution**: Set the attributes Langfuse expects -OpenTelemetry integrations use different attribute names. Langfuse maps these span attributes to observation input and output, in priority order: +Langfuse maps these OTEL span attributes to observation input/output (checked in this order): -| Observation field | OpenTelemetry attributes | +| Observation field | OTEL attributes (in priority order) | | ----------------- | ---------------------------------------------------------------------------------------- | | `input` | `langfuse.observation.input`, `gen_ai.prompt`, `input.value`, `mlflow.spanInputs` | | `output` | `langfuse.observation.output`, `gen_ai.completion`, `output.value`, `mlflow.spanOutputs` | -See the [OpenTelemetry property mapping](/integrations/native/opentelemetry#property-mapping) for the complete reference. +For trace-level input/output, Langfuse looks for: + +- `langfuse.trace.input` / `langfuse.trace.output`, OR +- The root span's observation input/output (using the attributes above) + +See the [complete property mapping reference](/integrations/native/opentelemetry#property-mapping) for all supported attributes. + +**Example**: Manually set the attributes Langfuse recognizes: @@ -162,14 +313,13 @@ import json tracer = trace.get_tracer(__name__) with tracer.start_as_current_span("my-operation") as span: - span.set_attribute( - "langfuse.observation.input", - json.dumps(input_data), - ) - span.set_attribute( - "langfuse.observation.output", - json.dumps(output_data), - ) + # Set attributes that Langfuse recognizes + span.set_attribute("input.value", str(input_data)) + span.set_attribute("output.value", str(output_data)) + + # Or use the langfuse namespace for guaranteed mapping + span.set_attribute("langfuse.observation.input", json.dumps(input_data)) + span.set_attribute("langfuse.observation.output", json.dumps(output_data)) ``` @@ -181,14 +331,13 @@ import { trace } from "@opentelemetry/api"; const tracer = trace.getTracer("my-service"); tracer.startActiveSpan("my-operation", (span) => { - span.setAttribute( - "langfuse.observation.input", - JSON.stringify(inputData), - ); - span.setAttribute( - "langfuse.observation.output", - JSON.stringify(outputData), - ); + // Set attributes that Langfuse recognizes + span.setAttribute("input.value", JSON.stringify(inputData)); + span.setAttribute("output.value", JSON.stringify(outputData)); + + // Or use the langfuse namespace for guaranteed mapping + span.setAttribute("langfuse.observation.input", JSON.stringify(inputData)); + span.setAttribute("langfuse.observation.output", JSON.stringify(outputData)); span.end(); }); @@ -198,81 +347,83 @@ tracer.startActiveSpan("my-operation", (span) => {
-**Which attributes does my OpenTelemetry provider use?** +**Which attributes does my OTEL provider use?** + +Different providers use different semantic conventions. Here's how to find out what your provider sends: -1. Enable debug logging in your OpenTelemetry exporter to inspect the raw span attributes. -2. Open the observation in Langfuse and check its metadata. -3. Review the provider's semantic conventions. +1. **Enable debug logging** in your OTEL exporter to see the raw span attributes +2. **Check the trace in Langfuse**: Open a trace, click on an observation, and look at the "Metadata" tab to see which attributes were received +3. **Consult your provider's documentation** for their semantic conventions -Common conventions include: +Common providers and their conventions: -- **OpenLLMetry:** `gen_ai.prompt` and `gen_ai.completion` -- **OpenInference:** `input.value` and `output.value` -- **MLflow:** `mlflow.spanInputs` and `mlflow.spanOutputs` +- **OpenLLMetry**: Uses `gen_ai.prompt` and `gen_ai.completion` +- **OpenInference**: Uses `input.value` and `output.value` +- **MLflow**: Uses `mlflow.spanInputs` and `mlflow.spanOutputs` +- **Pydantic Logfire**: Uses custom attributes (Langfuse has specific support since PR #5841) -If your provider uses different names, map them to the Langfuse observation attributes shown above. +If your provider uses different attribute names, you have two options: + +1. Manually set the attributes Langfuse expects (as shown above) +2. Open a [GitHub issue](https://github.com/langfuse/langfuse/issues) requesting support for your provider's conventions
-## Temporary compatibility for legacy evaluators +Many OTEL-specific issues have been fixed in recent Langfuse versions. If you're self-hosting, make sure you're on the latest version. - -The trace I/O methods are deprecated. Use them only while an existing trace-level evaluator still reads trace input or output. Do not add them to new instrumentation. - +--- + +## Still having issues? + +If none of the above solutions work: -After upgrading your SDK or ingestion path, trace input and output are no longer written automatically. Until you have [upgraded the evaluator](/faq/all/llm-as-a-judge-migration), you can continue supplying the legacy fields explicitly: +1. **Enable debug logging** to see what's being sent: ```python -from langfuse import get_client +from langfuse import Langfuse -langfuse = get_client() -user_input = "What's the weather like?" +langfuse = Langfuse(debug=True) +``` -with langfuse.start_as_current_observation( - as_type="span", - name="my-pipeline", -) as root_span: - result = process_request(user_input) + + - root_span.set_trace_io( - input={"query": user_input}, - output={"response": result}, - ) +```bash +# In your environment +export LANGFUSE_DEBUG="true" ``` - + -```typescript -import { - setActiveTraceIO, - startActiveObservation, -} from "@langfuse/tracing"; +2. **Check your SDK version** and update if needed: -await startActiveObservation("my-pipeline", async () => { - const userInput = "What's the weather like?"; - const result = await processRequest(userInput); + + - setActiveTraceIO({ - input: { query: userInput }, - output: { response: result }, - }); -}); +```bash +pip install --upgrade langfuse ``` - + -Once the observation-level evaluator is validated, remove these calls. +```bash +npm update @langfuse/tracing @langfuse/otel @langfuse/client +``` -## Still having issues? + + -1. Update to the latest Langfuse SDK version. -2. Open the trace and verify that the expected observation was exported. -3. Check that the observation itself has input and output. -4. For short-lived applications, confirm that the client flushes before exit. +3. **Look at the trace structure** in the Langfuse dashboard: + - Open a trace and look at the observation tree + - Is there a single root observation? + - Do the individual observations have input/output? -If the observation data is still missing, open a [GitHub discussion](https://github.com/langfuse/langfuse/discussions) with your SDK version, instrumentation snippet, and a screenshot of the observation tree. +4. **Ask for help**: Open a [GitHub discussion](https://github.com/langfuse/langfuse/discussions) with: + - Your SDK version + - A code snippet showing how you're creating traces + - A screenshot of the trace structure in the dashboard