diff --git a/content/integrations/frameworks/ag2.mdx b/content/integrations/frameworks/ag2.mdx new file mode 100644 index 0000000000..db70708664 --- /dev/null +++ b/content/integrations/frameworks/ag2.mdx @@ -0,0 +1,277 @@ +--- +title: Observability for AG2 with Langfuse Integration +sidebarTitle: AG2 +logo: /images/integrations/ag2_icon.svg +description: Integrate Langfuse with AG2 via native OpenTelemetry tracing for full observability into multi-agent conversations, LLM calls, tool executions, and costs. +category: Integrations +--- + +# Integrate Langfuse with AG2 + +This guide shows how to integrate **Langfuse** with **AG2** using AG2's built-in OpenTelemetry tracing for full observability into your multi-agent workflows. + +> **What is AG2?** [AG2](https://ag2.ai/) ([GitHub](https://github.com/ag2ai/ag2)) is an open-source Python framework for building multi-agent AI systems. AG2 provides tools for orchestrating collaborative agents, tool use, group chats, and distributed agent-to-agent (A2A) deployments. AG2 v0.11+ includes native [OpenTelemetry tracing](https://docs.ag2.ai/docs/user-guide/tracing/opentelemetry) that captures every conversation, agent turn, LLM call, tool execution, and speaker selection as structured spans. + +> **What is Langfuse?** [Langfuse](https://langfuse.com) is an open-source LLM engineering platform. It offers tracing and monitoring capabilities for AI applications. Langfuse helps developers debug, analyze, and optimize their AI systems by providing detailed insights and integrating with a wide array of tools and frameworks through native integrations, OpenTelemetry, and dedicated SDKs. + + + AG2 is the community-driven continuation of AutoGen. If you are using + Microsoft AutoGen instead, see the [AutoGen + integration](/integrations/frameworks/autogen), which traces via OpenLIT + rather than AG2's native OpenTelemetry support. + + +## Get started + +AG2's native OpenTelemetry tracing exports rich, hierarchical spans that Langfuse ingests directly β€” no intermediary libraries needed. + + +### Step 1: Install dependencies + +```bash +pip install "ag2[openai,tracing]" opentelemetry-exporter-otlp langfuse -U +``` + +### Step 2: Configure Langfuse SDK + +Set up your Langfuse API keys. You can get these keys by signing up for a free [Langfuse Cloud](https://cloud.langfuse.com/) account or by [self-hosting Langfuse](https://langfuse.com/self-hosting). + +```python +import os + +# Get keys for your project from the project settings page: https://cloud.langfuse.com +os.environ["LANGFUSE_PUBLIC_KEY"] = "pk-lf-..." +os.environ["LANGFUSE_SECRET_KEY"] = "sk-lf-..." +os.environ["LANGFUSE_BASE_URL"] = "https://cloud.langfuse.com" # πŸ‡ͺπŸ‡Ί EU region +# os.environ["LANGFUSE_BASE_URL"] = "https://us.cloud.langfuse.com" # πŸ‡ΊπŸ‡Έ US region + +# Your OpenAI key +os.environ["OPENAI_API_KEY"] = "sk-proj-..." +``` + +Initialize the Langfuse client and verify the connection: + +```python +from langfuse import get_client + +langfuse = get_client() + +if langfuse.auth_check(): + print("Langfuse client is authenticated and ready!") +``` + +### Step 3: Configure OpenTelemetry to export to Langfuse + +Set up an OpenTelemetry `TracerProvider` that exports spans directly to Langfuse's OTel endpoint: + +```python +import os +import base64 +from opentelemetry import trace +from opentelemetry.sdk.resources import Resource +from opentelemetry.sdk.trace import TracerProvider +from opentelemetry.sdk.trace.export import BatchSpanProcessor +from opentelemetry.exporter.otlp.proto.http.trace_exporter import OTLPSpanExporter + +LANGFUSE_PUBLIC_KEY = os.environ["LANGFUSE_PUBLIC_KEY"] +LANGFUSE_SECRET_KEY = os.environ["LANGFUSE_SECRET_KEY"] +LANGFUSE_BASE_URL = os.environ["LANGFUSE_BASE_URL"] + +auth = base64.b64encode( + f"{LANGFUSE_PUBLIC_KEY}:{LANGFUSE_SECRET_KEY}".encode() +).decode() + +resource = Resource.create({"service.name": "ag2-langfuse-demo"}) +tracer_provider = TracerProvider(resource=resource) + +exporter = OTLPSpanExporter( + endpoint=f"{LANGFUSE_BASE_URL}/api/public/otel/v1/traces", + headers={"Authorization": f"Basic {auth}"}, +) +tracer_provider.add_span_processor(BatchSpanProcessor(exporter)) +trace.set_tracer_provider(tracer_provider) +``` + +### Step 4: Instrument agents and run + +Create AG2 agents and instrument them with AG2's built-in tracing: + +```python +from autogen import ConversableAgent, LLMConfig +from autogen.opentelemetry import instrument_agent, instrument_llm_wrapper + +llm_config = LLMConfig({"model": "gpt-4o-mini"}) + +assistant = ConversableAgent( + name="assistant", + system_message="You are a helpful assistant.", + llm_config=llm_config, + human_input_mode="NEVER", +) + +user_proxy = ConversableAgent( + name="user_proxy", + human_input_mode="NEVER", + max_consecutive_auto_reply=1, +) + +# Instrument agents and LLM calls +instrument_llm_wrapper(tracer_provider=tracer_provider) +instrument_agent(assistant, tracer_provider=tracer_provider) +instrument_agent(user_proxy, tracer_provider=tracer_provider) + +# Run a chat β€” traces are sent to Langfuse automatically +result = user_proxy.run( + assistant, + message="What is the capital of France?", + max_turns=2, +) +result.process() + +# Flush spans before exit +tracer_provider.shutdown() +``` + +### Step 5: View traces in Langfuse + +After executing the application, navigate to your Langfuse Trace Table. You will see hierarchical traces showing the full conversation flow β€” each agent turn, LLM call (with model, tokens, and cost), and tool execution nested in a tree that mirrors your agent workflow. + +AG2's tracing emits 7 span types that map to Langfuse observations: + +| AG2 span type | What it captures | +| ------------------- | ------------------------------------------------- | +| `conversation` | Overall chat with total token usage and cost | +| `agent` | Individual agent turn with input/output messages | +| `llm` | LLM API call with model, tokens, cost, parameters | +| `tool` | Tool execution with arguments and return value | +| `code_execution` | Code execution with output | +| `human_input` | Human input prompt and response | +| `speaker_selection` | Group chat speaker selection with candidates | + + + +## Tool use example + +AG2 traces tool executions with full argument and return value capture. This example reuses the `tracer_provider` configured in Step 3 above: + +```python +from typing import Annotated +from autogen import ConversableAgent, LLMConfig +from autogen.opentelemetry import instrument_agent, instrument_llm_wrapper +from autogen.tools import tool + +@tool(description="Get weather information for a city") +def get_weather(city: Annotated[str, "The city name"]) -> str: + """Get weather information for a city.""" + weather_data = { + "new york": "Sunny, 72F", + "london": "Cloudy, 15C", + "tokyo": "Rainy, 18C", + } + return weather_data.get(city.lower(), f"Weather data not available for {city}") + +llm_config = LLMConfig({"model": "gpt-4o-mini"}) + +weather_agent = ConversableAgent( + name="weather", + system_message="Use the get_weather tool to answer weather questions.", + functions=[get_weather], + llm_config=llm_config, + human_input_mode="NEVER", +) + +instrument_llm_wrapper(tracer_provider=tracer_provider) +instrument_agent(weather_agent, tracer_provider=tracer_provider) + +result = weather_agent.run(message="What is the weather in Tokyo?", max_turns=2) +result.process() +``` + +In Langfuse, `execute_tool get_weather` spans appear nested under the agent span, with tool arguments and return values visible in the observation input/output. + +## Group chat example + +For group chats, use `instrument_pattern` to instrument all agents in a single call (again reusing the `tracer_provider` from Step 3): + +```python +from autogen import ConversableAgent, LLMConfig +from autogen.agentchat import run_group_chat +from autogen.agentchat.group.patterns import AutoPattern +from autogen.opentelemetry import instrument_llm_wrapper, instrument_pattern + +llm_config = LLMConfig({"model": "gpt-4o-mini"}) + +researcher = ConversableAgent( + name="researcher", + system_message="You research topics and provide factual information.", + llm_config=llm_config, + human_input_mode="NEVER", +) + +writer = ConversableAgent( + name="writer", + system_message="You write clear summaries. Say TERMINATE when done.", + llm_config=llm_config, + human_input_mode="NEVER", +) + +user = ConversableAgent(name="user", human_input_mode="NEVER", llm_config=False) + +pattern = AutoPattern( + initial_agent=researcher, + agents=[researcher, writer], + user_agent=user, + group_manager_args={"llm_config": llm_config}, +) + +instrument_llm_wrapper(tracer_provider=tracer_provider) +instrument_pattern(pattern, tracer_provider=tracer_provider) + +result = run_group_chat( + pattern=pattern, + messages="Explain quantum computing in simple terms.", + max_rounds=5, +) +result.process() +``` + +This produces hierarchical traces in Langfuse including `speaker_selection` spans that show which agent was chosen and why. + +## Distributed tracing with A2A + +AG2 supports the [A2A (Agent-to-Agent) protocol](https://a2a-protocol.org/) for distributed multi-service deployments. When agents run as separate services, AG2 propagates W3C Trace Context headers across HTTP calls, and Langfuse stitches the spans into a unified trace. + +```python +from autogen import ConversableAgent, LLMConfig +from autogen.a2a import A2aAgentServer +from autogen.opentelemetry import instrument_a2a_server + +agent = ConversableAgent( + name="assistant", + llm_config=LLMConfig({"model": "gpt-4o-mini"}), +) + +server = A2aAgentServer(agent, url="http://localhost:18123/") +instrument_a2a_server(server, tracer_provider=tracer_provider) +``` + +## Environment variables (alternative setup) + +Instead of configuring the exporter in code, you can use environment variables: + +```bash +export OTEL_EXPORTER_OTLP_ENDPOINT="https://cloud.langfuse.com/api/public/otel" +export OTEL_EXPORTER_OTLP_HEADERS="Authorization=Basic $(echo -n 'pk-lf-...:sk-lf-...' | base64)" +export OTEL_SERVICE_NAME="ag2-app" +``` + +## Learn more + +- [AG2 OpenTelemetry documentation](https://docs.ag2.ai/docs/user-guide/tracing/opentelemetry) +- [AG2 GitHub](https://github.com/ag2ai/ag2) +- [Langfuse OpenTelemetry integration](/integrations/native/opentelemetry) +- [OpenTelemetry GenAI Semantic Conventions](https://opentelemetry.io/docs/specs/semconv/gen-ai/gen-ai-agent-spans/) + +import LearnMore from "@/components-mdx/integration-learn-more.mdx"; + + diff --git a/content/integrations/frameworks/genkit.mdx b/content/integrations/frameworks/genkit.mdx new file mode 100644 index 0000000000..a7e04f74d6 --- /dev/null +++ b/content/integrations/frameworks/genkit.mdx @@ -0,0 +1,120 @@ +--- +title: Trace Genkit with Langfuse via OpenTelemetry +sidebarTitle: Genkit +logo: /images/integrations/genkit_icon.svg +description: Learn how to send Genkit traces to Langfuse from Go applications using OpenTelemetry. +category: Integrations +--- + +# Trace Genkit with Langfuse + +Langfuse can ingest Genkit traces via OpenTelemetry. Once your Genkit application exports OTLP spans, Langfuse maps Genkit spans into traces and observations with model input, output, and usage data when available. + +> **What is Genkit?** [Genkit](https://genkit.dev/) is an open-source framework for building AI features and agentic workflows with structured flows, tool calling, and model integrations. + +> **What is Langfuse?** [Langfuse](https://langfuse.com/) is an open-source LLM engineering platform for observability, prompt management, and evals. + + +This Langfuse integration is OpenTelemetry-based, not Go-specific. If your Genkit setup already exports OTLP traces, you can point that exporter at Langfuse. For Genkit Go, the common setup is the [Genkit OpenTelemetry plugin](https://github.com/genkit-ai/opentelemetry-go-plugin?tab=readme-ov-file#langfuse). + + + +## Step 1: Install the Genkit OpenTelemetry plugin + +```bash +go get github.com/xavidop/genkit-opentelemetry-go +``` + +The package is documented in the [`genkit-ai/opentelemetry-go-plugin` repository](https://github.com/genkit-ai/opentelemetry-go-plugin). + +## Step 2: Configure Langfuse credentials + +```bash +export LANGFUSE_BASE_URL="https://cloud.langfuse.com" # πŸ‡ͺπŸ‡Ί EU region +# export LANGFUSE_BASE_URL="https://us.cloud.langfuse.com" # πŸ‡ΊπŸ‡Έ US region + +export LANGFUSE_PUBLIC_KEY="pk-lf-..." +export LANGFUSE_SECRET_KEY="sk-lf-..." +``` + +## Step 3: Initialize the OpenTelemetry plugin + +Configure the plugin to send OTLP/HTTP traces to your Langfuse instance. Langfuse currently accepts OTLP over HTTP, so set `OTLPUseHTTP: true`. The example below disables metrics export because this setup is only sending traces to Langfuse. + +```go +package main + +import ( + "context" + "encoding/base64" + "log" + "os" + + "github.com/firebase/genkit/go/genkit" + opentelemetry "github.com/xavidop/genkit-opentelemetry-go" +) + +func main() { + baseURL := os.Getenv("LANGFUSE_BASE_URL") + if baseURL == "" { + baseURL = "https://cloud.langfuse.com" + } + + publicKey := os.Getenv("LANGFUSE_PUBLIC_KEY") + secretKey := os.Getenv("LANGFUSE_SECRET_KEY") + if publicKey == "" || secretKey == "" { + log.Fatal("LANGFUSE_PUBLIC_KEY and LANGFUSE_SECRET_KEY must be set") + } + + auth := base64.StdEncoding.EncodeToString([]byte(publicKey + ":" + secretKey)) + + plugin := opentelemetry.New(opentelemetry.Config{ + ServiceName: "my-genkit-app", + ForceExport: true, + DisableMetricsExporter: true, + OTLPEndpoint: baseURL + "/api/public/otel/v1/traces", + OTLPUseHTTP: true, + OTLPHeaders: map[string]string{ + "Authorization": "Basic " + auth, + "X-Langfuse-Ingestion-Version": "4", + }, + }) + + genkit.Init( + context.Background(), + genkit.WithPlugins(plugin), + ) +} +``` + +## Step 4: Run your Genkit flows + +After initializing the plugin, Genkit flows and model calls emit OpenTelemetry spans. Langfuse ingests those spans and displays them as traces with nested observations, including inputs, outputs, tool calls, and token usage when Genkit provides that metadata. + +If you already use an OpenTelemetry Collector, you can route Genkit spans through the Collector and export them to Langfuse from there. For endpoint and authentication details, see the [Langfuse OpenTelemetry guide](/integrations/native/opentelemetry). + +## Step 5: View traces in Langfuse + +Open **Traces** in Langfuse to inspect flow executions, model calls, latency, cost, and errors. If traces do not show up in local development, double-check `ForceExport: true`, the OTLP endpoint, and the Basic Auth credentials. + +![Genkit trace in Langfuse](/images/cookbook/integration-genkit/genkit-example-trace.png) + + + +## Troubleshooting + +- **No traces appear in Langfuse** + - The Genkit OpenTelemetry plugin is initialized before your flows run. + - `OTLPUseHTTP` is set to `true`. + - The OTLP endpoint points to `https:///api/public/otel/v1/traces`. + - The `Authorization` header contains the Base64-encoded `public_key:secret_key` pair. + - If the `X-Langfuse-Ingestion-Version: 4` header is not set, traces may still ingest successfully but can take up to 10 minutes to appear in Langfuse. + - `ForceExport` is enabled in short-lived local processes so spans are flushed before the program exits. + +## Next steps + +Once traces are flowing, you can use Langfuse to debug prompt inputs and outputs, add scores, and build dashboards on top of your Genkit workflows. + +import NextSteps from "@/components-mdx/get-started/next-steps.mdx"; + + diff --git a/content/integrations/frameworks/meta.json b/content/integrations/frameworks/meta.json index e5873c8d4e..0c456f1202 100644 --- a/content/integrations/frameworks/meta.json +++ b/content/integrations/frameworks/meta.json @@ -1,6 +1,7 @@ { "title": "Frameworks", "pages": [ + "ag2", "agno-agents", "ai-sdk-cpp", "amazon-agentcore", @@ -12,6 +13,7 @@ "dspy", "eve", "flue", + "genkit", "google-adk", "haystack", "instructor", diff --git a/content/integrations/gateways/agentgateway.mdx b/content/integrations/gateways/agentgateway.mdx new file mode 100644 index 0000000000..93a9a3327f --- /dev/null +++ b/content/integrations/gateways/agentgateway.mdx @@ -0,0 +1,435 @@ +--- +title: AgentGateway Integration +sidebarTitle: AgentGateway +logo: /images/integrations/agentgateway_icon.svg +description: Route LLM traffic through AgentGateway and get full observability in Langfuse via OpenTelemetry tracing +category: Integrations +--- + +# Trace AI traffic through AgentGateway with Langfuse + +This guide shows how to integrate **Langfuse** with **AgentGateway** to automatically capture and observe all LLM API calls routed through the gateway β€” without modifying your application code. + +> **What is AgentGateway?** [AgentGateway](https://agentgateway.dev) is an open source data plane built on AI-native protocols (A2A & MCP) to connect, secure, and observe agent-to-agent and agent-to-tool communication across any framework and environment. It routes traffic to LLM providers (OpenAI, Anthropic, Azure OpenAI, Bedrock, Gemini, and more), MCP tool servers, and AI agents. Open source ([CNCF](https://www.cncf.io/)) with an [Enterprise edition](https://docs.solo.io/agentgateway) from Solo.io. + +> **What is Langfuse?** [Langfuse](https://langfuse.com) is an open-source LLM observability platform that helps you trace, monitor, evaluate, and debug your LLM applications. + +## Features + +- **Zero-code instrumentation**: Automatic tracing for all LLM calls proxied through AgentGateway +- **Multi-provider support**: OpenAI, Anthropic, Azure OpenAI, AWS Bedrock, Google Gemini, Vertex AI, Ollama, and any OpenAI-compatible provider +- **Rich GenAI telemetry**: Model, token usage (input/output/total), streaming status, temperature, and other LLM parameters +- **Native OTLP export**: AgentGateway emits OpenTelemetry traces natively β€” no sidecar or SDK needed + +## Architecture + +``` +β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β” β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β” β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β” β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β” +β”‚ AI Agent │────▢│ AgentGateway │────▢│ OTEL Collector │────▢│ Langfuseβ”‚ +β”‚ β”‚ β”‚ (Gateway API) β”‚ β”‚ (otlphttp export)β”‚ β”‚ β”‚ +β”‚ β”‚ β”‚ β”‚ β”‚ β”‚ β”‚ β”‚ +β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜ β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜ β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜ β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜ + β”‚ + β–Ό + β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β” + β”‚ LLM Provider β”‚ + β”‚ (OpenAI, β”‚ + β”‚ Anthropic, β”‚ + β”‚ etc.) β”‚ + β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜ +``` + +AgentGateway emits OpenTelemetry traces for every request. An OTEL Collector receives the traces and forwards them to Langfuse's OTEL endpoint. + +## Prerequisites + +- Kubernetes cluster with [AgentGateway installed](https://agentgateway.dev/docs/kubernetes/latest/setup/) (OSS or Enterprise) +- Langfuse account ([self-hosted](/self-hosting) or [cloud](https://cloud.langfuse.com)) +- `kubectl` and `helm` CLI tools + +## Step 1: Get your Langfuse credentials + +From your Langfuse project settings, grab: + +- **Public Key** (`pk-lf-...`) +- **Secret Key** (`sk-lf-...`) +- **OTEL Endpoint** (e.g., `https://us.cloud.langfuse.com/api/public/otel` or your self-hosted URL) + +Create the Base64-encoded Basic auth header: + +```bash +export LANGFUSE_PUBLIC_KEY="pk-lf-..." +export LANGFUSE_SECRET_KEY="sk-lf-..." +export LANGFUSE_AUTH=$(echo -n "${LANGFUSE_PUBLIC_KEY}:${LANGFUSE_SECRET_KEY}" | base64) +echo $LANGFUSE_AUTH +``` + +## Step 2: Deploy an OpenTelemetry Collector + +Deploy an OTEL Collector that receives traces from AgentGateway and forwards them to Langfuse: + +```yaml +# otel-collector.yaml +apiVersion: v1 +kind: ConfigMap +metadata: + name: langfuse-otel-collector-config + namespace: agentgateway-system +data: + config.yaml: | + receivers: + otlp: + protocols: + grpc: + endpoint: 0.0.0.0:4317 + http: + endpoint: 0.0.0.0:4318 + exporters: + otlphttp/langfuse: + endpoint: https://us.cloud.langfuse.com/api/public/otel # Replace with your Langfuse OTEL endpoint + headers: + Authorization: "Basic " # Replace with your Base64-encoded credentials + retry_on_failure: + enabled: true + initial_interval: 5s + max_interval: 30s + max_elapsed_time: 300s + processors: + batch: + send_batch_size: 1000 + timeout: 5s + service: + pipelines: + traces: + receivers: [otlp] + processors: [batch] + exporters: [otlphttp/langfuse] +--- +apiVersion: apps/v1 +kind: Deployment +metadata: + name: langfuse-otel-collector + namespace: agentgateway-system + labels: + app: langfuse-otel-collector +spec: + replicas: 1 + selector: + matchLabels: + app: langfuse-otel-collector + template: + metadata: + labels: + app: langfuse-otel-collector + spec: + containers: + - name: otel-collector + image: docker.io/otel/opentelemetry-collector-contrib:0.132.1 + args: ["--config=/conf/config.yaml"] + ports: + - containerPort: 4317 + name: otlp-grpc + - containerPort: 4318 + name: otlp-http + volumeMounts: + - name: config + mountPath: /conf + resources: + requests: + cpu: 50m + memory: 128Mi + limits: + cpu: 200m + memory: 256Mi + volumes: + - name: config + configMap: + name: langfuse-otel-collector-config +--- +apiVersion: v1 +kind: Service +metadata: + name: langfuse-otel-collector + namespace: agentgateway-system + labels: + app: langfuse-otel-collector +spec: + selector: + app: langfuse-otel-collector + ports: + - name: otlp-grpc + port: 4317 + targetPort: 4317 + - name: otlp-http + port: 4318 + targetPort: 4318 +``` + +Apply it: + +```bash +kubectl apply -f otel-collector.yaml +``` + +## Step 3: Configure AgentGateway tracing + + + + +Create an `EnterpriseAgentgatewayParameters` resource to configure tracing with rich GenAI semantic conventions: + +```yaml +# tracing-params.yaml +apiVersion: enterpriseagentgateway.solo.io/v1alpha1 +kind: EnterpriseAgentgatewayParameters +metadata: + name: tracing + namespace: agentgateway-system +spec: + rawConfig: + config: + tracing: + otlpEndpoint: grpc://langfuse-otel-collector.agentgateway-system.svc.cluster.local:4317 + otlpProtocol: grpc + randomSampling: true + fields: + add: + # GenAI semantic conventions (maps to Langfuse fields) + gen_ai.operation.name: '"chat"' + gen_ai.system: "llm.provider" + gen_ai.request.model: "llm.requestModel" + gen_ai.response.model: "llm.responseModel" + gen_ai.streaming: "llm.streaming" + # Token usage + gen_ai.usage.input_tokens: "llm.inputTokens" + gen_ai.usage.output_tokens: "llm.outputTokens" + gen_ai.usage.total_tokens: "llm.totalTokens" + # LLM parameters + gen_ai.request.temperature: "llm.params.temperature" + gen_ai.request.top_p: "llm.params.top_p" + gen_ai.request.max_tokens: "llm.params.max_tokens" + # Prompt & completion content + gen_ai.prompt: "llm.prompt" + gen_ai.completion: "llm.completion" + # HTTP context + http.method: "request.method" + http.path: "request.path" + http.status_code: "response.code" +``` + +Apply and reference it from your Gateway: + +```yaml +# gateway.yaml +apiVersion: gateway.networking.k8s.io/v1 +kind: Gateway +metadata: + name: ai-gateway + namespace: agentgateway-system +spec: + gatewayClassName: enterprise-agentgateway + infrastructure: + parametersRef: + name: tracing + group: enterpriseagentgateway.solo.io + kind: EnterpriseAgentgatewayParameters + listeners: + - name: http + port: 8080 + protocol: HTTP + allowedRoutes: + namespaces: + from: All +``` + +```bash +kubectl apply -f tracing-params.yaml +kubectl apply -f gateway.yaml +``` + + + + +For the open source edition, configure tracing via Helm values when installing AgentGateway: + +```bash +helm upgrade -i agentgateway oci://ghcr.io/kgateway-dev/charts/agentgateway \ + --namespace agentgateway-system \ + --version v2.2.0 \ + --set "gateway.telemetry.tracing.otlp.endpoint=langfuse-otel-collector.agentgateway-system.svc.cluster.local:4317" +``` + +Then create a Gateway resource: + +```yaml +apiVersion: gateway.networking.k8s.io/v1 +kind: Gateway +metadata: + name: ai-gateway + namespace: agentgateway-system +spec: + gatewayClassName: agentgateway + listeners: + - name: http + port: 8080 + protocol: HTTP +``` + + + The open source edition provides basic OTLP tracing. The Enterprise edition adds rich GenAI semantic conventions with customizable field mappings. + + + + + +## Step 4: Set up an LLM route + +Create an `AgentgatewayBackend` and `HTTPRoute` to route traffic to an LLM provider: + +```yaml +# openai-backend.yaml +apiVersion: agentgateway.dev/v1alpha1 +kind: AgentgatewayBackend +metadata: + name: openai + namespace: agentgateway-system +spec: + ai: + provider: + openai: {} + policies: + auth: + secretRef: + name: openai-api-key + namespace: agentgateway-system +--- +apiVersion: gateway.networking.k8s.io/v1 +kind: HTTPRoute +metadata: + name: openai-route + namespace: agentgateway-system +spec: + parentRefs: + - name: ai-gateway + rules: + - matches: + - path: + type: PathPrefix + value: /openai + backendRefs: + - name: openai + group: agentgateway.dev + kind: AgentgatewayBackend +``` + +Create the API key secret: + +```bash +kubectl create secret generic openai-api-key \ + -n agentgateway-system \ + --from-literal="Authorization=Bearer $OPENAI_API_KEY" +``` + +Apply the route: + +```bash +kubectl apply -f openai-backend.yaml +``` + +## Step 5: Send a test request + +```bash +# Get the gateway address +export GATEWAY_IP=$(kubectl get gateway ai-gateway -n agentgateway-system \ + -o jsonpath='{.status.addresses[0].value}') + +# Or port-forward for local testing +kubectl port-forward -n agentgateway-system svc/ai-gateway 8080:8080 & + +# Send a request +curl http://${GATEWAY_IP:-localhost}:8080/openai/v1/chat/completions \ + -H "Content-Type: application/json" \ + -d '{ + "model": "gpt-4o", + "messages": [ + {"role": "system", "content": "You are a helpful assistant."}, + {"role": "user", "content": "What is Kubernetes?"} + ] + }' | jq . +``` + +## Step 6: View traces in Langfuse + +Open your Langfuse dashboard. You should see traces with: + +- **Model**: The LLM model used (e.g., `gpt-4o`) +- **Token usage**: Input, output, and total tokens +- **Latency**: End-to-end request duration +- **Prompt & completion**: Full request/response content (Enterprise) +- **Cost**: Automatically calculated from model and token usage + +Each trace includes the full GenAI semantic convention attributes, giving you deep visibility into every LLM call flowing through your gateway. + +## Advanced: Multiple exporters + +You can fan out traces to multiple backends (e.g., Langfuse + Jaeger + your own collector) by adding additional exporters to the OTEL Collector config: + +```yaml +exporters: + otlphttp/langfuse: + endpoint: https://us.cloud.langfuse.com/api/public/otel + headers: + Authorization: "Basic " + otlp/jaeger: + endpoint: jaeger-collector.observability:4317 + tls: + insecure: true +service: + pipelines: + traces: + receivers: [otlp] + processors: [batch] + exporters: [otlphttp/langfuse, otlp/jaeger] +``` + +## Advanced: Adding metadata + +Pass custom metadata through HTTP headers and map them to trace attributes in the tracing config. To enable Langfuse [user tracking](/docs/observability/features/users) and [session grouping](/docs/observability/features/sessions), map the headers to the attribute names Langfuse recognizes (`langfuse.user.id` and `langfuse.session.id`, or `user.id` and `session.id`): + +```yaml +# In the EnterpriseAgentgatewayParameters tracing config +fields: + add: + langfuse.user.id: 'request.headers["x-user-id"]' + langfuse.session.id: 'request.headers["x-session-id"]' +``` + +```bash +curl http://localhost:8080/openai/v1/chat/completions \ + -H "Content-Type: application/json" \ + -H "x-user-id: user-123" \ + -H "x-session-id: session-abc" \ + -d '{ + "model": "gpt-4o", + "messages": [{"role": "user", "content": "Hello"}] + }' +``` + +Headers that are not mapped to these attribute names are still captured as custom span attributes (Enterprise tracing config) and can be used for filtering, but they do not populate the Langfuse user and session views. + +## Troubleshooting + +| Issue | Check | +| --------------------------- | ---------------------------------------------------------------------------------------------------------- | +| No traces in Langfuse | Verify OTEL Collector is running: `kubectl get pods -n agentgateway-system -l app=langfuse-otel-collector` | +| Auth errors | Verify Base64 credentials: `echo -n "pk-lf-...:sk-lf-..." \| base64` | +| Missing token counts | Ensure Enterprise edition with `fields.add` config for `gen_ai.usage.*` | +| Traces but no cost | Langfuse calculates cost from `gen_ai.usage.*` and `gen_ai.response.model` β€” ensure both are present | +| Gateway not emitting traces | Check Gateway references the tracing `parametersRef` and the OTEL endpoint is reachable | + +## Learn more + +- [AgentGateway Documentation](https://agentgateway.dev) +- [Enterprise AgentGateway](https://docs.solo.io/agentgateway) +- [Langfuse OpenTelemetry integration](/integrations/native/opentelemetry) +- [OpenTelemetry GenAI Semantic Conventions](https://opentelemetry.io/docs/specs/semconv/gen-ai/) diff --git a/content/integrations/gateways/meta.json b/content/integrations/gateways/meta.json index ef8f656a04..aca9716ec1 100644 --- a/content/integrations/gateways/meta.json +++ b/content/integrations/gateways/meta.json @@ -1,6 +1,7 @@ { "title": "Gateways", "pages": [ + "agentgateway", "anannas", "helicone", "kong-ai-plugin", diff --git a/public/images/cookbook/integration-genkit/genkit-example-trace.png b/public/images/cookbook/integration-genkit/genkit-example-trace.png new file mode 100644 index 0000000000..fb67950bdb Binary files /dev/null and b/public/images/cookbook/integration-genkit/genkit-example-trace.png differ diff --git a/public/images/integrations/ag2_icon.svg b/public/images/integrations/ag2_icon.svg new file mode 100644 index 0000000000..e75885298e --- /dev/null +++ b/public/images/integrations/ag2_icon.svg @@ -0,0 +1,38 @@ + +AG2 + + + + + + + + + diff --git a/public/images/integrations/agentgateway_icon.svg b/public/images/integrations/agentgateway_icon.svg new file mode 100644 index 0000000000..0b3279c273 --- /dev/null +++ b/public/images/integrations/agentgateway_icon.svg @@ -0,0 +1,12 @@ + + + + + + + + + + + + diff --git a/public/images/integrations/genkit_icon.svg b/public/images/integrations/genkit_icon.svg new file mode 100644 index 0000000000..a41a538bde --- /dev/null +++ b/public/images/integrations/genkit_icon.svg @@ -0,0 +1,13 @@ + + Genkit + + +