diff --git a/en/docs/monitoring/api-analytics/moesif-analytics/moesif-data-capture.md b/en/docs/monitoring/api-analytics/moesif-analytics/moesif-data-capture.md new file mode 100644 index 0000000000..8f6e7c6bbe --- /dev/null +++ b/en/docs/monitoring/api-analytics/moesif-analytics/moesif-data-capture.md @@ -0,0 +1,227 @@ +# Capturing Request and Response Data + +By default, a Moesif analytics event describes an API invocation without carrying its content: you get the +API, the operation, the response code, the latencies and the identity fields, but not the headers or the +message bodies. + +WSO2 API Manager can publish both, and each is a separate opt-in. This page covers what is captured, what +is never captured, and how to configure each option. + +!!! note "Analytics must be enabled first" + Both options require analytics itself to be enabled and pointed at Moesif. See + [Moesif Analytics Integration]({{base_path}}/monitoring/api-analytics/moesif-analytics/moesif-integration-guide/) for the base configuration. + +## Capturing Request and Response Headers + +By default, request and response headers are **not** sent to Moesif. To include them, set +`send_headers` to `true`: + +```toml +[apim.analytics.properties] +send_headers = true +``` + +Headers are then published as the `requestHeaders` and `responseHeaders` fields of the analytics +event, and appear on the request and response in Moesif. + +### Headers That Are Never Published + +Regardless of your configuration, the following headers are excluded from the analytics event, in +both the request and the response direction. You do not need to configure anything to protect them: + +| **Header** | **Why it is excluded** | +|------------|------------------------| +| `Authorization` | Carries the bearer token or basic credentials used to invoke the API | +| `apikey` | Carries the API key used to invoke the API | +| `Cookie` | Carries client session state | +| `Set-Cookie` | Carries session state issued to the client | + +!!! note "This affects analytics only, not your API traffic" + These headers are dropped from the copy of the headers that is published to Moesif. The messages + themselves are not modified: the backend still receives all the request headers your client sent, and the client still receives all the response headers your backend + returned. + +Header names are matched **case-insensitively**, so `authorization`, `Authorization` and +`AUTHORIZATION` are all excluded. This also means headers sent by HTTP/2 clients, which lowercase all +header names, are captured and matched correctly. + +!!! warning "Custom authorization headers are not removed automatically" + If an API is configured to accept its credentials in a **custom** header name rather than the + default `Authorization` or `apikey`, that header is not recognised at this layer and is **not** + removed automatically. Mask it explicitly using + [`[apim.analytics.mask]`]({{base_path}}/monitoring/api-analytics/moesif-analytics/moesif-data-masking/). + +To hide the value of any other header without removing the header itself, see +[Privacy and Data Masking]({{base_path}}/monitoring/api-analytics/moesif-analytics/moesif-data-masking/). + +## Capturing Request and Response Bodies + +WSO2 API Manager can also publish the request and response **bodies** to Moesif, where they appear on +the request and response of each event and can be searched, filtered and inspected alongside the +rest of your analytics data. + +This is an opt-in feature. It is disabled by default because it publishes the full content of your +API traffic to Moesif and requires the gateway to hold each message in memory. + +### Enabling Body Capture + +Add the following to your `deployment.toml` file and restart the server: + +```toml +[apim.analytics.properties] +send_payloads = true +payload_size_limit = 100000 +capture_payloads_without_content_length = false +``` + +Body capture also requires analytics itself to be enabled (`[apim.analytics] enable = true`). When analytics is disabled, no body is captured and no message is built, even if `send_payloads` is `true`. + +### Body Capture Configuration Reference + +| **Name** | **Description** | **Default Value** | **Possible Data Types** | **Optional** | +|----------|-----------------|-------------------|-------------------------|--------------| +| send_payloads | Enables request and response body capture. | false | Boolean | Yes | +| payload_size_limit | Maximum size, in bytes, of a single captured body. A body larger than this is dropped from analytics; it is not truncated. The limit is applied separately to both the request body and the response body. | 100000 | Integer | Yes | +| capture_payloads_without_content_length | Whether to capture a body that does not declare a `Content-Length` header, for example a chunked response. | false | Boolean | Yes | + +If `payload_size_limit` is set to a value that is not a positive integer, the default of `100000` is +used instead and a warning is logged once. + +### What Is Captured + +The body is captured according to its content type: + +| **Payload** | **How it is published** | +|-------------|-------------------------| +| JSON | Parsed and published as a structured, searchable object. Because it is parsed rather than forwarded verbatim, whitespace and key formatting are not preserved | +| Plain text | Base64-encoded, and flagged to Moesif with a transfer encoding of `base64` | +| XML and SOAP | Serialized from the message body, then Base64-encoded and flagged with a transfer encoding of `base64` | +| Binary | Base64-encoded, and flagged to Moesif with a transfer encoding of `base64` | + +A body is treated as JSON when its content type contains `json`, or when the body itself begins with `{` +or `[`. A body that is declared as JSON but fails to parse is Base64-encoded instead, as is anything else. +Moesif decodes Base64 bodies for display, so this affects how the body is transported rather than whether +you can read it in the Moesif UI. + +The `Content-Type` of the captured body is published alongside it, so Moesif can label and parse the +body correctly even when `send_headers` is set to `false`. + +### What Is Not Captured + +A body is skipped in each of the following cases. In every one of them the full message is still +forwarded to the backend or the client; only the analytics copy is omitted. + +- **Requests with no body**, such as `GET` and `DELETE`. +- **Server-sent events** (`text/event-stream`), **multipart payloads** (`multipart/*`, including + file uploads), and **form submissions** (`application/x-www-form-urlencoded`). +- **Content types with no registered message builder.** The gateway consults the message builders + registered in `/repository/conf/axis2/axis2.xml` and skips any content type it does not + recognise, rather than risk corrupting a payload it cannot safely interpret. If you need a custom + content type captured, register a message builder for it in `axis2.xml`. +- **WebSocket APIs.** +- **Asynchronous and streaming APIs**, such as SSE and webhook APIs. +- **Bodies larger than `payload_size_limit`** - see [Size Limits](#size-limits). +- **Bodies with no `Content-Length` header**, unless `capture_payloads_without_content_length` is + enabled - see [Payloads Without a Content-Length](#payloads-without-a-content-length). + +### Size Limits + +`payload_size_limit` is measured in **bytes**, and is applied separately to the request body and the +response body. + +A body that exceeds the limit is **dropped in its entirety, not truncated**. This is deliberate: +Moesif only ever receives a whole, valid body or no body at all, so a partial payload can never be +mistaken for the real one. + +Where the payload declares its size through a `Content-Length` header, the check is applied *before* +the message is read into memory, so an oversized body is never buffered and the message is passed +straight through. A payload whose size only becomes known once it has been read is dropped after the +fact, so it is still subject to the re-serialization behaviour described in +[Impact on Request Forwarding](#impact-on-request-forwarding). + +### Payloads Without a Content-Length + +A body sent with chunked transfer encoding does not declare a `Content-Length`, so its size cannot be +checked before it is read. By default, such bodies are skipped, which keeps the default configuration +memory-safe. + +Set `capture_payloads_without_content_length = true` to capture them anyway. + +!!! warning "Memory impact" + With this setting enabled, a chunked body is read into memory in full and only then discarded if + it turns out to exceed `payload_size_limit`. A large chunked payload under load can therefore + exhaust the gateway's heap. Enable it only if you need these bodies and have verified you have + the memory headroom for them. + +### Bodies Are Never Masked + +!!! warning "Captured bodies are published in full" + The masking options under `[apim.analytics.mask]` apply to identity fields and to named headers. + They do **not** apply to request or response bodies. When `send_payloads` is enabled, every + captured body is published to Moesif exactly as it appeared, including any personal data, + credentials, payment details or other sensitive content it contains. + + There is no field-level redaction and no per-API opt-out, the setting is on or off for the + entire gateway. Before enabling it in production, confirm that publishing the full content of + your API traffic to Moesif is compatible with your organisation's data protection policies. + +### Impact on Request Forwarding + +!!! warning "Bodies are re-serialized when capture is enabled" + Capturing a body requires the gateway to build the message, which means the message is + re-serialized when it is forwarded. The forwarded body remains semantically equivalent, but it + is **not guaranteed to be byte-identical** to what the client sent. Whitespace, attribute and + namespace ordering, JSON key formatting and chunking may all differ. + + As a result, a signature computed over the raw bytes of the body (such as a JWS, an + HMAC-signed request body or a WS-Security signature) may fail to verify at the backend while + `send_payloads` is enabled. If any of your APIs rely on body signatures, do not enable body + capture for that gateway. + +### Performance and Memory Considerations + +Enabling `send_payloads` is more costly than the rest of the analytics pipeline: + +- Each captured message is held in memory in full and re-serialized when forwarded, rather than + being streamed straight through. +- Event sizes grow with your payload sizes, increasing the volume published to Moesif. + +Keep `payload_size_limit` no larger than you actually need, leave +`capture_payloads_without_content_length` disabled unless required, and validate the configuration +under representative load before rolling it out to production. + +### Troubleshooting Body Capture + +When a body is missing from Moesif, the gateway records the reason at debug level. Enable debug +logging for the capture utility by adding the following to +`/repository/conf/log4j2.properties`: + +```properties +logger.analytics-payload.name = org.wso2.carbon.apimgt.gateway.handlers.analytics.AnalyticsPayloadUtil +logger.analytics-payload.level = DEBUG +``` + +Add `analytics-payload` to the comma-separated `loggers` list at the top of the same file, then +invoke the API again and check `/repository/logs/wso2carbon.log`. Each skipped or dropped +body is logged with the reason and the direction, for example: + +``` +Dropping response body from analytics: 250000 bytes exceeds payload_size_limit of 100000. Increase payload_size_limit to capture it. +``` + +## Troubleshooting Missing Bodies + +If events reach Moesif but the bodies are missing: + +1. **Verify the Configuration**: Confirm that `send_payloads = true` is set under + `[apim.analytics.properties]` and that the server has been restarted since the change +2. **Check the Exclusions**: Confirm the payload is not one of the types that are never captured, + such as a multipart upload, a form submission or a server-sent event stream, see + [What Is Not Captured](#what-is-not-captured) +3. **Check the Size Limit**: A body larger than `payload_size_limit` is dropped rather than + truncated. Raise the limit if you need larger bodies captured +4. **Check for a Missing Content-Length**: A chunked payload is skipped unless + `capture_payloads_without_content_length` is enabled +5. **Enable Debug Logging**: The gateway logs the exact reason each body was skipped, see + [Troubleshooting Body Capture](#troubleshooting-body-capture) + diff --git a/en/docs/monitoring/api-analytics/moesif-analytics/moesif-data-masking.md b/en/docs/monitoring/api-analytics/moesif-analytics/moesif-data-masking.md new file mode 100644 index 0000000000..35499c94e1 --- /dev/null +++ b/en/docs/monitoring/api-analytics/moesif-analytics/moesif-data-masking.md @@ -0,0 +1,78 @@ +# Privacy and Data Masking + +Analytics events carry identity information about the client that invoked the API, and, if you have enabled +header capture, whatever your headers happen to contain. This page covers how to mask that information +before it is published to Moesif. + +!!! warning "Protect your Moesif API key" + Ensure that you **do not** expose your Moesif API Key in public repositories or logs, as it can lead + to unauthorized access to your analytics data. + +## Understanding Data Privacy Requirements + +WSO2 Analytics data may contain PII (Personally Identifiable Information) such as user IP addresses and usernames. Additionally, when `send_headers` is set to `true`, request and response headers may contain sensitive information. + +To comply with data privacy regulations (GDPR, CCPA, etc.) and protect user privacy, it is strongly recommended to mask or anonymize such sensitive information before sending it to Moesif. + +!!! warning "What masking covers" + Masking applies **only** to the identity fields listed below and to the request and response + headers you name explicitly. It does **not** apply to request or response bodies. If you have + enabled body capture with `send_payloads`, every captured body is published to Moesif in full, see [Bodies Are Never Masked]({{base_path}}/monitoring/api-analytics/moesif-analytics/moesif-data-capture/#bodies-are-never-masked). + +## Configuring Data Masking + +Add the following configuration to your `deployment.toml` file to enable data masking: + +```toml +[apim.analytics.mask] +"userIp" = "IPV4" +"userName" = "EMAIL" +"userId" = "EMAIL" +"userAgent" = "STRING" +"applicationOwner" = "EMAIL" +request_headers = ["X-Custom-Auth", "X-API-Key"] +response_headers = ["X-Account-Number"] +``` + +A masked header is published to Moesif with its value replaced by `*****`; the header name itself is +still visible. As with the headers excluded above, masking applies only to the published event, the +header reaches the backend or the client with its real value intact. Header names in +`request_headers` and `response_headers` are matched case-insensitively. + +!!! note + You do not need to list `Authorization`, `apikey`, `Cookie` or `Set-Cookie` here. Those headers + are removed from analytics events entirely, whether or not you configure masking, see + [Headers That Are Never Published]({{base_path}}/monitoring/api-analytics/moesif-analytics/moesif-data-capture/#headers-that-are-never-published). Use `request_headers` and + `response_headers` for headers specific to your deployment, such as a custom authorization + header name or a header carrying customer identifiers. + +## Masking Configuration Reference + +| **Name** | **Description** | **Accepted Values** | +|----------|-----------------|---------------------| +| userIp | Defines the format used to capture and store the user's IP address in analytics records | IPV4, IPV6 | +| userName | Specifies the format of the username field used for analytics or identification | EMAIL, STRING | +| userId | Identifies how the user ID is represented in analytics data | EMAIL, STRING | +| userAgent | Represents the type of the user agent string recorded from the client request | STRING | +| applicationOwner | Specifies the format of the application owner's identifier | EMAIL, STRING | +| response_headers | List of response headers to be masked for analytics or logging purposes | Header keys as strings | +| request_headers | List of request headers to be masked for analytics or logging purposes | Header keys as strings | + +## Masking Behavior Examples + +- **IPV4**: Masks the 3rd octet of an IPv4 address + - Original: `192.168.1.98` + - Masked: `192.168.***.98` + +- **IPV6**: Masks the 4th, 5th, 6th and 7th segments of an IPv6 address + - Original: `2001:0db8:85a3:0000:0000:8a2e:0370:7334` + - Masked: `2001:0db8:85a3:****:****:****:****:7334` + +- **EMAIL**: Masks the local part of an email address + - Original: `john.doe@gmail.com` + - Masked: `*****@gmail.com` + +- **STRING**: Masks the entire string value + - Original: `JohnDoe` + - Masked: `*****` + diff --git a/en/docs/monitoring/api-analytics/moesif-analytics/moesif-event-reference.md b/en/docs/monitoring/api-analytics/moesif-analytics/moesif-event-reference.md new file mode 100644 index 0000000000..2ec5862f56 --- /dev/null +++ b/en/docs/monitoring/api-analytics/moesif-analytics/moesif-event-reference.md @@ -0,0 +1,126 @@ +# Analytics Event Reference + +WSO2 API Manager generates two types of analytics events that are sent to Moesif. This page lists the +parameters carried by each. + +!!! note + Captured request and response bodies are not published as custom event metadata. They are mapped + onto Moesif's native request and response body fields, so they appear on the request and response + themselves in the Moesif UI and can be searched and filtered like any other Moesif payload. The + accompanying `requestContentType` and `responseContentType` values remain available as event + metadata. + +## apim_event_response + +This event is triggered for each successful API invocation. Even when an API-associated backend returns an error response, it will be logged through this event (as the gateway successfully processed the request). + +**Event Parameters:** + +| **Parameter** | **Type** | **Description** | +|---------------|----------|-----------------| +| apiCreator | String | Username of the API creator | +| apiCreatorTenantDomain | String | Tenant domain under which the API was created | +| apiId | String (UUID) | Unique identifier of the API | +| apiMethod | String | HTTP method used in the request (e.g., GET, POST) | +| apiName | String | Name of the API being invoked | +| apiResourceTemplate | String | Resource path template for the API | +| apiType | String | Type of the API (e.g., HTTP, SOAP, GRAPHQL) | +| apiVersion | String | Version of the API | +| applicationId | String (UUID) | Unique identifier of the invoking application | +| applicationName | String | Name of the invoking application | +| applicationOwner | String | Owner of the invoking application | +| backendLatency | Number | Time taken by the backend service to respond (in ms) | +| correlationId | String (UUID) | Unique identifier used to trace the request across components | +| destination | String | Backend endpoint URL to which the request was sent | +| eventType | String | Type of event (e.g., request, response) | +| gatewayType | String | Type of API Gateway handling the request (e.g., SYNAPSE, CHOREO) | +| keyType | String | Key type used for invoking the API (e.g., SANDBOX, PRODUCTION) | +| userName | String | Authenticated username of the API invoker | +| proxyResponseCode | Number | HTTP response code returned by the gateway | +| regionId | String | Identifier of the gateway region where the API was invoked | +| requestMediationLatency | Number | Latency introduced by mediation at the request flow (in ms) | +| requestTimestamp | String (ISO 8601) | Timestamp when the API request was initiated | +| responseCacheHit | Boolean | Indicates whether the response was served from cache | +| responseLatency | Number | Total latency for the response (in ms) | +| responseMediationLatency | Number | Latency introduced by mediation at the response flow (in ms) | +| targetResponseCode | Number | HTTP response code received from the backend service | +| userAgent | String | User agent string of the client (e.g., Chrome) | +| userIp | String | IP address of the client invoking the API | +| commonName | String | Common name extracted from certificate (if applicable) | +| responseContentType | String | Content type of the API response | +| subType | String | Subtype of the API event (e.g., DEFAULT) | +| isEgress | Boolean | Indicates whether the event occurred on the egress path | +| apiContext | String | Context path of the API | +| responseSize | Number | Size of the API response payload (in bytes) | +| requestHeaders | Object | Map of request headers sent to the backend. Present only when `send_headers` is enabled | +| responseHeaders | Object | Map of response headers received from the backend. Present only when `send_headers` is enabled | +| requestContentType | String | Content type of the API request. Present only when a request body was captured | +| requestBody | String | Captured request body. Present only when `send_payloads` is enabled and the body was captured | +| responseBody | String | Captured response body. Present only when `send_payloads` is enabled and the body was captured | +| requestBodyTransferEncoding | String | Set to `base64` when the request body is a Base64-encoded binary payload. Omitted otherwise | +| responseBodyTransferEncoding | String | Set to `base64` when the response body is a Base64-encoded binary payload. Omitted otherwise | +| vendorName | String | Name of the AI vendor (e.g., OpenAI) | +| vendorVersion | String | Version of the AI vendor API | +| model | String | Model identifier used (e.g., gpt-3.5-turbo) | +| promptTokens | Number | Number of tokens used for the input prompt | +| completionTokens | Number | Number of tokens used for the AI's generated response | +| totalTokens | Number | Total tokens consumed in the request | +| hour | String | Hour of the request, for usage tracking or analytics | + +## apim_event_faulty + +This event is triggered for each **failed** or **throttled** API invocation. This includes requests that failed due to authentication errors, authorization failures, rate limiting, or backend connectivity issues. + +Faulty events carry the same header and body fields as `apim_event_response`, where those were +captured. Note that a request rejected before it reaches the backend (for example, an authentication failure or a +throttled request) has no captured request body, because the request body is captured +immediately before the backend call. In that case the response body, if captured, is the error +response generated by the gateway rather than a backend response. + +**Event Parameters:** + +| **Parameter** | **Type** | **Description** | +|---------------|----------|-----------------| +| apiCreator | String | The creator of the API | +| apiCreatorTenantDomain | String | The tenant domain of the API creator | +| apiId | String | Unique identifier of the API | +| apiMethod | String | The HTTP method used by the API (e.g., GET, POST) | +| apiName | String | The name of the API | +| apiResourceTemplate | String | The template of the API resource accessed | +| apiType | String | The type of the API (e.g., HTTP, REST) | +| apiVersion | String | The version of the API | +| applicationId | String | Unique identifier of the application that makes the API call | +| applicationName | String | Name of the application that makes the API call | +| applicationOwner | String | Owner of the application that makes the API call | +| backendLatency | Long | The time taken by the backend to process the request | +| correlationId | String | Unique identifier for tracking API calls | +| destination | String | The backend URL to which the API call was redirected | +| eventType | String | The type of event | +| gatewayType | String | The type of the API gateway | +| keyType | String | Indicates whether the API key used was for SANDBOX or PRODUCTION | +| platform | String | Operating system used to access the API | +| properties | Object | Properties of the event | +| apiContext | String | The context of the API call | +| userName | String | The username of the individual who made the API call | +| proxyResponseCode | Int | The HTTP response code returned by the API gateway | +| regionId | String | The region identifier for the API call | +| requestMediationLatency | Int | Time taken for request mediation | +| requestTimestamp | Long | Timestamp when the request was made | +| responseCacheHit | Bool | Indicates if the response was served from cache | +| responseLatency | Long | Total time taken to respond to the request | +| responseMediationLatency | Long | Time taken for response mediation | +| targetResponseCode | Int | The HTTP response code received from the backend target | +| userAgent | String | The user agent of the client making the API call | +| userIp | String | The IP address of the user making the API call | +| errorCode | Int | The error code generated in a fault | +| errorMessage | String | The error message associated with the fault | +| errorType | String | The type of error (e.g., THROTTLED, AUTH_FAILURE, BACKEND_ERROR) | +| responseContentType | String | Content type of the API response | +| requestHeaders | Object | Map of request headers sent to the backend. Present only when `send_headers` is enabled | +| responseHeaders | Object | Map of response headers received from the backend. Present only when `send_headers` is enabled | +| requestContentType | String | Content type of the API request. Present only when a request body was captured | +| requestBody | String | Captured request body. Present only when `send_payloads` is enabled and the body was captured. A request rejected before it reaches the backend has no captured request body | +| responseBody | String | Captured response body. Present only when `send_payloads` is enabled and the body was captured. For a request rejected at the gateway, this is the gateway's error response | +| requestBodyTransferEncoding | String | Set to `base64` when the request body is a Base64-encoded binary payload. Omitted otherwise | +| responseBodyTransferEncoding | String | Set to `base64` when the response body is a Base64-encoded binary payload. Omitted otherwise | + diff --git a/en/docs/monitoring/api-analytics/moesif-analytics/moesif-integration-guide.md b/en/docs/monitoring/api-analytics/moesif-analytics/moesif-integration-guide.md index 2a44238bc2..681da058ee 100644 --- a/en/docs/monitoring/api-analytics/moesif-analytics/moesif-integration-guide.md +++ b/en/docs/monitoring/api-analytics/moesif-analytics/moesif-integration-guide.md @@ -25,6 +25,18 @@ The Moesif integration captures both **successful** and **failed** API invocatio +### In This Section + +This page covers the base setup. The rest of the Moesif documentation is organized as follows: + +| **Page** | **Covers** | +|----------|------------| +| [Capturing Request and Response Data]({{base_path}}/monitoring/api-analytics/moesif-analytics/moesif-data-capture/) | Publishing HTTP headers and message bodies | +| [Sampling and Reliability]({{base_path}}/monitoring/api-analytics/moesif-analytics/moesif-sampling-and-reliability/) | Publishing only a share of invocations, and holding events for retry when Moesif is unreachable | +| [Privacy and Data Masking]({{base_path}}/monitoring/api-analytics/moesif-analytics/moesif-data-masking/) | Masking IP addresses, usernames and sensitive headers before they reach Moesif | +| [Analytics Event Reference]({{base_path}}/monitoring/api-analytics/moesif-analytics/moesif-event-reference/) | Every parameter carried by the response and faulty event types | +| [Analytics Dashboards]({{base_path}}/monitoring/api-analytics/moesif-analytics/moesif-analytics-dashboards/) | Reading the dashboards Moesif provides for WSO2 API Manager | + ## Step 1: Set Up Your Moesif Account ### 1.1 Create an Account and Log In @@ -34,7 +46,9 @@ The Moesif integration captures both **successful** and **failed** API invocatio 3. Follow the onboarding wizard to get the Moesif Key 4. Copy the **Moesif API Key** from the **API Keys** section (you will need this in Step 2) -> **Note:** For more detailed instructions and advanced configuration options, refer to the official [Moesif Documentation](https://www.moesif.com/docs). +!!! note + For more detailed instructions and advanced configuration options, refer to the official + [Moesif Documentation](https://www.moesif.com/docs). ## Step 2: Configure WSO2 API Manager @@ -70,13 +84,27 @@ Replace `YOUR_MOESIF_API_KEY_HERE` with the actual API key you copied from Step | type | Type of Analytics platform. Set this to `moesif` to publish to Moesif. | - | String | No | | moesifKey | Moesif API Key | - | String | No | | moesif_base_url | Base URL of Moesif API | https://api.moesif.net | String | Yes | -| send_headers | Whether to send request and response headers to Moesif. See [Capturing Request and Response Headers](#capturing-request-and-response-headers). | false | Boolean | Yes | -| send_payloads | Whether to send request and response bodies to Moesif. See [Capturing Request and Response Bodies](#capturing-request-and-response-bodies). | false | Boolean | Yes | +| send_headers | Whether to send request and response headers to Moesif. See [Capturing Request and Response Headers]({{base_path}}/monitoring/api-analytics/moesif-analytics/moesif-data-capture/#capturing-request-and-response-headers). | false | Boolean | Yes | +| send_payloads | Whether to send request and response bodies to Moesif. See [Capturing Request and Response Bodies]({{base_path}}/monitoring/api-analytics/moesif-analytics/moesif-data-capture/#capturing-request-and-response-bodies). | false | Boolean | Yes | | payload_size_limit | Maximum size, in bytes, of a single request or response body captured for analytics. Applies only when `send_payloads` is `true`. | 100000 | Integer | Yes | | capture_payloads_without_content_length | Whether to capture bodies that do not declare a `Content-Length` header, such as chunked responses. Applies only when `send_payloads` is `true`. | false | Boolean | Yes | - -All of these properties are node-level: they apply to every API deployed on the gateway, and -changing any of them requires a restart. There is no per-API or per-resource override. +| sampling_enabled | Whether to publish only a sampled share of API invocations. Sample rates come from Moesif, not from this file. See [Dynamic Sampling]({{base_path}}/monitoring/api-analytics/moesif-analytics/moesif-sampling-and-reliability/#dynamic-sampling). | false | Boolean | Yes | +| sampling_refresh_interval_ms | How often, in milliseconds, the sampling configuration is re-fetched from Moesif. Applies only when `sampling_enabled` is `true`. | 60000 | Integer | Yes | +| sampling_fallback_rate | Percentage of events to publish when no sampling configuration has been fetched from Moesif yet. Applies only when `sampling_enabled` is `true`. | 100 | Integer (0-100) | Yes | +| retry_buffer_enabled | Whether to hold analytics events in memory while Moesif is unreachable and publish them on recovery. Enabled unless you set this to `false`. See [Reliability: The Retry Queue]({{base_path}}/monitoring/api-analytics/moesif-analytics/moesif-sampling-and-reliability/#reliability-the-retry-queue). | true | Boolean | Yes | +| retry_buffer_size | Maximum number of events held for retry, per Moesif API key. | 10000 | Integer | Yes | +| retry_interval_seconds | How often, in seconds, Moesif is probed and queued events are drained. | 5 | Integer | Yes | +| retry_log_multiplier | Multiplier applied to `retry_interval_seconds` to decide how often the repeated "still unreachable" error is logged. | 10 | Integer | Yes | +| retry_drain_burst_size | Maximum number of catch-up batches sent in quick succession once Moesif is reachable again. | 5 | Integer | Yes | +| retry_drain_batch_delay_ms | Delay, in milliseconds, between those catch-up batches. | 100 | Integer | Yes | + +Only `enable`, `type` and `moesifKey` are needed to get started. Every other property is optional and is +explained on the page it belongs to, linked from the table above. + +All of these properties are node-level: they apply to every API deployed on the gateway, and changing any +of them requires a restart. There is no per-API or per-resource override. The one exception is the +**sampling rates** themselves, which are defined in Moesif rather than here: those take effect within +`sampling_refresh_interval_ms` and need no restart. ### 2.2 Restart WSO2 API Manager @@ -88,388 +116,6 @@ cd /bin ./api-manager.sh start ``` -## Capturing Request and Response Headers - -By default, request and response headers are **not** sent to Moesif. To include them, set -`send_headers` to `true`: - -```toml -[apim.analytics.properties] -send_headers = true -``` - -Headers are then published as the `requestHeaders` and `responseHeaders` fields of the analytics -event, and appear on the request and response in Moesif. - -### Headers That Are Never Published - -Regardless of your configuration, the following headers are excluded from the analytics event, in -both the request and the response direction. You do not need to configure anything to protect them: - -| **Header** | **Why it is excluded** | -|------------|------------------------| -| `Authorization` | Carries the bearer token or basic credentials used to invoke the API | -| `apikey` | Carries the API key used to invoke the API | -| `Cookie` | Carries client session state | -| `Set-Cookie` | Carries session state issued to the client | - -!!! note "This affects analytics only, not your API traffic" - These headers are dropped from the copy of the headers that is published to Moesif. The messages - themselves are not modified: the backend still receives all the request headers your client sent, and the client still receives all the response headers your backend - returned. - -Header names are matched **case-insensitively**, so `authorization`, `Authorization` and -`AUTHORIZATION` are all excluded. This also means headers sent by HTTP/2 clients, which lowercase all -header names, are captured and matched correctly. - -!!! warning "Custom authorization headers are not removed automatically" - If an API is configured to accept its credentials in a **custom** header name rather than the - default `Authorization` or `apikey`, that header is not recognised at this layer and is **not** - removed automatically. Mask it explicitly using - [`[apim.analytics.mask]`](#privacy-masking-sensitive-information). - -To hide the value of any other header without removing the header itself, see -[Privacy: Masking Sensitive Information](#privacy-masking-sensitive-information). - -## Capturing Request and Response Bodies - -WSO2 API Manager can also publish the request and response **bodies** to Moesif, where they appear on -the request and response of each event and can be searched, filtered and inspected alongside the -rest of your analytics data. - -This is an opt-in feature. It is disabled by default because it publishes the full content of your -API traffic to Moesif and requires the gateway to hold each message in memory. - -### Enabling Body Capture - -Add the following to your `deployment.toml` file and restart the server: - -```toml -[apim.analytics.properties] -send_payloads = true -payload_size_limit = 100000 -capture_payloads_without_content_length = false -``` - -Body capture also requires analytics itself to be enabled (`[apim.analytics] enable = true`). When analytics is disabled, no body is captured and no message is built, even if `send_payloads` is `true`. - -### Body Capture Configuration Reference - -| **Name** | **Description** | **Default Value** | **Possible Data Types** | **Optional** | -|----------|-----------------|-------------------|-------------------------|--------------| -| send_payloads | Enables request and response body capture. | false | Boolean | Yes | -| payload_size_limit | Maximum size, in bytes, of a single captured body. A body larger than this is dropped from analytics; it is not truncated. The limit is applied separately to both the request body and the response body. | 100000 | Integer | Yes | -| capture_payloads_without_content_length | Whether to capture a body that does not declare a `Content-Length` header, for example a chunked response. | false | Boolean | Yes | - -If `payload_size_limit` is set to a value that is not a positive integer, the default of `100000` is -used instead and a warning is logged once. - -### What Is Captured - -The body is captured according to its content type: - -| **Payload** | **How it is published** | -|-------------|-------------------------| -| JSON | Sent as-is and rendered in Moesif as a structured, searchable object | -| Plain text | Sent as-is | -| XML and SOAP | Serialized from the message body. | -| Binary | Base64-encoded, and flagged to Moesif with a transfer encoding of `base64` | - -The `Content-Type` of the captured body is published alongside it, so Moesif can label and parse the -body correctly even when `send_headers` is set to `false`. - -### What Is Not Captured - -A body is skipped in each of the following cases. In every one of them the full message is still -forwarded to the backend or the client; only the analytics copy is omitted. - -- **Requests with no body**, such as `GET` and `DELETE`. -- **Server-sent events** (`text/event-stream`), **multipart payloads** (`multipart/*`, including - file uploads), and **form submissions** (`application/x-www-form-urlencoded`). -- **Content types with no registered message builder.** The gateway consults the message builders - registered in `/repository/conf/axis2/axis2.xml` and skips any content type it does not - recognise, rather than risk corrupting a payload it cannot safely interpret. If you need a custom - content type captured, register a message builder for it in `axis2.xml`. -- **WebSocket APIs.** -- **Asynchronous and streaming APIs**, such as SSE and webhook APIs. -- **Bodies larger than `payload_size_limit`** - see [Size Limits](#size-limits). -- **Bodies with no `Content-Length` header** - see - [Payloads Without a Content-Length](#payloads-without-a-content-length). - -### Size Limits - -`payload_size_limit` is measured in **bytes**, and is applied separately to the request body and the -response body. - -A body that exceeds the limit is **dropped in its entirety, not truncated**. This is deliberate: -Moesif only ever receives a whole, valid body or no body at all, so a partial payload can never be -mistaken for the real one. - -Where the payload declares its size through a `Content-Length` header, the check is applied *before* -the message is read into memory, so an oversized body is never buffered and the message is passed -straight through. A payload whose size only becomes known once it has been read is dropped after the -fact, so it is still subject to the re-serialization behaviour described in -[Impact on Request Forwarding](#impact-on-request-forwarding). - -### Payloads Without a Content-Length - -A body sent with chunked transfer encoding does not declare a `Content-Length`, so its size cannot be -checked before it is read. By default, such bodies are skipped, which keeps the default configuration -memory-safe. - -Set `capture_payloads_without_content_length = true` to capture them anyway. - -!!! warning "Memory impact" - With this setting enabled, a chunked body is read into memory in full and only then discarded if - it turns out to exceed `payload_size_limit`. A large chunked payload under load can therefore - exhaust the gateway's heap. Enable it only if you need these bodies and have verified you have - the memory headroom for them. - -### Bodies Are Never Masked - -!!! warning "Captured bodies are published in full" - The masking options under `[apim.analytics.mask]` apply to identity fields and to named headers. - They do **not** apply to request or response bodies. When `send_payloads` is enabled, every - captured body is published to Moesif exactly as it appeared, including any personal data, - credentials, payment details or other sensitive content it contains. - - There is no field-level redaction and no per-API opt-out, the setting is on or off for the - entire gateway. Before enabling it in production, confirm that publishing the full content of - your API traffic to Moesif is compatible with your organisation's data protection policies. - -### Impact on Request Forwarding - -!!! warning "Bodies are re-serialized when capture is enabled" - Capturing a body requires the gateway to build the message, which means the message is - re-serialized when it is forwarded. The forwarded body remains semantically equivalent, but it - is **not guaranteed to be byte-identical** to what the client sent. Whitespace, attribute and - namespace ordering, JSON key formatting and chunking may all differ. - - As a result, a signature computed over the raw bytes of the body (such as a JWS, an - HMAC-signed request body or a WS-Security signature) may fail to verify at the backend while - `send_payloads` is enabled. If any of your APIs rely on body signatures, do not enable body - capture for that gateway. - -### Performance and Memory Considerations - -Enabling `send_payloads` is more costly than the rest of the analytics pipeline: - -- Each captured message is held in memory in full and re-serialized when forwarded, rather than - being streamed straight through. -- Event sizes grow with your payload sizes, increasing the volume published to Moesif. - -Keep `payload_size_limit` no larger than you actually need, leave -`capture_payloads_without_content_length` disabled unless required, and validate the configuration -under representative load before rolling it out to production. - -### Troubleshooting Body Capture - -When a body is missing from Moesif, the gateway records the reason at debug level. Enable debug -logging for the capture utility by adding the following to -`/repository/conf/log4j2.properties`: - -```properties -logger.analytics-payload.name = org.wso2.carbon.apimgt.gateway.handlers.analytics.AnalyticsPayloadUtil -logger.analytics-payload.level = DEBUG -``` - -Add `analytics-payload` to the comma-separated `loggers` list at the top of the same file, then -invoke the API again and check `/repository/logs/wso2carbon.log`. Each skipped or dropped -body is logged with the reason and the direction, for example: - -``` -Dropping response body from analytics: 250000 bytes exceeds payload_size_limit of 100000. Increase payload_size_limit to capture it. -``` - -## Privacy: Masking Sensitive Information - -> **Warning:** Ensure that you **do not** expose your Moesif API Key in public repositories or logs, as it can lead to unauthorized access to your analytics data. - -### Understanding Data Privacy Requirements - -WSO2 Analytics data may contain PII (Personally Identifiable Information) such as user IP addresses and usernames. Additionally, when `send_headers` is set to `true`, request and response headers may contain sensitive information. - -To comply with data privacy regulations (GDPR, CCPA, etc.) and protect user privacy, it is strongly recommended to mask or anonymize such sensitive information before sending it to Moesif. - -!!! warning "What masking covers" - Masking applies **only** to the identity fields listed below and to the request and response - headers you name explicitly. It does **not** apply to request or response bodies. If you have - enabled body capture with `send_payloads`, every captured body is published to Moesif in full, see [Bodies Are Never Masked](#bodies-are-never-masked). - -### Configuring Data Masking - -Add the following configuration to your `deployment.toml` file to enable data masking: - -```toml -[apim.analytics.mask] -"userIp" = "IPV4" -"userName" = "EMAIL" -"userId" = "EMAIL" -"userAgent" = "STRING" -"applicationOwner" = "EMAIL" -request_headers = ["X-Custom-Auth", "X-API-Key"] -response_headers = ["X-Account-Number"] -``` - -A masked header is published to Moesif with its value replaced by `*****`; the header name itself is -still visible. As with the headers excluded above, masking applies only to the published event, the -header reaches the backend or the client with its real value intact. Header names in -`request_headers` and `response_headers` are matched case-insensitively. - -!!! note - You do not need to list `Authorization`, `apikey`, `Cookie` or `Set-Cookie` here. Those headers - are removed from analytics events entirely, whether or not you configure masking, see - [Headers That Are Never Published](#headers-that-are-never-published). Use `request_headers` and - `response_headers` for headers specific to your deployment, such as a custom authorization - header name or a header carrying customer identifiers. - -### Masking Configuration Reference - -| **Name** | **Description** | **Accepted Values** | -|----------|-----------------|---------------------| -| userIp | Defines the format used to capture and store the user's IP address in analytics records | IPV4, IPV6 | -| userName | Specifies the format of the username field used for analytics or identification | EMAIL, STRING | -| userId | Identifies how the user ID is represented in analytics data | EMAIL, STRING | -| userAgent | Represents the type of the user agent string recorded from the client request | STRING | -| applicationOwner | Specifies the format of the application owner's identifier | EMAIL, STRING | -| response_headers | List of response headers to be masked for analytics or logging purposes | Header keys as strings | -| request_headers | List of request headers to be masked for analytics or logging purposes | Header keys as strings | - -### Masking Behavior Examples - -- **IPV4**: Masks the 3rd octet of an IPv4 address - - Original: `192.168.1.98` - - Masked: `192.168.***.98` - -- **IPV6**: Masks the 4th, 5th, 6th and 7th segments of an IPv6 address - - Original: `2001:0db8:85a3:0000:0000:8a2e:0370:7334` - - Masked: `2001:0db8:85a3:****:****:****:****:7334` - -- **EMAIL**: Masks the local part of an email address - - Original: `john.doe@gmail.com` - - Masked: `*****@gmail.com` - -- **STRING**: Masks the entire string value - - Original: `JohnDoe` - - Masked: `*****` - -## Analytics Event Types - -WSO2 API Manager generates two types of analytics events that are sent to Moesif: - -!!! note - Captured request and response bodies are not published as custom event metadata. They are mapped - onto Moesif's native request and response body fields, so they appear on the request and response - themselves in the Moesif UI and can be searched and filtered like any other Moesif payload. The - accompanying `requestContentType` and `responseContentType` values remain available as event - metadata. - -### apim_event_response - -This event is triggered for each successful API invocation. Even when an API-associated backend returns an error response, it will be logged through this event (as the gateway successfully processed the request). - -**Event Parameters:** - -| **Parameter** | **Type** | **Description** | -|---------------|----------|-----------------| -| apiCreator | String | Username of the API creator | -| apiCreatorTenantDomain | String | Tenant domain under which the API was created | -| apiId | String (UUID) | Unique identifier of the API | -| apiMethod | String | HTTP method used in the request (e.g., GET, POST) | -| apiName | String | Name of the API being invoked | -| apiResourceTemplate | String | Resource path template for the API | -| apiType | String | Type of the API (e.g., HTTP, SOAP, GRAPHQL) | -| apiVersion | String | Version of the API | -| applicationId | String (UUID) | Unique identifier of the invoking application | -| applicationName | String | Name of the invoking application | -| applicationOwner | String | Owner of the invoking application | -| backendLatency | Number | Time taken by the backend service to respond (in ms) | -| correlationId | String (UUID) | Unique identifier used to trace the request across components | -| destination | String | Backend endpoint URL to which the request was sent | -| eventType | String | Type of event (e.g., request, response) | -| gatewayType | String | Type of API Gateway handling the request (e.g., SYNAPSE, CHOREO) | -| keyType | String | Key type used for invoking the API (e.g., SANDBOX, PRODUCTION) | -| userName | String | Authenticated username of the API invoker | -| proxyResponseCode | Number | HTTP response code returned by the gateway | -| regionId | String | Identifier of the gateway region where the API was invoked | -| requestMediationLatency | Number | Latency introduced by mediation at the request flow (in ms) | -| requestTimestamp | String (ISO 8601) | Timestamp when the API request was initiated | -| responseCacheHit | Boolean | Indicates whether the response was served from cache | -| responseLatency | Number | Total latency for the response (in ms) | -| responseMediationLatency | Number | Latency introduced by mediation at the response flow (in ms) | -| targetResponseCode | Number | HTTP response code received from the backend service | -| userAgent | String | User agent string of the client (e.g., Chrome) | -| userIp | String | IP address of the client invoking the API | -| commonName | String | Common name extracted from certificate (if applicable) | -| responseContentType | String | Content type of the API response | -| subType | String | Subtype of the API event (e.g., DEFAULT) | -| isEgress | Boolean | Indicates whether the event occurred on the egress path | -| apiContext | String | Context path of the API | -| responseSize | Number | Size of the API response payload (in bytes) | -| requestHeaders | Object | Map of request headers sent to the backend. Present only when `send_headers` is enabled | -| responseHeaders | Object | Map of response headers received from the backend. Present only when `send_headers` is enabled | -| requestContentType | String | Content type of the API request. Present only when a request body was captured | -| requestBody | String | Captured request body. Present only when `send_payloads` is enabled and the body was captured | -| responseBody | String | Captured response body. Present only when `send_payloads` is enabled and the body was captured | -| requestBodyTransferEncoding | String | Set to `base64` when the request body is a Base64-encoded binary payload. Omitted otherwise | -| responseBodyTransferEncoding | String | Set to `base64` when the response body is a Base64-encoded binary payload. Omitted otherwise | -| vendorName | String | Name of the AI vendor (e.g., OpenAI) | -| vendorVersion | String | Version of the AI vendor API | -| model | String | Model identifier used (e.g., gpt-3.5-turbo) | -| promptTokens | Number | Number of tokens used for the input prompt | -| completionTokens | Number | Number of tokens used for the AI's generated response | -| totalTokens | Number | Total tokens consumed in the request | -| hour | String | Hour of the request, for usage tracking or analytics | - -### apim_event_faulty - -This event is triggered for each **failed** or **throttled** API invocation. This includes requests that failed due to authentication errors, authorization failures, rate limiting, or backend connectivity issues. - -Faulty events carry the same header and body fields as `apim_event_response`, where those were -captured. Note that a request rejected before it reaches the backend (for example, an authentication failure or a -throttled request) has no captured request body, because the request body is captured -immediately before the backend call. In that case the response body, if captured, is the error -response generated by the gateway rather than a backend response. - -**Event Parameters:** - -| **Parameter** | **Type** | **Description** | -|---------------|----------|-----------------| -| apiCreator | String | The creator of the API | -| apiCreatorTenantDomain | String | The tenant domain of the API creator | -| apiId | String | Unique identifier of the API | -| apiMethod | String | The HTTP method used by the API (e.g., GET, POST) | -| apiName | String | The name of the API | -| apiResourceTemplate | String | The template of the API resource accessed | -| apiType | String | The type of the API (e.g., HTTP, REST) | -| apiVersion | String | The version of the API | -| applicationId | String | Unique identifier of the application that makes the API call | -| applicationName | String | Name of the application that makes the API call | -| applicationOwner | String | Owner of the application that makes the API call | -| backendLatency | Long | The time taken by the backend to process the request | -| correlationId | String | Unique identifier for tracking API calls | -| destination | String | The backend URL to which the API call was redirected | -| eventType | String | The type of event | -| gatewayType | String | The type of the API gateway | -| keyType | String | Indicates whether the API key used was for SANDBOX or PRODUCTION | -| platform | String | Operating system used to access the API | -| properties | Object | Properties of the event | -| apiContext | String | The context of the API call | -| userName | String | The username of the individual who made the API call | -| proxyResponseCode | Int | The HTTP response code returned by the API gateway | -| regionId | String | The region identifier for the API call | -| requestMediationLatency | Int | Time taken for request mediation | -| requestTimestamp | Long | Timestamp when the request was made | -| responseCacheHit | Bool | Indicates if the response was served from cache | -| responseLatency | Long | Total time taken to respond to the request | -| responseMediationLatency | Long | Time taken for response mediation | -| targetResponseCode | Int | The HTTP response code received from the backend target | -| userAgent | String | The user agent of the client making the API call | -| userIp | String | The IP address of the user making the API call | -| errorCode | Int | The error code generated in a fault | -| errorMessage | String | The error message associated with the fault | -| errorType | String | The type of error (e.g., THROTTLED, AUTH_FAILURE, BACKEND_ERROR) | - ## Troubleshooting ### Analytics Data Not Appearing in Moesif @@ -482,21 +128,21 @@ If you don't see data in your Moesif dashboard after configuration: 4. **Test API Invocation**: Make a test API call and wait 2-3 minutes for data to appear in Moesif 5. **Network Connectivity**: Verify that your WSO2 APIM server can reach `https://api.moesif.net` -### Request or Response Bodies Not Appearing in Moesif +### Fewer Events in Moesif Than API Calls + +If Moesif shows fewer events than you invoked, check whether dynamic sampling is enabled: -If events reach Moesif but the bodies are missing: +1. **Check `sampling_enabled`**: if it is `true`, only a share of invocations is published by design +2. **Check the rates in Moesif**: the effective rate comes from your Moesif application configuration, not + from `deployment.toml`, so a low global, per-user or per-company rate reduces what you see +3. **Check the aggregate metrics rather than the event list**: published events carry a weight, so request + counts and metrics still reflect full traffic even though individual calls are missing, see + [Sampling Weights and Metric Accuracy]({{base_path}}/monitoring/api-analytics/moesif-analytics/moesif-sampling-and-reliability/#sampling-weights-and-metric-accuracy) -1. **Verify the Configuration**: Confirm that `send_payloads = true` is set under - `[apim.analytics.properties]` and that the server has been restarted since the change -2. **Check the Exclusions**: Confirm the payload is not one of the types that are never captured, - such as a multipart upload, a form submission or a server-sent event stream, see - [What Is Not Captured](#what-is-not-captured) -3. **Check the Size Limit**: A body larger than `payload_size_limit` is dropped rather than - truncated. Raise the limit if you need larger bodies captured -4. **Check for a Missing Content-Length**: A chunked payload is skipped unless - `capture_payloads_without_content_length` is enabled -5. **Enable Debug Logging**: The gateway logs the exact reason each body was skipped, see - [Troubleshooting Body Capture](#troubleshooting-body-capture) +If sampling is disabled and events are still missing, confirm that Moesif is reachable. While the retry +queue is enabled, which it is by default, events that cannot be published are queued rather than sent and +appear once Moesif recovers. If you have set `retry_buffer_enabled = false`, they are dropped instead. See +[Verifying and Troubleshooting the Retry Queue]({{base_path}}/monitoring/api-analytics/moesif-analytics/moesif-sampling-and-reliability/#verifying-and-troubleshooting-the-retry-queue). ### Common Configuration Errors @@ -513,9 +159,15 @@ Enabling analytics introduces minimal overhead: - **Resource Usage**: Minimal CPU and memory impact due to efficient event batching - **Network**: Events are batched and sent in the background to minimize network calls -These figures apply to the default configuration. Enabling `send_payloads` adds a measurably larger -overhead, because each captured message is held in memory and re-serialized, see -[Performance and Memory Considerations](#performance-and-memory-considerations). +These figures apply to the default configuration. Two options change the picture: + +- Enabling `send_payloads` adds a measurably larger overhead, because each captured message is held in + memory and re-serialized, see + [Performance and Memory Considerations]({{base_path}}/monitoring/api-analytics/moesif-analytics/moesif-data-capture/#performance-and-memory-considerations). +- The retry queue is enabled by default and holds events **in heap** while Moesif is unreachable, up to + `retry_buffer_size` events per Moesif API key. Size that against your event size, especially with body + capture enabled, see + [Memory and Sizing Considerations]({{base_path}}/monitoring/api-analytics/moesif-analytics/moesif-sampling-and-reliability/#memory-and-sizing-considerations). ## Additional Resources diff --git a/en/docs/monitoring/api-analytics/moesif-analytics/moesif-sampling-and-reliability.md b/en/docs/monitoring/api-analytics/moesif-analytics/moesif-sampling-and-reliability.md new file mode 100644 index 0000000000..a068cfd244 --- /dev/null +++ b/en/docs/monitoring/api-analytics/moesif-analytics/moesif-sampling-and-reliability.md @@ -0,0 +1,260 @@ +# Sampling and Reliability + +Two independent features that control **how much** analytics data reaches Moesif and **what happens when Moesif +cannot be reached**: + +- **Dynamic sampling** publishes only a percentage of your API invocations, so you can keep analytics on + across high-traffic APIs without publishing every single call. It is disabled by default. +- **The retry queue** holds events in memory while Moesif is unreachable and sends them once it recovers, + instead of discarding them. It is **enabled by default**. + +!!! note "Analytics must be enabled first" + Both features require analytics itself to be enabled and pointed at Moesif. See + [Moesif Analytics Integration]({{base_path}}/monitoring/api-analytics/moesif-analytics/moesif-integration-guide/) + for the base configuration. + +## Dynamic Sampling + +With dynamic sampling enabled, the gateway publishes only a share of the analytics events it builds. Each +published event carries a **weight**, so Moesif can extrapolate counts and metrics back to your full +traffic volume rather than under-reporting it. + +!!! note "Sample rates are defined in Moesif, not in deployment.toml" + WSO2 API Manager does not decide the sample rate. It **opts in** to sampling and supplies a fallback + rate; the rates themselves come from your Moesif application configuration, where you define a global + sample rate and, optionally, per-user and per-company rates. Changing a rate in Moesif takes effect on + the gateway within `sampling_refresh_interval_ms` and does **not** require a restart. + +### Enabling Dynamic Sampling + +Add the following to your `deployment.toml` file and restart the server: + +```toml +[apim.analytics.properties] +sampling_enabled = true +sampling_refresh_interval_ms = 60000 +sampling_fallback_rate = 100 +``` + +### Sampling Configuration Reference + +| **Name** | **Description** | **Default Value** | **Possible Data Types** | **Optional** | +|----------|-----------------|-------------------|-------------------------|--------------| +| sampling_enabled | Enables dynamic sampling. Any value other than `true` leaves sampling disabled, and every event is published. | false | Boolean | Yes | +| sampling_refresh_interval_ms | How often, in milliseconds, the gateway re-fetches the sampling configuration from Moesif. Must be a positive integer. | 60000 | Integer | Yes | +| sampling_fallback_rate | Percentage of events to publish when no sampling configuration has been fetched from Moesif yet, or when the fetched configuration carries no rate. `100` publishes everything. | 100 | Integer (0-100) | Yes | + +If a numeric property is set to a value that cannot be parsed as a number, the default is used instead and +a warning is logged. Note that `sampling_refresh_interval_ms` is **not** range-checked: a value of `0` or +below is rejected by the scheduler and prevents the publisher from starting, so always set a positive +value. + +### How the Sample Rate Is Chosen + +For each event, the gateway applies the **most specific** rate that matches it: + +1. The **per-user** rate, if the event's user has one defined in Moesif. +2. Otherwise the **per-company** rate, if the event's company has one. +3. Otherwise the **global** sample rate from your Moesif application configuration. +4. Otherwise `sampling_fallback_rate`. + +The resulting rate is clamped to the range 0-100 and applied as follows: + +| **Rate** | **Behaviour** | +|----------|---------------| +| 100 or above | Every event is published | +| 1 to 99 | Each event is published with that probability, decided independently per event | +| 0 or below | No event is published | + +The decision is made per event, not per batch. When every event in a batch is sampled out, the gateway +publishes nothing for that batch and makes no event-publish call to Moesif. The periodic configuration +refresh described below still runs on its own schedule. + +### Sampling Weights and Metric Accuracy + +Each published event is stamped with a weight of approximately `100 / rate`. At a 10% sample rate, for +example, each published event carries a weight of 10, and Moesif multiplies it out so that request counts +and aggregate metrics still reflect your full traffic. When sampling is disabled, every event carries a +weight of 1. + +This means sampled metrics remain broadly accurate while individual invocations do not. Do not use a +sampled deployment to look for one specific API call, because that call has most likely not been published. + +### When the Sampling Configuration Cannot Be Fetched + +When no sampling configuration is available, `sampling_fallback_rate` decides what happens rather than the +event being dropped silently: + +- Before the first successful fetch, `sampling_fallback_rate` applies. At its default of `100`, everything + is published until Moesif's configuration arrives. Note that this rate is applied like any other, so + setting it to `0` **discards** every event until the configuration has been fetched. +- If a refresh fails or returns a non-successful status, the gateway logs a warning and **keeps the + previously fetched configuration** rather than falling back. +- Sampling begins only once the publisher has started tracking your Moesif key, which it does as it + publishes its first events after startup. Events handled before that point are published. + +### Dynamic Sampling and Body Capture + +Sampling is applied to the event **after** it has been built, which has two consequences when +[body capture]({{base_path}}/monitoring/api-analytics/moesif-analytics/moesif-data-capture/) is also +enabled: + +- An event that is sampled out is discarded together with its captured request and response bodies. +- An event that is sampled in keeps its bodies in full, and additionally carries the sampling weight. + +Sampling therefore reduces how much payload data reaches Moesif, but it does **not** reduce the gateway's +memory cost of capturing that data in the first place. Every message is still read and held in memory +before the sampling decision is made. + +!!! warning "Dynamic sampling applies to the standard gateway configuration only" + Sampling is wired into the direct-key publishing path, which is the one you get when you set + `type = "moesif"` and a `moesifKey` in `deployment.toml`. Deployments that resolve Moesif keys + per organization through the Moesif microservice path do not apply sampling. The retry queue described + below applies to both. + +### Verifying Dynamic Sampling + +On startup, confirm the following line in `/repository/logs/wso2carbon.log`: + +``` +Moesif dynamic sampling enabled (refresh=60000ms, fallbackRate=100) +``` + +If a configuration refresh fails, you will see the fetched status and a note that the previous +configuration is being retained: + +``` +Moesif app config fetch returned status 401 - keeping previous config +``` + +## Reliability: The Retry Queue + +When Moesif cannot be reached, the gateway holds the affected analytics events in memory and publishes them +once Moesif recovers, rather than discarding them. + +!!! info "This replaces the previous retry behaviour and is enabled by default" + Earlier releases retried a failed publish up to three times, ten seconds apart, on the publishing + thread, and then dropped the events. That has been replaced by the asynchronous retry queue described + here, which is **on by default**. No configuration change is needed to get it. To restore the previous + drop-on-failure behaviour, set `retry_buffer_enabled = false`. + +### Retry Queue Configuration Reference + +| **Name** | **Description** | **Default Value** | **Possible Data Types** | **Optional** | +|----------|-----------------|-------------------|-------------------------|--------------| +| retry_buffer_enabled | Enables the retry queue. Any value other than `true` disables it, and events are dropped when a publish fails. | true | Boolean | Yes | +| retry_buffer_size | Maximum number of **events** held for retry, counted **per Moesif API key**. | 10000 | Integer | Yes | +| retry_interval_seconds | How often, in seconds, the gateway probes Moesif and drains queued events. Must be a positive integer. | 5 | Integer | Yes | +| retry_log_multiplier | Controls how often the repeated "still unreachable" error is logged. The interval is `retry_interval_seconds` x `retry_log_multiplier`, so 50 seconds by default. | 10 | Integer | Yes | +| retry_drain_burst_size | Maximum number of catch-up batches sent in quick succession once Moesif becomes reachable again. | 5 | Integer | Yes | +| retry_drain_batch_delay_ms | Delay, in milliseconds, between those catch-up batches. | 100 | Integer | Yes | + +As with the sampling properties, a value that cannot be parsed as a number falls back to the default with a +warning. None of these properties is range-checked, so set each within its supported range: + +| **Name** | **Supported values** | **If set outside that range** | +|----------|----------------------|-------------------------------| +| retry_buffer_size | Positive integer | `0` or below means no batch ever fits, so every event is dropped instead of queued | +| retry_interval_seconds | Positive integer | `0` or below is rejected by the scheduler and prevents the publisher from starting | +| retry_log_multiplier | Positive integer | `0` or below removes the throttle, so every failed probe is logged | +| retry_drain_burst_size | Positive integer | `0` or below disables the catch-up burst; the queue still drains at the normal probe cadence | +| retry_drain_batch_delay_ms | `0` or a positive integer | A negative value is treated as `0`, so catch-up batches are sent with no delay between them | + +### How the Retry Queue Works + +The queue behaves like a circuit breaker, one per Moesif API key: + +1. **Open.** A retryable publish failure marks that key's queue unhealthy and stores the batch. +2. **Stash.** While unhealthy, subsequent batches are queued directly, with no HTTP call attempted. This + avoids hammering an endpoint that is known to be down. +3. **Probe.** Every `retry_interval_seconds`, the gateway attempts to send the oldest queued batch. Only + one probe is ever in flight. A failed probe puts the batch back at the front of the queue. +4. **Drain.** When a probe succeeds, the queue is marked healthy and catches up by sending up to + `retry_drain_burst_size` batches spaced `retry_drain_batch_delay_ms` apart, then returns to the normal + cadence until it is empty. + +All retry queues share a single two-thread scheduler, so the number of Moesif keys in play does not change +the thread count. + +### Which Failures Are Retried + +| **Outcome** | **Handling** | +|-------------|--------------| +| 5xx server errors | Retried | +| 408 Request Timeout, 429 Too Many Requests | Retried | +| Connection, timeout and other transport failures | Retried | +| Any other 4xx, such as 401 Unauthorized or 400 Bad Request | **Not retried.** Logged as an error and dropped | + +A 4xx other than 408 or 429 indicates a problem the gateway cannot resolve by waiting, most commonly an +invalid `moesifKey`, so those events are discarded rather than queued indefinitely. If events are +disappearing with an authorization error, correct the key rather than tuning the queue. + +### When the Queue Is Full + +`retry_buffer_size` is a cap on events, applied per Moesif API key. When a new batch does not fit, the +**oldest** queued batches are evicted until it does, so the queue always favours recent data. A single +batch larger than `retry_buffer_size` cannot be stored at all and is dropped whole, with a warning: + +``` +Cannot queue 20000 analytics events at once (max 10000 per Moesif key); these events are dropped +``` + +Evicted events are counted and reported in the periodic error log, so a sustained outage tells you how much +data was lost. + +### Memory and Sizing Considerations + +The queue is held **in heap**, so `retry_buffer_size` is effectively a memory budget. Size it against your +event size rather than picking a large number: with body capture enabled, each event can approach +`payload_size_limit` in size, so 10,000 queued events is a far larger footprint than it is with the default +configuration. Remember the cap is per Moesif API key, so a deployment using several keys can hold a +multiple of it. + +!!! warning "Queued events are not persisted" + The queue exists only in memory. Events still waiting for retry when the server stops are lost, and a + restart during a Moesif outage discards everything queued up to that point. The queue protects against + a Moesif or network outage, not against a gateway restart. + +### Disabling the Retry Queue + +To go back to dropping events when a publish fails, add the following and restart the server: + +```toml +[apim.analytics.properties] +retry_buffer_enabled = false +``` + +This is worth doing if you would rather lose analytics data than spend heap on it, for example on a +memory-constrained gateway. + +### Verifying and Troubleshooting the Retry Queue + +On startup, confirm the queue is active: + +``` +Moesif retry queue enabled (capacity=10000 events per Moesif key, check interval=5s, ...) +``` + +During an outage, the gateway logs the first failure, then throttles the repeat to once every +`retry_interval_seconds` x `retry_log_multiplier`. The Moesif key is masked to its last four characters: + +``` +Cannot reach Moesif (key ...abcd). Queueing analytics events for retry (50/10000 events queued). +Moesif (key ...abcd) still unreachable after 50000 ms. Queued: 500/10000 events, dropped 0 events so far, retry attempts: 10 +``` + +Recovery is logged with how long the outage lasted and how much is being flushed: + +``` +Moesif (key ...abcd) is reachable again after 65000 ms and 13 attempts. Sending 650 queued analytics events. +``` + +Some further pointers: + +- **Events queued but never sent.** Check outbound connectivity to `moesif_base_url` from the gateway. The + queue keeps probing indefinitely, so a queue that only grows means Moesif is still unreachable. +- **Events dropped with a status code.** A `Moesif rejected ... these events will not be retried` message + means a non-retryable response. Verify `moesifKey`. +- **Analytics threads failing.** Uncaught errors in the analytics publisher threads are now logged as + `Uncaught error in analytics publisher thread `. These indicate a publisher-side problem worth + reporting, not a configuration issue. diff --git a/en/docs/reference/config-catalog.md b/en/docs/reference/config-catalog.md index 8617ba7346..a4fa3a979b 100644 --- a/en/docs/reference/config-catalog.md +++ b/en/docs/reference/config-catalog.md @@ -2448,7 +2448,16 @@ send_headers = false build_response_message = false send_payloads = false payload_size_limit = 100000 -capture_payloads_without_content_length = false +capture_payloads_without_content_length = false +sampling_enabled = false +sampling_refresh_interval_ms = 60000 +sampling_fallback_rate = 100 +retry_buffer_enabled = true +retry_buffer_size = 10000 +retry_interval_seconds = 5 +retry_log_multiplier = 10 +retry_drain_burst_size = 5 +retry_drain_batch_delay_ms = 100
@@ -2653,6 +2662,195 @@ capture_payloads_without_content_length = false

If TRUE, capture bodies that do not declare a Content-Length header, such as chunked payloads. Such a body is read into memory in full before its size is checked. Applies only when send_payloads is TRUE.

+
+
+ sampling_enabled +
+
+
+

+ boolean + +

+
+ Default: false +
+
+ Possible Values: TRUE | FALSE +
+
+
+

If TRUE, publish only a sampled share of API invocations to Moesif. Sample rates are defined in the Moesif application configuration, not here. Applies only to the direct-key path, that is when type is 'moesif' and moesifKey is set; deployments that resolve Moesif keys through the Moesif microservice path do not apply sampling.

+
+
+
+
+ sampling_refresh_interval_ms +
+
+
+

+ integer + +

+
+ Default: 60000 +
+
+ Possible Values: Positive integer +
+
+
+

How often, in milliseconds, the sampling configuration is re-fetched from Moesif. Must be a positive integer. Applies only when sampling_enabled is TRUE.

+
+
+
+
+ sampling_fallback_rate +
+
+
+

+ integer + +

+
+ Default: 100 +
+
+ Possible Values: 0 - 100 +
+
+
+

Percentage of events to publish when no sampling configuration has been fetched from Moesif yet, or when the fetched configuration carries no rate. Applies only when sampling_enabled is TRUE.

+
+
+
+
+ retry_buffer_enabled +
+
+
+

+ boolean + +

+
+ Default: true +
+
+ Possible Values: TRUE | FALSE +
+
+
+

If TRUE, hold analytics events in memory while Moesif is unreachable and publish them on recovery. Enabled by default; set to FALSE to drop events when a publish fails.

+
+
+
+
+ retry_buffer_size +
+
+
+

+ integer + +

+
+ Default: 10000 +
+
+ Possible Values: Positive integer +
+
+
+

Maximum number of events held for retry, counted per Moesif API key. When the limit is reached the oldest queued batches are evicted whole until the new batch fits, and a single batch larger than this limit is dropped rather than queued.

+
+
+
+
+ retry_interval_seconds +
+
+
+

+ integer + +

+
+ Default: 5 +
+
+ Possible Values: Positive integer +
+
+
+

How often, in seconds, Moesif is probed and queued events are drained. Must be a positive integer.

+
+
+
+
+ retry_log_multiplier +
+
+
+

+ integer + +

+
+ Default: 10 +
+
+ Possible Values: Positive integer +
+
+
+

Multiplier applied to retry_interval_seconds to decide how often the repeated 'still unreachable' error is logged.

+
+
+
+
+ retry_drain_burst_size +
+
+
+

+ integer + +

+
+ Default: 5 +
+
+ Possible Values: Positive integer +
+
+
+

Maximum number of catch-up batches sent in quick succession once Moesif becomes reachable again.

+
+
+
+
+ retry_drain_batch_delay_ms +
+
+
+

+ integer + +

+
+ Default: 100 +
+
+ Possible Values: 0 or a positive integer +
+
+
+

Delay, in milliseconds, between catch-up batches during a burst drain.

+
+
diff --git a/en/mkdocs.yml b/en/mkdocs.yml index 9472128436..a949cd4940 100644 --- a/en/mkdocs.yml +++ b/en/mkdocs.yml @@ -569,6 +569,10 @@ nav: - Overview: monitoring/api-analytics/analytics-overview.md - Moesif Analytics: - Integration Guide: monitoring/api-analytics/moesif-analytics/moesif-integration-guide.md + - Capturing Request and Response Data: monitoring/api-analytics/moesif-analytics/moesif-data-capture.md + - Sampling and Reliability: monitoring/api-analytics/moesif-analytics/moesif-sampling-and-reliability.md + - Privacy and Data Masking: monitoring/api-analytics/moesif-analytics/moesif-data-masking.md + - Analytics Event Reference: monitoring/api-analytics/moesif-analytics/moesif-event-reference.md - Analytics Dashboards: monitoring/api-analytics/moesif-analytics/moesif-analytics-dashboards.md - Other Analytics Solutions: - ELK Based Analytics Installation Guide: monitoring/api-analytics/on-prem/elk-installation-guide.md diff --git a/en/tools/config-catalog-generator/data/apim.analytics.toml b/en/tools/config-catalog-generator/data/apim.analytics.toml index 05f48c3320..9aca129674 100644 --- a/en/tools/config-catalog-generator/data/apim.analytics.toml +++ b/en/tools/config-catalog-generator/data/apim.analytics.toml @@ -9,4 +9,13 @@ send_headers = false build_response_message = false send_payloads = false payload_size_limit = 100000 -capture_payloads_without_content_length = false \ No newline at end of file +capture_payloads_without_content_length = false +sampling_enabled = false +sampling_refresh_interval_ms = 60000 +sampling_fallback_rate = 100 +retry_buffer_enabled = true +retry_buffer_size = 10000 +retry_interval_seconds = 5 +retry_log_multiplier = 10 +retry_drain_burst_size = 5 +retry_drain_batch_delay_ms = 100 \ No newline at end of file diff --git a/en/tools/config-catalog-generator/data/configs.json b/en/tools/config-catalog-generator/data/configs.json index ef6d9e50ff..9c6c6cfb62 100755 --- a/en/tools/config-catalog-generator/data/configs.json +++ b/en/tools/config-catalog-generator/data/configs.json @@ -977,6 +977,78 @@ "default": false, "possible": "TRUE | FALSE", "description": "If TRUE, capture bodies that do not declare a Content-Length header, such as chunked payloads. Such a body is read into memory in full before its size is checked. Applies only when send_payloads is TRUE." + }, + { + "name": "sampling_enabled", + "type": "boolean", + "required": false, + "default": false, + "possible": "TRUE | FALSE", + "description": "If TRUE, publish only a sampled share of API invocations to Moesif. Sample rates are defined in the Moesif application configuration, not here. Applies only to the direct-key path, that is when type is 'moesif' and moesifKey is set; deployments that resolve Moesif keys through the Moesif microservice path do not apply sampling." + }, + { + "name": "sampling_refresh_interval_ms", + "type": "integer", + "required": false, + "default": 60000, + "possible": "Positive integer", + "description": "How often, in milliseconds, the sampling configuration is re-fetched from Moesif. Must be a positive integer. Applies only when sampling_enabled is TRUE." + }, + { + "name": "sampling_fallback_rate", + "type": "integer", + "required": false, + "default": 100, + "possible": "0 - 100", + "description": "Percentage of events to publish when no sampling configuration has been fetched from Moesif yet, or when the fetched configuration carries no rate. Applies only when sampling_enabled is TRUE." + }, + { + "name": "retry_buffer_enabled", + "type": "boolean", + "required": false, + "default": true, + "possible": "TRUE | FALSE", + "description": "If TRUE, hold analytics events in memory while Moesif is unreachable and publish them on recovery. Enabled by default; set to FALSE to drop events when a publish fails." + }, + { + "name": "retry_buffer_size", + "type": "integer", + "required": false, + "default": 10000, + "possible": "Positive integer", + "description": "Maximum number of events held for retry, counted per Moesif API key. When the limit is reached the oldest queued batches are evicted whole until the new batch fits, and a single batch larger than this limit is dropped rather than queued." + }, + { + "name": "retry_interval_seconds", + "type": "integer", + "required": false, + "default": 5, + "possible": "Positive integer", + "description": "How often, in seconds, Moesif is probed and queued events are drained. Must be a positive integer." + }, + { + "name": "retry_log_multiplier", + "type": "integer", + "required": false, + "default": 10, + "possible": "Positive integer", + "description": "Multiplier applied to retry_interval_seconds to decide how often the repeated 'still unreachable' error is logged." + }, + { + "name": "retry_drain_burst_size", + "type": "integer", + "required": false, + "default": 5, + "possible": "Positive integer", + "description": "Maximum number of catch-up batches sent in quick succession once Moesif becomes reachable again." + }, + { + "name": "retry_drain_batch_delay_ms", + "type": "integer", + "required": false, + "default": 100, + "possible": "0 or a positive integer", + "description": "Delay, in milliseconds, between catch-up batches during a burst drain." } ] }