Update data prepper parameters and diagram in the intro

.github/vale/styles/Vocab/OpenSearch/Products/accept.txt

@@ -27,6 +27,7 @@ Apache Hive
 Apache Kafka
 Apache Spark
 Application Default Credentials
+Azure
 arXiv
 Aryn
 Auditbeat
@@ -69,6 +70,7 @@ Dockerfile
 DoS
 Ehcache
 Elasticsearch
+Entra
 Eszett
 Faiss
 Filebeat

.github/vale/styles/Vocab/OpenSearch/Words/accept.txt

@@ -249,6 +249,7 @@ tera
 [Uu]ncomment
 [Uu]ndeploy
 [Uu]ndock(ed|ing)?
+[Uu]nencrypted
 [Uu]nescaped
 [Uu]ngroup(s|ed|ing)?
 [Uu]nhid(e|ing)

_data-prepper/common-use-cases/trace-analytics.md

@@ -25,9 +25,9 @@ To monitor trace analytics, you need to set up the following components in your
 
 ## Trace analytics pipeline
 
-To monitor trace analytics in Data Prepper, we provide three pipelines: `entry-pipeline`, `raw-trace-pipeline`, and `service-map-pipeline`. The following image provides an overview of how the pipelines work together to monitor trace analytics. 
+A trace analytics configuration consists of three pipelines: `entry-pipeline`, which receives trace data, and `raw-trace-pipeline` and `service-map-pipeline`, which process it. The following image shows how the pipelines work together.
 
-![Trace analytics pipeline overview]({{site.url}}{{site.baseurl}}/images/data-prepper/trace-analytics/trace-analytics-pipeline.jpg)
+![Trace analytics pipeline overview]({{site.url}}{{site.baseurl}}/images/data-prepper/trace-analytics/trace-analytics-pipeline.png)
 
 
 ### OpenTelemetry trace source

_data-prepper/pipelines/configuration/buffers/bounded-blocking.md

@@ -12,8 +12,8 @@ The `bounded_blocking` buffer is the default buffer and is memory based. The fol
 
 | Option | Required | Type | Description |
 | --- | --- | --- | --- |
-| buffer_size | No | Integer | The maximum number of records the buffer accepts. Default value is `12800`. |
-| batch_size | No | Integer | The maximum number of records the buffer drains after each read. Default value is `200`. |
+| `buffer_size` | No | Integer | The maximum number of records the buffer accepts. Default value is `12800`. |
+| `batch_size` | No | Integer | The maximum number of records the buffer drains after each read. Default value is `200`. |
 
 <!--- ## Configuration
 

_data-prepper/pipelines/configuration/processors/add-entries.md

@@ -25,6 +25,11 @@ You can configure the `add_entries` processor with the following options.
 | `add_when` | No | A [conditional expression]({{site.url}}{{site.baseurl}}/data-prepper/pipelines/expression-syntax/), such as `/some-key == "test"'`, that will be evaluated to determine whether the processor will be run on the event. |
 | `overwrite_if_key_exists` | No | When set to `true`, the existing value is overwritten if `key` already exists in the event. The default value is `false`. |
 | `append_if_key_exists` | No | When set to `true`, the existing value will be appended if a `key` already exists in the event. An array will be created if the existing value is not an array. Default is `false`. |
+| `iterate_on` | No | The key of a list of objects in the event. When specified, the processor adds the entry to each object in the list. Cannot be used with `metadata_key`. |
+| `add_to_element_when` | No | A [conditional expression]({{site.url}}{{site.baseurl}}/data-prepper/pipelines/expression-syntax/) that determines whether the entry is added to the objects in the `iterate_on` list. By default, the expression is evaluated once against the root event, and the entry is added to all objects in the list or to none of them. To evaluate the expression against each object in the list, set `evaluate_when_on_element` to `true`. Applies only when `iterate_on` is specified. |
+| `evaluate_when_on_element` | No | When set to `true`, the `add_to_element_when` expression is evaluated against each object in the `iterate_on` list, and the entry is added only to the objects for which the expression evaluates to `true`. Applies only when both `iterate_on` and `add_to_element_when` are specified. Default is `false`. |
+| `disable_root_keys` | No | When set to `false`, `value_expression` and `format` are resolved against the root event so that you can use root-level fields as values in each object in the `iterate_on` list. When set to `true`, they are resolved against each object in the list. Applies only when `iterate_on` is specified. Default is `true`. |
+| `flatten_key` | No | When set to `true`, `key` is treated as a literal key name in each object in the `iterate_on` list, so a key such as `a/b` creates a field named `a/b`. When set to `false`, `key` is treated as a JSON pointer, so `a/b` creates a nested `b` field inside an `a` object. Applies only when `iterate_on` is specified. Default is `true`. |
 
 
 ## Usage
@@ -223,6 +228,40 @@ The processed event will contain the following data:
 {"message": ["hello", "world"]}
 ```
 
+### Example: Add entries to objects in a list
+
+The following example shows you how to configure the processor to add entries to each object in the `items` list. The first entry is added only to objects in which `type` is `a`, and the second entry copies the root-level `env` field into every object:
+
+```yaml
+...
+  processor:
+    - add_entries:
+        entries:
+          - key: "tagged"
+            value: true
+            iterate_on: "items"
+            add_to_element_when: '/type == "a"'
+            evaluate_when_on_element: true
+          - key: "env"
+            value_expression: "/env"
+            iterate_on: "items"
+            disable_root_keys: false
+...
+```
+{% include copy.html %}
+
+When the input event contains the following data:
+
+```json
+{"env": "prod", "items": [{"type": "a"}, {"type": "b"}]}
+```
+
+The processed event will contain the following data:
+
+```json
+{"env": "prod", "items": [{"type": "a", "tagged": true, "env": "prod"}, {"type": "b", "env": "prod"}]}
+```
+
 ## Example
 
 The following pipeline performs these actions:

_data-prepper/pipelines/configuration/processors/aws-lambda.md

@@ -33,13 +33,17 @@ Field                | Type    | Required | Description
 `client.max_concurrency` | Integer | Optional | The maximum number of concurrent threads on the client. Default is `200`.
 `client.base_delay`  | Duration | Optional | The base delay for the exponential backoff. Default is `100ms`.
 `client.max_backoff` | Duration | Optional | The maximum backoff time for the exponential backoff. Default is `20s`.
-`client.retryable_status_codes` | List | Optional | A list of HTTP status codes that trigger a retry when returned by a Lambda invocation, for example `[500, 502, 503, 504]`. Default is an empty list (standard AWS SDK retry conditions only). Use for idempotent functions for which retrying transient server-side errors is safe.
+`client.retryable_status_codes` | List | Optional | A list of HTTP status codes that trigger a retry when returned by a Lambda invocation, for example `[500, 502, 503, 504]`. Default is an empty list, in which case Data Prepper retries only throttling errors, Lambda service errors, and network or connection errors. Use for idempotent functions for which retrying transient server-side errors is safe.
 `batch`              | Object  | Optional | The batch settings for the Lambda invocations. Contains `key_name` (default: `"events"`) and `threshold` object with `event_count` (default: `100`), `maximum_size` (default: `"5mb"`), and `event_collect_timeout` (default: `10s`). See [Batch processing](#batch-processing) for details.                            
 `lambda_when`        | String  | Optional | A conditional expression that determines when to invoke the Lambda processor.     
 `response_codec`     | Object  | Optional |  A codec configuration for parsing Lambda responses. Default is `json`.
 `tags_on_failure`    | List    | Optional |  A list of tags to add to events when the Lambda function fails or encounters an exception.
 `response_events_match` | Boolean | Optional | Specifies how Data Prepper interprets and processes Lambda function responses. Default is `false`.
 `response_mode`      | String  | Optional | The response handling mode, either `replace` or `merge`. Default is `replace`.
+`invocation_type`    | String  | Optional | The Lambda invocation type. Valid values are `request-response`, `event`, and `streaming-response`. Default is `request-response`. See [Usage](#usage).
+`streaming`          | Object  | Optional | The response streaming configuration. Response streaming is used only when `invocation_type` is `streaming-response` and `streaming.enabled` is `true`. When `streaming` is specified, `batch.threshold.event_count` must be `1`. See [Response streaming](#response-streaming).
+`streaming.enabled`  | Boolean | Optional | Whether to invoke the Lambda function using response streaming. Default is `false`.
+`streaming.response_handling` | String | Optional | Specifies how the streamed response is converted into events. The only valid value is `reconstruct-document`, which merges all parsed chunks of the streamed response into the original event. Default is `reconstruct-document`.
 `keys`               | List    | Optional | Keys to send to the Lambda function.
 `cache`              | Object  | Optional | The cache configuration. Only valid when `response_mode` is `merge` and `keys` are specified.
 `cache.ttl`          | Long    | Optional | The cache time-to-live.
@@ -50,7 +54,7 @@ Field                | Type    | Required | Description
 The following is an example configuration:
 
 ```yaml
-processors:
+processor:
   - aws_lambda:
       function_name: my-lambda-function
       response_events_match: false
@@ -101,6 +105,32 @@ The processor supports the following invocation types:
 
 - `request-response`: The processor waits for Lambda function completion before proceeding.
 - `event`: The function is triggered asynchronously without waiting for a response.
+- `streaming-response`: The processor invokes the function using Lambda response streaming and waits for the complete streamed response. Requires `streaming.enabled` to be `true`. See [Response streaming](#response-streaming).
+
+The `cache` setting is valid only when `invocation_type` is `request-response`.
+
+### Response streaming
+
+Use response streaming for Lambda functions that return large responses in multiple chunks. The processor collects all chunks of the streamed response, parses them using the `response_codec`, and merges the resulting data into the original event. Because each response is merged into a single original event, `batch.threshold.event_count` must be set to `1`.
+
+The following example configures response streaming:
+
+```yaml
+processor:
+  - aws_lambda:
+      function_name: my-streaming-function
+      invocation_type: streaming-response
+      streaming:
+        enabled: true
+        response_handling: reconstruct-document
+      batch:
+        threshold:
+          event_count: 1
+      aws:
+        region: us-east-1
+        sts_role_arn: arn:aws:iam::123456789012:role/my-lambda-role
+```
+{% include copy.html %}
 
 ### Batch processing
 

_data-prepper/pipelines/configuration/processors/convert-entry-type.md

@@ -28,10 +28,13 @@ This table is autogenerated. Do not edit it.
 
 | Option | Required | Description |
 | :--- | :--- | :--- |
-| `key`| Yes | Key whose value needs to be converted to a different type. |
-| `keys`| Yes | Keys whose value needs to be converted to a different type. |
+| `key`| Conditionally | The key of the value to convert to a different type. Required unless `keys` or `coerce_strings` is specified. Cannot be used with `keys`. |
+| `keys`| Conditionally | The keys of the values to convert to a different type. Required unless `key` or `coerce_strings` is specified. Cannot be used with `key`. |
 | `type` | No | Target type for the key-value pair. Possible values are `integer`, `long`, `double`, `big_decimal`, `string`, and `boolean`. Default value is `integer`. |
-| `null_values` | No | String representation of what constitutes a `null` value. If the field value equals one of these strings, then the value is considered `null` and is converted to `null`. |
+| `null_conversion_values` | No | A list of strings that represent a `null` value. If the value of a field specified in `key` or `keys` equals one of these strings, then the field is removed from the event. |
+| `iterate_on` | No | The key of a list of objects in the event. When specified, the processor converts the `key` or `keys` fields in each object in the list. |
+| `coerce_strings` | No | When specified, the processor converts the string values of all fields in the event, including fields in nested objects and lists, to the type that the string represents. The strings `true` and `false` (case insensitive) are converted to Booleans. Strings that contain a colon (`:`) and match one of the `coerce_strings.time_formats` patterns are converted to epoch milliseconds. Numeric strings that contain a decimal point or an exponent are converted to doubles, and other numeric strings are converted to integers or, if they are out of the integer range, to longs. Strings that cannot be converted are not changed. Specify `coerce_strings` as an empty object (`{}`) to use the default time formats. |
+| `coerce_strings.time_formats` | No | A list of [Java DateTimeFormatter](https://docs.oracle.com/en/java/javase/17/docs/api/java.base/java/time/format/DateTimeFormatter.html) patterns used to convert time strings to epoch milliseconds. Time strings that do not include a time zone offset are interpreted in the system time zone, and patterns that do not include a year use the current year. Default is `["MMM d HH:mm:ss", "yyyy-MM-dd'T'HH:mm:ss.SSSXXX", "yyyy-MM-dd'T'HH:mm:ss.SSSSSSX", "dd/MMM/yyyy:HH:mm:ss Z"]`. |
 | `scale` | No | Modifies the scale of the `big_decimal` when converting to a `big_decimal`. The default value is `0`. |
 | `tags_on_failure` | No | A list of tags to be added to the event metadata when the event fails to convert. |
 | `convert_when` | No | Specifies a condition using a [Data Prepper expression]({{site.url}}{{site.baseurl}}/data-prepper/pipelines/expression-syntax/) for performing the `convert_type` operation. If specified, the `convert_type` operation runs only when the expression evaluates to `true`. |

_data-prepper/pipelines/configuration/processors/csv.md

@@ -23,6 +23,9 @@ Option | Required | Type | Description
 `column_names_source_key` | No | String | The field in the event that specifies the CSV column names, which will be automatically detected. If there need to be extra column names, the column names are automatically generated according to their index. If `column_names` is also defined, the header in `column_names_source_key` can also be used to generate the event fields. If too few columns are specified in this field, the remaining column names are automatically generated. If too many column names are specified in this field, the CSV processor omits the extra column names.
 `column_names` | No | List | User-specified names for the CSV columns. Default is `[column1, column2, ..., columnN]` if there are no columns of data in the CSV record and `column_names_source_key` is not defined. If `column_names_source_key` is defined, the header in `column_names_source_key` generates the event fields. If too few columns are specified in this field, the remaining column names are automatically generated. If too many column names are specified in this field, the CSV processor omits the extra column names.
 `delete_source` | No | Boolean | If `true`, deletes the configured `source` field (by default, `message`) after CSV parsing. This configuration option improves memory pressure if the `source` field is not going to be used because the processing is done in batches. Default is `false`.
+`normalize_keys` | No | Boolean | If `true`, replaces characters that are not valid in event keys with an underscore (`_`) in the column names. For example, the column name `user:name` becomes `user_name`. Default is `false`.
+`multiline` | No | Boolean | If `true`, the `source` field is treated as multiline CSV data in which the first line contains the column names and each following line contains a row of data. The processor creates a new event for each data row, containing the fields of the original event and the parsed columns, and drops the original event. Events that the processor does not parse, such as events in which the `source` field contains only one line, are also dropped. When `multiline` is `true`, the `column_names`, `column_names_source_key`, `delete_header`, and `delete_source` options are not applied. Default is `false`.
+`csv_when` | No | String | A [conditional expression]({{site.url}}{{site.baseurl}}/data-prepper/pipelines/expression-syntax/), such as `/some_key == "test"`. If specified, the `csv` processor runs only on events for which the expression evaluates to `true`.
 
 ## Usage
 

_data-prepper/pipelines/configuration/processors/grok.md

@@ -23,17 +23,19 @@ This table is autogenerated. Do not edit it.
 
 Option | Required | Type | Description
 :--- | :--- |:--- | :---
-`break_on_match` | No | Boolean | Specifies whether to match all patterns (`true`) or stop once the first successful match is found (`false`). Default is `true`.
+`break_on_match` | No | Boolean | Specifies whether to stop once the first successful match is found (`true`) or match all patterns (`false`). Default is `true`.
 `grok_when` | No | String  | Specifies under what condition the `grok` processor should perform matching. For information about this expression, see [Expression syntax]({{site.url}}{{site.baseurl}}/data-prepper/pipelines/expression-syntax/). Default is no condition.
 `keep_empty_captures` | No | Boolean | Enables the preservation of `null` captures from the processed output. Default is `false`.
 `keys_to_overwrite` | No | List | Specifies which existing keys will be overwritten if there is a capture with the same key value. Default is `[]`.
 `match` | No | Map | Specifies which keys should match specific patterns. Default is an empty response body.
 `named_captures_only` | No | Boolean | Specifies whether to keep only named captures. Default is `true`.
 `pattern_definitions` | No | Map | Allows for a custom pattern that can be used inline inside the response body. Default is an empty response body.
 `patterns_directories` | No | List | Specifies which directory paths contain the custom pattern files. Default is an empty list.
-`pattern_files_glob` | No | String | Specifies which pattern files to use from the directories specified for `pattern_directories`. Default is `*`.
+`patterns_files_glob` | No | String | Specifies which pattern files to use from the directories specified for `patterns_directories`. Default is `*`.
 `target_key` | No | String | Specifies a parent-level key used to store all captures. Default value is `null`.
-`timeout_millis` | No | Integer | The maximum amount of time during which matching occurs. Setting to `0` prevents any matching from occurring. Default is `30,000`.
+`timeout_millis` | No | Integer | The maximum amount of time, in milliseconds, during which matching occurs. Setting to `0` disables the timeout. Default is `30,000`.
+`tags_on_match_failure` | No | List | A list of tags to add to the event metadata when the event does not match any pattern or an exception occurs during matching.
+`tags_on_timeout` | No | List | A list of tags to add to the event metadata when matching exceeds `timeout_millis`. If not specified, the `tags_on_match_failure` tags are added when matching times out.
 `performance_metadata` | No | Boolean | Whether or not to add the performance metadata to events. Default is `false`. For more information, see [Grok performance metadata](#grok-performance-metadata).
 
 ## Grok performance metadata