Update data prepper parameters and diagram in the intro by kolchfa-aws · Pull Request #13393 · opensearch-project/documentation-website

kolchfa-aws · 2026-10-26T16:32:14Z

This PR brings parameter tables in data prepper up to date with the current release. It adds more example configuration and udpates the trace analytics diagram so it follows the intro sentence.

Checklist

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license and subject to the Developers Certificate of Origin.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

Signed-off-by: Fanit Kolchina <kolchfa@amazon.com>

github-actions · 2026-10-26T16:32:23Z

Thank you for submitting your PR. The PR states are In progress (or Draft) -> Tech review -> Doc review -> Merged.

Before you submit your PR for doc review, make sure the content is technically accurate. If you need help finding a tech reviewer, tag a maintainer.

When you're ready for doc review, tag the assignee of this PR. The doc reviewer may push edits to the PR directly or leave comments and editorial suggestions for you to address (let us know in a comment if you have a preference).

dlvenable

Thank you @kolchfa-aws for this great update to our documentation!

I have a few comments to improve.

dlvenable · 2026-10-29T17:15:19Z

 `sts_role_arn` | No | String | The AWS Security Token Service (AWS STS) role to assume for requests to Amazon Simple Queue Service (Amazon SQS) and Amazon Simple Storage Service (Amazon S3). Default is `null`, which will use the [standard SDK behavior for credentials](https://docs.aws.amazon.com/sdk-for-java/latest/developer-guide/credentials.html).
 `msk` | No | JSON object | The [MSK](#msk) configuration settings.
+`sts_header_overrides` | No | Map | A map of headers to add to the AWS STS `AssumeRole` request. You can specify up to 5 headers.
+`role_session_name` | No | String | The role session name to use when authenticating to Amazon MSK using the role specified in `sts_role_arn`. Applies only when `aws_msk_iam` is set to `role`.


It appears that this is unused. Maybe remove it for now.

Sure. This was used in 2.16 but I see it was removed for 2.17 so removing.

dlvenable · 2026-10-29T17:15:47Z

 `sts_role_arn` | No | String | The AWS STS role to assume for requests to Amazon Simple Queue Service (Amazon SQS) and Amazon Simple Storage Service (Amazon S3). Default is `null`, which will use the [standard SDK behavior for credentials](https://docs.aws.amazon.com/sdk-for-java/latest/developer-guide/credentials.html).
 `msk` | No | JSON object | The [Amazon MSK](#msk) configuration settings.
+`sts_header_overrides` | No | Map | A map of headers to add to the AWS STS `AssumeRole` request. You can specify up to 5 headers.
+`role_session_name` | No | String | The role session name to use when authenticating to Amazon MSK using the role specified in `sts_role_arn`. Applies only when `aws_msk_iam` is set to `role`.


This also appears unused.

dlvenable · 2026-10-29T17:18:04Z

+`path_prefix` | No | String | A path prefix that the sink adds to the path of every request sent to OpenSearch, for example, `/opensearch`. Use this option when OpenSearch is accessible under a subpath, such as behind a reverse proxy.
 `index` | Conditionally | String | The name of the export index. Only required when the `index_type` is `custom`. The index can be a plain string, such as `my-index-name`, contain [Java date-time patterns](https://docs.oracle.com/javase/8/docs/api/java/time/format/DateTimeFormatter.html), such as `my-index-%{yyyy.MM.dd}` or `my-%{yyyy-MM-dd-HH}-index`, be formatted using field values, such as `my-index-${/my_field}`, or use [Data Prepper expressions]({{site.url}}{{site.baseurl}}/data-prepper/pipelines/expression-syntax/), such as `my-index-${getMetadata(\"my_metadata_field\"}`. All formatting options can be combined to provide flexibility when creating static, dynamic, and rolling indexes. 
-`index_type` | No | String | Specifies the type of data the sink plugin handles. Valid values include `custom`, `trace-analytics-raw`, `trace-analytics-plain-raw`, `trace-analytics-service-map`, `log-analytics`, `log-analytics-plain`, `metric-analytics`, `metric-analytics-plain`, and `management-disabled`. <br><br>To produce Amazon Security Lake–compliant data from the `otel_logs_source` with `output_format: otel`, set `index_type` to `log-analytics-plain`. <br>For `otel_metrics_source` with `output_format: otel`, set `index_type` to `metric-analytics-plain`. <br>For `otel_trace_source` with `output_format: otel`, set `index_type` to `trace-analytics-plain-raw`. <br><br>Default is `custom`.
+`index_type` | No | String | Specifies the type of data the sink plugin handles. Valid values include `custom`, `trace-analytics-raw`, `trace-analytics-plain-raw`, `trace-analytics-service-map`, `otel-v2-apm-service-map`, `log-analytics`, `log-analytics-plain`, `metric-analytics`, `metric-analytics-plain`, `tsdb`, and `management_disabled`. <br><br>To produce Amazon Security Lake–compliant data from the `otel_logs_source` with `output_format: otel`, set `index_type` to `log-analytics-plain`. <br>For `otel_metrics_source` with `output_format: otel`, set `index_type` to `metric-analytics-plain`. <br>For `otel_trace_source` with `output_format: otel`, set `index_type` to `trace-analytics-plain-raw`. <br><br>Default is `custom`.


We recently added: trace-analytics-plain-raw-v2 and it will release in 2.17. Please include this here as well.

Sure, adding these. Will wait until DP 2.17 is out to merge this PR.

dlvenable · 2026-10-29T17:19:37Z

+The following example configures response streaming:
+
+```yaml
+processors:


This should be processor:. Singular, not plural.

dlvenable · 2026-10-29T17:21:07Z

+`paths` | Conditionally | List | A list of absolute file paths or glob patterns that specify the files to read, for example, `/var/log/app/*.log` or `/var/log/**/*.log`. Either `path` or `paths` is required.
+`exclude_paths` | No | List | A list of absolute file paths or glob patterns that specify files to skip among the files matched by `paths`.
+`format` | No | String | Specifies how to interpret the file content. Valid values are `json` and `plain`. Use `json` when your file has one JSON object per line or a JSON array. Use `plain` for raw text lines. Applies only when `path` is used without `paths`, `exclude_paths`, `codec`, or `tail`. Default is `plain`.
+`record_type` | No | String | The type of output record produced by the source. Valid values are `event` and `string`. Use `event` to produce structured events expected by downstream processors and the OpenSearch sink. Applies only when `path` is used without `paths`, `exclude_paths`, `codec`, or `tail`. Default is `string`.


When a codec is configured, the customer must set record_type to event. We have an assertion on this currently.

dlvenable · 2026-10-29T17:21:38Z

+The source reads files in one of the following ways, depending on its configuration:
+
+- If you specify `path` and none of the `paths`, `exclude_paths`, `codec`, or `tail` settings, the source reads the file once, line by line, and creates events according to the `format` and `record_type` settings.
+- If you specify `paths`, `exclude_paths`, `codec`, or `tail`, the source ignores the `format` and `record_type` settings. It parses each file using the configured `codec` or, if no `codec` is configured, creates one event for each line. For the structure of these events, see [Event structure](#event-structure).


When a codec is configured, the customer must set record_type to event. We have an assertion on this currently. Please reword to clarify that this value must be set to event.

dlvenable · 2026-10-29T17:22:37Z

+The source reads files in one of the following ways, depending on its configuration:
+
+- If you specify `path` and none of the `paths`, `exclude_paths`, `codec`, or `tail` settings, the source reads the file once, line by line, and creates events according to the `format` and `record_type` settings.
+- If you specify `paths`, `exclude_paths`, `codec`, or `tail`, the source ignores the `format` and `record_type` settings. It parses each file using the configured `codec` or, if no `codec` is configured, creates one event for each line. For the structure of these events, see [Event structure](#event-structure).


It may be good to add that the recommended configuration is to use record_type: event and provide a codec:. The format and record_type: string are legacy configurations.

dlvenable · 2026-10-29T17:23:31Z

+`fetch_min_bytes` | No | String | The minimum amount of data the server returns during a fetch request through Kafka's `fetch.min.bytes` setting. Default is `2kb`.
+`retry_backoff` | No | Duration | The amount of time to wait before attempting to retry a failed request to a given topic partition. Default is `10s`.
+`reconnect_backoff` | No | Duration | The amount of time to wait before attempting to reconnect to a broker through Kafka's `reconnect.backoff.ms` setting. Default is `10s`.
+`max_poll_interval` | No | Duration | The maximum delay between invocations of a `poll()` when using group management through Kafka's `max.poll.interval.ms` option. Default is `300s`.


We have recently added:

bounded_buffer_size

bounded_batch_size

It may be good to include these. They will release in 2.17.

Signed-off-by: Fanit Kolchina <kolchfa@amazon.com>

kolchfa-aws · 2026-10-29T17:44:02Z

Thank you for reviewing, @dlvenable! I addressed your comments.

dlvenable · 2026-10-29T21:19:50Z

This will be a great improvement to the public docs!

kolchfa-aws · 2026-10-29T21:53:30Z

Waiting for DP 2.17 to merge

Signed-off-by: Fanit Kolchina <kolchfa@amazon.com>

dlvenable

Thank you @kolchfa-aws !

Signed-off-by: Fanit Kolchina <kolchfa@amazon.com>

kolchfa-aws added 2 commits October 22, 2026 13:59

Update data prepper parameters and examples

698c39d

Signed-off-by: Fanit Kolchina <kolchfa@amazon.com>

Update the diagram according to intro sentence

aa7c4b7

Signed-off-by: Fanit Kolchina <kolchfa@amazon.com>

kolchfa-aws requested review from cwperks, dlvenable, epugh, peterzhuamazon and sumobrian as code owners October 26, 2026 16:32

github-actions Bot added the Tech review PR: Tech review in progress label Oct 26, 2026

github-actions Bot assigned kolchfa-aws Oct 26, 2026

kolchfa-aws added data-prepper backport 3.9 labels Oct 26, 2026

dlvenable reviewed Oct 29, 2026

View reviewed changes

Address comments

a5a4fea

Signed-off-by: Fanit Kolchina <kolchfa@amazon.com>

dlvenable approved these changes Oct 29, 2026

View reviewed changes

kolchfa-aws added Done but waiting to merge PR: The work is done and ready to merge and removed Tech review PR: Tech review in progress labels Oct 29, 2026

Document new Data Prepper 2.17 parameters

77291a7

Signed-off-by: Fanit Kolchina <kolchfa@amazon.com>

dlvenable approved these changes Nov 5, 2026

View reviewed changes

kolchfa-aws mentioned this pull request Nov 5, 2026

[DOC] Add shard_discovery_interval and shard_acquisition_backoff_max to the dynamodb source stream options #13430

Open

1 task done

Add Vale terms and reword Lambda retry default

e8467c5

Signed-off-by: Fanit Kolchina <kolchfa@amazon.com>

Conversation

kolchfa-aws commented Oct 26, 2026

Checklist

Uh oh!

github-actions Bot commented Oct 26, 2026

Uh oh!

dlvenable left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

kolchfa-aws commented Oct 29, 2026

Uh oh!

dlvenable commented Oct 29, 2026

Uh oh!

kolchfa-aws commented Oct 29, 2026

Uh oh!

dlvenable left a comment

Choose a reason for hiding this comment

Uh oh!

This branch has not been deployed

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants