Skip to content

DocTest - elasticsearch-output - DO NOT MERGE #2038

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
karenzone wants to merge 1 commit into
base: 8.19
Choose a base branch
from
Open

DocTest - elasticsearch-output - DO NOT MERGE #2038

Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
169 changes: 63 additions & 106 deletions docs/plugins/outputs/elasticsearch.asciidoc
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -66,9 +66,10 @@ Set the value to port :443 instead.
For more info on sending data from {ls} to {es-serverless}, check out the {serverless-docs}/elasticsearch/what-is-elasticsearch-serverless[{es-serverless} docs].

[id="plugins-{type}s-{plugin}-ess"]
==== Hosted {es} Service on Elastic Cloud
==== {ls} to {ech}

{ess-leadin}
You can run Elasticsearch on your own hardware or use Elastic Cloud Hosted, available on AWS, GCP, and Azure.
Try Elastic Cloud Hosted for free: https://cloud.elastic.co/registration.

==== Compatibility with the Elastic Common Schema (ECS)

Expand Down Expand Up @@ -139,6 +140,15 @@ output {

==== Writing to different indices: best practices

.Important considerations for time-based routing
****
Time-based routing in this plugin follows *one* model at a time. _Do not_ mix models.

* *Data streams.* Use <<plugins-{type}s-{plugin}-data_stream,`data_stream`>> and related settings.
* *Index lifecycle management (ILM) rollover.* The value you set using <<plugins-{type}s-{plugin}-ilm_pattern,`ilm_pattern`>> is used for naming, and should align with {ref}/indices-rollover-index.html#_using_date_math_with_the_rollover_api[Elasticsearch rollover naming].
* *Explicit index names.* Use the <<plugins-{type}s-{plugin}-index,`index`>> setting and `%{+...}` syntax for daily (or other) rotation.
****

NOTE: You cannot use dynamic variable substitution when `ilm_enabled` is `true`
and when using `ilm_rollover_alias`.

Expand Down Expand Up @@ -190,7 +200,7 @@ Example:
}


==== Retry Policy
==== Retry policy

The retry policy has changed significantly in the 8.1.1 release.
This plugin uses the Elasticsearch bulk API to optimize its imports into Elasticsearch. These requests may experience
Expand Down Expand Up @@ -223,16 +233,16 @@ Note that 409 exceptions are no longer retried. Please set a higher `retry_on_co
It is more performant for Elasticsearch to retry these exceptions than this plugin.

[id="plugins-{type}s-{plugin}-dlq-policy"]
==== DLQ Policy
==== DLQ policy

Mapping (404) errors from Elasticsearch can lead to data loss. Unfortunately
mapping errors cannot be handled without human intervention and without looking
at the field that caused the mapping mismatch. If the DLQ is enabled, the
original events causing the mapping errors are stored in a file that can be
processed at a later time. Often times, the offending field can be removed and
re-indexed to Elasticsearch. If the DLQ is not enabled, and a mapping error
happens, the problem is logged as a warning, and the event is dropped. See
<<dead-letter-queues>> for more information about processing events in the DLQ.
happens, the problem is logged as a warning, and the event is dropped.
Check out https://www.elastic.co/guide/en/logstash/current/dead-letter-queues.html[Dead letter queues (DLQ)] for more information about processing events in the DLQ.
The list of error codes accepted for DLQ could be customized with <<plugins-{type}s-{plugin}-dlq_custom_codes>>
but should be used only in motivated cases.

Expand Down Expand Up @@ -303,12 +313,17 @@ single request.
==== DNS Caching

This plugin uses the JVM to lookup DNS entries and is subject to the value of
https://docs.oracle.com/javase/7/docs/technotes/guides/net/properties.html[networkaddress.cache.ttl],
a global setting for the JVM.
https://docs.oracle.com/en/java/javase/21/docs/api/java.base/java/net/doc-files/net-properties.html#address-cache-heading[Address Cache settings]
such as `networkaddress.cache.ttl` and `networkaddress.cache.negative.ttl`, global settings for the JVM.

As an example, to set your DNS TTL to 1 second you would set
the `LS_JAVA_OPTS` environment variable to `-Dnetworkaddress.cache.ttl=1`.

The default value for `networkaddress.cache.ttl` depends on the JVM implementation,
which is 30 seconds for the JDK bundled with Logstash.
The `networkaddress.cache.negative.ttl` setting, that controls how long Java caches
the result of failed DNS lookups, defaults to 10 seconds.

Keep in mind that a connection with keepalive enabled will
not reevaluate its DNS value while the keepalive is in effect.

Expand Down Expand Up @@ -341,8 +356,10 @@ When a string value on an event contains one or more byte sequences that are not
[id="plugins-{type}s-{plugin}-options"]
==== Elasticsearch Output Configuration Options

This plugin supports the following configuration options plus the
<<plugins-{type}s-{plugin}-common-options>> and the <<plugins-{type}s-{plugin}-deprecated-options>> described later.
This plugin supports these configuration options plus the <<plugins-{type}s-{plugin}-common-options>> described later.

NOTE: As of version 12.0.0 of this plugin, a number of previously deprecated SSL settings have been removed.
Please check out <<plugins-{type}s-{plugin}-obsolete-options>> for details.

[cols="<,<,<",options="header",]
|=======================================================================
Expand All @@ -366,6 +383,7 @@ This plugin supports the following configuration options plus the
| <<plugins-{type}s-{plugin}-doc_as_upsert>> |<<boolean,boolean>>|No
| <<plugins-{type}s-{plugin}-document_id>> |<<string,string>>|No
| <<plugins-{type}s-{plugin}-document_type>> |<<string,string>>|No
| <<plugins-{type}s-{plugin}-drop_error_types>> |<<array,array>>|No
| <<plugins-{type}s-{plugin}-ecs_compatibility>> | <<string,string>>|No
| <<plugins-{type}s-{plugin}-failure_type_logging_whitelist>> |<<array,array>>|No
| <<plugins-{type}s-{plugin}-healthcheck_path>> |<<string,string>>|No
Expand Down Expand Up @@ -457,7 +475,7 @@ For more details on actions, check out the {ref}/docs-bulk.html[Elasticsearch bu
* There is no default value for this setting.

Authenticate using Elasticsearch API key.
Note that this option also requires SSL/TLS, which can be enabled by supplying a <<plugins-{type}s-{plugin}-cloud_id>>, a list of HTTPS <<plugins-{type}s-{plugin}-hosts>>, or by setting <<plugins-{type}s-{plugin}-ssl,`ssl_enabled => true`>>.
Note that this option also requires SSL/TLS, which can be enabled by supplying a <<plugins-{type}s-{plugin}-cloud_id>>, a list of HTTPS <<plugins-{type}s-{plugin}-hosts>>, or by setting <<plugins-{type}s-{plugin}-ssl_enabled,`ssl_enabled => true`>>.

Format is `id:api_key` where `id` and `api_key` are as returned by the
Elasticsearch {ref}/security-api-create-api-key.html[Create API key API].
Expand Down Expand Up @@ -626,8 +644,7 @@ Elasticsearch with the same ID.

NOTE: This option is deprecated due to the
https://www.elastic.co/guide/en/elasticsearch/reference/6.0/removal-of-types.html[removal
of types in Elasticsearch 6.0]. It will be removed in the next major version of
Logstash.
of types in Elasticsearch 6.0].

NOTE: This value is ignored and has no effect for Elasticsearch clusters `8.x`.

Expand All @@ -636,9 +653,25 @@ similar events to the same 'type'. String expansion `%{foo}` works here.
If you don't set a value for this option:

- for elasticsearch clusters 8.x: no value will be used;
- for elasticsearch clusters 7.x: the value of '_doc' will be used;
- for elasticsearch clusters 6.x: the value of 'doc' will be used;
- for elasticsearch clusters 5.x and below: the event's 'type' field will be used, if the field is not present the value of 'doc' will be used.
- for elasticsearch clusters 7.x: the value of '_doc' will be used.

[id="plugins-{type}s-{plugin}-drop_error_types"]
===== `drop_error_types`

* Value type is <<array,array>>
* Default value is `[]`

Lists the set of error types for which individual bulk request actions will not be retried. Unless an individual - document level - action returns 409 or an error from this list, failures will be retried indefinitely.
A warning message will be logged indicating that the action failed, unless the error type is
listed in the <<plugins-{type}s-{plugin}-silence_errors_in_log>> config option.
Note that the events are not added to the Dead Letter Queue (DLQ), regardless of whether it is enabled.

[source,ruby]
output {
elasticsearch {
drop_error_types => ["index_closed_exception"]
}
}

[id="plugins-{type}s-{plugin}-ecs_compatibility"]
===== `ecs_compatibility`
Expand Down Expand Up @@ -1053,8 +1086,6 @@ NOTE: Deprecates <<plugins-{type}s-{plugin}-failure_type_logging_whitelist>>.

This setting asks Elasticsearch for the list of all cluster nodes and adds them
to the hosts list.
For Elasticsearch 5.x and 6.x any nodes with `http.enabled` (on by default) will
be added to the hosts list, excluding master-only nodes.

[id="plugins-{type}s-{plugin}-sniffing_delay"]
===== `sniffing_delay`
Expand Down Expand Up @@ -1340,99 +1371,25 @@ https://www.elastic.co/blog/elasticsearch-versioning-support[versioning support
blog] and {ref}/docs-index_.html#_version_types[Version types] in the
Elasticsearch documentation.

[id="plugins-{type}s-{plugin}-deprecated-options"]
==== Elasticsearch Output Deprecated Configuration Options
[id="plugins-{type}s-{plugin}-obsolete-options"]
==== Elasticsearch Output Obsolete Configuration Options

This plugin supports the following deprecated configurations.
WARNING: As of version `12.0.0` of this plugin, some configuration options have been replaced.
The plugin will fail to start if it contains any of these obsolete options.

WARNING: Deprecated options are subject to removal in future releases.

[cols="<,<,<",options="header",]
[cols="<,<",options="header",]
|=======================================================================
|Setting|Input type|Replaced by
| <<plugins-{type}s-{plugin}-cacert>> |a valid filesystem path|<<plugins-{type}s-{plugin}-ssl_certificate_authorities>>
| <<plugins-{type}s-{plugin}-keystore>> |a valid filesystem path|<<plugins-{type}s-{plugin}-ssl_keystore_path>>
| <<plugins-{type}s-{plugin}-keystore_password>> |<<password,password>>|<<plugins-{type}s-{plugin}-ssl_keystore_password>>
| <<plugins-{type}s-{plugin}-ssl>> |<<boolean,boolean>>|<<plugins-{type}s-{plugin}-ssl_enabled>>
| <<plugins-{type}s-{plugin}-ssl_certificate_verification>> |<<boolean,boolean>>|<<plugins-{type}s-{plugin}-ssl_verification_mode>>
| <<plugins-{type}s-{plugin}-truststore>> |a valid filesystem path|<<plugins-{type}s-{plugin}-ssl_truststore_path>>
| <<plugins-{type}s-{plugin}-truststore_password>> |<<password,password>>|<<plugins-{type}s-{plugin}-ssl_truststore_password>>
|Setting|Replaced by
| cacert | <<plugins-{type}s-{plugin}-ssl_certificate_authorities>>
| keystore | <<plugins-{type}s-{plugin}-ssl_keystore_path>>
| keystore_password | <<plugins-{type}s-{plugin}-ssl_keystore_password>>
| ssl | <<plugins-{type}s-{plugin}-ssl_enabled>>
| ssl_certificate_verification | <<plugins-{type}s-{plugin}-ssl_verification_mode>>
| truststore | <<plugins-{type}s-{plugin}-ssl_truststore_path>>
| truststore_password | <<plugins-{type}s-{plugin}-ssl_truststore_password>>
|=======================================================================


[id="plugins-{type}s-{plugin}-cacert"]
===== `cacert`
deprecated[11.14.0, Replaced by <<plugins-{type}s-{plugin}-ssl_certificate_authorities>>]

* Value type is a list of <<path,path>>
* There is no default value for this setting.

The .cer or .pem file to validate the server's certificate.

[id="plugins-{type}s-{plugin}-keystore"]
===== `keystore`
deprecated[11.14.0, Replaced by <<plugins-{type}s-{plugin}-ssl_keystore_path>>]

* Value type is <<path,path>>
* There is no default value for this setting.

The keystore used to present a certificate to the server.
It can be either .jks or .p12

NOTE: You cannot use this setting and <<plugins-{type}s-{plugin}-ssl_certificate>> at the same time.

[id="plugins-{type}s-{plugin}-keystore_password"]
===== `keystore_password`
deprecated[11.14.0, Replaced by <<plugins-{type}s-{plugin}-ssl_keystore_password>>]

* Value type is <<password,password>>
* There is no default value for this setting.

Set the keystore password

[id="plugins-{type}s-{plugin}-ssl"]
===== `ssl`
deprecated[11.14.0, Replaced by <<plugins-{type}s-{plugin}-ssl_enabled>>]

* Value type is <<boolean,boolean>>
* There is no default value for this setting.

Enable SSL/TLS secured communication to Elasticsearch cluster.
Leaving this unspecified will use whatever scheme is specified in the URLs listed in <<plugins-{type}s-{plugin}-hosts>> or extracted from the <<plugins-{type}s-{plugin}-cloud_id>>.
If no explicit protocol is specified plain HTTP will be used.

[id="plugins-{type}s-{plugin}-ssl_certificate_verification"]
===== `ssl_certificate_verification`
deprecated[11.14.0, Replaced by <<plugins-{type}s-{plugin}-ssl_verification_mode>>]

* Value type is <<boolean,boolean>>
* Default value is `true`

Option to validate the server's certificate. Disabling this severely compromises security.
For more information on disabling certificate verification please read
https://www.cs.utexas.edu/~shmat/shmat_ccs12.pdf

[id="plugins-{type}s-{plugin}-truststore"]
===== `truststore`
deprecated[11.14.0, Replaced by <<plugins-{type}s-{plugin}-ssl_truststore_path>>]

* Value type is <<path,path>>
* There is no default value for this setting.

The truststore to validate the server's certificate.
It can be either `.jks` or `.p12`.
Use either `:truststore` or `:cacert`.

[id="plugins-{type}s-{plugin}-truststore_password"]
===== `truststore_password`
deprecated[11.14.0, Replaced by <<plugins-{type}s-{plugin}-ssl_truststore_password>>]

* Value type is <<password,password>>
* There is no default value for this setting.

Set the truststore password

[id="plugins-{type}s-{plugin}-common-options"]
include::{include_path}/{type}.asciidoc[]

:no_codec!:
:no_codec!: