diff --git a/docs/plugins/outputs/elasticsearch.asciidoc b/docs/plugins/outputs/elasticsearch.asciidoc index 4ae90295..f766953c 100644 --- a/docs/plugins/outputs/elasticsearch.asciidoc +++ b/docs/plugins/outputs/elasticsearch.asciidoc @@ -66,9 +66,10 @@ Set the value to port :443 instead. For more info on sending data from {ls} to {es-serverless}, check out the {serverless-docs}/elasticsearch/what-is-elasticsearch-serverless[{es-serverless} docs]. [id="plugins-{type}s-{plugin}-ess"] -==== Hosted {es} Service on Elastic Cloud +==== {ls} to {ech} -{ess-leadin} +You can run Elasticsearch on your own hardware or use Elastic Cloud Hosted, available on AWS, GCP, and Azure. +Try Elastic Cloud Hosted for free: https://cloud.elastic.co/registration. ==== Compatibility with the Elastic Common Schema (ECS) @@ -139,6 +140,15 @@ output { ==== Writing to different indices: best practices +.Important considerations for time-based routing +**** +Time-based routing in this plugin follows *one* model at a time. _Do not_ mix models. + +* *Data streams.* Use <> and related settings. +* *Index lifecycle management (ILM) rollover.* The value you set using <> is used for naming, and should align with {ref}/indices-rollover-index.html#_using_date_math_with_the_rollover_api[Elasticsearch rollover naming]. +* *Explicit index names.* Use the <> setting and `%{+...}` syntax for daily (or other) rotation. +**** + NOTE: You cannot use dynamic variable substitution when `ilm_enabled` is `true` and when using `ilm_rollover_alias`. @@ -190,7 +200,7 @@ Example: } -==== Retry Policy +==== Retry policy The retry policy has changed significantly in the 8.1.1 release. This plugin uses the Elasticsearch bulk API to optimize its imports into Elasticsearch. These requests may experience @@ -223,7 +233,7 @@ Note that 409 exceptions are no longer retried. Please set a higher `retry_on_co It is more performant for Elasticsearch to retry these exceptions than this plugin. [id="plugins-{type}s-{plugin}-dlq-policy"] -==== DLQ Policy +==== DLQ policy Mapping (404) errors from Elasticsearch can lead to data loss. Unfortunately mapping errors cannot be handled without human intervention and without looking @@ -231,8 +241,8 @@ at the field that caused the mapping mismatch. If the DLQ is enabled, the original events causing the mapping errors are stored in a file that can be processed at a later time. Often times, the offending field can be removed and re-indexed to Elasticsearch. If the DLQ is not enabled, and a mapping error -happens, the problem is logged as a warning, and the event is dropped. See -<> for more information about processing events in the DLQ. +happens, the problem is logged as a warning, and the event is dropped. +Check out https://www.elastic.co/guide/en/logstash/current/dead-letter-queues.html[Dead letter queues (DLQ)] for more information about processing events in the DLQ. The list of error codes accepted for DLQ could be customized with <> but should be used only in motivated cases. @@ -303,12 +313,17 @@ single request. ==== DNS Caching This plugin uses the JVM to lookup DNS entries and is subject to the value of -https://docs.oracle.com/javase/7/docs/technotes/guides/net/properties.html[networkaddress.cache.ttl], -a global setting for the JVM. +https://docs.oracle.com/en/java/javase/21/docs/api/java.base/java/net/doc-files/net-properties.html#address-cache-heading[Address Cache settings] +such as `networkaddress.cache.ttl` and `networkaddress.cache.negative.ttl`, global settings for the JVM. As an example, to set your DNS TTL to 1 second you would set the `LS_JAVA_OPTS` environment variable to `-Dnetworkaddress.cache.ttl=1`. +The default value for `networkaddress.cache.ttl` depends on the JVM implementation, +which is 30 seconds for the JDK bundled with Logstash. +The `networkaddress.cache.negative.ttl` setting, that controls how long Java caches +the result of failed DNS lookups, defaults to 10 seconds. + Keep in mind that a connection with keepalive enabled will not reevaluate its DNS value while the keepalive is in effect. @@ -341,8 +356,10 @@ When a string value on an event contains one or more byte sequences that are not [id="plugins-{type}s-{plugin}-options"] ==== Elasticsearch Output Configuration Options -This plugin supports the following configuration options plus the -<> and the <> described later. +This plugin supports these configuration options plus the <> described later. + +NOTE: As of version 12.0.0 of this plugin, a number of previously deprecated SSL settings have been removed. +Please check out <> for details. [cols="<,<,<",options="header",] |======================================================================= @@ -366,6 +383,7 @@ This plugin supports the following configuration options plus the | <> |<>|No | <> |<>|No | <> |<>|No +| <> |<>|No | <> | <>|No | <> |<>|No | <> |<>|No @@ -457,7 +475,7 @@ For more details on actions, check out the {ref}/docs-bulk.html[Elasticsearch bu * There is no default value for this setting. Authenticate using Elasticsearch API key. -Note that this option also requires SSL/TLS, which can be enabled by supplying a <>, a list of HTTPS <>, or by setting < true`>>. +Note that this option also requires SSL/TLS, which can be enabled by supplying a <>, a list of HTTPS <>, or by setting < true`>>. Format is `id:api_key` where `id` and `api_key` are as returned by the Elasticsearch {ref}/security-api-create-api-key.html[Create API key API]. @@ -626,8 +644,7 @@ Elasticsearch with the same ID. NOTE: This option is deprecated due to the https://www.elastic.co/guide/en/elasticsearch/reference/6.0/removal-of-types.html[removal -of types in Elasticsearch 6.0]. It will be removed in the next major version of -Logstash. +of types in Elasticsearch 6.0]. NOTE: This value is ignored and has no effect for Elasticsearch clusters `8.x`. @@ -636,9 +653,25 @@ similar events to the same 'type'. String expansion `%{foo}` works here. If you don't set a value for this option: - for elasticsearch clusters 8.x: no value will be used; -- for elasticsearch clusters 7.x: the value of '_doc' will be used; -- for elasticsearch clusters 6.x: the value of 'doc' will be used; -- for elasticsearch clusters 5.x and below: the event's 'type' field will be used, if the field is not present the value of 'doc' will be used. +- for elasticsearch clusters 7.x: the value of '_doc' will be used. + +[id="plugins-{type}s-{plugin}-drop_error_types"] +===== `drop_error_types` + + * Value type is <> + * Default value is `[]` + +Lists the set of error types for which individual bulk request actions will not be retried. Unless an individual - document level - action returns 409 or an error from this list, failures will be retried indefinitely. +A warning message will be logged indicating that the action failed, unless the error type is +listed in the <> config option. +Note that the events are not added to the Dead Letter Queue (DLQ), regardless of whether it is enabled. + +[source,ruby] + output { + elasticsearch { + drop_error_types => ["index_closed_exception"] + } + } [id="plugins-{type}s-{plugin}-ecs_compatibility"] ===== `ecs_compatibility` @@ -1053,8 +1086,6 @@ NOTE: Deprecates <>. This setting asks Elasticsearch for the list of all cluster nodes and adds them to the hosts list. -For Elasticsearch 5.x and 6.x any nodes with `http.enabled` (on by default) will -be added to the hosts list, excluding master-only nodes. [id="plugins-{type}s-{plugin}-sniffing_delay"] ===== `sniffing_delay` @@ -1340,99 +1371,25 @@ https://www.elastic.co/blog/elasticsearch-versioning-support[versioning support blog] and {ref}/docs-index_.html#_version_types[Version types] in the Elasticsearch documentation. -[id="plugins-{type}s-{plugin}-deprecated-options"] -==== Elasticsearch Output Deprecated Configuration Options +[id="plugins-{type}s-{plugin}-obsolete-options"] +==== Elasticsearch Output Obsolete Configuration Options -This plugin supports the following deprecated configurations. +WARNING: As of version `12.0.0` of this plugin, some configuration options have been replaced. +The plugin will fail to start if it contains any of these obsolete options. -WARNING: Deprecated options are subject to removal in future releases. - -[cols="<,<,<",options="header",] +[cols="<,<",options="header",] |======================================================================= -|Setting|Input type|Replaced by -| <> |a valid filesystem path|<> -| <> |a valid filesystem path|<> -| <> |<>|<> -| <> |<>|<> -| <> |<>|<> -| <> |a valid filesystem path|<> -| <> |<>|<> +|Setting|Replaced by +| cacert | <> +| keystore | <> +| keystore_password | <> +| ssl | <> +| ssl_certificate_verification | <> +| truststore | <> +| truststore_password | <> |======================================================================= - -[id="plugins-{type}s-{plugin}-cacert"] -===== `cacert` -deprecated[11.14.0, Replaced by <>] - -* Value type is a list of <> -* There is no default value for this setting. - -The .cer or .pem file to validate the server's certificate. - -[id="plugins-{type}s-{plugin}-keystore"] -===== `keystore` -deprecated[11.14.0, Replaced by <>] - -* Value type is <> -* There is no default value for this setting. - -The keystore used to present a certificate to the server. -It can be either .jks or .p12 - -NOTE: You cannot use this setting and <> at the same time. - -[id="plugins-{type}s-{plugin}-keystore_password"] -===== `keystore_password` -deprecated[11.14.0, Replaced by <>] - -* Value type is <> -* There is no default value for this setting. - -Set the keystore password - -[id="plugins-{type}s-{plugin}-ssl"] -===== `ssl` -deprecated[11.14.0, Replaced by <>] - -* Value type is <> -* There is no default value for this setting. - -Enable SSL/TLS secured communication to Elasticsearch cluster. -Leaving this unspecified will use whatever scheme is specified in the URLs listed in <> or extracted from the <>. -If no explicit protocol is specified plain HTTP will be used. - -[id="plugins-{type}s-{plugin}-ssl_certificate_verification"] -===== `ssl_certificate_verification` -deprecated[11.14.0, Replaced by <>] - -* Value type is <> -* Default value is `true` - -Option to validate the server's certificate. Disabling this severely compromises security. -For more information on disabling certificate verification please read -https://www.cs.utexas.edu/~shmat/shmat_ccs12.pdf - -[id="plugins-{type}s-{plugin}-truststore"] -===== `truststore` -deprecated[11.14.0, Replaced by <>] - -* Value type is <> -* There is no default value for this setting. - -The truststore to validate the server's certificate. -It can be either `.jks` or `.p12`. -Use either `:truststore` or `:cacert`. - -[id="plugins-{type}s-{plugin}-truststore_password"] -===== `truststore_password` -deprecated[11.14.0, Replaced by <>] - -* Value type is <> -* There is no default value for this setting. - -Set the truststore password - [id="plugins-{type}s-{plugin}-common-options"] include::{include_path}/{type}.asciidoc[] -:no_codec!: +:no_codec!: \ No newline at end of file