Skip to content

Improves a visibility the case events routed to DLQ. #1253

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
mashhurs merged 6 commits into from
Apr 16, 2026
+15 −2
Merged

Improves a visibility the case events routed to DLQ. #1253

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
6 commits
Select commit Hold shift + click to select a range
ef59f21
Improves a visibility about how many failed events went to DLQ. If er…
mashhurs Mar 24, 2026
2199b89
Oops, block closure wasn't committed.
mashhurs Mar 24, 2026
bad8c0c
Version bump, add changelog, put back the origin exception throw afte…
mashhurs Mar 25, 2026
c72668b
Capture a sample event for each status to show what is happening with…
mashhurs Mar 25, 2026
a330470
Rollback the dlq writer begin-end statement which sample events logge…
mashhurs Mar 25, 2026
b306761
Fix the typo wrong naming
mashhurs Apr 14, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 3 additions & 0 deletions CHANGELOG.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,3 +1,6 @@
## 12.1.3
- Improves the logging experience when DLQ used [#1253](https://github.com/logstash-plugins/logstash-output-elasticsearch/pull/1253)

## 12.1.2
- Fix: replace deprecated `File.exists?` with `File.exist?` for Ruby 3.4 (JRuby 10) compatibility [#1238](https://github.com/logstash-plugins/logstash-output-elasticsearch/pull/1238)

Expand Down
12 changes: 11 additions & 1 deletion lib/logstash/plugin_mixins/elasticsearch/common.rb
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -241,7 +241,6 @@ def handle_dlq_response(message, action, status, response)
@dlq_writer.write(event, "#{detailed_message}")
else
log_level = dig_value(response, 'index', 'error', 'type') == 'invalid_index_name_exception' ? :error : :warn

@logger.public_send(log_level, message, status: status, action: action_params, response: response)
end
end
Expand Down Expand Up @@ -273,6 +272,7 @@ def submit(actions)
end

actions_to_retry = []
dlq_routed_stats = {}
responses.each_with_index do |response,idx|
action_type, action_props = response.first

Expand All @@ -296,6 +296,11 @@ def submit(actions)
elsif @dlq_codes.include?(status)
handle_dlq_response("Could not index event to Elasticsearch.", action, status, response)
Copy link
Copy Markdown
Member

@robbavey robbavey Mar 26, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

WDYT about DLQ'd events going to the debug log?

Copy link
Copy Markdown
Contributor Author

@mashhurs mashhurs Mar 26, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

kk good question! Yesterday it took me ~15min to decide to go with warning. Why?!- I consider this case need an ATTENTION. But I am open to change if you have strong point.

Copy link
Copy Markdown
Member

@robbavey robbavey Mar 26, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sorry if there is a misunderstanding - my comment was whether we should consider the reason for all events going to the DLQ at a debug level, not the "summary" message, which I think a warn is appropriate, although I could be persuaded that it is an info level

@document_level_metrics.increment(:dlq_routed)
if dlq_routed_stats.key?(status)
dlq_routed_stats[status][:count] += 1
else
dlq_routed_stats[status] = { :count => 1, :sample_event => { :action => action, :response => response } }
end
next
else
# only log what the user whitelisted
Expand All @@ -305,6 +310,11 @@ def submit(actions)
end
end

if @dlq_writer && !dlq_routed_stats.empty?
total = dlq_routed_stats.values.sum { |entry| entry[:count] }
@logger.warn("Events could not be indexed and routing to DLQ", :count => total, :dlq_routed_stats => dlq_routed_stats)
end

actions_to_retry
end

Expand Down
2 changes: 1 addition & 1 deletion logstash-output-elasticsearch.gemspec
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
Gem::Specification.new do |s|
s.name = 'logstash-output-elasticsearch'
s.version = '12.1.2'
s.version = '12.1.3'
s.licenses = ['apache-2.0']
s.summary = "Stores logs in Elasticsearch"
s.description = "This gem is a Logstash plugin required to be installed on top of the Logstash core pipeline using $LS_HOME/bin/logstash-plugin install gemname. This gem is not a stand-alone program"
Expand Down
Loading