Add an opt-in partial-result mode for aggregations on text/keyword mapping conflicts

[ahkcs]

Description

On the Calcite PPL path, an aggregation grouped on a field that is mapped keyword in some indices of a wildcard pattern and text in others cannot use native pushdown. The multi-index type merge collapses the field to text-without-.keyword, which has no doc values, so the aggregation runs as a per-document _source script over every document — correct, but a full-index scan that is orders of magnitude slower on a wide pattern.

This PR adds an opt-in mode that returns a fast, partial answer instead: it aggregates over only the subset of indices where the field is natively aggregatable (keyword) and attaches a warning naming the ones it excluded.

So the choice becomes complete-but-slow (default) vs fast-but-partial (opt-in) — both correct, differing in coverage and speed.

How it works

1. Warning channel. Successful PPL JSON responses gain an optional warnings: [{type, message, detail}] array, emitted only when non-empty (existing responses are byte-for-byte unchanged):

   "warnings": [{
     "type": "PARTIAL_RESULT",
     "message": "Results exclude 1 of 2 indices due to a text/keyword mapping conflict on [applicationid].",
     "detail": "[applicationid] is not mapped as keyword in every queried index, so these indices were excluded from the aggregation: [logs-text]. Map [applicationid] as keyword across all indices to include them."
   }]

2. Partial-result plan (PartialResultAggregatePushdown). When the mode is on and the group key is a text/keyword conflict, the scan is narrowed to the aggregatable index subset and the aggregation pushed down over just that subset (size = 0, no PIT). The partitioning logic is unit-tested in isolation.
3. Per-request override. A partial_result boolean in the query body (mirroring profile) overrides the cluster setting for one query; absent → cluster setting decides.

Behavior

Cluster setting plugins.query.partial_result.on_mapping_conflict.enabled (default false):

Query	Off (default)	On
stats count() by <conflict field>	complete result, slow (_source scan of all docs)	fast result over the keyword subset + PARTIAL_RESULT warning
no-conflict / single-index aggregation	complete, no warning	complete, no warning (unchanged)
any of the above with format=csv	as above	falls through to the complete result (CSV has no warning channel)

Key points

• Opt-in, default off. A partial result is knowingly incomplete, so it never happens silently; with the setting off the change is behavior-preserving.
• Never degrades silently. Only the JSON shape carries warnings, so CSV/RAW/VIZ fall through to the complete result rather than dropping data unannounced.
• Deterministic selection. Keep the keyword group whenever one exists; otherwise the text-with-.keyword group; always exclude bare text. The result never depends on how many indices of each type match.
• Calcite PPL path only; V2/legacy untouched.

Not in scope: recovering an excluded but aggregatable group (text-with-.keyword alongside a keyword group) — that needs a per-group split-and-union, a larger separate change. This is why the warning recommends mapping the field as keyword everywhere.

Related Issues

Check List

• New functionality includes testing (unit + integration).
• New functionality has been documented (docs/user/admin/settings.rst).
• New functionality has javadoc added.
• API changes companion pull request created.
• Commits are signed per the DCO using --signoff or -s.
• Public documentation issue/PR created.

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.

[github-actions]

PR Reviewer Guide 🔍

(Review updated until commit d647281)

Here are some key observations to aid the review process:

🧪 PR contains tests

🔒 No security concerns identified

✅ No TODO sections

🔀 No multiple PR themes

⚡ Recommended focus areas for review Possible Issue The stringOf helper catches all RuntimeException to handle non-string content, but this is too broad. If content.stringValue() throws an unexpected runtime exception (e.g., NullPointerException from a bug in the content implementation), the catch block silently converts it to a string representation of the underlying object, masking the real error. This could hide legitimate bugs in the content-handling code. The catch should be narrowed to the specific exception type that signals "not a string value" (e.g., ClassCastException or a domain-specific exception), or the method should document why swallowing all runtime exceptions is safe here. private static String stringOf(Content content) { try { return content.stringValue(); } catch (RuntimeException e) { // Not a string value (e.g. a numeric aggregation bucket key landing in a text column via a // partial-result narrowing) -- render its string form instead of failing the cast to null. return String.valueOf(content.objectValue()); } } Possible Issue resolvePartitionFields returns null when any group key is a pure constant (no field references). The caller tryPartialResultAggregate then skips partial-result mode entirely. However, a query like stats count() by 'literal', city has one constant key and one field key (city). The current logic returns null because the constant key contributes no refs, so the entire aggregation is left un-pushed even though city alone could be partitioned on. The method should either skip constant keys and return the non-empty field list, or the caller should handle mixed cases. As written, a single constant in a multi-field group key disables partial mode unnecessarily. @Nullable private List<String> resolvePartitionFields(Aggregate aggregate, @Nullable Project project) { List<String> scanFields = getRowType().getFieldNames(); List<String> fields = new ArrayList<>(); for (int group : aggregate.getGroupSet()) { Set<Integer> refs = new LinkedHashSet<>(); if (project == null) { refs.add(group); // group key indexes directly into the scan } else { project .getProjects() .get(group) .accept( new RexVisitorImpl<Void>(true) { @Override public Void visitInputRef(RexInputRef ref) { refs.add(ref.getIndex()); return null; } }); } if (refs.isEmpty()) { return null; // constant group key -> nothing to partition on } for (int ref : refs) { String name = scanFields.get(ref); if (!fields.contains(name)) { fields.add(name); } } } return fields; } Possible Issue resolveBucketSignature returns null if any field in bucketNames is absent from the index mapping. This means an index where one group field is missing (but others are present and aggregatable) is excluded entirely. For a multi-field group key like stats count() by city, region, if an index has city as keyword but no region field at all, that index is excluded even though city alone is aggregatable. The logic conflates "field is non-aggregatable" with "field is absent," but absence is a schema mismatch that might warrant a different treatment (e.g., a separate warning or allowing the index if the present fields are aggregatable). As written, a missing field silently excludes the index without distinguishing the reason. static String resolveBucketSignature( Map<String, OpenSearchDataType> flatMapping, List<String> bucketNames) { List<String> tokens = new ArrayList<>(); for (String field : bucketNames) { OpenSearchDataType type = flatMapping.get(field); if (type == null) { return null; // field absent here -> not aggregatable } MappingType mappingType = type.getMappingType(); if (mappingType == MappingType.Text || mappingType == MappingType.MatchOnlyText) { return null; // text family (incl. text-with-.keyword) collapses to bare text on merge } tokens.add("t:" + mappingType); // aggregatable type (keyword, numeric, date, boolean, ip) } return String.join("|", tokens); }

[github-actions]

PR Code Suggestions ✨

Latest suggestions up to d647281

Explore these optional code suggestions:

Category	Suggestion	Impact
General	Validate kept indices before use Verify that plan.keptIndices() is not empty before constructing the narrowed index. An empty kept-indices list would create an invalid index pattern and should be treated as a planning failure rather than proceeding to construct an index. opensearch/src/main/java/org/opensearch/sql/opensearch/storage/scan/CalciteLogicalIndexScan.java [511-554] private AbstractRelNode tryPartialResultAggregate( Aggregate aggregate, @Nullable Project project, List<String> partitionFields) { if (!QueryContext.isPartialResultEnabled(osIndex.getSettings())) { return null; } // A format with no warnings channel (CSV/RAW/VIZ) must not silently drop indices. if (!QueryContext.isWarningsSupported()) { return null; } try { Map<String, IndexMapping> mappings = osIndex.getIndexMappings(); PartialResultAggregatePushdown.Plan plan = PartialResultAggregatePushdown.plan(partitionFields, mappings); - if (plan == null) { + if (plan == null || plan.keptIndices().isEmpty()) { return null; } OpenSearchIndex narrowedIndex = new OpenSearchIndex( osIndex.getClient(), osIndex.getSettings(), String.join(",", plan.keptIndices())); - CalciteLogicalIndexScan narrowedScan = - new CalciteLogicalIndexScan( - getCluster(), - traitSet, - hints, - table, - narrowedIndex, - getRowType(), - pushDownContext.cloneWithOsIndex(narrowedIndex)); - // allowPartialFallback=false: the subset is already narrowed, so keep this one-shot. - AbstractRelNode pushed = narrowedScan.pushDownAggregate(aggregate, project, false); - if (pushed == null) { - return null; // narrowed subset still can't push down -> leave un-pushed - } - - CalcitePlanContext.addWarning(plan.warning()); - return pushed; + ... } catch (Exception e) { if (LOG.isDebugEnabled()) { LOG.debug("Cannot apply partial-result aggregate pushdown for {}", aggregate, e); } return null; } } Suggestion importance[1-10]: 7 __ Why: The suggestion to verify plan.keptIndices() is not empty before constructing the narrowed index is a valid defensive check. However, the plan() method already returns null when there's no valid partition (including when kept indices would be empty), so this check is somewhat redundant but still adds clarity and safety.	Medium
	Narrow exception catch scope Catching all RuntimeException is too broad and may hide unexpected errors. Consider catching only the specific exception type that content.stringValue() throws when the value is not a string, or add logging to track when this fallback path is taken. opensearch/src/main/java/org/opensearch/sql/opensearch/data/value/OpenSearchExprValueFactory.java [248-256] private static String stringOf(Content content) { try { return content.stringValue(); - } catch (RuntimeException e) { + } catch (ClassCastException e) { // Not a string value (e.g. a numeric aggregation bucket key landing in a text column via a // partial-result narrowing) -- render its string form instead of failing the cast to null. return String.valueOf(content.objectValue()); } } Suggestion importance[1-10]: 6 __ Why: Catching RuntimeException is broad, but the suggestion to narrow to ClassCastException may not be accurate without knowing the exact exception thrown by content.stringValue(). The suggestion is reasonable for improving error handling specificity, though it assumes the exception type without verification.	Low

---

Previous suggestions

Suggestions up to commit 97aa640

Category	Suggestion	Impact
General	Clear all request context state The clearRequestScopedState method does not clear the request ID set by QueryContext.addRequestId(). This could cause request IDs to leak across pooled threads, leading to incorrect request tracking. plugin/src/main/java/org/opensearch/sql/plugin/transport/TransportPPLQueryAction.java [452-456] private static void clearRequestScopedState() { QueryProfiling.clear(); QueryContext.setPartialResultOverride(null); QueryContext.setWarningsSupported(false); + QueryContext.clear(); } Suggestion importance[1-10]: 8 __ Why: This identifies a potential resource leak. The method clears partial result and warnings state but does not clear the request ID set by QueryContext.addRequestId(). On pooled threads, this could cause request IDs to leak to subsequent queries, leading to incorrect tracking and potential confusion in logs or monitoring.	Medium
	Catch specific exception type Catching RuntimeException is too broad and may hide unexpected errors. Consider catching a more specific exception type that content.stringValue() throws when the value is not a string, or add logging to track when this fallback is triggered. opensearch/src/main/java/org/opensearch/sql/opensearch/data/value/OpenSearchExprValueFactory.java [248-256] private static String stringOf(Content content) { try { return content.stringValue(); - } catch (RuntimeException e) { + } catch (ClassCastException e) { // Not a string value (e.g. a numeric aggregation bucket key landing in a text column via a // partial-result narrowing) -- render its string form instead of failing the cast to null. return String.valueOf(content.objectValue()); } } Suggestion importance[1-10]: 6 __ Why: Catching RuntimeException is overly broad and could mask unexpected errors. However, the suggestion to catch ClassCastException may not be the correct specific exception type without knowing what content.stringValue() actually throws. The suggestion is reasonable but may need verification of the actual exception type.	Low
Possible issue	Validate kept indices not empty Verify that plan.keptIndices() is not empty before constructing the narrowed index. An empty kept-indices list would create an invalid index pattern and could cause downstream failures. opensearch/src/main/java/org/opensearch/sql/opensearch/storage/scan/CalciteLogicalIndexScan.java [511-543] private AbstractRelNode tryPartialResultAggregate( Aggregate aggregate, @Nullable Project project, List<String> partitionFields) { if (!QueryContext.isPartialResultEnabled(osIndex.getSettings())) { return null; } - // A format with no warnings channel (CSV/RAW/VIZ) must not silently drop indices. if (!QueryContext.isWarningsSupported()) { return null; } try { Map<String, IndexMapping> mappings = osIndex.getIndexMappings(); PartialResultAggregatePushdown.Plan plan = PartialResultAggregatePushdown.plan(partitionFields, mappings); - if (plan == null) { + if (plan == null || plan.keptIndices().isEmpty()) { return null; } OpenSearchIndex narrowedIndex = new OpenSearchIndex( osIndex.getClient(), osIndex.getSettings(), String.join(",", plan.keptIndices())); ... } catch (Exception e) { ... } } Suggestion importance[1-10]: 7 __ Why: This is a valid defensive check. While PartialResultAggregatePushdown.plan likely returns null when no indices can be kept, explicitly checking for an empty keptIndices() list prevents potential issues with constructing an invalid index pattern (empty string from String.join).	Medium

Suggestions up to commit 3ffd741

Category	Suggestion	Impact
General	Reset profiling flag in cleanup Ensure that QueryContext.setProfile(false) is also called to reset the profiling flag, since QueryContext.setProfile(transformedRequest.profile()) is set at the start of the request. This prevents the profiling state from leaking to subsequent requests on the same thread. plugin/src/main/java/org/opensearch/sql/plugin/transport/TransportPPLQueryAction.java [452-456] private static void clearRequestScopedState() { QueryProfiling.clear(); + QueryContext.setProfile(false); QueryContext.setPartialResultOverride(null); QueryContext.setWarningsSupported(false); } Suggestion importance[1-10]: 8 __ Why: This is a valid concern about thread-local state leakage. Since QueryContext.setProfile() is set at the start of the request, it should be reset in cleanup to prevent the profiling state from affecting subsequent requests on pooled threads. This is an important correctness issue for thread safety.	Medium
	Avoid catching broad RuntimeException Catching RuntimeException is too broad and may mask unexpected errors. Catch the specific exception type that content.stringValue() throws when the value is not a string, or verify the content type before attempting the cast. opensearch/src/main/java/org/opensearch/sql/opensearch/data/value/OpenSearchExprValueFactory.java [248-256] private static String stringOf(Content content) { - try { + if (content.isString()) { return content.stringValue(); - } catch (RuntimeException e) { - // Not a string value (e.g. a numeric aggregation bucket key landing in a text column via a - // partial-result narrowing) -- render its string form instead of failing the cast to null. - return String.valueOf(content.objectValue()); } + // Not a string value (e.g. a numeric aggregation bucket key landing in a text column via a + // partial-result narrowing) -- render its string form instead of failing the cast to null. + return String.valueOf(content.objectValue()); } Suggestion importance[1-10]: 7 __ Why: The suggestion correctly identifies that catching RuntimeException is overly broad. Using a type check (content.isString()) before calling stringValue() is a cleaner approach that avoids masking unexpected errors. However, the impact is moderate since the current code does handle the case correctly, just less elegantly.	Medium
	Validate mappings before use Verify that osIndex.getIndexMappings() returns a non-null and non-empty map before passing it to PartialResultAggregatePushdown.plan(). If the mappings are empty or null, the method should return early to avoid unnecessary processing. opensearch/src/main/java/org/opensearch/sql/opensearch/storage/scan/CalciteLogicalIndexScan.java [511-526] private AbstractRelNode tryPartialResultAggregate( Aggregate aggregate, @Nullable Project project, List<String> partitionFields) { if (!QueryContext.isPartialResultEnabled(osIndex.getSettings())) { return null; } // A format with no warnings channel (CSV/RAW/VIZ) must not silently drop indices. if (!QueryContext.isWarningsSupported()) { return null; } try { Map<String, IndexMapping> mappings = osIndex.getIndexMappings(); + if (mappings == null || mappings.isEmpty()) { + return null; + } PartialResultAggregatePushdown.Plan plan = PartialResultAggregatePushdown.plan(partitionFields, mappings); if (plan == null) { return null; } Suggestion importance[1-10]: 6 __ Why: Adding a null/empty check for mappings before passing to plan() is defensive programming. However, PartialResultAggregatePushdown.plan() already handles the case where mappings.size() < 2 returns null, so this check is somewhat redundant. The suggestion improves clarity but has limited practical impact.	Low

Suggestions up to commit 77fa72d

Category	Suggestion	Impact
Possible issue	Validate non-empty kept indices list The narrowedIndex is created with a comma-joined list of index names without validation. If plan.keptIndices() is empty, this will create an index with an empty string name, which could cause unexpected behavior. Verify that keptIndices() is non-empty before creating the narrowed index. opensearch/src/main/java/org/opensearch/sql/opensearch/storage/scan/CalciteLogicalIndexScan.java [520-530] -private AbstractRelNode tryPartialResultAggregate( - Aggregate aggregate, @Nullable Project project, List<String> partitionFields) { - if (!QueryContext.isPartialResultEnabled(osIndex.getSettings())) { - return null; - } - // A format with no warnings channel (CSV/RAW/VIZ) must not silently drop indices. - if (!QueryContext.isWarningsSupported()) { - return null; - } - try { - Map<String, IndexMapping> mappings = osIndex.getIndexMappings(); - PartialResultAggregatePushdown.Plan plan = - PartialResultAggregatePushdown.plan(partitionFields, mappings); - if (plan == null) { - return null; - } - - OpenSearchIndex narrowedIndex = - new OpenSearchIndex( - osIndex.getClient(), osIndex.getSettings(), String.join(",", plan.keptIndices())); - ... - } +if (plan == null) { + return null; +} +if (plan.keptIndices().isEmpty()) { + return null; } +OpenSearchIndex narrowedIndex = + new OpenSearchIndex( + osIndex.getClient(), osIndex.getSettings(), String.join(",", plan.keptIndices())); + Suggestion importance[1-10]: 7 __ Why: Valid concern about potential empty keptIndices() list leading to an invalid index name. However, the suggestion asks to verify/ensure a condition rather than fixing a definite bug, so it should not score above 7.	Medium
	Add null check for mapping type The method assumes type.getMappingType() never returns null, but if it does, the subsequent checks will fail with a NullPointerException. Add a null check for mappingType before using it in comparisons to prevent potential crashes. opensearch/src/main/java/org/opensearch/sql/opensearch/storage/scan/PartialResultAggregatePushdown.java [110-125] -static String resolveBucketSignature( - Map<String, OpenSearchDataType> flatMapping, List<String> bucketNames) { - List<String> tokens = new ArrayList<>(); - for (String field : bucketNames) { - OpenSearchDataType type = flatMapping.get(field); - if (type == null) { - return null; // field absent here -> not aggregatable - } - MappingType mappingType = type.getMappingType(); - if (mappingType == MappingType.Keyword) { - tokens.add("kw"); - } else if (mappingType == MappingType.Text || mappingType == MappingType.MatchOnlyText) { - return null; // text family (incl. text-with-.keyword) collapses to bare text on merge - } else { - tokens.add("t:" + mappingType); // other aggregatable type (numeric, date, boolean, ip) - } - } - return String.join("|", tokens); +MappingType mappingType = type.getMappingType(); +if (mappingType == null) { + return null; // mapping type unavailable -> not aggregatable +} +if (mappingType == MappingType.Keyword) { + tokens.add("kw"); +} else if (mappingType == MappingType.Text || mappingType == MappingType.MatchOnlyText) { + return null; +} else { + tokens.add("t:" + mappingType); } Suggestion importance[1-10]: 7 __ Why: The suggestion addresses a potential NullPointerException if getMappingType() returns null. This is a reasonable defensive check, though it's unclear if this scenario can actually occur in practice. As error handling, it should not score above 8, and since it's asking to verify a condition, it caps at 7.	Medium

Suggestions up to commit 0db5aa7

Category	Suggestion	Impact
General	Pre-flatten mappings to avoid redundant traversals The traverseAndFlatten call inside the loop is invoked once per index, which can be expensive for large wildcard patterns. Consider pre-flattening all mappings once before the loop, or caching the flattened result in IndexMapping itself to avoid redundant traversals. opensearch/src/main/java/org/opensearch/sql/opensearch/storage/scan/PartialResultAggregatePushdown.java [71-76] static Plan plan(List<String> bucketNames, Map<String, IndexMapping> mappings) { if (bucketNames.isEmpty() || mappings.size() < 2) { return null; } + // Pre-flatten all mappings once + Map<String, Map<String, OpenSearchDataType>> flatMappings = new LinkedHashMap<>(); + for (Map.Entry<String, IndexMapping> entry : mappings.entrySet()) { + flatMappings.put(entry.getKey(), + OpenSearchDataType.traverseAndFlatten(entry.getValue().getFieldMappings())); + } ... - for (Map.Entry<String, IndexMapping> entry : mappings.entrySet()) { - Map<String, OpenSearchDataType> flatMapping = - OpenSearchDataType.traverseAndFlatten(entry.getValue().getFieldMappings()); - switch (resolveBucketMapping(flatMapping, bucketNames)) { + for (Map.Entry<String, Map<String, OpenSearchDataType>> entry : flatMappings.entrySet()) { + switch (resolveBucketMapping(entry.getValue(), bucketNames)) { Suggestion importance[1-10]: 5 __ Why: Valid optimization to avoid repeated traverseAndFlatten calls in the loop. However, the impact is moderate since this only runs when partial mode applies (opt-in setting, multi-index conflict), and the flattening cost is typically small compared to the query execution itself.	Low
	Guard against stale thread-local state outside request lifecycle The method reads from ThreadContext without verifying that the context is properly initialized for the current request. If called outside a request lifecycle (e.g., during initialization or in a background thread), it may return stale data from a previous request on the same pooled thread. Add a guard or document the lifecycle requirement. common/src/main/java/org/opensearch/sql/common/utils/QueryContext.java [134-140] public static boolean isPartialResultEnabled(Settings settings) { - String override = ThreadContext.get(PARTIAL_RESULT_OVERRIDE_KEY); - if (override != null) { - return Boolean.parseBoolean(override); + // Only check override if a request ID is set (indicating active request context) + if (ThreadContext.get(REQUEST_ID_KEY) != null) { + String override = ThreadContext.get(PARTIAL_RESULT_OVERRIDE_KEY); + if (override != null) { + return Boolean.parseBoolean(override); + } } return settings.getSettingValue(Settings.Key.PARTIAL_RESULT_ON_MAPPING_CONFLICT); } Suggestion importance[1-10]: 4 __ Why: The concern about stale state is valid, but the PR already addresses cleanup in TransportPPLQueryAction.clearRequestScopedState() (lines 406-408 in TransportPPLQueryAction.java), which clears the override after each request. The suggested guard adds defensive depth but may be unnecessary given the existing cleanup.	Low
	Defer partial-result attempt until after normal pushdown fails The partial-result path is attempted before the normal aggregation analysis, which means it may fire even when the aggregation would have succeeded normally. Consider moving this check after the initial AggregateAnalyzer attempt, so partial mode only activates when a conflict actually prevents pushdown. This avoids unnecessary partitioning overhead for queries that don't need it. opensearch/src/main/java/org/opensearch/sql/opensearch/storage/scan/CalciteLogicalIndexScan.java [420-427] -List<String> partitionFields = resolvePartitionFields(aggregate, project); -if (partitionFields != null) { - AbstractRelNode partial = tryPartialResultAggregate(aggregate, project, partitionFields); - if (partial != null) { - return partial; +try { + // ... existing analyze logic ... + AggregateAnalyzer.AggregateBuilderHelper helper = ...; + // If analysis succeeds, return the pushed aggregate +} catch (Exception e) { + // Only try partial mode if normal pushdown failed + List<String> partitionFields = resolvePartitionFields(aggregate, project); + if (partitionFields != null && allowPartialFallback) { + AbstractRelNode partial = tryPartialResultAggregate(aggregate, project, partitionFields); + if (partial != null) { + return partial; + } } + throw e; } Suggestion importance[1-10]: 3 __ Why: The suggestion misunderstands the PR's design. The comment at line 418-419 explicitly states partial mode must be tried before analyze because since #5646 a text/keyword conflict pushes down as a slow script instead of failing, so a post-failure fallback would never fire. Moving the check after analysis would break the feature.	Low

Suggestions up to commit cafac5f

Category	Suggestion	Impact
Possible issue	Clear all thread-local state consistently The clearRequestScopedState method clears thread-local state but does not clear the CalcitePlanContext warnings that are also thread-local. This could cause warnings from one request to leak into the next request on a pooled thread. Ensure all thread-local state is cleared consistently. plugin/src/main/java/org/opensearch/sql/plugin/transport/TransportPPLQueryAction.java [404-408] private static void clearRequestScopedState() { QueryProfiling.clear(); QueryContext.setPartialResultOverride(null); QueryContext.setWarningsSupported(false); + CalcitePlanContext.drainWarnings(); } Suggestion importance[1-10]: 8 __ Why: Valid concern: CalcitePlanContext.drainWarnings() is called in OpenSearchExecutionEngine.buildResultSet (line 495), but if an exception occurs before that point, warnings could leak onto the next pooled thread. Adding drainWarnings() to clearRequestScopedState ensures warnings are always cleared, preventing potential cross-request contamination.	Medium
General	Cache flattened mappings to avoid redundant operations The traverseAndFlatten operation is called for every index mapping in the loop, which could be expensive for large wildcard patterns with many indices. Consider caching the flattened mappings or performing this operation once per unique mapping structure to improve performance. opensearch/src/main/java/org/opensearch/sql/opensearch/storage/scan/PartialResultAggregatePushdown.java [68-74] static Plan plan(List<String> bucketNames, Map<String, IndexMapping> mappings) { if (bucketNames.isEmpty() || mappings.size() < 2) { return null; } - ... + + Map<String, Map<String, OpenSearchDataType>> flattenedMappings = new LinkedHashMap<>(); for (Map.Entry<String, IndexMapping> entry : mappings.entrySet()) { - Map<String, OpenSearchDataType> flatMapping = - OpenSearchDataType.traverseAndFlatten(entry.getValue().getFieldMappings()); - String signature = resolveBucketSignature(flatMapping, bucketNames); + flattenedMappings.put( + entry.getKey(), + OpenSearchDataType.traverseAndFlatten(entry.getValue().getFieldMappings())); + } + + Map<String, List<String>> aggregatableGroups = new LinkedHashMap<>(); + List<String> nonAggregatable = new ArrayList<>(); + for (Map.Entry<String, Map<String, OpenSearchDataType>> entry : flattenedMappings.entrySet()) { + String signature = resolveBucketSignature(entry.getValue(), bucketNames); Suggestion importance[1-10]: 4 __ Why: The suggestion correctly identifies that traverseAndFlatten is called in a loop, but the optimization is minor: the method is already efficient for typical wildcard patterns, and the added complexity of caching may not justify the marginal performance gain. The existing code is clear and correct.	Low
	Move partial-result attempt after normal pushdown The partial-result fallback is attempted before the main aggregation analysis, which means it may fire even when the aggregation would have succeeded normally. Consider moving the partial-result attempt into the catch block or after detecting a specific failure condition to avoid unnecessary overhead when the aggregation can be pushed down cleanly. opensearch/src/main/java/org/opensearch/sql/opensearch/storage/scan/CalciteLogicalIndexScan.java [390-428] private AbstractRelNode pushDownAggregate( Aggregate aggregate, @Nullable Project project, boolean allowPartialFallback) { try { CalciteLogicalIndexScan newScan = new CalciteLogicalIndexScan( getCluster(), ... + // Attempt normal pushdown first + AggregateAnalyzer.AggregateBuilderHelper helper = ... + // Only try partial fallback if normal pushdown fails + } catch (Exception e) { if (allowPartialFallback) { List<String> partitionFields = resolvePartitionFields(aggregate, project); if (partitionFields != null) { AbstractRelNode partial = tryPartialResultAggregate(aggregate, project, partitionFields); if (partial != null) { return partial; } } } + if (LOG.isDebugEnabled()) { + LOG.debug("Cannot pushdown the aggregate {}", aggregate, e); + } + } + return null; +} Suggestion importance[1-10]: 3 __ Why: The suggestion misunderstands the design: the comment at line 418 explicitly states partial mode must be tried before analyze because since #5646 a text/keyword conflict pushes down as a slow script instead of failing. Moving it to the catch block would defeat the purpose, as the fallback would never fire. The current placement is intentional.	Low

[github-actions]

Persistent review updated to latest commit dad3bb3

[github-actions]

Persistent review updated to latest commit 078c949

[github-actions]

Persistent review updated to latest commit 83fd527

[github-actions]

Persistent review updated to latest commit 2a3eab8

[github-actions]

PR Code Analyzer ❗

AI-powered 'Code-Diff-Analyzer' found issues on commit d647281.

Path	Line	Severity	Description
integ-test/src/test/resources/expectedOutput/calcite/explain_partial_filter_script_push.json	4	low	A segment of the long base64-encoded serialized Java object string changed ('cJjvMoC+gU1' → 'qHvcFFLzXT') in the expected physical-plan fixture. The same substitution appears in the YAML fixture as well, consistent with a serialVersionUID shift caused by the new cloneDeep() method on OpenSearchDataType. The blobs are opaque without decoding; the change is plausibly explained by the class modification but cannot be fully verified from diff context alone.
opensearch/src/main/java/org/opensearch/sql/opensearch/data/value/OpenSearchExprValueFactory.java	248	low	The new stringOf() helper silently catches all RuntimeException and falls back to String.valueOf(content.objectValue()), broadly suppressing unexpected type conversion failures. This is a defensive coding pattern rather than a deliberate threat, but the wide catch could mask unintended data coercions in edge cases not covered by partial-result narrowing.

The table above displays the top 10 most important findings.

Total: 2 | Critical: 0 | High: 0 | Medium: 0 | Low: 2

---

Pull Requests Author(s): Please update your Pull Request according to the report above.

Repository Maintainer(s): You can bypass diff analyzer by adding label skip-diff-analyzer after reviewing the changes carefully, then re-run failed actions. To re-enable the analyzer, remove the label, then re-run all actions.

---

⚠️ Note: The Code-Diff-Analyzer helps protect against potentially harmful code patterns. Please ensure you have thoroughly reviewed the changes beforehand.

Thanks.

[github-actions]

Persistent review updated to latest commit 19b6187

[github-actions]

Persistent review updated to latest commit 51220fe

[anasalkouz]

1. Is this only applicable for non-mustang?
2. Is this only limited to text vs keyward use-case? can we extend the scope?
3. Shall we have a role on the inspect query feature to suggest customer to enable this parital result flag to optimize performance if the query fails to push down?
4. Can we have performance benchmark for the 3 cases? with no pushdown, with text pushdown, and with partial results?

[ahkcs]

Performance benchmark: partial results vs. today vs. scripted text pushdown (#5646)

Comparing three responses to the mapping-conflict PIT-exhaustion case (an aggregation groups on a field mapped keyword in some indices of a wildcard pattern and text in others):

• A — today. The type merge collapses the field to text-without-doc-values, aggregate pushdown is lost, and the engine scans every document per shard — opening a Point-In-Time (PIT) context on every shard — and aggregates client-side. Complete answer; trips search.max_open_pit_context on wide patterns.
• B — scripted text pushdown (Push down aggregation on text field without .keyword sub-field #5646). Routes the group key through a Calcite _source script pushed down with size=0. No PIT. Complete answer.
• C — partial results (this PR). Narrows the scan to the aggregatable (keyword) subset, pushes down natively (size=0, no PIT), returns a partial answer plus a PARTIAL_RESULT warning naming the excluded indices.

These do not compute the same thing, so every latency figure is paired with a completeness column.

Test setup

Cluster	Single node, OpenSearch 3.8.0-SNAPSHOT, 2 GB heap (raised from the 512 MB dev default so A fails on PIT, not the query memory circuit breaker)
Engine	Calcite path (plugins.calcite.enabled=true) — the only path in scope for this PR
A & C	Same build (this PR); A = partial_result:false, C = partial_result:true
B	#5646 build, separate run on identical seeded data
Iterations	30 measured + 5 warmup per query, serial (clean per-query latency + exact PIT deltas)
Latency	Client-side wall-clock of the _plugins/_ppl call
PIT/query	Delta of the cumulative point_in_time_total node stat

Datasets (deterministic, seed = 42):

• wide — 40 keyword + 4 bare-text indices, 2 shards each (88 shards), 5,000 docs/index (220,000 total). A wide wildcard pattern where 88 shards exceeds any realistic PIT limit.
• small — 1 keyword + 1 text, 1 shard each, 20,000 docs/index. Control below the PIT limit.
• flat — same as small but the conflict field is top-level (appid), not nested. (See the note on B.)

Conflict field for wide/small is a nested resource.attributes.applicationid; for flat it is top-level appid.

Latency p50 / p90 / p99 (ms)

"Today" (A) has two modes on the same query, decided by whether the shard count exceeds search.max_open_pit_context:

Query	A: PIT opened, under limit (no 500)	A: PIT limit exceeded	B: #5646 (script)	C: partial (this PR)	Completeness of C	PIT/query (A)
wide stats	441 / 463 / 494	FAIL — 500	112 / 150 / 271	11 / 12 / 13	91.2% (200k/219k)	88
wide top	439 / 455 / 486	FAIL — 500	123 / 149 / 293	16 / 18 / 21	91.2%	88
small stats	90 / 110 / 115	92 / 111 / 114	32 / 36 / 43	5 / 6 / 7	50% (20k/40k)	2
small top	96 / 116 / 122	94 / 102 / 113	37 / 44 / 53	11 / 14 / 15	50%	2
flat stats	58 / 78 / 82	56 / 63 / 75	20 / 23 / 42	4 / 5 / 5	50% (20k/40k)	4
flat top	60 / 67 / 86	57 / 62 / 80	27 / 30 / 31	10 / 12 / 14	50%	4

The "under limit" column used max_open_pit_context=500; "exceeded" used =10. A only fails where shard count crosses the limit (wide, 88 shards). Small/flat stay under and complete — but still open PITs and run 8–20× slower than C. Error rate in the exceeded regime: A = 100% on wide, C = 0% everywhere (never opens a PIT).

Completeness (sum of count() across all buckets)

Dataset	A (complete)	B	C (partial)	C completeness
wide, nested field	219,328	220,000 but 1 null bucket (grouping lost)	200,000	91.2%
small, nested field	40,000	40,000 but 1 null bucket	20,000	50%
flat field	40,000	40,000, 50 buckets (correct)	20,000	50%

Why the latencies differ (mechanism)

A leaves the aggregate above the scan (explain shows requestedTotalSize=2147483647): every matching document is streamed out of every shard over PIT cursors into the coordinator JVM and counted there — cost scales with document count. B and C fuse the aggregate into the scan (size=0), so the count runs inside each shard and only bucket results cross the wire — cost scales with bucket count, and no PIT is opened. B groups on a per-document _source script; C groups on native keyword doc values, which is why C stays ~2–5× ahead of B even where both push down.

Takeaways

1. When it runs, C is fastest (~34× vs A, ~10× vs B on wide) — but that speed is the partial answer: it excludes the non-aggregatable indices. On wide that is an 8.8% undercount; where the text indices hold half the data, 50%. Always accompanied by the PARTIAL_RESULT warning.
2. In the low-PIT-budget regime, A fails outright (100% errors on wide). B and C never open a PIT.
3. B is complete and PIT-free on flat fields and is the natural default there. On the nested dotted field, B in its current state grouped all documents into a single null bucket (complete count, grouping lost) — worth verifying whether the scripted _source reader resolves nested dotted paths. This PR's producer resolves the nested path.

C is intended as an opt-in escape hatch (default off) for the widest patterns / lowest PIT budgets where a knowingly-partial, clearly-warned answer is preferable to a slow scan or a 500 — complementary to, not competing with, a complete-answer pushdown fix.

_{Single-node, laptop-scale absolutes; the ratios and the PIT / completeness / error-rate columns are the transferable results.}

[ahkcs]

1. Is this only applicable for non-mustang?
2. Is this only limited to text vs keyward use-case? can we extend the scope?
3. Shall we have a role on the inspect query feature to suggest customer to enable this parital result flag to optimize performance if the query fails to push down?
4. Can we have performance benchmark for the 3 cases? with no pushdown, with text pushdown, and with partial results?

1. Yes, currently it's only applicable for Calcite path.
2. Today it's deliberately scoped to the text/keyword conflict, it can be extended, and the shape generalizes cleanly if we have more partial result use cases.
3. We can add that recommendation/suggestion
4. link for performance benchmarking: Add an opt-in partial-result mode for aggregations on text/keyword mapping conflicts #5657 (comment)

[github-actions]

Persistent review updated to latest commit 97aa640

[github-actions]

Persistent review updated to latest commit d647281