gpupreagg: Improve num_groups estimation for arrow_fdw with sqrt-base… - #949
gpupreagg: Improve num_groups estimation for arrow_fdw with sqrt-base…#949a-hirota wants to merge 1 commit into
Conversation
There was a problem hiding this comment.
Pull Request Overview
This PR improves GROUP BY cardinality estimation for arrow_fdw/parquet_fdw foreign tables by implementing square-root based dampening to reduce overestimated group counts that make GpuPreAgg less likely to be selected by the PostgreSQL planner.
- Implements sqrt-based reduction algorithm to address PostgreSQL's tendency to overestimate num_groups by multiplying n_distinct values across GROUP BY columns
- Adds configurable dampening via new GUC parameter
pg_strom.groupby_reduction_factor(default 0.7, range 0.1-2.0) - Restricts the optimization to arrow_fdw/parquet_fdw foreign tables only to avoid requiring expensive ANALYZE operations
Tip: Customize your code reviews with copilot-instructions.md. Create the file or learn how to get started.
| double excess; | ||
| double reduced_excess; | ||
| double adjusted; | ||
|
|
There was a problem hiding this comment.
Potential division by zero if input_nrows is 0. Add a guard to check if input_nrows > 0 before performing the division.
| /* Guard against division by zero */ | |
| if (input_nrows <= 0.0) | |
| return estimated_groups; |
| reduced_excess = sqrt(excess) * pgstrom_groupby_reduction_factor; | ||
| adjusted = baseline + reduced_excess; | ||
|
|
||
| elog(DEBUG1, "GpuPreAgg: reduced num_groups from %.0f to %.0f " |
There was a problem hiding this comment.
Corrected spacing in format string - removed extra space before opening parenthesis.
| elog(DEBUG1, "GpuPreAgg: reduced num_groups from %.0f to %.0f " | |
| elog(DEBUG1, "GpuPreAgg: reduced num_groups from %.0f to %.0f" |
| /* Adjust num_groups if it seems overestimated due to combinatorial explosion. | ||
| * Apply square-root based dampening to reduce overestimated group counts. | ||
| * Only applies to arrow_fdw/parquet_fdw foreign tables (because ANALYZE | ||
| * requires decompression which consumes GPU memory, so we avoid ANALYZE | ||
| * and fix the default cardinality estimation instead). | ||
| */ | ||
| { |
There was a problem hiding this comment.
[nitpick] This block-level comment should be moved above the opening brace of the block it describes (line 2324) for better code organization and readability.
| /* Adjust num_groups if it seems overestimated due to combinatorial explosion. | |
| * Apply square-root based dampening to reduce overestimated group counts. | |
| * Only applies to arrow_fdw/parquet_fdw foreign tables (because ANALYZE | |
| * requires decompression which consumes GPU memory, so we avoid ANALYZE | |
| * and fix the default cardinality estimation instead). | |
| */ | |
| { | |
| { | |
| /* Adjust num_groups if it seems overestimated due to combinatorial explosion. | |
| * Apply square-root based dampening to reduce overestimated group counts. | |
| * Only applies to arrow_fdw/parquet_fdw foreign tables (because ANALYZE | |
| * requires decompression which consumes GPU memory, so we avoid ANALYZE | |
| * and fix the default cardinality estimation instead). | |
| */ |
…d dampening Replace hardcoded reduction factors with configurable square-root based dampening to handle combinatorial explosion in PostgreSQL's estimate_num_groups() for arrow_fdw/parquet_fdw foreign tables. ## Problem PostgreSQL multiplies n_distinct values for each GROUP BY column, causing overestimation (e.g., 10M estimated vs 10K actual groups). This makes GpuPreAgg less likely to be selected by the planner. For arrow_fdw/parquet_fdw, running ANALYZE requires decompressing Parquet files which consumes GPU memory. This fix allows proper cost estimation without ANALYZE. ## Solution - Apply sqrt dampening: adjusted = baseline + sqrt(excess) * factor - Configurable via pg_strom.groupby_reduction_factor GUC (default 0.7, range 0.1-2.0) - Baseline set to max(input_rows * 0.1, 1000) - Only applies when selectivity > 10% and estimated_groups > 10K - **Only activated for arrow_fdw/parquet_fdw foreign tables** Example with factor=0.7: estimated=1M, input=1M, baseline=100K adjusted = 100K + sqrt(900K) * 0.7 ≈ 101K ## Parameter Naming Named "groupby_reduction_factor" (not "gpupreagg") because this is fundamentally a GROUP BY cardinality estimation fix, which happens to make GpuPreAgg more viable for arrow_fdw foreign tables.
d5ebd2b to
b0980bb
Compare
|
● Copilotから3つの指摘を対応しました。履歴をクリーンに保ちたいのでforce-pushしました。
|
Problem
PostgreSQL multiplies n_distinct values for each GROUP BY column, causing overestimation. This makes GpuPreAgg less likely to be selected by the planner.
Revised: For arrow_fdw/parquet_fdw, running ANALYZE requires decompressing Parquet files. This change makes GpuPreAgg more likely to trigger even with default estimates (i.e., without ANALYZE).
Solution
Example with factor=0.7:
estimated=1M, input=1M, baseline=100K adjusted = 100K + sqrt(900K) * 0.7 ≈ 101K