Skip to content

gpupreagg: Improve num_groups estimation for arrow_fdw with sqrt-base… - #949

Open
a-hirota wants to merge 1 commit into
heterodb:masterfrom
a-hirota:feature/preagg-cost-adjustment
Open

gpupreagg: Improve num_groups estimation for arrow_fdw with sqrt-base…#949
a-hirota wants to merge 1 commit into
heterodb:masterfrom
a-hirota:feature/preagg-cost-adjustment

Conversation

@a-hirota

Copy link
Copy Markdown

Problem

PostgreSQL multiplies n_distinct values for each GROUP BY column, causing overestimation. This makes GpuPreAgg less likely to be selected by the planner.

Revised: For arrow_fdw/parquet_fdw, running ANALYZE requires decompressing Parquet files. This change makes GpuPreAgg more likely to trigger even with default estimates (i.e., without ANALYZE).

Solution

  • Apply sqrt dampening: adjusted = baseline + sqrt(excess) * factor
  • Configurable via pg_strom.groupby_reduction_factor GUC (default 0.7, range 0.1-2.0)
  • Baseline set to max(input_rows * 0.1, 1000)
  • Only applies when selectivity > 10% and estimated_groups > 10K
  • Only activated for arrow_fdw/parquet_fdw foreign tables

Example with factor=0.7:
estimated=1M, input=1M, baseline=100K adjusted = 100K + sqrt(900K) * 0.7 ≈ 101K

Copilot AI review requested due to automatic review settings October 14, 2025 02:10

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull Request Overview

This PR improves GROUP BY cardinality estimation for arrow_fdw/parquet_fdw foreign tables by implementing square-root based dampening to reduce overestimated group counts that make GpuPreAgg less likely to be selected by the PostgreSQL planner.

  • Implements sqrt-based reduction algorithm to address PostgreSQL's tendency to overestimate num_groups by multiplying n_distinct values across GROUP BY columns
  • Adds configurable dampening via new GUC parameter pg_strom.groupby_reduction_factor (default 0.7, range 0.1-2.0)
  • Restricts the optimization to arrow_fdw/parquet_fdw foreign tables only to avoid requiring expensive ANALYZE operations

Tip: Customize your code reviews with copilot-instructions.md. Create the file or learn how to get started.

Comment thread src/gpu_preagg.c
double excess;
double reduced_excess;
double adjusted;

Copilot AI Oct 14, 2025

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Potential division by zero if input_nrows is 0. Add a guard to check if input_nrows > 0 before performing the division.

Suggested change
/* Guard against division by zero */
if (input_nrows <= 0.0)
return estimated_groups;

Copilot uses AI. Check for mistakes.
Comment thread src/gpu_preagg.c Outdated
reduced_excess = sqrt(excess) * pgstrom_groupby_reduction_factor;
adjusted = baseline + reduced_excess;

elog(DEBUG1, "GpuPreAgg: reduced num_groups from %.0f to %.0f "

Copilot AI Oct 14, 2025

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Corrected spacing in format string - removed extra space before opening parenthesis.

Suggested change
elog(DEBUG1, "GpuPreAgg: reduced num_groups from %.0f to %.0f "
elog(DEBUG1, "GpuPreAgg: reduced num_groups from %.0f to %.0f"

Copilot uses AI. Check for mistakes.
Comment thread src/gpu_preagg.c
Comment on lines +2318 to +2324
/* Adjust num_groups if it seems overestimated due to combinatorial explosion.
* Apply square-root based dampening to reduce overestimated group counts.
* Only applies to arrow_fdw/parquet_fdw foreign tables (because ANALYZE
* requires decompression which consumes GPU memory, so we avoid ANALYZE
* and fix the default cardinality estimation instead).
*/
{

Copilot AI Oct 14, 2025

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[nitpick] This block-level comment should be moved above the opening brace of the block it describes (line 2324) for better code organization and readability.

Suggested change
/* Adjust num_groups if it seems overestimated due to combinatorial explosion.
* Apply square-root based dampening to reduce overestimated group counts.
* Only applies to arrow_fdw/parquet_fdw foreign tables (because ANALYZE
* requires decompression which consumes GPU memory, so we avoid ANALYZE
* and fix the default cardinality estimation instead).
*/
{
{
/* Adjust num_groups if it seems overestimated due to combinatorial explosion.
* Apply square-root based dampening to reduce overestimated group counts.
* Only applies to arrow_fdw/parquet_fdw foreign tables (because ANALYZE
* requires decompression which consumes GPU memory, so we avoid ANALYZE
* and fix the default cardinality estimation instead).
*/

Copilot uses AI. Check for mistakes.
…d dampening

Replace hardcoded reduction factors with configurable square-root based
dampening to handle combinatorial explosion in PostgreSQL's
estimate_num_groups() for arrow_fdw/parquet_fdw foreign tables.

## Problem

PostgreSQL multiplies n_distinct values for each GROUP BY column,
causing overestimation (e.g., 10M estimated vs 10K actual groups).
This makes GpuPreAgg less likely to be selected by the planner.

For arrow_fdw/parquet_fdw, running ANALYZE requires decompressing
Parquet files which consumes GPU memory. This fix allows proper
cost estimation without ANALYZE.

## Solution

- Apply sqrt dampening: adjusted = baseline + sqrt(excess) * factor
- Configurable via pg_strom.groupby_reduction_factor GUC (default 0.7, range 0.1-2.0)
- Baseline set to max(input_rows * 0.1, 1000)
- Only applies when selectivity > 10% and estimated_groups > 10K
- **Only activated for arrow_fdw/parquet_fdw foreign tables**

Example with factor=0.7:
  estimated=1M, input=1M, baseline=100K
  adjusted = 100K + sqrt(900K) * 0.7 ≈ 101K

## Parameter Naming

Named "groupby_reduction_factor" (not "gpupreagg") because this is
fundamentally a GROUP BY cardinality estimation fix, which happens
to make GpuPreAgg more viable for arrow_fdw foreign tables.
@a-hirota
a-hirota force-pushed the feature/preagg-cost-adjustment branch from d5ebd2b to b0980bb Compare October 14, 2025 02:22
@a-hirota

Copy link
Copy Markdown
Author

● Copilotから3つの指摘を対応しました。履歴をクリーンに保ちたいのでforce-pushしました。

  1. Division by Zero防止: input_nrows > 0 のチェックを追加
  2. フォーマット修正: debug logの末尾の余分なスペースを削除
  3. コメント配置: ブロックコメントを開き括弧の上に移動

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants