Skip to content

Perf: Fewer and lighter SQL queries for uncached category pages - #1346

Merged
Hlavtox merged 6 commits into
PrestaShop:devfrom
LameuleFR:perf/faceted-search
Oct 24, 2026
Merged

Hlavtox merged 6 commits into
PrestaShop:devfrom
LameuleFR:perf/faceted-search

Conversation

@LameuleFR

@LameuleFR LameuleFR commented Oct 8, 2026 •

Copy link
Copy Markdown
Contributor
Questions Answers
Description? Reduce the SQL load of category pages when the facet blocks are not cached yet (every new filter combination, which is what crawlers hit). Same products; 5 to 10 times less database time on a 10k-product catalog. Also fixes facet counts that ignored the price filter. Details below.
Type? improvement
BC breaks? no
Deprecations? no
Fixed ticket?
How to test? See below.
Sponsor company

Changes

  1. Category subtree filter (Search.php, Block.php categories block): add nleft <= nright of the browsed category next to nleft >= nleft / nright <= nright. Redundant in the nested set, but with only nleft >= the index range covers every following category, so MySQL drove the population from all the shop products (10k products, 33k category rows) instead of the subtree. Listing query of a 50-product category: 70 ms → 0.6 ms.
  2. Stock join (MySQL.php mapping of quantity / out_of_stock): join stock_available on pa.id_product_attribute instead of going through product_attribute_combination, which repeated each combination stock row once per attribute.
  3. Attribute and feature counts (Block.php): the groups/features without selected value (and the AND features) share the same filters, so their values are counted in one query instead of one query per group/feature. A facet with a selected value keeps its own query, as it ignores its own selection. Feature values are loaded in one query instead of one per feature (DataAccessor::loadFeatureValues()).
  4. Not available count: the quantity and out_of_stock items of one operation now share their stock_available join (shareOperationJoin mapping flag) instead of joining the same row twice. Only these stock fields are flagged: the unique key of stock_available guarantees a single row; operations on other tables keep one join per item.
  5. Price filter lost after an empty weight block (Block::getWeightRangeBlock()): when no product of the population has a weight, the block returned before restoring the price and weight filters it had removed from the population. Every facet computed after it was then counted without the price filter.

Results

Test catalog: 10k products, 31.7k combinations, 9 attribute groups, 42 features (1084 values), 152 brands, filter template with 58 filters; facet block cache emptied before each cold request.

Page Before (cold) After (cold)
Category, 2492 products 6.2 s, 58 facet queries 0.62 s, 9 facet queries
Same, sorted by price, 1 filter 7.2 s 1.05 s
1 feature filter 3.9 s 0.72 s
2 / 4 filters 2.9 s / 2.2 s 0.53 s / 0.51 s
"Not available" filter 105 s 1.2 s
Color + "In stock" filters 38.6 s 2.4 s

Warm pages (facet blocks cached): 263 ms → about 205 ms on the big category (listing query, change 1).

The rendered pages (product list, every facet value and count) were compared with dev on 15 URLs, including availability filters, AND features, attribute + availability and the root category: identical.

Behaviour changes

  • Facet counts with a price filter. On dev, when the template has a weight filter and no product of the page has a weight, every facet placed after the weight filter in the template is counted as if the price filter were not set (a value can claim more products than the page lists). They are now counted within the price range (change 5, covered by BlockTest::testGetFiltersBlockWithoutWeight).
  • Stock ordering. PS_LAYERED_FILTER_SHOW_OUT_OF_STOCK_LAST and the quantity sort order on the sum of the stock of the product combinations. That sum counted each combination once per attribute; it is now the real sum. The order changes for products whose combinations have different numbers of attributes.

How to test

  1. On a catalog with many products, features and attribute groups, enable the filters in a template and empty the facet cache (Modules > Faceted search > Clear cache).
  2. Open a category page, then the same page with several filters (attributes, features, brand, availability): products, facet values and counts must be the same as before the change, and the page must be faster.
  3. With an AND feature filter, a selected value must keep the other values counted within the current selection.
  4. With products without weight and a template where the weight filter comes before attribute or feature filters, filter on a price range: the attribute and feature counts must not exceed the number of listed products.

* @param int[] $idFeatures
* @param int $idLang
*/
public function loadFeatureValues(array $idFeatures, $idLang)

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Unnecessary abstraction, why?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

it removes an N+1: getFeaturesBlock() ran one feature value query per displayed feature. Measured on 42 features, cold page: 42 queries / ~9.5 ms of SQL → 1 query / ~2.3 ms. getFeatureValues() keeps its signature for the other callers (Converter) and only loads what isn't loaded yet.

Comment thread src/Filters/Block.php Outdated
With only nleft >= and nright <= on the browsed category, the range read on
the category index covers every following category, so MySQL computed the
population from all the shop products and filtered the subtree last.
Bounding nleft by the category nright is redundant in the nested set but
makes the index range match the subtree.
The stock_available join went through product_attribute_combination, which
repeated each combination stock row once per attribute. Joining it on
product_attribute selects the same rows without the duplicates.

The quantity summed to order the out of stock products last no longer counts
a combination once per attribute: it is now the actual stock of the product.
Each attribute group and each feature facet ran its own count query, and
each one recomputed the population of the category. The facets without
selected value, and the AND features, share the current filters: their
values are now counted in one query per facet type. A facet with a selected
value keeps its own query, as it ignores its own selection.

The values of the displayed features are also loaded in one query.
The "Not available" count filters on quantity and out_of_stock in the same
operation, and each item got its own stock_available join on the same row.
The stock fields are now flagged to share the join of their operation: the
unique key of stock_available guarantees a single row. Operations on other
tables keep one join per item, needed when they require several values.
@LameuleFR
LameuleFR force-pushed the perf/faceted-search branch from 7bbd38d to f49a1cc Compare October 8, 2026 20:57
@kpodemski

Copy link
Copy Markdown
Contributor

Review — ps_facetedsearch PR #1346 · Perf: Fewer and lighter SQL queries for uncached category pages

  • Repo / branch: PrestaShop/ps_facetedsearch → base dev · Author: LameuleFR · Size: +193/−115, 7 files
  • Reviewed: 2026-10-09 · head f49a1cce, baseline dev@98fa494 (merge base = current dev tip)
  • Verdict: ✅ Big, consistent speed-up with identical product results. One facet-count difference turned up,
    and the PR is the side that is right: dev ignores the price filter when counting attribute and feature
    values. Please add a test that pins this, since the PR changes it without saying so.

Environment

PrestaShop 9.2.0 (prestashop/prestashop:9.2.0-8.3), PHP 8.3.28, Hummingbird theme
DB MariaDB 11.4 (native arm64), Docker on Apple Silicon
Data 100 000 products, 2 434 categories, combinations of 3 attributes each, 8 features, ~1.2 GB DB
Module ps_facetedsearch 5.1.0 on both sides, composer install --no-dev, layered cache off

Method

  • ABBA order: base → PR → PR → base, which exposes drift and warm-up effects.
  • Each round runs three measurements:
    • SQL bench: module only, Search + Block on scenarios A–M, best of 2.
    • HTTP bench: 12 storefront URLs, 3 runs each, best reported. Two of the URLs (home and a product page) are
      controls.
  • Correctness record, once per side:
    • Combinations: filter set × sort × PS_LAYERED_FILTER_SHOW_OUT_OF_STOCK_LAST 0/1 × leaf/100k category.
    • It includes a pass with one feature switched to the AND checkbox mode (filter_type = 4).
    • Each case records total count, product ids in order, and every facet's value counts.
  • Base round 1 was a cold first run on the freshly seeded stack (HTTP /100-benchmark 32.4 s, against 21.7 s in
    round 2). The comparison below uses base round 2: base gets its warmest conditions, so the gains are conservative.

Results

HTTP (best of 3)

URL dev PR Δ
/ (control) 2.18 s 2.14 s −2%
/1000-bench-product-1000.html (control) 0.177 s 0.176 s 0%
/101-bench-cat-1 (4 166 products) 1.19 s 0.78 s −34%
…?q=Availability-Not available 6.92 s 1.39 s −80%
…?q=Availability-In stock 8.03 s 1.70 s −79%
…?q=Bench Colour-Colour 6/Availability-In stock 4.38 s 1.87 s −57%
…?q=Bench Colour-Colour 6/Bench Size-Size 1 1.13 s 0.79 s −30%
…?q=Bench Feature 1-F1V6 0.95 s 0.72 s −24%
…?q=Bench Feature 1-F1V6/Bench Feature 2-F2V6 0.91 s 0.74 s −19%
…?order=product.price.asc&q=Bench Feature 1-F1V6 0.97 s 0.73 s −25%
/100-benchmark (100 000 products) 21.69 s 11.17 s −48%
…?q=Bench Colour-Colour 6/Availability-In stock 22.16 s 9.39 s −58%

The controls stay flat, so the gains come from the module. The PR's times repeat within 3% between its two
rounds. All 12 URLs return 200 with the PR.

SQL bench, module only (best of both rounds, list + facets)

Scenario Products dev PR Δ
A. big cat, no filter 100 000 21.68 s 10.93 s −50%
B. big cat, 1 feature 5 573 2.36 s 1.64 s −30%
C. big cat, 2 features 309 0.634 s 0.537 s −15%
D. big cat, 3 features 17 0.436 s 0.410 s −6%
E. big cat, 2 feat + colour 84 0.652 s 0.604 s −7%
F. big cat, 2 feat + in stock 309 3.05 s 1.30 s −57%
G. leaf cat, no filter 4 166 0.820 s 0.396 s −52%
H. leaf cat, 1 feature 253 0.602 s 0.370 s −39%
I. leaf cat, not available 666 6.58 s 1.01 s −85%
J. leaf cat, colour + in stock 0 2.36 s 1.21 s −49%
K. big cat, colour + in stock 24 000 16.44 s 8.39 s −49%
L. big cat, colour + size 0 2.07 s 1.83 s −12%
M. big cat, feature + not available 863 8.08 s 4.60 s −43%

Almost all of the time saved is in facet building; product listing times barely change. The availability scenarios
(F, I, J, K, M) gain the most, which fits the stock join moving from pac to pa. Narrow multi-feature filters
(C–E) were already cheap and gain little.

Correctness

The record covers 560 cases on each side. 546 of them could be compared:

  • Product totals: 0 differences.
  • Product ids, in order: 0 differences, under all 8 sorts and both out-of-stock-last settings.
  • AND-mode feature facet: identical.
  • Facet counts: 2 cases differ, the same price-filter case with each out-of-stock-last setting. Details below.

Facet counts with a price filter: dev is wrong, this PR is right

On /101-bench-cat-1?q=Price-€-10-200, 1 878 products are listed on both sides. On dev, though, every
attribute and feature facet is counted as if the price filter were not there:

Facet value dev PR SQL count over the 1 878 listed products
Size 1 1 666 626 626
Size 2 1 666 834 834
Colour 2 2 083 1 043 1 043
Bench attr. group 102, every value 4 166 1 878 1 878
F1V6 253 122 —

On dev, then, a value can claim more products than the page lists: 4 166 against 1 878. The category,
availability, condition and extras facets were already correct on both sides. I confirmed this on the storefront
HTML, not only through the CLI harness, and the SQL ground truth matches the PR exactly.

I haven't traced which part of the change fixes it: computing the counts up front in countFacetValues() or the
MySQL.php join changes. Either way it is a visible behaviour change that the description doesn't mention.
Could you:

  1. add it to the description;

  2. add a test (unit or Behat) that pins the attribute/feature counts while a price filter is active, so that
    neither dev nor a later refactor can regress it.

    Two findings from testing this PR on PrestaShop 9.2.0 (100k products, MariaDB 11.4), compared with dev@98fa494. Neither one is blocking, and neither needs a code change. In both cases the PR's behaviour is the correct one. What is missing is the description and a test.

1. Facet counts under a price filter: dev is wrong, this PR fixes it (undocumented)

What happens. When a price filter is active, dev counts attribute and feature values as if the price filter were not there. This PR counts them within the price range.

Example. On /101-bench-cat-1?q=Price-€-10-200, 1 878 products are listed on both sides:

Facet value dev this PR SQL count over the 1 878 listed products
Size 1 1 666 626 626
Colour 2 2 083 1 043 1 043
A Material value 4 166 1 878 1 878

On dev, then, a value can claim more products than the page lists (4 166 against 1 878). The category, availability, condition and extras facets are correct on both sides. Only attribute and feature facets are affected. I confirmed this on the storefront HTML, and the PR's numbers match a direct SQL count exactly.

Is it blocking? No. This is a pre-existing bug on dev, and the PR fixes it, most likely as a side effect of counting all facets up front in countFacetValues(). I haven't traced which part of the change fixes it.

What to do:

  • Mention the fix in the description and changelog: it is a visible change to the counts merchants and customers see.
  • Add a test (unit or Behat) that checks attribute/feature counts while a price filter is active. The fix is accidental, so nothing stops a later refactor from bringing the bug back.

2. The stock-ordering change is correct, but the description doesn't match what it does

What happens. As documented, the stock sum used for ordering used to count each combination once per attribute, and it is now the real sum. To check this I seeded products with negative stock and with combinations of different attribute counts. Then I compared the whole ordered product list of a 4 166-product category, under 60 combinations of filter, sort and out-of-stock-last setting:

  • Product totals, product sets and facet counts are identical.
  • Where the order changes, it is identical again once the affected products are removed. Nothing else moves.
  • The affected products end up where their real stock puts them. The PR is right and dev is wrong.

The description is inaccurate in two ways:

  1. The trigger is narrower than described. The description says the order changes for "a product mixing combinations in stock and combinations with a negative stock". In fact it only changes when those combinations have different numbers of attributes.
    • When all combinations have the same attribute count, the old sum is just a multiple of the real one, with the same sign, so nothing moves.
    • I had 166 mixed-stock products with 3 attributes on every combination in the tested category, and none of them moved.
    • So the trigger is a product mixing in-stock and negative-stock combinations with different attribute counts. The back-office combination generator gives every combination of a product the same attribute groups, so this mostly happens with combinations created or imported one by one.
  2. The quantity sort changes too, even with PS_LAYERED_FILTER_SHOW_OUT_OF_STOCK_LAST off. product.quantity.desc orders on the same SUM, so the same products move there. Hummingbird doesn't offer this sort in its menu, so it only matters to themes or modules that request it.

Is it blocking? No. The new behaviour is a fix, since the old sum was wrong, and it affects few products.

What to do: update the "Behaviour change" section, for example:

PS_LAYERED_FILTER_SHOW_OUT_OF_STOCK_LAST and the quantity sort order on the sum of the stock of the product combinations. That sum counted each combination once per attribute; it is now the real sum. The order changes for products whose combinations have different numbers of attributes.

kpodemski
kpodemski previously approved these changes Oct 9, 2026

@kpodemski kpodemski left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

To me it looks good, it would be great to have @Hlavtox green light as well. Thank you @LameuleFR

@ps-jarvis ps-jarvis added the Waiting for QA Status: Action required, Waiting for test feedback label Oct 9, 2026
@LameuleFR

Copy link
Copy Markdown
Contributor Author

Thanks for the review, the ABBA runs and the 560-case record are very useful.

1. Counts under a price filter. Root cause found: getWeightRangeBlock() removes the price and weight filters from the initial population, and returns early when the population has no weight, before restoring them. Every facet computed after the weight block then ignores the price filter. Your bench products probably have no weight, with the weight filter placed before the attribute and feature filters in the template, which also explains why category, availability and condition (placed before it) were right on both sides.

This PR fixed it by accident for attributes and features only, because their counts now run before the loop over the blocks; facets placed after the weight block were still wrong. Commit 04f7e9f "Restore the price filter after an empty weight block" moves the restore before the early return, and BlockTest::testGetFiltersBlockWithoutWeight now checks that the price filters are still on the population afterwards (it fails without the fix).

2. Stock ordering. Right on both points, the "Behaviour changes" section now uses your wording, including the quantity sort.

PululuK
PululuK previously approved these changes Oct 11, 2026
Comment thread src/Product/Search.php
// we add this one specific category ID, otherwise, we will add everything using nleft and nright
if (Configuration::get('PS_LAYERED_FULL_TREE')) {
$this->getSearchAdapter()->addFilter('nleft', [$category->nleft], '>=');
$this->getSearchAdapter()->addFilter('nleft', [$category->nright], '<=');

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hello guys, just one question, what this line does?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Logically this changes nothing, in the nested set, every subcategory already has its nleft between the category's nleft and nright. The extra condition is only there for MySQL. Without an upper bound on nleft, the index range runs to the end of the tree, so MySQL was starting from all of the shop's products instead of just the subtree.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@Hlavtox is this PR ok for you ?

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Oh, now I understand.

@LameuleFR Can you add some comment for next generations? :D Something like:

// Explicitly bound nleft on both sides to limit the index scan
 // to the current category subtree. The upper bound is logically
 // redundant but helps MySQL use a much smaller index range.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done in 988e00d, on both places using this bound (Search.php and the categories block in Block.php).

Touxten
Touxten previously approved these changes Oct 23, 2026
@LameuleFR
LameuleFR dismissed stale reviews from Touxten and PululuK via 988e00d October 24, 2026 09:42
@LameuleFR

Copy link
Copy Markdown
Contributor Author

@Hlavtox comment added in 988e00d. @PululuK @Touxten @kpodemski the push dismissed your approvals, could you re-approve so we can merge? CI is green.

@Hlavtox
Hlavtox merged commit 23e94e8 into PrestaShop:dev Oct 24, 2026
17 checks passed
@github-project-automation github-project-automation Bot moved this from To be tested to Merged in PR Dashboard Oct 24, 2026
@ps-jarvis

Copy link
Copy Markdown

PR merged, well done!

Message to @PrestaShop/committers: do not forget to milestone it before the merge.

@kpodemski kpodemski added QA ✔️ Status: Check done, Code approved and removed Waiting for QA Status: Action required, Waiting for test feedback labels Oct 26, 2026
@kpodemski kpodemski added this to the 5.1.1 milestone Oct 26, 2026
jf-viguier added a commit to jf-viguier/ps_facetedsearch that referenced this pull request Oct 26, 2026
Base 6187d3b. Apporte notamment la PR PrestaShop#1346 de performance, le support des
conditions produit de PrestaShop 9.2, le correctif PHP 8.5 et le passage de
webpack a esbuild.

Rebase sans conflit, les 14 commits locaux rejoues tels quels : la liste des
fichiers divergents du §2 reste exacte, et src/Adapter/MySQL.php est identique
a upstream, ce qui laisse entrer les optimisations SQL sans retouche.

Compte de tests attendu actualise, l'amont en a ajoute deux.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

QA ✔️ Status: Check done, Code approved

Projects

Status: Merged

Development

Successfully merging this pull request may close these issues.

7 participants