Skip to content

perf(quality-gates): Variance reduction - outlier filtering, warmup exclusion#41961

Merged
MajorLift merged 25 commits intomainfrom
jongsun/perf/quality-gates/variance-reduction
Apr 24, 2026
Merged

perf(quality-gates): Variance reduction - outlier filtering, warmup exclusion#41961
MajorLift merged 25 commits intomainfrom
jongsun/perf/quality-gates/variance-reduction

Conversation

@MajorLift
Copy link
Copy Markdown
Contributor

@MajorLift MajorLift commented Apr 20, 2026 •
edited by cursor Bot
Loading

Description

The variance audit found 5 metrics with CV 30–50% and 2 borderline at 25–30%. Raw per-run samples contain outliers from cold-start JIT compilation, GC pauses, and CI machine variance. These inflate stdDev and produce false positives across all three verdict layers — not just Layer 3 (Mann-Whitney U), but Layer 1 (absolute thresholds) and Layer 2 (historical baselines) today.

This PR adds sample preparation that benefits every layer downstream:

IQR-based outlier trimming. trimOutliers() in statistics.ts removes values outside [Q1 − 1.5·IQR, Q3 + 1.5·IQR] before stats computation. At n=15 independent sessions, this removes 0–3 extreme values. trimmedCount is exposed per-metric in TimerStatistics, BenchmarkResults, and as a Sentry tag for observability.

Warm-up run exclusion. The first WARMUP_RUNS (default: 1) browser-load sessions are discarded before computing stats. The first cold-start iteration is a known outlier source (JIT compilation, cache priming). Applied before IQR trimming so the trim operates on warm samples only.

PowerUser iteration rebalance. startupPowerUserHome changes from 10 browser loads × 10 page loads to 15 × 7 (105 total samples, same CI time). For outlier detection and Mann-Whitney U, independent session count matters more than within-session page reloads. Trading 3 page loads per session for 5 additional sessions increases independent n by 50% at negligible CI time cost. After warm-up exclusion: 14 effective sessions × 6 page loads = 84 effective samples.

Minimum sample gate constant. MIN_SAMPLES_FOR_VERDICT = 5 is defined in constants.ts and wired into the Mann-Whitney U verdict logic in #41520.

Expected CV impact

Metric Current CV Projected (trimmed, 15 sessions)
startupPowerUser.uiStartup 34.3% ~18–24%
startupPowerUser.load 30.5% ~17–23%
startupPowerUser.loadScripts 31.8% ~17–24%
openAccountMenuToAccountListLoaded 36.7% ~28–32%
solanaAssetDetails.assetClickToPriceChart 33.6% ~26–30%

Metrics with tailRatio > 1.3 benefit most from trimming (the top three above).

Changelog

CHANGELOG entry: No user-facing changes. Benchmark infrastructure only (outlier trimming, warm-up exclusion, PowerUser iteration rebalance, trimmedCount Sentry tag).

Related issues

Fixes: MetaMask/MetaMask-planning#7185

Depends on: #40729 (Layer 1+2 comparison CLI)
Related: #41520 (Mann-Whitney U — trimming improves statistical power; wires MIN_SAMPLES_FOR_VERDICT into verdict logic)

Manual testing steps

  1. Run the IQR trimming unit tests: yarn jest test/e2e/benchmarks/utils/outlier-trimming.test.ts
  2. Run the full benchmark utils suite: yarn jest test/e2e/benchmarks/utils/
  3. Verify all cases pass: empty array, n=1, n=2, n=3, identical distribution, single high outlier, single low outlier, both-end outliers, n=15 benchmark scenario (0–3 trimmed), low-variance run (0 trimmed), input-order invariance, no input mutation.

Screenshots/Recordings

N/A — CI tooling change with no visual output.

Pre-merge author checklist

Pre-merge reviewer checklist

  • I've manually tested the PR (e.g. pull and build branch, run the app, test code being changed).
  • I confirm that this PR addresses all acceptance criteria described in the ticket it closes and includes the necessary testing evidence such as recordings and or screenshots.

Note

Medium Risk
Changes benchmark aggregation and reporting (warm-up run exclusion, IQR outlier trimming, and new derived metrics), which can materially shift quality-gate/Sentry signals even though it’s CI-only.

Overview
Reduces benchmark variance by discarding warm-up browser sessions (WARMUP_RUNS) and applying IQR-based outlier trimming before computing page-load statistics, with new per-metric trimmedCount/outliers included in BenchmarkResults.

Rebalances the startupPowerUserHome preset to 15 browser loads × 7 page loads (and defaults power-user runs to the higher session count), and extends Sentry reporting to include derived reliability metrics (cv, dataQuality, tailRatio) plus the new trimming/outlier counts.

Reviewed by Cursor Bugbot for commit f1fc8fe. Bugbot is set up for automated code reviews on this repo. Configure here.

@MajorLift MajorLift self-assigned this Apr 20, 2026
@MajorLift MajorLift added the team-extension-platform Extension Platform team label Apr 20, 2026
@github-actions
Copy link
Copy Markdown
Contributor

github-actions Bot commented Apr 20, 2026

CLA Signature Action: All authors have signed the CLA. You may need to manually re-run the blocking PR check if it doesn't pass in a few minutes.

@MajorLift MajorLift force-pushed the jongsun/perf/quality-gates/variance-reduction branch from ce2c331 to 775bb15 Compare April 21, 2026 18:53
@metamaskbotv2
Copy link
Copy Markdown
Contributor

metamaskbotv2 Bot commented Apr 21, 2026

Builds ready [775bb15]
⚡ Performance Benchmarks (Total: 🟢 7 pass · 🟡 8 warn · 🔴 0 fail)

Baseline (latest main): 71bd826 | Date: 10/14/58243 | Pipeline: 24740598518 | Baseline logs

Interaction Benchmarks · Samples: 5
Benchmarkchrome-browserify
loadNewAccount🟡 [Show logs]
confirmTx🟡 [Show logs]
bridgeUserActions🟡 [Show logs]

📈 Results compared to the previous 5 runs on main

  • ↓ loadNewAccount/load_new_account: -66%
  • ↓ loadNewAccount/total: -66%
  • ↓ bridgeUserActions/bridge_load_page: -27%
  • ↓ bridgeUserActions/bridge_load_asset_picker: -35%
  • ↓ bridgeUserActions/bridge_search_token: -28%
  • ↓ bridgeUserActions/total: -30%

🌐 Core Web Vitals — 🟢 good · 🟡 needs improvement · 🔴 poor (web.dev thresholds)

  • 🟡 loadNewAccount/FCP: p75 2.8s
  • 🟡 confirmTx/FCP: p75 2.5s
  • 🟡 bridgeUserActions/FCP: p75 2.5s
Startup Benchmarks · Samples: 100
Benchmarkchrome-browserifychrome-webpackfirefox-browserifyfirefox-webpack
startupStandardHome🟢 [Show logs]🟢 [Show logs]🟢 [Show logs]🟢 [Show logs]

📈 Results compared to the previous 5 runs on main

  • ↓ startupStandardHome/uiStartup: -26%
  • ↓ startupStandardHome/load: -15%
  • ↓ startupStandardHome/domContentLoaded: -17%
  • ↓ startupStandardHome/firstPaint: -13%
  • ↓ startupStandardHome/firstReactRender: -10%
  • ↓ startupStandardHome/initialActions: -33%
  • ↓ startupStandardHome/loadScripts: -20%
  • ↓ startupStandardHome/numNetworkReqs: -37%
  • ↓ startupStandardHome/uiStartup: -22%
  • ↓ startupStandardHome/load: -17%
  • ↓ startupStandardHome/domContentLoaded: -16%
  • ↓ startupStandardHome/firstPaint: -12%
  • ↓ startupStandardHome/backgroundConnect: -38%
  • ↓ startupStandardHome/firstReactRender: -27%
  • ↓ startupStandardHome/loadScripts: -16%
  • ↓ startupStandardHome/setupStore: -13%
  • ↓ startupStandardHome/numNetworkReqs: -44%
  • ↓ startupStandardHome/uiStartup: -13%
  • ↓ startupStandardHome/domInteractive: -46%
  • ↑ startupStandardHome/backgroundConnect: +13%
  • ↑ startupStandardHome/initialActions: +33%
  • ↓ startupStandardHome/setupStore: -17%
  • ↓ startupStandardHome/numNetworkReqs: -37%
  • ↓ startupStandardHome/domInteractive: -28%
  • ↑ startupStandardHome/initialActions: +14%
  • ↓ startupStandardHome/setupStore: -64%
  • ↓ startupStandardHome/numNetworkReqs: -37%
User Journey Benchmarks · Samples: 5 · mock API
Benchmarkchrome-browserify
onboardingImportWallet🟢 [Show logs]
onboardingNewWallet🟢 [Show logs]
assetDetails🟡 [Show logs]
solanaAssetDetails🟡 [Show logs]
importSrpHome🟡 [Show logs]
sendTransactions🟡 [Show logs]
swap🟡 [Show logs]

📈 Results compared to the previous 5 runs on main

  • ↓ onboardingImportWallet/srpButtonToSrpForm: -85%
  • ↓ onboardingImportWallet/metricsToWalletReadyScreen: -49%
  • ↓ onboardingImportWallet/doneButtonToHomeScreen: -79%
  • ↑ onboardingImportWallet/openAccountMenuToAccountListLoaded: +27%
  • ↓ onboardingImportWallet/total: -44%
  • ↓ onboardingNewWallet/srpButtonToPwForm: -78%
  • ↓ onboardingNewWallet/skipBackupToMetricsScreen: -69%
  • ↓ onboardingNewWallet/doneButtonToAssetList: -26%
  • ↓ onboardingNewWallet/total: -27%
  • ↓ assetDetails/assetClickToPriceChart: -51%
  • ↓ assetDetails/total: -51%
  • ↓ solanaAssetDetails/assetClickToPriceChart: -67%
  • ↓ solanaAssetDetails/total: -67%
  • ↓ importSrpHome/openAccountMenuAfterLogin: -71%
  • ↓ importSrpHome/homeAfterImportWithNewWallet: -63%
  • ↓ importSrpHome/total: -56%
  • ↓ sendTransactions/openSendPageFromHome: -16%
  • ↓ sendTransactions/selectTokenToSendFormLoaded: -20%
  • ↑ sendTransactions/reviewTransactionToConfirmationPage: +34%
  • ↑ sendTransactions/total: +32%
  • ↓ swap/openSwapPageFromHome: -96%
  • ↑ swap/fetchAndDisplaySwapQuotes: +31%
  • ↑ swap/total: +11%

🌐 Core Web Vitals — 🟢 good · 🟡 needs improvement · 🔴 poor (web.dev thresholds)

  • 🟡 assetDetails/FCP: p75 2.5s
  • 🟡 solanaAssetDetails/FCP: p75 2.5s
  • 🟡 importSrpHome/FCP: p75 2.5s
  • 🟡 sendTransactions/FCP: p75 2.4s
  • 🟡 swap/FCP: p75 2.4s
Dapp Page Load Benchmarks · Samples: 100
Benchmarkchrome-browserify
dappPageLoad🟢 [Show logs]
Bundle size diffs
  • background: 58 Bytes (0%)
  • ui: 5 Bytes (0%)
  • common: 20 Bytes (0%)

@metamaskbotv2
Copy link
Copy Markdown
Contributor

metamaskbotv2 Bot commented Apr 21, 2026

Builds ready [897ae1d] [reused from 775bb15]
⚡ Performance Benchmarks (Total: 🟢 7 pass · 🟡 8 warn · 🔴 0 fail)

Baseline (latest main): 71bd826 | Date: 10/14/58243 | Pipeline: 24741621937 | Baseline logs

Interaction Benchmarks · Samples: 5
Benchmarkchrome-browserify
loadNewAccount🟡 [Show logs]
confirmTx🟡 [Show logs]
bridgeUserActions🟡 [Show logs]

📈 Results compared to the previous 5 runs on main

  • ↓ loadNewAccount/load_new_account: -66%
  • ↓ loadNewAccount/total: -66%
  • ↓ bridgeUserActions/bridge_load_page: -27%
  • ↓ bridgeUserActions/bridge_load_asset_picker: -35%
  • ↓ bridgeUserActions/bridge_search_token: -28%
  • ↓ bridgeUserActions/total: -30%

🌐 Core Web Vitals — 🟢 good · 🟡 needs improvement · 🔴 poor (web.dev thresholds)

  • 🟡 loadNewAccount/FCP: p75 2.8s
  • 🟡 confirmTx/FCP: p75 2.5s
  • 🟡 bridgeUserActions/FCP: p75 2.5s
Startup Benchmarks · Samples: 100
Benchmarkchrome-browserifychrome-webpackfirefox-browserifyfirefox-webpack
startupStandardHome🟢 [Show logs]🟢 [Show logs]🟢 [Show logs]🟢 [Show logs]

📈 Results compared to the previous 5 runs on main

  • ↓ startupStandardHome/uiStartup: -26%
  • ↓ startupStandardHome/load: -15%
  • ↓ startupStandardHome/domContentLoaded: -17%
  • ↓ startupStandardHome/firstPaint: -13%
  • ↓ startupStandardHome/firstReactRender: -10%
  • ↓ startupStandardHome/initialActions: -33%
  • ↓ startupStandardHome/loadScripts: -20%
  • ↓ startupStandardHome/numNetworkReqs: -37%
  • ↓ startupStandardHome/uiStartup: -22%
  • ↓ startupStandardHome/load: -17%
  • ↓ startupStandardHome/domContentLoaded: -16%
  • ↓ startupStandardHome/firstPaint: -12%
  • ↓ startupStandardHome/backgroundConnect: -38%
  • ↓ startupStandardHome/firstReactRender: -27%
  • ↓ startupStandardHome/loadScripts: -16%
  • ↓ startupStandardHome/setupStore: -13%
  • ↓ startupStandardHome/numNetworkReqs: -44%
  • ↓ startupStandardHome/uiStartup: -13%
  • ↓ startupStandardHome/domInteractive: -46%
  • ↑ startupStandardHome/backgroundConnect: +13%
  • ↑ startupStandardHome/initialActions: +33%
  • ↓ startupStandardHome/setupStore: -17%
  • ↓ startupStandardHome/numNetworkReqs: -37%
  • ↓ startupStandardHome/domInteractive: -28%
  • ↑ startupStandardHome/initialActions: +14%
  • ↓ startupStandardHome/setupStore: -64%
  • ↓ startupStandardHome/numNetworkReqs: -37%
User Journey Benchmarks · Samples: 5 · mock API
Benchmarkchrome-browserify
onboardingImportWallet🟢 [Show logs]
onboardingNewWallet🟢 [Show logs]
assetDetails🟡 [Show logs]
solanaAssetDetails🟡 [Show logs]
importSrpHome🟡 [Show logs]
sendTransactions🟡 [Show logs]
swap🟡 [Show logs]

📈 Results compared to the previous 5 runs on main

  • ↓ onboardingImportWallet/srpButtonToSrpForm: -85%
  • ↓ onboardingImportWallet/metricsToWalletReadyScreen: -49%
  • ↓ onboardingImportWallet/doneButtonToHomeScreen: -79%
  • ↑ onboardingImportWallet/openAccountMenuToAccountListLoaded: +27%
  • ↓ onboardingImportWallet/total: -44%
  • ↓ onboardingNewWallet/srpButtonToPwForm: -78%
  • ↓ onboardingNewWallet/skipBackupToMetricsScreen: -69%
  • ↓ onboardingNewWallet/doneButtonToAssetList: -26%
  • ↓ onboardingNewWallet/total: -27%
  • ↓ assetDetails/assetClickToPriceChart: -51%
  • ↓ assetDetails/total: -51%
  • ↓ solanaAssetDetails/assetClickToPriceChart: -67%
  • ↓ solanaAssetDetails/total: -67%
  • ↓ importSrpHome/openAccountMenuAfterLogin: -71%
  • ↓ importSrpHome/homeAfterImportWithNewWallet: -63%
  • ↓ importSrpHome/total: -56%
  • ↓ sendTransactions/openSendPageFromHome: -16%
  • ↓ sendTransactions/selectTokenToSendFormLoaded: -20%
  • ↑ sendTransactions/reviewTransactionToConfirmationPage: +34%
  • ↑ sendTransactions/total: +32%
  • ↓ swap/openSwapPageFromHome: -96%
  • ↑ swap/fetchAndDisplaySwapQuotes: +31%
  • ↑ swap/total: +11%

🌐 Core Web Vitals — 🟢 good · 🟡 needs improvement · 🔴 poor (web.dev thresholds)

  • 🟡 assetDetails/FCP: p75 2.5s
  • 🟡 solanaAssetDetails/FCP: p75 2.5s
  • 🟡 importSrpHome/FCP: p75 2.5s
  • 🟡 sendTransactions/FCP: p75 2.4s
  • 🟡 swap/FCP: p75 2.4s
Dapp Page Load Benchmarks · Samples: 100
Benchmarkchrome-browserify
dappPageLoad🟢 [Show logs]
Bundle size diffs
  • background: 58 Bytes (0%)
  • ui: 5 Bytes (0%)
  • common: 20 Bytes (0%)

@MajorLift MajorLift marked this pull request as ready for review April 21, 2026 19:51
@MajorLift MajorLift requested review from a team, HowardBraham and itsyoboieltr as code owners April 21, 2026 19:51
@github-project-automation github-project-automation Bot moved this to Needs dev review in PR review queue Apr 21, 2026
@MajorLift MajorLift changed the title perf(quality-gates): variance reduction - outlier filtering, warmup exclusion perf(quality-gates): Variance reduction - outlier filtering, warmup exclusion Apr 21, 2026
cursor[bot]
cursor Bot reviewed Apr 21, 2026
View reviewed changes
Comment thread test/e2e/benchmarks/utils/runner.ts Outdated
@MajorLift
fix: filter measuredWebVitalsRuns by iteration instead of slice
7cd855d 
Web vitals collection can fail silently inside try-catch, making
allWebVitalsRuns sparse. Slicing by warmupSize = WARMUP_RUNS * pageLoads
assumes every warmup page load produced an entry; filtering by wv.iteration
>= warmupSize is correct regardless of collection gaps.

Reported by Cursor Bugbot (review #4150432255).
@metamaskbotv2
Copy link
Copy Markdown
Contributor

metamaskbotv2 Bot commented Apr 23, 2026

Builds ready [451bc12] [reused from fc2dcd7]
⚡ Performance Benchmarks (Total: 🟢 7 pass · 🟡 8 warn · 🔴 0 fail)

Baseline (latest main): 71bd826 | Date: 10/14/58243 | Pipeline: 24843983297 | Baseline logs

Interaction Benchmarks · Samples: 5
Benchmarkchrome-browserify
loadNewAccount🟡 [Show logs]
confirmTx🟡 [Show logs]
bridgeUserActions🟡 [Show logs]

📈 Results compared to the previous 5 runs on main

  • ↓ loadNewAccount/load_new_account: -57%
  • ↓ loadNewAccount/total: -57%
  • ↓ bridgeUserActions/bridge_load_page: -15%
  • ↓ bridgeUserActions/bridge_load_asset_picker: -33%
  • ↓ bridgeUserActions/bridge_search_token: -27%
  • ↓ bridgeUserActions/total: -26%

🌐 Core Web Vitals — 🟢 good · 🟡 needs improvement · 🔴 poor (web.dev thresholds)

  • 🟡 loadNewAccount/FCP: p75 2.6s
  • 🟡 confirmTx/FCP: p75 2.6s
  • 🟡 bridgeUserActions/FCP: p75 2.6s
Startup Benchmarks · Samples: 100
Benchmarkchrome-browserifychrome-webpackfirefox-browserifyfirefox-webpack
startupStandardHome🟢 [Show logs]🟢 [Show logs]🟢 [Show logs]🟢 [Show logs]

📈 Results compared to the previous 5 runs on main

  • ↓ startupStandardHome/uiStartup: -22%
  • ↓ startupStandardHome/load: -11%
  • ↓ startupStandardHome/domContentLoaded: -13%
  • ↑ startupStandardHome/domInteractive: +14%
  • ↑ startupStandardHome/backgroundConnect: +15%
  • ↓ startupStandardHome/firstReactRender: -10%
  • ↓ startupStandardHome/initialActions: -33%
  • ↓ startupStandardHome/loadScripts: -17%
  • ↓ startupStandardHome/numNetworkReqs: -37%
  • ↓ startupStandardHome/uiStartup: -17%
  • ↓ startupStandardHome/load: -12%
  • ↓ startupStandardHome/domContentLoaded: -11%
  • ↓ startupStandardHome/backgroundConnect: -35%
  • ↓ startupStandardHome/firstReactRender: -30%
  • ↓ startupStandardHome/loadScripts: -12%
  • ↓ startupStandardHome/setupStore: -13%
  • ↓ startupStandardHome/numNetworkReqs: -44%
  • ↓ startupStandardHome/uiStartup: -14%
  • ↓ startupStandardHome/domInteractive: -64%
  • ↓ startupStandardHome/initialActions: -33%
  • ↓ startupStandardHome/setupStore: -25%
  • ↓ startupStandardHome/numNetworkReqs: -37%
  • ↓ startupStandardHome/uiStartup: -20%
  • ↓ startupStandardHome/load: -12%
  • ↓ startupStandardHome/domContentLoaded: -12%
  • ↓ startupStandardHome/domInteractive: -73%
  • ↓ startupStandardHome/firstReactRender: -12%
  • ↓ startupStandardHome/initialActions: -43%
  • ↓ startupStandardHome/loadScripts: -12%
  • ↓ startupStandardHome/setupStore: -67%
  • ↓ startupStandardHome/numNetworkReqs: -37%
User Journey Benchmarks · Samples: 5 · mock API
Benchmarkchrome-browserify
onboardingImportWallet🟢 [Show logs]
onboardingNewWallet🟢 [Show logs]
assetDetails🟡 [Show logs]
solanaAssetDetails🟡 [Show logs]
importSrpHome🟡 [Show logs]
sendTransactions🟡 [Show logs]
swap🟡 [Show logs]

📈 Results compared to the previous 5 runs on main

  • ↓ onboardingImportWallet/srpButtonToSrpForm: -84%
  • ↓ onboardingImportWallet/metricsToWalletReadyScreen: -32%
  • ↓ onboardingImportWallet/doneButtonToHomeScreen: -71%
  • ↑ onboardingImportWallet/openAccountMenuToAccountListLoaded: +19%
  • ↓ onboardingImportWallet/total: -41%
  • ↓ onboardingNewWallet/srpButtonToPwForm: -78%
  • ↓ onboardingNewWallet/skipBackupToMetricsScreen: -67%
  • ↓ onboardingNewWallet/agreeButtonToOnboardingSuccess: -21%
  • ↓ onboardingNewWallet/doneButtonToAssetList: -34%
  • ↓ onboardingNewWallet/total: -33%
  • ↓ assetDetails/assetClickToPriceChart: -48%
  • ↓ assetDetails/total: -48%
  • ↓ solanaAssetDetails/assetClickToPriceChart: -65%
  • ↓ solanaAssetDetails/total: -65%
  • ↓ importSrpHome/loginToHomeScreen: -14%
  • ↓ importSrpHome/openAccountMenuAfterLogin: -83%
  • ↓ importSrpHome/homeAfterImportWithNewWallet: -69%
  • ↓ importSrpHome/total: -62%
  • ↓ sendTransactions/selectTokenToSendFormLoaded: -27%
  • ↑ sendTransactions/reviewTransactionToConfirmationPage: +35%
  • ↑ sendTransactions/total: +32%
  • ↓ swap/openSwapPageFromHome: -97%
  • ↑ swap/fetchAndDisplaySwapQuotes: +31%
  • ↑ swap/total: +11%

🌐 Core Web Vitals — 🟢 good · 🟡 needs improvement · 🔴 poor (web.dev thresholds)

  • 🟡 assetDetails/FCP: p75 2.5s
  • 🟡 solanaAssetDetails/FCP: p75 2.5s
  • 🟡 importSrpHome/FCP: p75 2.4s
  • 🟡 sendTransactions/FCP: p75 2.5s
  • 🟡 swap/FCP: p75 2.5s
Dapp Page Load Benchmarks · Samples: 100
Benchmarkchrome-browserify
dappPageLoad🟢 [Show logs]
Bundle size diffs [🚨 Warning! Bundle size has increased!]
  • background: 1.15 KiB (0.02%)
  • ui: 12.46 KiB (0.15%)
  • common: 1.49 KiB (0.01%)

@MajorLift
Fix JSDoc lint errors on \runPageLoadBenchmark\
0827815 
Add missing \`@param options.*\` declarations and remove markdown bold syntax
that triggered the \`jsdoc/check-indentation\` rule.
cursor[bot]
cursor Bot reviewed Apr 23, 2026
View reviewed changes
Comment thread test/e2e/benchmarks/utils/runner.ts
@MajorLift
Populate \outliers\ in \runPageLoadBenchmark\ return for Sentry c…
a8c5df8 
…onsistency

Page-load benchmarks use IQR-only trimming, so \`outliers === trimmedCount\`
(no z-score pass). Setting both fields ensures Sentry telemetry is consistent
across page-load and iteration-based benchmark paths. Dashboards can still
derive z-score count as \`outliers - trimmedCount\`; for page loads that
difference is always zero, which is correct.
cursor[bot]
cursor Bot reviewed Apr 23, 2026
View reviewed changes
Comment thread test/e2e/benchmarks/utils/runner.ts Outdated
@MajorLift
Spread \outliers\ copy in \runPageLoadBenchmark\ to avoid shared …
4c40363 
…reference

\`outliers\` and \`trimmedCount\` pointed at the same object. For page loads
(IQR-only) the values are equal, but aliasing the reference means a mutation
to one silently corrupts the other. Spread produces an independent copy with
the correct values.
@metamaskbotv2
Copy link
Copy Markdown
Contributor

metamaskbotv2 Bot commented Apr 23, 2026

Builds ready [a8c5df8]
⚡ Performance Benchmarks (Total: 🟢 7 pass · 🟡 8 warn · 🔴 0 fail)

Baseline (latest main): 71bd826 | Date: 10/14/58243 | Pipeline: 24852396817 | Baseline logs

Interaction Benchmarks · Samples: 5
Benchmarkchrome-browserify
loadNewAccount🟡 [Show logs]
confirmTx🟡 [Show logs]
bridgeUserActions🟡 [Show logs]

📈 Results compared to the previous 5 runs on main

  • ↓ loadNewAccount/load_new_account: -53%
  • ↓ loadNewAccount/total: -53%
  • ↓ bridgeUserActions/bridge_load_asset_picker: -22%
  • ↓ bridgeUserActions/bridge_search_token: -24%
  • ↓ bridgeUserActions/total: -23%

🌐 Core Web Vitals — 🟢 good · 🟡 needs improvement · 🔴 poor (web.dev thresholds)

  • 🟡 loadNewAccount/FCP: p75 2.7s
  • 🟡 confirmTx/FCP: p75 2.7s
  • 🟡 bridgeUserActions/FCP: p75 2.8s
Startup Benchmarks · Samples: 100
Benchmarkchrome-browserifychrome-webpackfirefox-browserifyfirefox-webpack
startupStandardHome🟢 [Show logs]🟢 [Show logs]🟢 [Show logs]🟢 [Show logs]

📈 Results compared to the previous 5 runs on main

  • ↓ startupStandardHome/uiStartup: -20%
  • ↓ startupStandardHome/domContentLoaded: -11%
  • ↑ startupStandardHome/backgroundConnect: +18%
  • ↓ startupStandardHome/firstReactRender: -10%
  • ↓ startupStandardHome/initialActions: -33%
  • ↓ startupStandardHome/loadScripts: -14%
  • ↓ startupStandardHome/numNetworkReqs: -37%
  • ↓ startupStandardHome/uiStartup: -24%
  • ↓ startupStandardHome/load: -19%
  • ↓ startupStandardHome/domContentLoaded: -19%
  • ↓ startupStandardHome/domInteractive: -15%
  • ↓ startupStandardHome/firstPaint: -29%
  • ↓ startupStandardHome/backgroundConnect: -38%
  • ↓ startupStandardHome/firstReactRender: -30%
  • ↓ startupStandardHome/loadScripts: -19%
  • ↓ startupStandardHome/setupStore: -13%
  • ↓ startupStandardHome/numNetworkReqs: -44%
  • ↓ startupStandardHome/uiStartup: -14%
  • ↓ startupStandardHome/domInteractive: -64%
  • ↓ startupStandardHome/initialActions: -33%
  • ↓ startupStandardHome/setupStore: -17%
  • ↓ startupStandardHome/numNetworkReqs: -37%
  • ↓ startupStandardHome/uiStartup: -15%
  • ↓ startupStandardHome/domInteractive: -61%
  • ↑ startupStandardHome/initialActions: +14%
  • ↓ startupStandardHome/setupStore: -60%
  • ↓ startupStandardHome/numNetworkReqs: -37%
User Journey Benchmarks · Samples: 5 · mock API
Benchmarkchrome-browserify
onboardingImportWallet🟢 [Show logs]
onboardingNewWallet🟢 [Show logs]
assetDetails🟡 [Show logs]
solanaAssetDetails🟡 [Show logs]
importSrpHome🟡 [Show logs]
sendTransactions🟡 [Show logs]
swap🟡 [Show logs]

📈 Results compared to the previous 5 runs on main

  • ↓ onboardingImportWallet/srpButtonToSrpForm: -85%
  • ↓ onboardingImportWallet/metricsToWalletReadyScreen: -47%
  • ↓ onboardingImportWallet/doneButtonToHomeScreen: -76%
  • ↑ onboardingImportWallet/openAccountMenuToAccountListLoaded: +17%
  • ↓ onboardingImportWallet/total: -43%
  • ↓ onboardingNewWallet/srpButtonToPwForm: -79%
  • ↓ onboardingNewWallet/skipBackupToMetricsScreen: -68%
  • ↓ onboardingNewWallet/doneButtonToAssetList: -25%
  • ↓ onboardingNewWallet/total: -26%
  • ↓ assetDetails/assetClickToPriceChart: -80%
  • ↓ assetDetails/total: -80%
  • ↓ solanaAssetDetails/assetClickToPriceChart: -73%
  • ↓ solanaAssetDetails/total: -73%
  • ↓ importSrpHome/loginToHomeScreen: -15%
  • ↓ importSrpHome/openAccountMenuAfterLogin: -58%
  • ↓ importSrpHome/homeAfterImportWithNewWallet: -69%
  • ↓ importSrpHome/total: -61%
  • ↓ sendTransactions/selectTokenToSendFormLoaded: -16%
  • ↑ sendTransactions/reviewTransactionToConfirmationPage: +35%
  • ↑ sendTransactions/total: +32%
  • ↓ swap/openSwapPageFromHome: -96%
  • ↑ swap/fetchAndDisplaySwapQuotes: +31%
  • ↑ swap/total: +11%

🌐 Core Web Vitals — 🟢 good · 🟡 needs improvement · 🔴 poor (web.dev thresholds)

  • 🟡 assetDetails/FCP: p75 2.6s
  • 🟡 solanaAssetDetails/FCP: p75 2.6s
  • 🟡 importSrpHome/FCP: p75 2.5s
  • 🟡 sendTransactions/INP: p75 288ms
  • 🟡 sendTransactions/FCP: p75 2.5s
  • 🟡 swap/FCP: p75 2.5s
Dapp Page Load Benchmarks · Samples: 100
Benchmarkchrome-browserify
dappPageLoad🟢 [Show logs]

📈 Results compared to the previous 5 runs on main

  • ↓ dappPageLoad/pageLoadTime: -20%
  • ↓ dappPageLoad/domContentLoaded: -21%
  • ↓ dappPageLoad/firstPaint: -18%
  • ↓ dappPageLoad/firstContentfulPaint: -18%
Bundle size diffs [🚨 Warning! Bundle size has increased!]
  • background: 31.3 KiB (0.52%)
  • ui: 8.29 KiB (0.1%)
  • common: 8.84 KiB (0.07%)

cursor[bot]
cursor Bot reviewed Apr 23, 2026
View reviewed changes
Copy link
Copy Markdown

@cursor cursor Bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Cursor Bugbot has reviewed your changes and found 1 potential issue.

Fix All in Cursor

❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, have a team admin enable autofix in the Cursor dashboard.

Reviewed by Cursor Bugbot for commit 4c40363. Configure here.

Comment thread test/e2e/benchmarks/utils/runner.ts Outdated
@sonarqubecloud
Copy link
Copy Markdown

sonarqubecloud Bot commented Apr 23, 2026

Quality Gate Passed Quality Gate passed

Issues
0 New issues
0 Accepted issues

Measures
0 Security Hotspots
0.0% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarQube Cloud

@metamaskbotv2
Copy link
Copy Markdown
Contributor

metamaskbotv2 Bot commented Apr 23, 2026

Builds ready [f1fc8fe]
⚡ Performance Benchmarks (Total: 🟢 6 pass · 🟡 8 warn · 🔴 0 fail)

Baseline (latest main): 71bd826 | Date: 10/14/58243 | Pipeline: 24855206774 | Baseline logs

Interaction Benchmarks · Samples: 5
Benchmarkchrome-browserify
loadNewAccount🟡 [Show logs]
confirmTx🟡 [Show logs]
bridgeUserActions🟡 [Show logs]

📈 Results compared to the previous 5 runs on main

  • ↓ loadNewAccount/load_new_account: -54%
  • ↓ loadNewAccount/total: -54%
  • ↓ bridgeUserActions/bridge_load_page: -19%
  • ↓ bridgeUserActions/bridge_load_asset_picker: -32%
  • ↓ bridgeUserActions/bridge_search_token: -27%
  • ↓ bridgeUserActions/total: -27%

🌐 Core Web Vitals — 🟢 good · 🟡 needs improvement · 🔴 poor (web.dev thresholds)

  • 🟡 loadNewAccount/FCP: p75 2.5s
  • 🟡 confirmTx/FCP: p75 2.4s
  • 🟡 bridgeUserActions/FCP: p75 2.5s
Startup Benchmarks · Samples: 100
Benchmarkchrome-browserifychrome-webpackfirefox-browserifyfirefox-webpack
startupStandardHome🟢 [Show logs]🟢 [Show logs]🟢 [Show logs]🟢 [Show logs]

📈 Results compared to the previous 5 runs on main

  • ↓ startupStandardHome/uiStartup: -26%
  • ↓ startupStandardHome/load: -15%
  • ↓ startupStandardHome/domContentLoaded: -17%
  • ↓ startupStandardHome/firstReactRender: -10%
  • ↓ startupStandardHome/initialActions: -33%
  • ↓ startupStandardHome/loadScripts: -20%
  • ↓ startupStandardHome/numNetworkReqs: -37%
  • ↓ startupStandardHome/uiStartup: -20%
  • ↓ startupStandardHome/load: -14%
  • ↓ startupStandardHome/domContentLoaded: -14%
  • ↑ startupStandardHome/domInteractive: +13%
  • ↑ startupStandardHome/firstPaint: +13%
  • ↓ startupStandardHome/backgroundConnect: -39%
  • ↓ startupStandardHome/firstReactRender: -27%
  • ↓ startupStandardHome/loadScripts: -14%
  • ↓ startupStandardHome/setupStore: -13%
  • ↓ startupStandardHome/numNetworkReqs: -44%
  • ↓ startupStandardHome/uiStartup: -10%
  • ↓ startupStandardHome/domInteractive: -58%
  • ↑ startupStandardHome/initialActions: +33%
  • ↓ startupStandardHome/numNetworkReqs: -37%
  • ↓ startupStandardHome/uiStartup: -13%
  • ↓ startupStandardHome/domInteractive: -42%
  • ↓ startupStandardHome/backgroundConnect: -11%
  • ↓ startupStandardHome/initialActions: -43%
  • ↓ startupStandardHome/setupStore: -64%
  • ↓ startupStandardHome/numNetworkReqs: -37%
User Journey Benchmarks · Samples: 5 · mock API
Benchmarkchrome-browserify
onboardingImportWallet🟢 [Show logs]
onboardingNewWallet🟢 [Show logs]
assetDetails🟡 [Show logs]
solanaAssetDetails🟡 [Show logs]
importSrpHome🟡 [Show logs]
sendTransactions🟡 [Show logs]
swap🟡 [Show logs]

📈 Results compared to the previous 5 runs on main

  • ↓ onboardingImportWallet/srpButtonToSrpForm: -89%
  • ↓ onboardingImportWallet/confirmSrpToPwForm: -23%
  • ↓ onboardingImportWallet/pwFormToMetricsScreen: -26%
  • ↓ onboardingImportWallet/metricsToWalletReadyScreen: -57%
  • ↓ onboardingImportWallet/doneButtonToHomeScreen: -77%
  • ↑ onboardingImportWallet/openAccountMenuToAccountListLoaded: +35%
  • ↓ onboardingImportWallet/total: -48%
  • ↓ onboardingNewWallet/srpButtonToPwForm: -77%
  • ↓ onboardingNewWallet/skipBackupToMetricsScreen: -67%
  • ↓ onboardingNewWallet/agreeButtonToOnboardingSuccess: -20%
  • ↓ onboardingNewWallet/doneButtonToAssetList: -31%
  • ↓ onboardingNewWallet/total: -31%
  • ↓ assetDetails/assetClickToPriceChart: -41%
  • ↓ assetDetails/total: -41%
  • ↓ solanaAssetDetails/assetClickToPriceChart: -72%
  • ↓ solanaAssetDetails/total: -72%
  • ↓ importSrpHome/openAccountMenuAfterLogin: -71%
  • ↓ importSrpHome/homeAfterImportWithNewWallet: -69%
  • ↓ importSrpHome/total: -61%
  • ↓ sendTransactions/openSendPageFromHome: -35%
  • ↓ sendTransactions/selectTokenToSendFormLoaded: -27%
  • ↑ sendTransactions/reviewTransactionToConfirmationPage: +35%
  • ↑ sendTransactions/total: +33%
  • ↓ swap/openSwapPageFromHome: -96%
  • ↑ swap/fetchAndDisplaySwapQuotes: +31%
  • ↑ swap/total: +11%

🌐 Core Web Vitals — 🟢 good · 🟡 needs improvement · 🔴 poor (web.dev thresholds)

  • 🟡 assetDetails/FCP: p75 2.4s
  • 🟡 solanaAssetDetails/FCP: p75 2.4s
  • 🟡 importSrpHome/FCP: p75 2.6s
  • 🟡 sendTransactions/FCP: p75 2.5s
  • 🟡 swap/FCP: p75 2.6s
Dapp Page Load Benchmarks · Samples: 100

⚠️ Missing data: chrome/browserify/pageLoadBenchmark

✅ No regressions detected

Bundle size diffs [🚨 Warning! Bundle size has increased!]
  • background: 31.2 KiB (0.52%)
  • ui: 8.33 KiB (0.1%)
  • common: 8.22 KiB (0.06%)

@metamaskbotv2
Copy link
Copy Markdown
Contributor

metamaskbotv2 Bot commented Apr 23, 2026

Builds ready [f1fc8fe]
⚡ Performance Benchmarks (Total: 🟢 7 pass · 🟡 8 warn · 🔴 0 fail)

Baseline (latest main): 71bd826 | Date: 10/14/58243 | Pipeline: 24855206774 | Baseline logs

Interaction Benchmarks · Samples: 5
Benchmarkchrome-browserify
loadNewAccount🟡 [Show logs]
confirmTx🟡 [Show logs]
bridgeUserActions🟡 [Show logs]

📈 Results compared to the previous 5 runs on main

  • ↓ loadNewAccount/load_new_account: -54%
  • ↓ loadNewAccount/total: -54%
  • ↓ bridgeUserActions/bridge_load_page: -19%
  • ↓ bridgeUserActions/bridge_load_asset_picker: -32%
  • ↓ bridgeUserActions/bridge_search_token: -27%
  • ↓ bridgeUserActions/total: -27%

🌐 Core Web Vitals — 🟢 good · 🟡 needs improvement · 🔴 poor (web.dev thresholds)

  • 🟡 loadNewAccount/FCP: p75 2.5s
  • 🟡 confirmTx/FCP: p75 2.4s
  • 🟡 bridgeUserActions/FCP: p75 2.5s
Startup Benchmarks · Samples: 100
Benchmarkchrome-browserifychrome-webpackfirefox-browserifyfirefox-webpack
startupStandardHome🟢 [Show logs]🟢 [Show logs]🟢 [Show logs]🟢 [Show logs]

📈 Results compared to the previous 5 runs on main

  • ↓ startupStandardHome/uiStartup: -26%
  • ↓ startupStandardHome/load: -15%
  • ↓ startupStandardHome/domContentLoaded: -17%
  • ↓ startupStandardHome/firstReactRender: -10%
  • ↓ startupStandardHome/initialActions: -33%
  • ↓ startupStandardHome/loadScripts: -20%
  • ↓ startupStandardHome/numNetworkReqs: -37%
  • ↓ startupStandardHome/uiStartup: -20%
  • ↓ startupStandardHome/load: -14%
  • ↓ startupStandardHome/domContentLoaded: -14%
  • ↑ startupStandardHome/domInteractive: +13%
  • ↑ startupStandardHome/firstPaint: +13%
  • ↓ startupStandardHome/backgroundConnect: -39%
  • ↓ startupStandardHome/firstReactRender: -27%
  • ↓ startupStandardHome/loadScripts: -14%
  • ↓ startupStandardHome/setupStore: -13%
  • ↓ startupStandardHome/numNetworkReqs: -44%
  • ↓ startupStandardHome/uiStartup: -10%
  • ↓ startupStandardHome/domInteractive: -58%
  • ↑ startupStandardHome/initialActions: +33%
  • ↓ startupStandardHome/numNetworkReqs: -37%
  • ↓ startupStandardHome/uiStartup: -13%
  • ↓ startupStandardHome/domInteractive: -42%
  • ↓ startupStandardHome/backgroundConnect: -11%
  • ↓ startupStandardHome/initialActions: -43%
  • ↓ startupStandardHome/setupStore: -64%
  • ↓ startupStandardHome/numNetworkReqs: -37%
User Journey Benchmarks · Samples: 5 · mock API
Benchmarkchrome-browserify
onboardingImportWallet🟢 [Show logs]
onboardingNewWallet🟢 [Show logs]
assetDetails🟡 [Show logs]
solanaAssetDetails🟡 [Show logs]
importSrpHome🟡 [Show logs]
sendTransactions🟡 [Show logs]
swap🟡 [Show logs]

📈 Results compared to the previous 5 runs on main

  • ↓ onboardingImportWallet/srpButtonToSrpForm: -89%
  • ↓ onboardingImportWallet/confirmSrpToPwForm: -23%
  • ↓ onboardingImportWallet/pwFormToMetricsScreen: -26%
  • ↓ onboardingImportWallet/metricsToWalletReadyScreen: -57%
  • ↓ onboardingImportWallet/doneButtonToHomeScreen: -77%
  • ↑ onboardingImportWallet/openAccountMenuToAccountListLoaded: +35%
  • ↓ onboardingImportWallet/total: -48%
  • ↓ onboardingNewWallet/srpButtonToPwForm: -77%
  • ↓ onboardingNewWallet/skipBackupToMetricsScreen: -67%
  • ↓ onboardingNewWallet/agreeButtonToOnboardingSuccess: -20%
  • ↓ onboardingNewWallet/doneButtonToAssetList: -31%
  • ↓ onboardingNewWallet/total: -31%
  • ↓ assetDetails/assetClickToPriceChart: -41%
  • ↓ assetDetails/total: -41%
  • ↓ solanaAssetDetails/assetClickToPriceChart: -72%
  • ↓ solanaAssetDetails/total: -72%
  • ↓ importSrpHome/openAccountMenuAfterLogin: -71%
  • ↓ importSrpHome/homeAfterImportWithNewWallet: -69%
  • ↓ importSrpHome/total: -61%
  • ↓ sendTransactions/openSendPageFromHome: -35%
  • ↓ sendTransactions/selectTokenToSendFormLoaded: -27%
  • ↑ sendTransactions/reviewTransactionToConfirmationPage: +35%
  • ↑ sendTransactions/total: +33%
  • ↓ swap/openSwapPageFromHome: -96%
  • ↑ swap/fetchAndDisplaySwapQuotes: +31%
  • ↑ swap/total: +11%

🌐 Core Web Vitals — 🟢 good · 🟡 needs improvement · 🔴 poor (web.dev thresholds)

  • 🟡 assetDetails/FCP: p75 2.4s
  • 🟡 solanaAssetDetails/FCP: p75 2.4s
  • 🟡 importSrpHome/FCP: p75 2.6s
  • 🟡 sendTransactions/FCP: p75 2.5s
  • 🟡 swap/FCP: p75 2.6s
Dapp Page Load Benchmarks · Samples: 100
Benchmarkchrome-browserify
dappPageLoad🟢 [Show logs]
Bundle size diffs [🚨 Warning! Bundle size has increased!]
  • background: 31.2 KiB (0.52%)
  • ui: 8.33 KiB (0.1%)
  • common: 8.22 KiB (0.06%)

@MajorLift MajorLift enabled auto-merge April 24, 2026 09:46
@MajorLift MajorLift added this pull request to the merge queue Apr 24, 2026
@github-project-automation github-project-automation Bot moved this from Review in progress to Review finalised - Ready to be merged in PR review queue Apr 24, 2026
Merged via the queue into main with commit f750c30 Apr 24, 2026
398 of 399 checks passed
@MajorLift MajorLift deleted the jongsun/perf/quality-gates/variance-reduction branch April 24, 2026 12:00
@github-project-automation github-project-automation Bot moved this from Review finalised - Ready to be merged to Merged, Closed or Archived in PR review queue Apr 24, 2026
@github-actions github-actions Bot locked and limited conversation to collaborators Apr 24, 2026
@metamaskbot metamaskbot added the release-13.30.0 Issue or pull request that will be included in release 13.30.0 label Apr 24, 2026
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.

Reviewers

@cursor cursor[bot] cursor[bot] left review comments

@gauthierpetetin gauthierpetetin gauthierpetetin approved these changes

@itsyoboieltr itsyoboieltr itsyoboieltr approved these changes

@HowardBraham HowardBraham Awaiting requested review from HowardBraham HowardBraham is a code owner

Assignees

@MajorLift MajorLift

Labels

release-13.30.0 Issue or pull request that will be included in release 13.30.0 size-M team-extension-platform Extension Platform team

Projects

PR review queue
Archived in project

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

4 participants

@MajorLift @gauthierpetetin @itsyoboieltr @metamaskbot