Fork test: PR #1384 - Refactor profiler column metrics into an extensible registry - #1529
Closed
mwojtyczka wants to merge 31 commits into
Closed
Fork test: PR #1384 - Refactor profiler column metrics into an extensible registry#1529mwojtyczka wants to merge 31 commits into
mwojtyczka wants to merge 31 commits into
Conversation
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
…nKurchenko/dqx into feature/profiler_additional_metrics
…nKurchenko/dqx into feature/profiler_additional_metrics
…nKurchenko/dqx into feature/profiler_additional_metrics
DataFrame.summary() emits percentile rows labelled 25%/50%/75%, and the
profiler stores them verbatim (_summarize: metric = row_dict["summary"]).
The reference table listed them as 25/50/75, so a custom builder doing
profiler_metrics.get("50") always got None. Document the literal keys.
Co-authored-by: Isaac <no-reply@databricks.com>
…ferences - Call register_profile_column_metric / register_profile_builder decorators - Drop misleading 'at no extra cost'; note each metric adds work per run - Tie profile_column_metric_type to the concrete decorator argument - Document the metric and builder function interfaces (inputs/outputs) - Frame generated profiles as data quality rule suggestions - Cross-reference the reference page's Custom Column Metrics section Co-authored-by: Isaac <no-reply@databricks.com>
…overwrite - Add integration test for the full extension path: a custom column metric consumed by a custom register_profile_builder builder, producing a DQProfile via the public profile() API (also asserts the built-in count_distinct metric) - Add a builder-registry snapshot fixture for test isolation - Add a unit test for the column-metric overwrite-and-warn path Co-authored-by: Isaac <no-reply@databricks.com>
mwojtyczka
added
do-not-merge
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## main #1529 +/- ##
==========================================
+ Coverage 92.81% 92.96% +0.14%
==========================================
Files 141 142 +1
Lines 13690 13864 +174
Branches 151 151
==========================================
+ Hits 12707 12888 +181
+ Misses 914 906 -8
- Partials 69 70 +1
Flags with carried forward coverage won't be shown. Click here to find out more. ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
Contributor
|
✅ 1004/1004 passed, 3 flaky, 63 skipped, 6h25m32s total Flaky tests:
Running from acceptance #5880 |
Contributor
|
✅ 1/1 passed, 31m59s total Running from mcp #629 |
Contributor
|
✅ 195/195 passed, 1 flaky, 1 skipped, 7h4m6s total Flaky tests:
Running from anomaly #1994 |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Automated sync from fork PR for CI testing.
Original PR: #1384
All tests, including unit and integration tests run on this PR (they are skipped for fork PRs).