Performance Enhancement: Significant Optimization of compare_dataset() Function

[shaikh-ma]

This Pull Request introduces substantial performance improvements to the compare_dataset() function, dramatically enhancing its efficiency when processing large-scale data comparisons.

Key Improvements: 🥇

The optimization delivers exceptional speed gains, reducing execution time from several minutes to approximately 2 seconds when comparing large datasets. Benchmark testing demonstrates consistent performance even in edge cases, such as comparing a dataset against itself with zero modifications.

Impact: 📈

This enhancement represents a fundamental improvement in dataset comparison capabilities, enabling:

• Near-instantaneous processing of previously time-intensive operations
• Improved scalability for large-scale data workflows
• Enhanced user experience through significantly reduced wait times
• The performance metrics indicate an order-of-magnitude improvement, transforming compare_dataset() from a potential bottleneck into a highly efficient operation suitable for production environments at scale.

Performance Improvement ⚡

• 99.95% reduction in execution time
• Performance improvement - 660X to 2,190X faster
• More than three orders of magnitude faster.

Snapshots 📷

Master branch	This branch	Performance improvement
		•From 73 minutes to 2 seconds •2,190x performance improvement •Nearly three orders of magnitude faster •99.95% reduction in execution time
		• From 22 minutes to 2 seconds • 99.85% reduction in execution time • 660X performance improvement • Two orders of magnitude faster

Tests

Testing compare_dataset() on multiple dataset with 2200+ vairables

[shaikh-ma]

Hi @jjdelc, @aless10, 👋

I've just submitted this PR with some significant improvements to the compare_dataset() function that we think will make a real difference.

The Data Processing users will now be able to handle their workflows more efficiently directly - the performance enhancements mean they can consolidate their tools and work faster, even with larger datasets, without needing to rely on other alternatives like R, etc., for dataset comparision.

I've tested it against typical datasets, and the results are pretty promising. Would appreciate it if you could take a look when you have a moment and let me know your thoughts. 😄

Cheers!
Aamir

[alexbuchhammer]

Hey team - hope all is well! Could anyone take a look at the rewrite of the method? It would help our python-based DP teams a ton as it offers functionality and performance that is matching the R/rcrunch equivalent.

Cheers!
Alex

[jjdelc]

I am looking at the testing for this function and it's very minimal only checking the empty case.

Would it be helpful to move this out of the dataset entity and factor out the logic as compare_datasets(metadata_a, metadata_b) then we can test it easily without having to mock the MutableDataset instances

[shaikh-ma]

Thanks @jjdelc ,

I've updated as per your feedback; the previous function will now give DeprecationWarning and will suggest to use the new standalone function compare_datasets().