[AMORO-4229] Refactor synchronizing Hive tables process via ProcessFactory plugin#4230
Conversation
Codecov Report❌ Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## master #4230 +/- ##
============================================
- Coverage 29.91% 23.09% -6.82%
+ Complexity 4317 2706 -1611
============================================
Files 677 463 -214
Lines 55037 42826 -12211
Branches 7028 6044 -984
============================================
- Hits 16464 9891 -6573
+ Misses 37337 32076 -5261
+ Partials 1236 859 -377
Flags with carried forward coverage won't be shown. Click here to find out more. ☔ View full report in Codecov by Sentry. 🚀 New features to boost your workflow:
|
ef170cc to
821cd71
Compare
821cd71 to
72578cc
Compare
| return Optional.empty(); | ||
| } | ||
|
|
||
| return Optional.of(new HiveCommitSyncProcess(tableRuntime, localEngine)); |
There was a problem hiding this comment.
triggerHiveCommitSync is missing a table format check before creating the process.
The IcebergProcessFactory.formats list includes ICEBERG, MIXED_ICEBERG, and MIXED_HIVE, so this trigger will be invoked for all three formats. Without an early guard, a HiveCommitSyncProcess is created and submitted to the thread pool for every table, only to discover inside run() that the table is not a Hive table and return early — wasted scheduling overhead on every cycle.
For comparison, triggerAutoCreateTag already does this correctly:
if (localEngine == null
|| tableRuntime.getFormat() != TableFormat.ICEBERG
|| ...) {
return Optional.empty();
}Suggested fix — filter at the trigger layer:
private Optional<TableProcess> triggerHiveCommitSync(TableRuntime tableRuntime) {
if (localEngine == null
|| tableRuntime.getFormat() != TableFormat.MIXED_HIVE) {
return Optional.empty();
}
return Optional.of(new HiveCommitSyncProcess(tableRuntime, localEngine));
}The TableTypeUtil.isHive() guard inside HiveCommitSyncProcess.run() can remain as a defensive check, but the trigger layer should be the first gate.
4368de4 to
16d1064
Compare
zhoujinsong
left a comment
There was a problem hiding this comment.
LGTM.
Thanks for the work!
Why are the changes needed?
Close #4229.
Brief change log
How was this patch tested?
Add some test cases that check the changes thoroughly including negative and positive cases if possible
Add screenshots for manual tests if appropriate
Run test locally before making a pull request
Documentation