rptest: Enable fast uploads in datalake test#28492
Conversation
The test expects uplads to happen in 10 minutes. The problem is that it create different number of partitions in docker and on a dedicated machine (100 vs 1000). With 1000 partitions the chance that one partition will receive not enough data to roll a segment is relatively high. The fix is to force uploads and to increase the amount of data on a dedicated node. Signed-off-by: Evgeny Lazin <[email protected]>
There was a problem hiding this comment.
Pull Request Overview
This PR fixes a timing issue in the datalake delayed translation test where uploads were not completing within the expected 10-minute window. The root cause was that the test creates different numbers of partitions (100 vs 1000) depending on whether it runs in Docker or on dedicated machines, leading to insufficient data per partition to trigger segment rolls on dedicated nodes.
Key changes:
- Enables fast uploads to ensure timely completion of data uploads
- Increases data volume on dedicated nodes to guarantee segment rolls across all partitions
| self.TOPIC_NAME, | ||
| self.SCHEMA_NAME, | ||
| {"weight": "range(0, 10000)"}, | ||
| {"weight": f"range(0, {count})"}, |
There was a problem hiding this comment.
The use of an f-string to construct a range string (e.g., 'range(0, 50000)') suggests this is being passed as a string to be evaluated elsewhere. Consider whether passing the actual count value directly would be clearer and safer than constructing a string representation of Python code.
| {"weight": f"range(0, {count})"}, | |
| {"weight": count}, |
bashtanov
left a comment
bashtanov
left a comment
There was a problem hiding this comment.
LGTM
there's a typo in the commit message: "uplads"
CI test resultstest results on build#76122
|
|
/backport v25.3.x |
|
/backport v25.2.x |
|
Branch name "v25.3.x" not found. |
The test expects uplads to happen in 10 minutes. The problem is that it create different number of partitions in docker and on a dedicated machine (100 vs 1000). With 1000 partitions the chance that one partition will receive not enough data to roll a segment is relatively high.
The fix is to force uploads and to increase the amount of data on a dedicated node.
Backports Required
Release Notes