KAFKA-19193 Support rack-aware partitioning for Kafka producer by ivanyu · Pull Request #19850 · apache/kafka

ivanyu · 2025-05-29T17:02:10Z

According to
KIP-1123,
this commit adds the support for rack-aware partitioning to
BuiltInPartitioner. It comes with two new configs for the producer:
partitioner.rack.aware and client.rack, which allows enabling the
new behavior.

Apart from the added unit tests, the desired behavior was tested by
kafka-producer-perf-test.sh with an existing and a non-existing rack
against a 4 node cluster with two racks and 12-partition topic:

./kafka_2.13-4.1.0-SNAPSHOT/bin/kafka-producer-perf-test.sh \
  --topic test-topic --num-records 100000 --throughput -1 --record-size
1 \
  --producer-props bootstrap.servers=127.0.0.1:9092 \
  client.rack=rack0 partitioner.rack.aware=true

Reviewers: Viktor Somogyi-Vass viktorsomogyi@gmail.com, Liam
Clarke-Hutchinson liam@steelsky.co.nz

github-actions · 2025-06-06T03:28:00Z

A label of 'needs-attention' was automatically added to this PR in order to raise the
attention of the committers. Once this issue has been triaged, the triage label
should be removed to prevent this automation from happening again.

marcospassos · 2025-08-21T18:56:01Z

Is there any chance this feature will be included in the next release?

ivanyu · 2025-08-22T14:24:50Z

@marcospassos I'm waiting for 4.1 release to be cut, after that I'll ping the mailing list for review.

github-actions · 2025-08-23T03:21:42Z

A label of 'needs-attention' was automatically added to this PR in order to raise the
attention of the committers. Once this issue has been triaged, the triage label
should be removed to prevent this automation from happening again.

github-actions · 2025-09-18T03:17:50Z

A label of 'needs-attention' was automatically added to this PR in order to raise the
attention of the committers. Once this issue has been triaged, the triage label
should be removed to prevent this automation from happening again.

LiamClarkeNZ

Love the tests.

LiamClarkeNZ · 2025-10-04T04:30:32Z

+    private final String rack;

-    private volatile PartitionLoadStats partitionLoadStats = null;
+    private volatile PartitionLoadStatsHolder partitionLoadStats = null;


Suggested change

private volatile PartitionLoadStatsHolder partitionLoadStats = null;

private volatile PartitionLoadStatsHolder partitionLoadStatsHolder = null;

As it got a bit confusing when later the code is retrieving partitionLoadStats.inThisRack which is a member of PartitionLoadStatsHolder of type PartitionLoadStats. :D

Thanks, renamed

LiamClarkeNZ · 2025-10-04T04:37:03Z


        // Create partitions with "sticky" batch size to accommodate 3 records.
-        BuiltInPartitioner builtInPartitionerA = new SequentialPartitioner(logContext, TOPIC_A, 3);
+        BuiltInPartitioner builtInPartitionerA = new SequentialPartitioner(logContext, TOPIC_A, 3, false, "");


Suggested change

BuiltInPartitioner builtInPartitionerA = new SequentialPartitioner(logContext, TOPIC_A, 3, false, "");

final boolean rackAware = false;

final String clientRackId = "";

BuiltInPartitioner builtInPartitionerA = new SequentialPartitioner(logContext, TOPIC_A, 3, rackAware, clientRackId);

LiamClarkeNZ · 2025-10-04T04:37:52Z


        // Check that switching works even when there is one partition.
-        BuiltInPartitioner builtInPartitionerB = new SequentialPartitioner(logContext, TOPIC_B, 1);
+        BuiltInPartitioner builtInPartitionerB = new SequentialPartitioner(logContext, TOPIC_B, 1, false, "");


Suggested change

BuiltInPartitioner builtInPartitionerB = new SequentialPartitioner(logContext, TOPIC_B, 1, false, "");

BuiltInPartitioner builtInPartitionerB = new SequentialPartitioner(logContext, TOPIC_B, 1, rackAware, clientRackId);

Just to make the meaning of false and "" obvious at a glance :)

LiamClarkeNZ · 2025-10-04T07:57:18Z

+                partitionRacks[i] = leader.rack();
+            }
+            allPartitions.add(new PartitionInfo(TOPIC_A, i, leader, NODES, NODES));
            expectedFrequencies[i] = 6 - queueSizes[i];  // 6 is max(queueSizes) + 1


Love the comment :)

ivanyu · 2026-03-19T19:51:33Z

@kirktrue sure, I will review the PR soon!

ivanyu · 2026-03-28T16:25:33Z

For some reason, I cannot reopen the PR (the button isn't even there). Maybe someone else can?

ivanyu · 2026-03-30T08:45:52Z

Thank you @LiamClarkeNZ, I addressed your comments

viktorsomogyi · 2026-04-20T15:39:38Z

+            this.rack = rack;
+
+            if (rackAware && Utils.isBlank(rack)) {
+                throw new IllegalArgumentException("client.rack must be provided if partitioner.rack.aware is enabled");


The KIP states:

Select the next partition from all partitions following the current algorithm in the following cases:

if the client rack is not specified;

if there are no partitions with leaders in the same rack as the client;

if all the “nearby” partitions are unavailable.

if partitioner.rack.aware is false.

Based on this, is it correct to throw an exception here?

I think this is OK, for the sake of the user. This exception will be thrown at the producer constructor time, preventing inconsistent config from silently sneaking in.

Should we use ConfigException instead for consistency?

@chia7712 done, please have a look: #22433

According to KIP-1123, this commit adds the support for rack-aware partitioning to `BuiltInPartitioner`. It comes with two new configs for the producer: `partitioner.rack.aware` and `client.rack`, which allows enabling the new behavior. Apart from the added unit tests, the desired behavior was validated by `kafka-producer-perf-test.sh` with an existing and a non-existing rack against a 4 node cluster with two racks and 12-partition topic: ```shell ./kafka_2.13-4.1.0-SNAPSHOT/bin/kafka-producer-perf-test.sh \ --topic test-topic --num-records 100000 --throughput -1 --record-size 1 \ --producer-props bootstrap.servers=127.0.0.1:9092 \ client.rack=rack0 partitioner.rack.aware=true ```

viktorsomogyi

LGTM, let's merge it when the CI is green.

ivanyu · 2026-04-21T14:25:31Z

The failed tests are marked as flaky and pass locally

chia7712

@ivanyu thanks for driving this patch! I left a few minor suggestions for you. On a related note, have you considered adding a metric like rack-aware-fallback-rate? If it's out of scope for your KIP, I'd gladly draft a follow-up KIP to add it. A silent fallback might lead to an unpleasant surprise for users when they see their cloud networking bills!

chia7712 · 2026-04-22T13:51:40Z

+            this.rack = rack;
+
+            if (rackAware && Utils.isBlank(rack)) {
+                throw new IllegalArgumentException("client.rack must be provided if partitioner.rack.aware is enabled");


Should we use ConfigException instead for consistency?

chia7712 · 2026-04-22T14:07:17Z

        // Invert and fold the queue size, so that they become separator values in the CFT.
+        invertAndFoldQueueSizeArray(queueSizes, maxSizePlus1, length);
+
+        log.trace("Partition load stats for topic {}: CFT={}, IDs={}, length={}",


Should we include the rack-specific load stats (partitionLoadStatsInThisRack) in this trace log when rackAware is true?

Please have a look: #22434

ivanyu · 2026-04-26T07:50:52Z

Hi @chia7712! Thanks for your feedback. This KIP will land only in Kafka 4.4.0 (due to 4.3.0 KIP freeze already in effect). I'll implement what you suggest by then, including the proposed observability KIP.

junrao

@ivanyu : Thanks for the PR. Just a minor comment.

junrao · 2026-05-01T20:37:15Z

     * @param stickyBatchSize How much to produce to partition before switch
     */
-    public BuiltInPartitioner(LogContext logContext, String topic, int stickyBatchSize) {
+    public BuiltInPartitioner(LogContext logContext, String topic, int stickyBatchSize, boolean rackAware, String rack) {


Could we add the new params in the javadoc?

@junrao sure, please have a look: #22432

As a follow-up to comments in apache#19850

github-actions Bot added triage PRs from the community producer clients labels May 29, 2025

ivanyu force-pushed the kip-1123-rack-aware-partitioning branch 11 times, most recently from 1af9189 to 564e60f Compare May 31, 2025 06:14

ivanyu marked this pull request as ready for review May 31, 2025 06:20

github-actions Bot added the needs-attention label Jun 6, 2025

zornhsu mentioned this pull request Jun 16, 2025

feat(producer): rack-aware partitioner IBM/sarama#3198

Closed

github-actions Bot removed the needs-attention label Aug 22, 2025

github-actions Bot added the needs-attention label Aug 23, 2025

showuon self-assigned this Sep 17, 2025

github-actions Bot removed the needs-attention label Sep 17, 2025

github-actions Bot added the needs-attention label Sep 18, 2025

LiamClarkeNZ approved these changes Oct 4, 2025

View reviewed changes

github-actions Bot removed needs-attention triage PRs from the community labels Oct 5, 2025

viktorsomogyi reopened this Mar 30, 2026

github-actions Bot added the triage PRs from the community label Mar 30, 2026

viktorsomogyi removed stale Stale PRs closed-stale PRs that were closed due to inactivity labels Mar 30, 2026

ivanyu force-pushed the kip-1123-rack-aware-partitioning branch from 564e60f to 6816a21 Compare March 30, 2026 08:45

viktorsomogyi added ci-approved and removed triage PRs from the community labels Mar 30, 2026

ivanyu force-pushed the kip-1123-rack-aware-partitioning branch 3 times, most recently from 1c6c786 to 520eeed Compare March 30, 2026 13:49

viktorsomogyi reviewed Mar 31, 2026

View reviewed changes

Comment thread clients/src/main/java/org/apache/kafka/clients/producer/internals/BuiltInPartitioner.java

ivanyu force-pushed the kip-1123-rack-aware-partitioning branch from 520eeed to 1a47d94 Compare April 2, 2026 06:03

viktorsomogyi reviewed Apr 20, 2026

View reviewed changes

ivanyu force-pushed the kip-1123-rack-aware-partitioning branch from 1a47d94 to 7404aa9 Compare April 21, 2026 07:25

ivanyu force-pushed the kip-1123-rack-aware-partitioning branch from 7404aa9 to e8965db Compare April 21, 2026 09:15

viktorsomogyi approved these changes Apr 21, 2026

View reviewed changes

viktorsomogyi merged commit a3f1732 into apache:trunk Apr 22, 2026
34 of 36 checks passed

chia7712 reviewed Apr 22, 2026

View reviewed changes

ivanyu deleted the kip-1123-rack-aware-partitioning branch April 26, 2026 07:51

junrao reviewed May 1, 2026

View reviewed changes

This was referenced May 31, 2026

KAFKA-19193: add Javadoc for rack-aware params in BuiltInPartitioner #22432

Open

KAFKA-19193: throw ConfigException if rack is empty in rack-aware mode #22433

Open

KAFKA-19193: trace-log rack-specific load stats in rack-aware mode #22434

Open

ivanyu added a commit to ivanyu/kafka that referenced this pull request May 31, 2026

KAFKA-19193: trace-log rack-specific load stats in rack-aware mode

a4f26a2

As a follow-up to comments in apache#19850

	private volatile PartitionLoadStatsHolder partitionLoadStats = null;
	private volatile PartitionLoadStatsHolder partitionLoadStatsHolder = null;

-        BuiltInPartitioner builtInPartitionerA = new SequentialPartitioner(logContext, TOPIC_A, 3, false, "");
+        final boolean rackAware = false;
+        final String clientRackId = "";
+        BuiltInPartitioner builtInPartitionerA = new SequentialPartitioner(logContext, TOPIC_A, 3, rackAware, clientRackId);

	BuiltInPartitioner builtInPartitionerB = new SequentialPartitioner(logContext, TOPIC_B, 1, false, "");
	BuiltInPartitioner builtInPartitionerB = new SequentialPartitioner(logContext, TOPIC_B, 1, rackAware, clientRackId);

Conversation

ivanyu commented May 29, 2025 • edited by github-actions Bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actions Bot commented Jun 6, 2025

Uh oh!

marcospassos commented Aug 21, 2025

Uh oh!

ivanyu commented Aug 22, 2025

Uh oh!

github-actions Bot commented Aug 23, 2025

Uh oh!

github-actions Bot commented Sep 18, 2025

Uh oh!

LiamClarkeNZ left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

ivanyu commented Mar 19, 2026

Uh oh!

ivanyu commented Mar 28, 2026

Uh oh!

ivanyu commented Mar 30, 2026

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

viktorsomogyi left a comment

Choose a reason for hiding this comment

Uh oh!

ivanyu commented Apr 21, 2026

Uh oh!

Uh oh!

chia7712 left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

ivanyu commented Apr 26, 2026

Uh oh!

junrao left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

9 participants

ivanyu commented May 29, 2025 •

edited by github-actions Bot

Loading