Java 是否可以将分区添加到 Kafka 0.8.2 中的现有主题

Is it possible to add partitions to an existing topic in Kafka 0.8.2

提问by Asif Iqbal

I have a Kafka cluster running with 2 partitions. I was looking for a way to increase the partition count to 3. However, I don't want to lose existing messages in the topic. I tried stopping Kafka, modifying the server.propertiesfile to increase the number of partitions to 3 and restart Kafka. However, that does not seem to change anything. Using Kafka ConsumerOffsetChecker, I still see it is using only 2 partitions. The Kafka version I am using is 0.8.2.2. In version 0.8.1, there used to be a script called kafka-add-partitions.sh, which I guess might do the trick. However, I don't see any such script in 0.8.2. Is there any way of accomplishing this? I did experiment with creating a whole new topic and for that one it does seem to use 3 partitions as per the change in the server.propertiesfile. However, for existing topics, it doesn't seem to care.

我有一个运行 2 个分区的 Kafka 集群。我一直在寻找将分区数增加到 3 的方法。但是，我不想丢失主题中的现有消息。我尝试停止 Kafka，修改server.properties文件以将分区数增加到 3 并重新启动 Kafka。然而，这似乎并没有改变任何事情。使用 Kafka ConsumerOffsetChecker，我仍然看到它只使用了 2 个分区。我使用的 Kafka 版本是 0.8.2.2。在 0.8.1 版本中，曾经有一个名为 的脚本kafka-add-partitions.sh，我想它可能会起作用。但是，我在 0.8.2 中没有看到任何这样的脚本。有没有办法做到这一点？我确实尝试过创建一个全新的主题，对于那个主题，它似乎根据server.properties文件中的更改使用了 3 个分区。但是，对于现有主题，它似乎并不关心。

采纳答案by blockR

Looks like you can use thisscript instead:

看起来您可以改用此脚本：

bin/kafka-topics.sh --zookeeper zk_host:port/chroot --alter --topic my_topic_name 
   --partitions 40 

In the code it looks like they do same thing:

在代码中看起来他们做同样的事情：

 AdminUtils.createOrUpdateTopicPartitionAssignmentPathInZK(topic, partitionReplicaList, zkClient, true)

kafka-topics.shexecutes thispiece of code as well as AddPartitionsCommandused by kafka-add-partition script.

kafka-topics.sh执行这段代码以及kafka-add-partition 脚本使用的AddPartitionsCommand。

However you have to be aware of re-partitioning when using key:

但是，使用密钥时必须注意重新分区：

Be aware that one use case for partitions is to semantically partition data, and adding partitions doesn't change the partitioning of existing dataso this may disturb consumers if they rely on that partition. That is if data is partitioned by hash(key) % number_of_partitionsthen this partitioning will potentially be shuffled by adding partitions but Kafka will not attempt to automatically redistribute data in any way.

请注意，分区的一个用例是对数据进行语义分区，并且添加分区不会更改现有数据的分区，因此如果消费者依赖该分区，这可能会打扰他们。也就是说，如果数据被分区，hash(key) % number_of_partitions那么这个分区可能会通过添加分区进行混洗，但 Kafka 不会尝试以任何方式自动重新分配数据。

回答by Chandan Kumar

In my case the value zk_host:port/chrootfor parameter --zookeeperthrew the following exception:

在我的情况下zk_host:port/chroot，参数的值--zookeeper引发了以下异常：

ERROR java.lang.IllegalArgumentException: Topic my_topic_name does not exist on ZK path zk_host:port/chroot.

错误 java.lang.IllegalArgumentException：主题 my_topic_name 在 ZK 路径 zk_host:port/chroot 上不存在。

So, I tried the following and it worked:

所以，我尝试了以下方法，它奏效了：

 bin/kafka-topics.sh --alter --zookeeper zk_host:port --topic my_topic_name --partitions 10

回答by Aarya

If you are using Kafka in Windows try this code for alter or add partition in topic

如果您在 Windows 中使用 Kafka，请尝试使用此代码在主题中更改或添加分区

.\bin\windows\kafka-topics.bat --alter --zookeeper localhost:2181 --topic TopicName --partitions 20

.\bin\windows\kafka-topics.bat --alter --zookeeper localhost:2181 --topic TopicName --partitions 20

or

或者

.\bin\windows\kafka-topics.bat --alter --zookeeper localhost:2181 --topic TopicName --replica-assignment 0:1:2,0:1:2,0:1:2,2:1:0 --partitions 10

.\bin\windows\kafka-topics.bat --alter --zookeeper localhost:2181 --topic TopicName --replica-assignment 0:1:2,0:1:2,0:1:2,2:1:0 --partitions 10

回答by c0der512

For anyone who wants solution for newer Kafka versions.Please follow this method.

对于任何想要更新 Kafka 版本的解决方案的人。请遵循此方法。

Kafka's entire data retention and transfer policy depends on partitions so be careful about effects of increasing partitions. (Kafka's newer versions display warning regarding this) Try to avoid configuration in which one broker has too many leader partitions.

Kafka 的整个数据保留和传输策略取决于分区，因此请注意增加分区的影响。（Kafka 的较新版本会显示有关此问题的警告）尽量避免一个代理具有过多领导分区的配置。

There is simple 3 stage approach to this.

对此有简单的 3 阶段方法。

Step 1: Increase the partitions in topics

第一步：增加topics中的partition

./bin/kafka-topics.sh --zookeeper localhost:9092 --alter --topic testKafka_5 --partitions 6

./bin/kafka-topics.sh --zookeeper localhost:9092 --alter --topic testKafka_5 --partitions 6

Step 2: Create a partitioning json file for given topic

第 2 步：为给定主题创建分区 json 文件

{ "version":1, "partitions":[ {"topic":"testKafka_5","partition":0,"replicas":[0,1,2]}, {"topic":"testKafka_5","partition":1,"replicas":[2,1,0]}, {"topic":"testKafka_5","partition":2,"replicas":[1,2,0]}, {"topic":"testKafka_5","partition":3,"replicas":[0,1,2]}, {"topic":"testKafka_5","partition":4,"replicas":[2,1,0]}, {"topic":"testKafka_5","partition":5,"replicas":[1,2,0]} ]}

{ "version":1, "partitions":[ {"topic":"testKafka_5","partition":0,"replicas":[0,1,2]}, {"topic":"testKafka_5"," partition":1,"replicas":[2,1,0]}, {"topic":"testKafka_5","partition":2,"replicas":[1,2,0]}, {"topic" :"testKafka_5","partition":3,"replicas":[0,1,2]}, {"topic":"testKafka_5","partition":4,"replicas":[2,1,0] }, {"topic":"testKafka_5","partition":5,"replicas":[1,2,0]} ]}

Create file with newer partition and replicas. It's better to expand replicas to different brokers but they should be present within same cluster. Take latency into consideration for distant replicas. Transfer the given file to your Kafka.

使用较新的分区和副本创建文件。最好将副本扩展到不同的代理，但它们应该存在于同一个集群中。考虑远程副本的延迟。将给定的文件传输到您的 Kafka。

Step 3: Reassign partitions and verify

步骤 3：重新分配分区并验证

./bin/kafka-reassign-partitions.sh --zookeeper localhost:9092 --reassignment-json-file bin/increase-replication-factor.json  --execute

./bin/kafka-reassign-partitions.sh --zookeeper localhost:9092 --reassignment-json-file bin/increase-replication-factor.json --verify

You can check the effects of your change using --describecommand.

您可以使用--describe命令检查更改的效果。