java 消费者。如何指定要读取的分区?[卡夫卡]
声明:本页面是StackOverFlow热门问题的中英对照翻译,遵循CC BY-SA 4.0协议,如果您需要使用它,必须同样遵循CC BY-SA许可,注明原文地址和作者信息,同时你必须将它归于原作者(不是我):StackOverFlow
原文地址: http://stackoverflow.com/questions/46492707/
Warning: these are provided under cc-by-sa 4.0 license. You are free to use/share it, But you must attribute it to the original authors (not me):
StackOverFlow
consumer.How to specify partition to read? [kafka]
提问by gstackoverflow
I am introducing with kafka and I want to know how to specify partition when I consume messages from topic.
我正在使用 kafka 进行介绍,我想知道在使用来自主题的消息时如何指定分区。
I have found several picture like this:
我找到了几张这样的图片:
It means that 1 consumer can consume messages from several partitions but 1 partition can be read by single consumer(within consumer group)
这意味着 1 个消费者可以消费来自多个分区的消息,但单个消费者(在消费者组内)可以读取 1 个分区
Also I have read several examples for consumer and it looks like this:
我还为消费者阅读了几个示例,它看起来像这样:
Properties props = new Properties();
props.put("bootstrap.servers", "localhost:9092");
props.put("group.id", "consumer-tutorial");
props.put("key.deserializer", StringDeserializer.class.getName());
props.put("value.deserializer", StringDeserializer.class.getName());
KafkaConsumer<String, String> consumer = new KafkaConsumer<>(props);
and:
和:
1.subscribe:
1.订阅:
consumer.subscribe(Arrays.asList(“foo”, “bar”));
2. poll
2. 投票
try {
while (running) {
ConsumerRecords<String, String> records = consumer.poll(1000);
for (ConsumerRecord<String, String> record : records)
System.out.println(record.offset() + ": " + record.value());
}
} finally {
consumer.close();
}
How does this work? From which partition will I read messages?
这是如何运作的?我将从哪个分区读取消息?
回答by Treziac
There are two ways to tell what topic/partitions you want to consume: KafkaConsumer#assign()(you specify the partition you want and the offset where you begin) and subscribe
(you join a consumer group, and partition/offset will be dynamically assigned by group coordinator depending of consumers in the same consumer group, and may change during runtime)
有两种方法可以告诉您要消费的主题/分区:KafkaConsumer#assign()(您指定所需的分区和开始的偏移量)和subscribe
(您加入消费者组,分区/偏移量将被动态分配由组协调器取决于同一消费者组中的消费者,并且可能在运行时发生变化)
In both case, you need to poll
to receive data.
在这两种情况下,您都需要poll
接收数据。
See https://kafka.apache.org/0110/javadoc/index.html?org/apache/kafka/clients/consumer/KafkaConsumer.html, especially paragraphs Consumer Groups and Topic Subscriptions
and Manual Partition Assignment
参见https://kafka.apache.org/0110/javadoc/index.html?org/apache/kafka/clients/consumer/KafkaConsumer.html,尤其是段落Consumer Groups and Topic Subscriptions
和 Manual Partition Assignment