java 消费者。如何指定要读取的分区?[卡夫卡]

声明:本页面是StackOverFlow热门问题的中英对照翻译,遵循CC BY-SA 4.0协议,如果您需要使用它,必须同样遵循CC BY-SA许可,注明原文地址和作者信息,同时你必须将它归于原作者(不是我):StackOverFlow 原文地址: http://stackoverflow.com/questions/46492707/
Warning: these are provided under cc-by-sa 4.0 license. You are free to use/share it, But you must attribute it to the original authors (not me): StackOverFlow

提示:将鼠标放在中文语句上可以显示对应的英文。显示中英文
时间:2020-11-03 09:14:39  来源:igfitidea点击:

consumer.How to specify partition to read? [kafka]

javaapache-kafkapartitionconsumer

提问by gstackoverflow

I am introducing with kafka and I want to know how to specify partition when I consume messages from topic.

我正在使用 kafka 进行介绍,我想知道在使用来自主题的消息时如何指定分区。

I have found several picture like this:

我找到了几张这样的图片:

enter image description here

在此处输入图片说明

It means that 1 consumer can consume messages from several partitions but 1 partition can be read by single consumer(within consumer group)

这意味着 1 个消费者可以消费来自多个分区的消息,但单个消费者(在消费者组内)可以读取 1 个分区

Also I have read several examples for consumer and it looks like this:

我还为消费者阅读了几个示例,它看起来像这样:

Properties props = new Properties();
props.put("bootstrap.servers", "localhost:9092");
props.put("group.id", "consumer-tutorial");
props.put("key.deserializer", StringDeserializer.class.getName());
props.put("value.deserializer", StringDeserializer.class.getName());
KafkaConsumer<String, String> consumer = new KafkaConsumer<>(props); 

and:

和:

1.subscribe:

1.订阅:

consumer.subscribe(Arrays.asList(“foo”, “bar”)); 

2. poll

2. 投票

 try {
      while (running) {
        ConsumerRecords<String, String> records = consumer.poll(1000);
        for (ConsumerRecord<String, String> record : records)
          System.out.println(record.offset() + ": " + record.value());
      }
    } finally {
      consumer.close();
    }

How does this work? From which partition will I read messages?

这是如何运作的?我将从哪个分区读取消息?

回答by Treziac

There are two ways to tell what topic/partitions you want to consume: KafkaConsumer#assign()(you specify the partition you want and the offset where you begin) and subscribe(you join a consumer group, and partition/offset will be dynamically assigned by group coordinator depending of consumers in the same consumer group, and may change during runtime)

有两种方法可以告诉您要消费的主题/分区:KafkaConsumer#assign()(您指定所需的分区和开始的偏移量)和subscribe(您加入消费者组,分区/偏移量将被动态分配由组协调器取决于同一消费者组中的消费者,并且可能在运行时发生变化)

In both case, you need to pollto receive data.

在这两种情况下,您都需要poll接收数据。

See https://kafka.apache.org/0110/javadoc/index.html?org/apache/kafka/clients/consumer/KafkaConsumer.html, especially paragraphs Consumer Groups and Topic Subscriptionsand Manual Partition Assignment

参见https://kafka.apache.org/0110/javadoc/index.html?org/apache/kafka/clients/consumer/KafkaConsumer.html,尤其是段落Consumer Groups and Topic SubscriptionsManual Partition Assignment