java Kafka突然重置消费者Offset
声明：本页面是StackOverFlow热门问题的中英对照翻译，遵循CC BY-SA 4.0协议，如果您需要使用它，必须同样遵循CC BY-SA许可，注明原文地址和作者信息，同时你必须将它归于原作者（不是我）：StackOverFlow
原文地址: http://stackoverflow.com/questions/29276912/
Warning: these are provided under cc-by-sa 4.0 license. You are free to use/share it, But you must attribute it to the original authors (not me):
StackOverFlow
Kafka suddenly reset the consumer Offset
提问by Gnarik
I'm working with Kafka 0.8 & zookeeper 3.3.5. Actually, we have a dozen of topic we are consuming without any issue.
我正在使用 Kafka 0.8 和 zookeeper 3.3.5。实际上，我们有十几个主题正在消费，没有任何问题。
Recently, we started to feed and consume a new topic that has a weird behavior. The consumed offset was suddenly reset. It respects the auto.offset.reset policy we set (actually smallest) but I cannot understand why the topic suddenly reset its offset.
最近，我们开始喂食和消费一个行为怪异的新话题。消耗的偏移量突然重置。它尊重我们设置的 auto.offset.reset 策略（实际上是最小的），但我不明白为什么这个话题突然重置了它的偏移量。
I'm using the high-level consumer.
我正在使用高级消费者。
Here are some ERROR log I found: We have a bunch of this error log:
这是我发现的一些错误日志： 我们有一堆这样的错误日志：
[2015-03-26 05:21:17,789] INFO Fetching metadata from broker id:1,host:172.16.23.1,port:9092 with correlation id 47 for 1 topic(s) Set(MyTopic) (kafka.cl
ient.ClientUtils$)
[2015-03-26 05:21:17,789] ERROR Producer connection to 172.16.23.1:9092 unsuccessful (kafka.producer.SyncProducer)
java.nio.channels.ClosedByInterruptException
at java.nio.channels.spi.AbstractInterruptibleChannel.end(AbstractInterruptibleChannel.java:202)
at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:681)
at kafka.network.BlockingChannel.connect(BlockingChannel.scala:57)
at kafka.producer.SyncProducer.connect(SyncProducer.scala:141)
at kafka.producer.SyncProducer.getOrMakeConnection(SyncProducer.scala:156)
at kafka.producer.SyncProducer.kafka$producer$SyncProducer$$doSend(SyncProducer.scala:68)
at kafka.producer.SyncProducer.send(SyncProducer.scala:112)
at kafka.client.ClientUtils$.fetchTopicMetadata(ClientUtils.scala:53)
at kafka.client.ClientUtils$.fetchTopicMetadata(ClientUtils.scala:88)
at kafka.consumer.ConsumerFetcherManager$LeaderFinderThread.doWork(ConsumerFetcherManager.scala:66)
at kafka.utils.ShutdownableThread.run(ShutdownableThread.scala:51)
Each time this issue happened I see that WARN log:
每次发生此问题时，我都会看到 WARN 日志：
[2015-03-26 05:21:30,596] WARN Reconnect due to socket error: null (kafka.consumer.SimpleConsumer)
And then the real problem happens:
然后真正的问题发生了：
[2015-03-26 05:21:47,551] INFO Connected to 172.16.23.5:9092 for producing (kafka.producer.SyncProducer) [2015-03-26 05:21:47,552] INFO Disconnecting from 172.16.23.5:9092 (kafka.producer.SyncProducer) [2015-03-26 05:21:47,553] INFO [ConsumerFetcherManager-1427047649942] Added fetcher for partitions ArrayBuffer([[MyTopic,0], initOffset 45268422051 to br oker id:5,host:172.16.23.5,port:9092] ) (kafka.consumer.ConsumerFetcherManager) [2015-03-26 05:21:47,553] INFO [ConsumerFetcherThread-MyTopic_group-1427047649884-699191d4-0-5], Starting (kafka.consumer.Cons umerFetcherThread) [2015-03-26 05:21:50,388] ERROR [ConsumerFetcherThread-MyTopic_group-1427047649884-699191d4-0-5], Current offset 45268422051 for partition [MyTopic,0] out of range; reset offset to 1948447612 (kafka.consumer.ConsumerFetcherThread) [2015-03-26 05:21:50,490] ERROR [ConsumerFetcherThread-MyTopic_group-1427047649884-699191d4-0-5], Current offset 1948447612 for partition [MyTopic,0] out of range; reset offset to 1948447612 (kafka.consumer.ConsumerFetcherThread) [2015-03-26 05:21:50,591] ERROR [ConsumerFetcherThread-MyTopic_group-1427047649884-699191d4-0-5], Current offset 1948447612 for partition [MyTopic,0] out of range; reset offset to 1948447612 (kafka.consumer.ConsumerFetcherThread) [2015-03-26 05:21:50,692] ERROR [ConsumerFetcherThread-MyTopic_group-1427047649884-699191d4-0-5], Current offset 1948447612 for partition [MyTopic,0] out of range; reset offset to 1948447612 (kafka.consumer.ConsumerFetcherThread)
Now the questions: Is there someone who has already experienced this behavior ? Is there someone who can tell me when Kafka decides to reset its offset whether the auto.offset.reset is largest or smallest ?
现在的问题是：是否有人已经经历过这种行为？有没有人可以告诉我 Kafka 何时决定重置其偏移量 auto.offset.reset 是最大还是最小？
Thank you.
谢谢你。
回答by C4stor
What's happening is you are depiling your topic too slowly for a while.
发生的事情是您在一段时间内过慢地处理您的主题。
Kafka has a retention model which is not based on whether consumer got the data, but on disk usage and/or period. At some point, you get too late, and the next message you need has already been wiped out, and isn't available anymore, due to kafka having cleaned the data. Hence the Current offset 45268422051 for partition [MyTopic,0] out of range; reset offset to 1948447612messages.
Kafka 的保留模型不是基于消费者是否获得数据，而是基于磁盘使用情况和/或周期。在某些时候，您为时已晚，并且您需要的下一条消息已经被清除，并且不再可用，因为 kafka 已经清理了数据。因此Current offset 45268422051 for partition [MyTopic,0] out of range; reset offset to 1948447612消息。
Your consumer then applies your reset policy to bootstrap itself again, in your case smallest.
然后您的消费者应用您的重置策略再次引导自己，在您的情况下是最小的。
It's a common issue when you have bursty workflows, and sometimes fall out of the data retention range. It probably disappeared because you improved your depiling speed, or increased your retention policy to be able to survive bursts.
当您有突发工作流时，这是一个常见问题，有时会超出数据保留范围。它可能消失了，因为您提高了脱毛速度，或者增加了保留策略以能够在爆发中幸存下来。
