windows Kafka：无法启动 Kafka - 进程无法访问文件 00000000000000000000.timeindex

Kafka: unable to start Kafka - process can not access file 00000000000000000000.timeindex

提问by a_a

Kafka enthusiast, need little help here. I am unable to start kafka because the file \00000000000000000000.timeindexis being used by another process. Below are the logs:

卡夫卡爱好者，这里几乎不需要帮助。我无法启动 kafka，因为\00000000000000000000.timeindex另一个进程正在使用该文件。以下是日志：

[2017-08-09 22:49:22,811] FATAL [Kafka Server 0], Fatal error during KafkaServer startup. Prepare to shutdown (kafka.server.KafkaServer)
java.nio.file.FileSystemException: \installation\kafka_2.11-0.11.0.0\log\test-0
java.nio.file.FileSystemException: \installation\kafka_2.11-0.11.0.0\log\test-0
$ ps aux | grep zookeeper
$ sudo kill -9 <PID> 
000000000000000000.timeindex: The process cannot access the file because it is being used by another process.
000000000000000000.timeindex: The process cannot access the file because it is being used by another process.

        at sun.nio.fs.WindowsException.translateToIOException(WindowsException.java:86)
        at sun.nio.fs.WindowsException.rethrowAsIOException(WindowsException.java:97)
        at sun.nio.fs.WindowsException.rethrowAsIOException(WindowsException.java:102)
        at sun.nio.fs.WindowsFileSystemProvider.implDelete(WindowsFileSystemProvider.java:269)
        at sun.nio.fs.AbstractFileSystemProvider.deleteIfExists(AbstractFileSystemProvider.java:108)
        at java.nio.file.Files.deleteIfExists(Files.java:1165)
        at kafka.log.Log$$anonfun$loadSegmentFiles.apply(Log.scala:311)
        at kafka.log.Log$$anonfun$loadSegmentFiles.apply(Log.scala:272)
        at scala.collection.TraversableLike$WithFilter$$anonfun$foreach.apply(TraversableLike.scala:733)
        at scala.collection.IndexedSeqOptimized$class.foreach(IndexedSeqOptimized.scala:33)
        at scala.collection.mutable.ArrayOps$ofRef.foreach(ArrayOps.scala:186)
        at scala.collection.TraversableLike$WithFilter.foreach(TraversableLike.scala:732)
        at kafka.log.Log.loadSegmentFiles(Log.scala:272)
        at kafka.log.Log.loadSegments(Log.scala:376)
        at kafka.log.Log.<init>(Log.scala:179)
        at kafka.log.Log$.apply(Log.scala:1580)
        at kafka.log.LogManager$$anonfun$loadLogs$$anonfun$$anonfun$apply$$anonfun$apply.apply$mcV$sp(LogManager.scala:172)
        at kafka.utils.CoreUtils$$anon.run(CoreUtils.scala:57)
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
        at java.util.concurrent.FutureTask.run(FutureTask.java:266)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:745)
[2017-08-09 22:49:22,826] INFO [Kafka Server 0], shutting down (kafka.server.KafkaServer)

回答by patelb

I had the same issue. The only way I could figure it out that was just delete the C:\tmp\kafka-logsdirectory. After that i was able to start up the kafka server.

我遇到过同样的问题。我能弄清楚的唯一方法就是删除C:\tmp\kafka-logs目录。之后我就可以启动kafka服务器了。

You will lose your data and the offset will start from 0.

您将丢失数据，偏移量将从0开始。

回答by SkyWalker

log.dirs=/tmp/logs/kafka**

00000000000000000000.timeindex is being used by another process. So you can delete the process by using following command

00000000000000000000.timeindex 正在被另一个进程使用。因此，您可以使用以下命令删除该进程

log.dirs=/tmp/logs/kafka1

Here PID is the zookeeper's process ID.

这里的PID是zookeeper的进程ID。

---

---

The problem is not fixed yet. It is described here: https://issues.apache.org/jira/browse/KAFKA-1194

问题还没有解决。它在这里描述：https: //issues.apache.org/jira/browse/KAFKA-1194

There are 2 ways for temporary solutiongiven by ephemeral972:

ephemeral972给出了两种临时解决方法：

1. [Recommended] You need to clean up the broker ids in the zookeeper path /brokers/ids/[]. Use the zk-cli tool delete command to clean up the paths. Start your brokers and verify it registers with the coordinator.
2. The other way of resolving this is to change your broker-id from kafka server config and restarting the broker. However, this would corrupt your partitions and data is not recommended

1. 【推荐】需要清理zookeeper路径/brokers/ids/[]中的broker id。使用 zk-cli tool delete 命令清理路径。启动您的经纪人并验证它向协调器注册。
2. 解决此问题的另一种方法是从 kafka 服务器配置更改您的代理 ID 并重新启动代理。但是，这会损坏您的分区，不建议使用数据

回答by spock

I got this error too while running kafka on windows. You can avoid this error by changing the default config in sever.properties file.

在 Windows 上运行 kafka 时我也遇到了这个错误。您可以通过更改 sever.properties 文件中的默认配置来避免此错误。

Please follow these steps:

请按照以下步骤操作：

1. Go to the config folder of kafka installation.
2. Open the Server.properties file
3. you will see the config

1. 进入kafka安装的config文件夹。
2. 打开 Server.properties 文件
3. 你会看到配置

A comma separated list of directories under which to store log files:

以逗号分隔的目录列表，用于存储日志文件：

kafka-server-stop.sh

zookeeper-server-stop.sh

Change the value of log.dirs=/tmp/logs/kafkato some other value, for example:

将 的值更改为log.dirs=/tmp/logs/kafka其他值，例如：

00000000000000000000.index
00000000000000000000.log
00000000000000000000.timeindex
leader-epoch-checkpoint

1. Now start your kafka-server again.

4. 现在再次启动您的 kafka 服务器。

This should solve the issue.

这应该可以解决问题。

回答by Surasin Tancharoen

All answers give you a same solution by remove data, not how to prevent the problem.

所有答案都通过删除数据为您提供相同的解决方案，而不是如何预防问题。

Which actually, you just need to stop Kafka and Zookeepter properly.

实际上，您只需要正确停止 Kafka 和 Zookeeper。

You just have run these two commands in order

您只需按顺序运行这两个命令

# The minimum age of a log file to be eligible for deletion due to age
log.retention.hours=-1

Then next time when you start, you will see no problems.

然后下次开始时，您将看不到任何问题。

回答by ryry

I faced the same issue and restarting kafka and zook then windows didn't work for me. what works for me (Don't reproduce that in Production mode, I'm not sure it will works fine but it could be acceptable with a DEVELOPMENT kafka server.

我遇到了同样的问题并重新启动了 kafka 和 zook，然后 Windows 对我不起作用。什么对我有用（不要在生产模式下复制它，我不确定它是否会正常工作，但对于 DEVELOPMENT kafka 服务器来说是可以接受的。

on a dev kafka server: go to the concerned directory (for instance \installation\kafka_2.11-0.11.0.0\log\test-0) and delete all files other than :

在 dev kafka 服务器上：转到相关目录（例如 \installation\kafka_2.11-0.11.0.0\log\test-0）并删除除以下内容之外的所有文件：

log.cleaner.enable=false

Then restart kafka, it was ok for me, after restarting (zookeeper then kafka), kafka add a .snapshotfile and everything was ok.

然后重启kafka，对我来说没问题，重启后（zookeeper然后kafka），kafka添加一个.snapshot文件，一切正常。

回答by Kiran K

This seems to be a known issue that gets trigerred on Windows after 168 hours have elapsed since you last published the message. Apparently this issue is being tracked and worked on here: KAFKA-8145

这似乎是一个已知问题，在您上次发布消息后 168 小时后会在 Windows 上触发。显然这个问题正在这里被跟踪和处理：KAFKA-8145

There are 2 workarounds for this:

有两种解决方法：

1. As suggested by others here you can clean up your directory containing your log files (or take a back up and have log.dirs point to another directory). However by this way you will loose your data.
2. Go to you server.properties file and make following changes to it. Note: This is temporary solution to allow your consumers to come up and consume any remaining data so that there is no data loss. After having got all the data you need you should revert to Step 1 to clean up your data folder once and for all.

1. 正如其他人在这里建议的那样，您可以清理包含日志文件的目录（或备份并让 log.dirs 指向另一个目录）。但是，通过这种方式，您将丢失数据。
2. 转到您的 server.properties 文件并对其进行以下更改。注意：这是临时解决方案，允许您的消费者上来并使用任何剩余数据，以免数据丢失。获得所需的所有数据后，您应该返回到步骤 1 以一劳永逸地清理您的数据文件夹。

Update below property to prescribed value

将以下属性更新为规定值

log.dirs=d:\tmp\kafka-logs

Add this property at the end of your properties file.

在属性文件的末尾添加此属性。

log.dirs=d:/tmp/kafka-logs

Essentially what you are doing is that you are saying to the Kafka broker to not bother deleting old messages and that the age of all messages is now infinite i.e they will never be deleted. As you can see this is obviously not a desirable state and hence you should only do this in order for you to be able to consume whatever you need and then clean up your files / directory (Step 1). IMHO that the JIRA issue mentioned above is worked on soon and as per this commentlooks like it may soon be resolved.

本质上，您正在做的是告诉 Kafka 代理不要打扰删除旧消息，并且所有消息的年龄现在是无限的，即它们永远不会被删除。正如您所看到的，这显然不是一个理想的状态，因此您应该这样做，以便您能够使用所需的任何内容，然后清理您的文件/目录（步骤 1）。恕我直言，上面提到的 JIRA 问题 很快就会得到解决，根据此评论，它似乎很快就会得到解决。

回答by Homayoun Behzadian

I configure tmp path as below: (in file ./config/server.properties)

我配置 tmp 路径如下：（在文件 ./config/server.properties 中）

Change the logs.dir in config/server.properties to new location
Change the dataDir in config/zookeeper.properties to new location
Restart zookeeper and kafka

then I changed from backslash '\' to '/':

然后我从反斜杠 '\' 改为 '/'：

##代码##

and create folder to solve the problem

并创建文件夹来解决问题

回答by sivaganti

For me it worked after renaming the log files log.dirs=D:/kafka_2.13-2.4.0/data/kafka to kafka1.

对我来说，它在将日志文件 log.dirs=D:/kafka_2.13-2.4.0/data/kafka 重命名为 kafka1 后起作用了。

Also modified the log.retention.hours=1 , to avoid repetition of issue

还修改了 log.retention.hours=1 ，以避免问题重复

回答by user1775015

I faced the same problem and this is how i resolved it.

我遇到了同样的问题，这就是我解决它的方法。

Change the log.dirs path in server.properties log.dirs=C:\kafka\logs

更改 server.properties 中的 log.dirs 路径 log.dirs=C:\kafka\logs

Another solution which worked : delete all files from the below dir wherever configured kafkalogs\test-0

另一个有效的解决方案：从下面的目录中删除所有配置 kafkalogs\test-0 的文件

回答by fatcook

I had similar issue on windows , partly because i had deleted couple of topics ( since i found no other way to just flush only the messages from those topics ). This is what worked for me.

我在 Windows 上遇到了类似的问题，部分原因是我删除了几个主题（因为我找不到其他方法只刷新这些主题中的消息）。这对我有用。

##代码##

The above obviously will work when you have no other topics other than what you deleted on the zookeeper/kafka to cater for , if there are other topics which you still want to retain configuration for , i believe the solution proposed by @Sumit Das might work. I had issues starting zkCli on my windows and i had only those topics which i deleted on my brokers , so i could safely do the above steps and get away with it.

当您除了在zookeeper / kafka上删除的内容之外没有其他主题可以满足时，上述显然会起作用，如果您仍然希望保留其他主题的配置，我相信@Sumit Das提出的解决方案可能会奏效. 我在 Windows 上启动 zkCli 时遇到问题，而且我只有那些我在经纪人上删除的主题，所以我可以安全地执行上述步骤并摆脱它。