scala 如何构建高效的Kafka broker健康检查？

How to build efficient Kafka broker healthcheck?

提问by codejitsu

In my app I will perform some kind of health check of my Kafka cluster.

在我的应用程序中，我将对我的 Kafka 集群执行某种健康检查。

Currently I make a TopicMetadataRequestto detect dead brokers:

目前我做了一个TopicMetadataRequest来检测死经纪人：

  Future {
    // this will fail if Kafka is unavailable
    consumer.send(new TopicMetadataRequest(Seq("health-check-topic"), 1))
  }

Unfortunately this call produces a huge network traffic, because of Cluster topology/settings.

不幸的是，由于集群拓扑/设置，此调用会产生巨大的网络流量。

Is there a better way to check kafka brokers? What I need is something simple like true/falseindicator.

有没有更好的方法来检查 kafka 经纪人？我需要的是像true/false指标这样简单的东西。

采纳答案by Harvinder Singh

I would strongly recommend you to use Yahoo Kafka Manager, which provides all the information related to Kafka setup. (e.g. bytes sent/consumed over a time interval). This tool can also be used for managing your Kafka Cluster.

我强烈建议您使用 Yahoo Kafka Manager，它提供了与 Kafka 设置相关的所有信息。（例如，在一个时间间隔内发送/消耗的字节数）。此工具还可用于管理您的 Kafka 集群。

It also exposes Restful API and you can consume these API in your own application, if needed. Follow the following link to access it.

它还公开了 Restful API，如果需要，您可以在自己的应用程序中使用这些 API。按照以下链接访问它。

https://github.com/yahoo/kafka-manager

https://github.com/yahoo/kafka-manager

回答by sm4rk0

If you want to build your own health check, this is a current (January 2020) list of KIPscovering health checks:

如果您想构建自己的健康检查，这是当前（2020 年 1 月）涵盖健康检查的KIP列表：

• KIP-143: Controller Health Metrics
• KIP-188: Add new metrics to support health checks
• KIP-237: More Controller Health Metrics

• KIP-143：控制器健康指标
• KIP-188：添加新指标以支持健康检查
• KIP-237：更多控制器健康指标

Regarding Harvinder Singh's currently accepted answer:

关于哈文德·辛格的目前接受的答案：

Kafka Manageris great but it's evolving slowly. There's of course Confluent Control Center- a part of Confluent Platform, but you'll need a license for it. Confluent is a company founded by the team that built Apache Kafka. I've heard about akHQ (ex KafkaHQ)(HackerNews story). Here's a list of management consoles maintained on Apache Kafka Confluence page(check URLs there):

Kafka Manager很棒，但发展缓慢。当然还有Confluent Control Center- Confluent Platform 的一部分，但您需要获得许可证。Confluent 是由构建 Apache Kafka 的团队创立的公司。我听说过akHQ（前 KafkaHQ）（HackerNews 故事）。这是在Apache Kafka Confluence 页面上维护的管理控制台列表（检查那里的 URL）：

• Kafka Manager - A tool for managing Apache Kafka.
• kafkat - Simplified command-line administration for Kafka brokers.
• Kafka Web Console - Displays information about your Kafka cluster including which nodes are up and what topics they host data for.
• Kafka Offset Monitor - Displays the state of all consumers and how far behind the head of the stream they are.
• Capillary - Displays the state and deltas of Kafka-based Apache Storm topologies. Supports Kafka >= 0.8. It also provides an API for fetching this information for monitoring purposes.
• Doctor Kafka - Service for cluster auto healing and workload balancing.
• Cruise Control - Fully automate the dynamic workload rebalance and self-healing of a Kafka cluster.
• Burrow - Monitoring companion that provides consumer lag checking as a service without the need for specifying thresholds.
• Chaperone - An audit system that monitors the completeness and latency of data stream.

• Kafka Manager - 用于管理 Apache Kafka 的工具。
• kafkat - Kafka 代理的简化命令行管理。
• Kafka Web 控制台 - 显示有关 Kafka 集群的信息，包括哪些节点已启动以及它们为哪些主题托管数据。
• Kafka Offset Monitor - 显示所有消费者的状态以及他们落后于流头部的程度。
• Capillary - 显示基于 Kafka 的 Apache Storm 拓扑的状态和增量。支持 Kafka >= 0.8。它还提供了一个 API，用于获取此信息以进行监控。
• Doctor Kafka - 用于集群自动修复和工作负载平衡的服务。
• Cruise Control - 完全自动化 Kafka 集群的动态工作负载重新平衡和自我修复。
• Burrow - 监控伴侣，无需指定阈值即可将消费者滞后检查作为服务提供。
• Chaperone - 监控数据流完整性和延迟的审计系统。

If you don't need GUI, there are also:

如果您不需要 GUI，还有：

• https://github.com/andreas-schroeder/kafka-health-check
• and its fork https://github.com/ustream/kafka-health-check

• https://github.com/andreas-schroeder/kafka-health-check
• 及其分支https://github.com/ustream/kafka-health-check

回答by Harvinder Singh

You can also use Zookeeper API to get the broker list as follows:

您还可以使用 Zookeeper API 获取代理列表，如下所示：

ZooKeeper zk = new ZooKeeper(KafkaContextLookupUtil.getZookeeperConnect().getZkConnect(), 10000, null);
    List<String> ids = zk.getChildren("/brokers/ids", false);
    List<Map> brokerList = new ArrayList<>();
    ObjectMapper objectMapper = new ObjectMapper();

    for (String id : ids) {
        Map map = objectMapper.readValue(zk.getData("/brokers/ids/" + id, false, null), Map.class);
        brokerList.add(map);
    }
    return brokerList;