java 如何监控 hbase 的主要压缩

How to monitor major compaction of hbase

提问by Julias

I want to monitor major compaction of the hbase. For each table I would like to know

我想监视 hbase 的主要压缩。对于我想知道的每张桌子

• when it starts (by hbase or by user)
• when it finished

• 启动时（由 hbase 或由用户）
• 当它完成时

How can I monitor this?

我该如何监控？

• via JMX ?
• via UI ?

• 通过 JMX ?
• 通过用户界面？

In general what additional parameters should I monitor to detect that the major compaction is needed?

一般来说，我应该监控哪些额外的参数来检测需要主要压缩？

采纳答案by eclark

HBase exports lots of metrics through Apache Hadoop's metrics systems. These metrics are also exposed through jmx. So gangliaor jmxare pretty easy ways to get started on watching the metrics that HBase exports. Other options include OpenTSDB and propriety entries.

HBase 通过 Apache Hadoop 的指标系统导出大量指标。这些指标也通过 jmx 公开。因此，ganglia或jmx是开始观察 HBase 导出的指标的非常简单的方法。其他选项包括 OpenTSDB 和专有条目。

Inside of ganglia or jmx the metric you will want to look at is the hbase.regionserver.compactionQueueSize. That is the number of compactions that have been scheduled.

在 ganglia 或 jmx 内部，您要查看的指标是 hbase.regionserver.compactionQueueSize。这是已安排的压缩次数。

The RegionServer UI also includes a copy of some of the metrics, including compactionQueueSize. Later versions of HBase will look better but this gives an example.

RegionServer UI 还包括一些指标的副本，包括 compactionQueueSize。更高版本的 HBase 看起来会更好，但这给出了一个示例。

回答by user43552

I know the answer is a bit late, but instead of manually looking at the Log-Files you could also give Hannibal a try. It is a open-source tool that provides History-Graphs for each region. The graphs show when compactions happened and how long they took.

我知道答案有点晚了，但是除了手动查看日志文件之外，您还可以尝试一下 Hannibal。它是一个开源工具，可为每个区域提供历史图表。图表显示了压缩发生的时间以及它们花费了多长时间。

https://github.com/sentric/hannibal/

https://github.com/sentric/hannibal/

回答by Engineiro

This is the case for hbase versions 0.96 and 0.98

这是 hbase 版本 0.96 和 0.98 的情况

When a compaction started and Status of HBase Compactions can be gleaned from: <hbase.master.ip>:<hbase.master.info.port>/dumpi.e. 10.4.4.40:60010/dump

当压缩开始时，可​​以从以下位置收集 HBase 压缩的状态： <hbase.master.ip>:<hbase.master.info.port>/dump即10.4.4.40:60010/dump

under the section titled

在标题为的部分下

servers:

服务器：

    <hostname>,60020,1406845954503: ...<other stuff here>...
    totalCompactingKVs=9121040186, currentCompactedKVs=14351361679,
    compactionProgressPct=1.5734347,...

compactionProgressPctgives the percentage of compaction completed that you are currently churning through.

compactionProgressPct给出您当前正在搅拌的已完成压缩的百分比。

A much friendlier way of viewing compactions is here:

一种更友好的查看压缩的方式在这里：

http://<hbase.regionserver.ip>:<hbase.regionserver.info.port>/rs-status#queueStats

http://<hbase.regionserver.ip>:<hbase.regionserver.info.port>/rs-status#queueStats

It will give you stats that look like this:

它将为您提供如下所示的统计信息：

回答by karthikp

Compaction status can be viewed from Hbase log or Regionservers GUI But if we've huge transaction, It is tough to track compacting region status from log. GUI only shows compacting status on the same node and tedious to view in GUI If we've 'n' regionservers

可以从 Hbase 日志或 Regionservers GUI 中查看压缩状态但是如果我们有大量事务，则很难从日志中跟踪压缩区域状态。GUI 仅显示同一节点上的压缩状态，并且在 GUI 中查看很乏味 如果我们有“n”个区域服务器

Download the utility that collect online compacting region status across Regionservers to a given table. Github linkto StatusUtil.java

下载将跨 Regionservers 的在线压缩区域状态收集到给定表的实用程序。Github链接到StatusUtil.java

回答by karthikp

We can get compaction status via the JMX.

我们可以通过 JMX 获取压缩状态。

compactionQueueSize, compactionSizeNumOps, compactionSizeAvgTime, compactionSizeMinTime, compactionSizeMaxTime, compactionTimeNumOps, compactionTimeAvgTime, compactionTimeMinTime, compactionTimeMaxTime

compactionQueueSize、compactionSizeNumOps、compactionSizeAvgTime、compactionSizeMinTime、compactionSizeMaxTime、compactionTimeNumOps、compactionTimeAvgTime、compactionTimeMinTime、compactionTimeMaxTime

Enable JMX metrics as shown at http://hbase.apache.org/metrics.html

启用 JMX 指标，如http://hbase.apache.org/metrics.html所示

You can download the JMX Example here

您可以在此处下载 JMX 示例