无需解析日志即可从 Java 应用程序登录到 ELK
声明：本页面是StackOverFlow热门问题的中英对照翻译，遵循CC BY-SA 4.0协议，如果您需要使用它，必须同样遵循CC BY-SA许可，注明原文地址和作者信息，同时你必须将它归于原作者（不是我）：StackOverFlow
原文地址: http://stackoverflow.com/questions/32302421/
Warning: these are provided under cc-by-sa 4.0 license. You are free to use/share it, But you must attribute it to the original authors (not me):
StackOverFlow
Logging from Java app to ELK without need for parsing logs
提问by Rob
I want to send logs from a Java app to ElasticSearch, and the conventional approach seems to be to set up Logstash on the server running the app, and have logstash parse the log files (with regex...!) and load them into ElasticSearch.
我想将日志从 Java 应用程序发送到 ElasticSearch，而传统方法似乎是在运行该应用程序的服务器上设置 Logstash，并让 logstash 解析日志文件（使用正则表达式...！）并将它们加载到 ElasticSearch 中.
Is there a reason it's done this way, rather than just setting up log4J (or logback) to log things in the desired format directly into a log collector that can then be shipped to ElasticSearch asynchronously? It seems crazy to me to have to fiddle with grok filters to deal with multiline stack traces (and burn CPU cycles on log parsing) when the app itself could just log it the desired format in the first place?
这样做是否有原因，而不仅仅是设置 log4J（或 logback）以将所需格式的内容直接记录到日志收集器中，然后可以异步发送到 ElasticSearch？当应用程序本身可以首先以所需的格式记录它时，不得不摆弄 grok 过滤器来处理多行堆栈跟踪（并在日志解析时消耗 CPU 周期）对我来说似乎很疯狂？
On a tangentially related note, for apps running in a Docker container, is best practice to log directly to ElasticSearch, given the need to run only one process?
顺便提一下，对于在 Docker 容器中运行的应用程序，鉴于只需要运行一个进程，直接登录到 ElasticSearch 是最佳实践吗？
采纳答案by Magnus B?ck
I think it's usually ill-advised to log directly to Elasticsearch from a Log4j/Logback/whatever appender, but I agree that writing Logstash filters to parse a "normal" human-readable Java log is a bad idea too. I use https://github.com/logstash/log4j-jsonevent-layouteverywhere I can to have Log4j's regular file appenders produce JSON logs that don't require any further parsing by Logstash.
我认为从 Log4j/Logback/任何 appender 直接登录到 Elasticsearch 通常是不明智的，但我同意编写 Logstash 过滤器来解析“正常”人类可读的 Java 日志也是一个坏主意。我在任何地方都使用https://github.com/logstash/log4j-jsonevent-layout让 Log4j 的常规文件附加程序生成不需要 Logstash 进一步解析的 JSON 日志。
回答by Val
If you really want to go down that path, the idea would be to use something like an Elasticsearch appender(or this oneor this other one) which would ship your logs directly to your ES cluster.
如果你真的想沿着这条路走下去，想法是使用类似Elasticsearch appender（或这个或另一个）的东西，它将你的日志直接发送到你的 ES 集群。
However, I'd advise against it for the same reasons mentioned by @Vineeth Mohan. You'd also need to ask yourself a couple questions, but mainly what would happen if your ES cluster goes down for any reason (OOM, network down, ES upgrade, etc)?
但是，出于@Vineeth Mohan 提到的相同原因，我建议不要这样做。您还需要问自己几个问题，但主要是如果您的 ES 集群因任何原因（OOM、网络故障、ES 升级等）宕机会发生什么？
There are many reasons why asynchronicity exists, one of which is robustness of your architecture and most of the time that's much more important than burning a few more CPU cycles on log parsing.
存在异步性的原因有很多，其中之一是架构的健壮性，而且在大多数情况下，这比在日志解析时消耗更多 CPU 周期要重要得多。
Also note that there is an ongoing discussionabout this very subject going on in the official ES discussion forum.
另请注意，官方 ES 讨论论坛中正在进行有关此主题的讨论。
回答by Marcelo Grossi
If you need a quick solution I've written this appender here Log4J2 Elastic REST Appenderif you want to use it. It has the ability to buffer log events based on time and/or number of events before sending it to Elastic (using the _bulk API so that it sends it all in one go). It has been published to Maven Central so it's pretty straight forward.
如果您需要一个快速解决方案，我已经在此处编写了这个 appender Log4J2 Elastic REST Appender如果您想使用它。它能够在将日志事件发送到 Elastic 之前根据时间和/或事件数量缓冲日志事件（使用 _bulk API 以便一次性发送所有事件）。它已发布到 Maven Central，因此非常简单。
As the other folks have already mentioned the best way to do it would be to save it to file, and then ship it to ES separately. However I think that there is value if you need to get something running quickly until you have time/resources implement the optimal way.
正如其他人已经提到的那样，最好的方法是将其保存到文件中，然后将其单独发送到 ES。但是，我认为如果您需要让某些东西快速运行，直到您有时间/资源以最佳方式实施，这是有价值的。
回答by Felix
There is also https://github.com/elastic/java-ecs-loggingwhich provides a layout for log4j, log4j2 and Logback. It's quite efficient and the Filebeat configuration is very minimal.
还有https://github.com/elastic/java-ecs-logging为 log4j、log4j2 和 Logback 提供布局。它非常高效，而且 Filebeat 配置非常少。
Disclaimer: I'm the author of this library.
免责声明：我是这个库的作者。
