将 JSON 文件导入 Logstash + Elasticsearch + Kibana
声明：本页面是StackOverFlow热门问题的中英对照翻译，遵循CC BY-SA 4.0协议，如果您需要使用它，必须同样遵循CC BY-SA许可，注明原文地址和作者信息，同时你必须将它归于原作者（不是我）：StackOverFlow
原文地址: http://stackoverflow.com/questions/25977423/
Warning: these are provided under cc-by-sa 4.0 license. You are free to use/share it, But you must attribute it to the original authors (not me):
StackOverFlow
Import JSON Files into Logstash + Elasticsearch + Kibana
提问by Pedro M. Silva
So, I have a web platform that prints a JSON file per request containing some log data about that request. I can configure several rules about when should it log stuff, only at certain levels, etc...
因此，我有一个 Web 平台，可以为每个请求打印一个 JSON 文件，其中包含有关该请求的一些日志数据。我可以配置几个关于何时应该记录内容的规则，仅在某些级别等......
Now, I've been toying with the Logstash + Elasticsearch + Kibana3 stack, and I'd love to find a way to see those logs in Kibana. My question is, is there a way to make Logstash import these kind of files, or would I have to write a custom input plugin for it? I've searched around and for what I've seen, plugins are written in Ruby, a language I don't have experience with.
现在，我一直在玩弄 Logstash + Elasticsearch + Kibana3 堆栈，我很想找到一种在 Kibana 中查看这些日志的方法。我的问题是，有没有办法让 Logstash 导入这些类型的文件，还是我必须为它编写自定义输入插件？我四处搜索，发现插件是用 Ruby 编写的，我没有使用过这种语言。
回答by griffon vulture
Logstash is a very good tool for processing dynamic files.
Logstash 是一个非常好的处理动态文件的工具。
Here is the way to import your json file into elasticsearch using logstash:
以下是使用 logstash 将 json 文件导入 elasticsearch 的方法：
configuration file:
配置文件：
input
{
file
{
path => ["/path/to/json/file"]
start_position => "beginning"
sincedb_path => "/dev/null"
exclude => "*.gz"
}
}
filter
{
mutate
{
replace => [ "message", "%{message}" ]
gsub => [ 'message','\n','']
}
if [message] =~ /^{.*}$/
{
json { source => message }
}
}
output
{
elasticsearch {
protocol => "http"
codec => json
host => "localhost"
index => "json"
embedded => true
}
stdout { codec => rubydebug }
}
example of json file:
json文件示例：
{"foo":"bar", "bar": "foo"}
{"hello":"world", "goodnight": "moon"}
Note the json need to be in one line. if you want to parse a multiline json file, replace relevant fields in your configuration file:
请注意 json 需要在一行中。如果要解析多行 json 文件，请替换配置文件中的相关字段：
input
{
file
{
codec => multiline
{
pattern => '^\{'
negate => true
what => previous
}
path => ["/opt/mount/ELK/json/*.json"]
start_position => "beginning"
sincedb_path => "/dev/null"
exclude => "*.gz"
}
}
filter
{
mutate
{
replace => [ "message", "%{message}}" ]
gsub => [ 'message','\n','']
}
if [message] =~ /^{.*}$/
{
json { source => message }
}
}
回答by Fred the Magic Wonder Dog
Logstash is just a tool for converting various kinds of syslog files into JSON and loading them into elasticsearch (or graphite, or... ).
Logstash 只是一个将各种 syslog 文件转换为 JSON 并将它们加载到 elasticsearch（或石墨，或...）的工具。
Since your files are already in JSON, you don't need logstash. You can upload them directly into elasticsearch using curl.
由于您的文件已经在 JSON 中，因此您不需要 logstash。您可以使用 curl 将它们直接上传到 elasticsearch。
See Import/Index a JSON file into Elasticsearch
请参阅将 JSON 文件导入/索引到 Elasticsearch
However, in order to work well with Kibana, your JSON files need to be at a minimum.
但是，为了与 Kibana 配合使用，您的 JSON 文件必须最少。
Flat - Kibana does not grok nested JSON structs. You need a simple hash of key/value pairs.
Have a identifiable timestamp.
Flat - Kibana 不了解嵌套的 JSON 结构。您需要一个简单的键/值对散列。
有一个可识别的时间戳。
What I would suggest is looking the JSON files logstash outputs and seeing if you can massage your JSON files to match that structure. You can do this in any language you
like that supports JSON. The program jqis very handy for filtering json from one format to another.
我的建议是查看 JSON 文件 logstash 输出，看看是否可以调整 JSON 文件以匹配该结构。您可以使用任何您喜欢的支持 JSON 的语言来执行此操作。该程序jq对于将 json 从一种格式过滤到另一种格式非常方便。
Logstash format - https://gist.github.com/jordansissel/2996677
Logstash 格式 - https://gist.github.com/jordansissel/2996677
回答by b0ti
Logstash can import different formats and sources as it provides a lot of plugins. There are also other log collector and forwarder tools that can send logs to logstash such as nxlog, rsyslog, syslog-ng, flume, kafka, fluentd, etc. From what I've heard most people use nxlog on windows (though it works on linux equally well) in combination with the ELK stack because of its low resource footprint. (Disclaimer: I'm affiliated with the project)
Logstash 可以导入不同的格式和来源，因为它提供了很多插件。还有其他日志收集器和转发器工具可以将日志发送到 logstash，例如nxlog、rsyslog、syslog-ng、flume、kafka、fluentd 等。据我所知，大多数人在 Windows 上使用 nxlog（尽管它适用于linux 同样好）与 ELK 堆栈结合使用，因为它的资源占用低。（免责声明：我隶属于该项目）
