带有 JSON 字段的 Amazon Cloudwatch Logs Insights
声明：本页面是StackOverFlow热门问题的中英对照翻译，遵循CC BY-SA 4.0协议，如果您需要使用它，必须同样遵循CC BY-SA许可，注明原文地址和作者信息，同时你必须将它归于原作者（不是我）：StackOverFlow
原文地址: http://stackoverflow.com/questions/53725133/
Warning: these are provided under cc-by-sa 4.0 license. You are free to use/share it, But you must attribute it to the original authors (not me):
StackOverFlow
Amazon Cloudwatch Logs Insights with JSON fields
提问by Cyril Duchon-Doris
I am trying to use Logs Insights with data containing JSON in one of the fields, and to parse the JSON fields
我正在尝试将 Logs Insights 与其中一个字段中包含 JSON 的数据一起使用，并解析 JSON 字段
My data looks like the following when I put it in insights with the starter code
当我将数据与入门代码一起放入见解中时，我的数据如下所示
fields @timestamp, @message
| sort @timestamp desc
| limit 25
How can I easily extract the pathvariable in my nested JSON to perform aggregations on it ? By looking at some documentation, I thought @message.pathwould work but it does not seem so. Has anyone successfully interpreted JSON logs in Insights
如何轻松提取path嵌套 JSON 中的变量以对其执行聚合？通过查看一些文档，我认为@message.path会起作用，但似乎并非如此。有没有人在 Insights 中成功解释过 JSON 日志
EDIT : Sample of what my data looks like
编辑：我的数据示例
#
@timestamp
@message
1
2018-12-19 23:42:52.000
I, [2018-12-19T23:42:52.629855 #23447] INFO -- : [2ce588f1-c27d-4a55-ac05-62a75b39e762] {"method":"GET","path":"/api/v1/professionals/ID","format":"json","controller":"API::V1::Public::ProfessionalsController","action":"show","status":200,"duration":285.27,"view":222.36,"time":"2018-12-19T23:42:52.344+00:00","params":{"include":"user,tags,promotions,company_sector,similar_professionals.tags,similar_professionals.user","format":"json","compress":false,"id":"ID"},"@timestamp":"2018-12-19T23:42:52.629Z","@version":"1","message":"[200] GET /api/v1/professionals/ID (API::V1::Public::ProfessionalsController#show)"}
@logStream i-05d1d61ab853517a0
@message I, [2018-12-19T23:42:52.629855 #23447] INFO -- : [2ce588f1-c27d-4a55-ac05-62a75b39e762] {"method":"GET","path":"/api/v1/professionals/ID","format":"json","controller":"API::V1::Public::ProfessionalsController","action":"show","status":200,"duration":285.27,"view":222.36,"time":"2018-12-19T23:42:52.344+00:00","params":{"include":"xxx","format":"json","compress":false,"id":"ID"},"@timestamp":"2018-12-19T23:42:52.629Z","@version":"1","message":"[200] GET /api/v1/professionals/ID (API::V1::Public::ProfessionalsController#show)"}
@timestamp 1545262972000
2
2018-12-19 23:42:16.000
I, [2018-12-19T23:42:16.723472 #851] INFO -- : [ea712503-eb86-4a6e-ab38-ddbcd6c2b4d0] {"method":"GET","path":"/api/v1/heartbeats/new","format":"json","controller":"API::V1::Public::HeartbeatsController","action":"new","status":201,"duration":9.97,"view":3.2,"time":"2018-12-19T23:42:16.712+00:00","params":{"format":"json","compress":false},"@timestamp":"2018-12-19T23:42:16.722Z","@version":"1","message":"[201] GET /api/v1/heartbeats/new (API::V1::Public::HeartbeatsController#new)"}
回答by pyb
CloudWatch Insights Logs automatically discovers fields for the following log types:
CloudWatch Insights Logs 会自动发现以下日志类型的字段：
Lambda logs
CloudWatch Logs Insights automatically discovers log fields in Lambda logs, but only for the first embedded JSON fragment in each log event(Note: emphasis mine). If a Lambda log event contains multiple JSON fragments, you can parse and extract the log fields by using the parse command. For more information, see Fields in JSON Logs.
CloudTrail logs
See fields in JSON logs.
Lambda 日志
CloudWatch Logs Insights 会自动发现 Lambda 日志中的日志字段，但仅针对每个日志事件中的第一个嵌入的 JSON 片段（注意：重点是我的）。如果 Lambda 日志事件包含多个 JSON 片段，您可以使用 parse 命令解析和提取日志字段。有关更多信息，请参阅 JSON 日志中的字段。
CloudTrail 日志
查看JSON 日志中的字段。
Source:Supported Logs and Discovered Fields
来源：支持的日志和发现的字段
If @messageis I, [2018-12-11T13:20:27] INFO -- : {"method":"GET"}
如果@message是I, [2018-12-11T13:20:27] INFO -- : {"method":"GET"}
Then you can select and filter the fields like so:
然后您可以像这样选择和过滤字段：
fields @timestamp, @message, method
| filter method = "GET"
| sort @timestamp desc
It works with nested fields too, i.e. params.format = "json"or results.0.firstName = "Paul".
它也适用于嵌套字段，即params.format = "json"或results.0.firstName = "Paul"。
回答by GusGold
Building on @pyb insights, I was able to use parse @message '"path":"*"' as pathto extract the path from any place in the @message.
以@pyb 见解为基础，我能够用来parse @message '"path":"*"' as path从@message.
You can go on to get your method by piping another parse @message '"method":"*"' as methodwithout concern for ordering as it is a second global plain text search on @message
您可以继续通过管道另一个方法来获取您的方法，parse @message '"method":"*"' as method而无需担心订购，因为它是第二个全局纯文本搜索@message
In the case where your @messageis:
如果您@message是：
I, [2018-12-19T23:42:52.629855 #23447] INFO -- : [2ce588f1-c27d-4a55-ac05-62a75b39e762] {"method":"GET","path":"/api/v1/professionals/ID","format":"json","controller":"API::V1::Public::ProfessionalsController","action":"show","status":200,"duration":285.27,"view":222.36,"time":"2018-12-19T23:42:52.344+00:00","params":{"include":"xxx","format":"json","compress":false,"id":"ID"},"@timestamp":"2018-12-19T23:42:52.629Z","@version":"1","message":"[200] GET /api/v1/professionals/ID (API::V1::Public::ProfessionalsController#show)"}
Using:
使用：
parse @message '"path":"*"' as path | parse @message '"method":"*"' as method
will result in the fields: path = '/api/v1/professionals/ID'and method = 'GET'
将导致字段：path = '/api/v1/professionals/ID'和method = 'GET'
Note that this is still simply string parsing and as such, it has no concept of nested keys like params.formatwould not find json, however using just formatwould, so long as there wasn't another formatstring anywhere in your @message.
请注意，这仍然只是字符串解析，因此，它没有像params.formatwould not find这样的嵌套键的概念json，但是只使用format会，只要format您的@message.
Also note that this is for the case where Insights is not discovering your JSON in the message. I belive this is the case that @pyb was referring to in this answer. My logs aren't being discovered either using the following format
另请注意，这是针对 Insights 未在消息中发现您的 JSON 的情况。我相信这是@pyb 在这个答案中所指的情况。使用以下格式未发现我的日志
info - Request: {"method":"POST","path":"/auth/login/","body":{"login":{"email":"[email protected]","password":"********"}},"uuid":"36d76df2-aec4-4549-8b73-f237e8f14e23","ip":"*.*.*.*"}
回答by pyb
You can use the parsecommand to extract fields.
您可以使用该parse命令提取字段。
If @messageis
如果@message是
I, [2018-12-11T13:20:27] INFO -- : {"method":"GET"}
Then you extract the fields like so:
然后像这样提取字段：
fields @timestamp, @message
| parse "I, [*T*] INFO -- : {"method":"*"}" as @date, @time, @method
| filter method=GET
| sort @timestamp desc
| limit 20
The documentation is rather light for now. I am able to get results by replacing the wildcard *with a regular expression, but then the parsing fails.
文档目前相当简单。我可以通过用*正则表达式替换通配符来获得结果，但解析失败。
回答by R0B0T-B0BA
Would this regex query within the parse command help you?
parse 命令中的这个正则表达式查询对你有帮助吗？
filter @message like / \"path\":\"/
| parse @message /(?<@endpt>((\/[a-zA-Z0-9_{}()-?]+){1,}))/
Good luck!
祝你好运！
回答by Neyma
Also another parsewith help of regex
也是在正则表达式的帮助下的另一个解析
Assume your @messageis:
假设你@message是：
I, [2018-12-19T23:42:52.629855 #23447] INFO -- : [2ce588f1-c27d-4a55-ac05-62a75b39e762] {"method":"GET","path":"/api/v1/professionals/"}
You can extract method by
您可以通过以下方式提取方法
fields @timestamp, @message
| parse @message /\"method\":\"(?<method_type>.*?)\"/
