Amazon Cloudwatch使用JSON字段记录数据洞察

Question

我试图在其中一个字段中使用包含JSON的数据的Logs Insights，并解析JSON字段

当我使用入门代码将其放入数据时，我的数据如下所示

fields @timestamp, @message
| sort @timestamp desc
| limit 25

如何在嵌套的JSON中轻松提取path变量以对其执行聚合？通过查看一些文档，我认为@message.path会起作用，但似乎并非如此。有没有人成功解释了Insights中的JSON日志

编辑：我的数据的样本

#
@timestamp
@message
1
2018-12-19 23:42:52.000
I, [2018-12-19T23:42:52.629855 #23447] INFO -- : [2ce588f1-c27d-4a55-ac05-62a75b39e762] {"method":"GET","path":"/api/v1/professionals/ID","format":"json","controller":"API::V1::Public::ProfessionalsController","action":"show","status":200,"duration":285.27,"view":222.36,"time":"2018-12-19T23:42:52.344+00:00","params":{"include":"user,tags,promotions,company_sector,similar_professionals.tags,similar_professionals.user","format":"json","compress":false,"id":"ID"},"@timestamp":"2018-12-19T23:42:52.629Z","@version":"1","message":"[200] GET /api/v1/professionals/ID (API::V1::Public::ProfessionalsController#show)"}
@logStream  i-05d1d61ab853517a0
@message  I, [2018-12-19T23:42:52.629855 #23447] INFO -- : [2ce588f1-c27d-4a55-ac05-62a75b39e762] {"method":"GET","path":"/api/v1/professionals/ID","format":"json","controller":"API::V1::Public::ProfessionalsController","action":"show","status":200,"duration":285.27,"view":222.36,"time":"2018-12-19T23:42:52.344+00:00","params":{"include":"xxx","format":"json","compress":false,"id":"ID"},"@timestamp":"2018-12-19T23:42:52.629Z","@version":"1","message":"[200] GET /api/v1/professionals/ID (API::V1::Public::ProfessionalsController#show)"}
@timestamp  1545262972000
2
2018-12-19 23:42:16.000
I, [2018-12-19T23:42:16.723472 #851] INFO -- : [ea712503-eb86-4a6e-ab38-ddbcd6c2b4d0] {"method":"GET","path":"/api/v1/heartbeats/new","format":"json","controller":"API::V1::Public::HeartbeatsController","action":"new","status":201,"duration":9.97,"view":3.2,"time":"2018-12-19T23:42:16.712+00:00","params":{"format":"json","compress":false},"@timestamp":"2018-12-19T23:42:16.722Z","@version":"1","message":"[201] GET /api/v1/heartbeats/new (API::V1::Public::HeartbeatsController#new)"}

Answer 1

在@pyb insights的基础上，我能够使用parse @message '"path":"*"' as path从@message的任何地方提取路径。

您可以通过管道另一个parse @message '"method":"*"' as method继续获取您的方法而无需担心订购，因为它是@message上的第二个全球纯文本搜索

在您的@message是：

I, [2018-12-19T23:42:52.629855 #23447] INFO -- : [2ce588f1-c27d-4a55-ac05-62a75b39e762] {"method":"GET","path":"/api/v1/professionals/ID","format":"json","controller":"API::V1::Public::ProfessionalsController","action":"show","status":200,"duration":285.27,"view":222.36,"time":"2018-12-19T23:42:52.344+00:00","params":{"include":"xxx","format":"json","compress":false,"id":"ID"},"@timestamp":"2018-12-19T23:42:52.629Z","@version":"1","message":"[200] GET /api/v1/professionals/ID (API::V1::Public::ProfessionalsController#show)"}

使用：

parse @message '"path":"*"' as path | parse @message '"method":"*"' as method

将导致字段：path = '/api/v1/professionals/ID'和method = 'GET'

请注意，这仍然只是字符串解析，因此，它没有嵌套键的概念，如params.format找不到json，但只使用format，只要你的format中没有其他任何@message字符串。

另请注意，这适用于Insights未在消息中发现您的JSON的情况。我相信这是@pyb在this answer中提到的情况。使用以下格式未发现我的日志

info - Request: {"method":"POST","path":"/auth/login/","body":{"login":{"email":"[email protected]","password":"********"}},"uuid":"36d76df2-aec4-4549-8b73-f237e8f14e23","ip":"*.*.*.*"}

Answer 2

CloudWatch Insights Logs自动发现以下日志类型的字段：

Lambda logs

CloudWatch Logs Insights会自动发现Lambda日志中的日志字段，但仅针对每个日志事件中的第一个嵌入式JSON片段（注意：强调我的）。如果Lambda日志事件包含多个JSON片段，则可以使用parse命令解析和提取日志字段。有关更多信息，请参阅JSON日志中的字段。

CloudTrail logs

见fields in JSON logs。

资料来源：Supported Logs and Discovered Fields

如果@message是I, [2018-12-11T13:20:27] INFO -- : {"method":"GET"}

然后您可以选择并过滤字段，如下所示：

fields @timestamp, @message, method
| filter method = "GET"
| sort @timestamp desc

它也适用于嵌套字段，即params.format = "json"或results.0.firstName = "Paul"。

Answer 3

您可以使用parse命令提取字段。

如果@message是

I, [2018-12-11T13:20:27] INFO -- : {"method":"GET"}

然后你像这样提取字段：

fields @timestamp, @message
| parse "I, [*T*] INFO -- : {"method":"*"}" as @date, @time, @method
| filter method=GET
| sort @timestamp desc
| limit 20

文档目前很轻松。我可以通过用正则表达式替换通配符*来获得结果，但随后解析失败。

Answer 4

这个在parse命令中的正则表达式查询会帮助你吗？

filter @message like / \"path\":\"/
| parse @message /(?<@endpt>((\/[a-zA-Z0-9_{}()-?]+){1,}))/

祝好运！