Amazon Cloudwatch Logs Insights with JSON fields

Question

I am trying to use Logs Insights with data containing JSON in one of the fields, and to parse the JSON fields

My data looks like the following when I put it in insig

Answer 1

Building on @pyb insights, I was able to use parse @message '"path":"*"' as path to extract the path from any place in the @message.

You can go on to get your method by piping another parse @message '"method":"*"' as method without concern for ordering as it is a second global plain text search on @message

In the case where your @message is:

I, [2018-12-19T23:42:52.629855 #23447] INFO -- : [2ce588f1-c27d-4a55-ac05-62a75b39e762] {"method":"GET","path":"/api/v1/professionals/ID","format":"json","controller":"API::V1::Public::ProfessionalsController","action":"show","status":200,"duration":285.27,"view":222.36,"time":"2018-12-19T23:42:52.344+00:00","params":{"include":"xxx","format":"json","compress":false,"id":"ID"},"@timestamp":"2018-12-19T23:42:52.629Z","@version":"1","message":"[200] GET /api/v1/professionals/ID (API::V1::Public::ProfessionalsController#show)"}

Using:

parse @message '"path":"*"' as path | parse @message '"method":"*"' as method

will result in the fields: path = '/api/v1/professionals/ID' and method = 'GET'

Note that this is still simply string parsing and as such, it has no concept of nested keys like params.format would not find json, however using just format would, so long as there wasn't another format string anywhere in your @message.

Also note that this is for the case where Insights is not discovering your JSON in the message. I belive this is the case that @pyb was referring to in this answer. My logs aren't being discovered either using the following format

info - Request: {"method":"POST","path":"/auth/login/","body":{"login":{"email":"email@example.com","password":"********"}},"uuid":"36d76df2-aec4-4549-8b73-f237e8f14e23","ip":"*.*.*.*"}