如何忽略 jq 中损坏的 JSON 行？

Question

当使用

jq

处理日志文件时，一些行可能被破坏，因此

jq

抛出错误并停止处理。

例如完整日志：

{"level":"debug","time":"2021-09-24T19:42:47.140+0800","message":"sent send binary to ws server1","pid":41491,"cid":"32likw","num":1,"count":5120}
{"level":"debug","time":"2021-09-24T19:42:47.305+0800","message":"sent send binary to ws server2","pid":41491,"cid":"32likw","num":1,"count":5120}
{"level":"debug","time":"2021-09-24T19:42:47.469+0800","message":"sent send binary to ws server3","pid":41491,"cid":"32likw","num":1,"count":5120}
{"level":"debug","time":"2021-09-24T19:42:47.499+0800","message":"sent send binary to ws server4","pid":41491,"cid":"32likw","num":1,"count":5120}
{"level":"debug","time":"2021-09-24T19:42:47.581+0800","message":"sent send binary to ws server5","pid":41491,"cid":"32likw","num":1,"count":5120}

jq 处理得很好：

< snippet1.json jq -C -r '.message'
sent send binary to ws server1
sent send binary to ws server2
sent send binary to ws server3
sent send binary to ws server4
sent send binary to ws server5

破损的（第 3 行的最后一部分缺失）：

{"level":"debug","time":"2021-09-24T19:42:47.140+0800","message":"sent send binary to ws server1","pid":41491,"cid":"32likw","num":1,"count":5120}
{"level":"debug","time":"2021-09-24T19:42:47.305+0800","message":"sent send binary to ws server2","pid":41491,"cid":"32likw","num":1,"count":5120}
{"level":"debug","time":"2021-09-24T19:42:47.469+0800","message":"sent send binary to ws server3","pi
{"level":"debug","time":"2021-09-24T19:42:47.499+0800","message":"sent send binary to ws server4","pid":41491,"cid":"32likw","num":1,"count":5120}
{"level":"debug","time":"2021-09-24T19:42:47.581+0800","message":"sent send binary to ws server5","pid":41491,"cid":"32likw","num":1,"count":5120}

jq

停在虚线处：

< snippet2.json jq -C -r '.message'
sent send binary to ws server1
sent send binary to ws server2
parse error: Invalid string: control characters from U+0000 through U+001F must be escaped at line 4, column 2

我希望

jq

可以忽略第三行并继续，就像这样：

< snippet2.json jq -C -r '.message'
sent send binary to ws server1
sent send binary to ws server2
sent send binary to ws server4
sent send binary to ws server5

我尝试使用

另一篇文章

中提到的-R，它对这种情况没有帮助。

< snippet2.json jq -C -R -r '.message'
jq: error (at <stdin>:1): Cannot index string with string "message"
jq: error (at <stdin>:2): Cannot index string with string "message"
jq: error (at <stdin>:3): Cannot index string with string "message"
jq: error (at <stdin>:4): Cannot index string with string "message"
jq: error (at <stdin>:5): Cannot index string with string "message"

你能告诉我是否有任何解决方案/技巧可以忽略/跳过/抑制这样的错误并获得其余的结果吗？

Answer 1

要跳过虚线，您可以使用：

jq -Rr 'fromjson? | .message'

如果你想用它们做些别的事情，你可以从这样的事情开始：

jq -R '. as $line | try fromjson catch $line'

有关其他选项，请参阅：

𝑸：有没有办法让 jq 在遇到输入文件错误后继续运行？ jq 可以处理损坏的 JSON 吗？

在 jq FAQ.

Answer 2

对peak的回答再做一些解释。 （学分达到顶峰）

解决方案#1：

❯ cat bad.json | jq -r -R 'fromjson? | .message'
sent send binary to ws server1
sent send binary to ws server2
sent send binary to ws server4
sent send binary to ws server5

解决方案#2：

❯ cat bad.json | jq -r -R '. as $line | try fromjson catch $line | .message'
sent send binary to ws server1
sent send binary to ws server2
jq: error (at <stdin>:3): Cannot index string with string "message"
sent send binary to ws server4
sent send binary to ws server5

jq 仍然输出错误，但是它在 stderr 上，你可以重定向它：

❯ cat bad.json | jq -r -R '. as $line | try fromjson catch $line | .message' 2>/dev/null
sent send binary to ws server1
sent send binary to ws server2
sent send binary to ws server4
sent send binary to ws server5

值得注意的是

-R

和

-r

可以一起使用。 （感谢@peak！）

Answer 3

我也遇到了这个问题。简而言之，这是我们的设置：

• Kubernets/AKS
• 日志收集的Datadog
• Maven for Java Spring Boot 具有

  logstash-logback-encoder

  依赖性

现在

kubectl

日志在命令行上变得非常不可读。不过，我觉得在终端调试还是有它的优势的。因此，这是我最喜欢的单行解决方案，用于从 kubernetes pod 或多或少地进行弹性实时跟踪和快速调试：

kubectl logs -f --tail 200 <podname> | jq -R 'fromjson? | . | "\(.["@timestamp"]) -- \(.level) -- \(.logger_name) -- \(.message) \(.stack_trace)"' -r

这将忽略/隐藏带有解析错误的行，并使用上面的格式字符串显示每个日志行的一些 JSON 字段。

它并不完美，但我想可以作为一个起点。适应您的需求:-)