将日志文件数据提取到csv文件中

Question

我有一个application.log文件，它同时包含以下数据和其他数据。此日志文件总共包含7000个条目。从此，我试图仅提取Params : {"file_list":和request : {"method":"registerSender"数据。

2020-04-14 13:34:08 INFO  app-proxy:53 - Params : {"file_list":["index.html"],"conn_id":"35d20d60","Token":"m.usha","proxy_name":"India","hash":"03870bc9df9"}

2020-04-14 06:09:57 INFO  app-proxy:53 - Params : {"file_list":["DU_SAMPLE_TERT_BD1234.7z"],"conn_id":"5e8e2cb9","Token":"m.usha","proxy_name":"India","hash":"2a357570"}

2020-04-14 10:32:04 INFO  app-proxy:53 - Params : {"file_list":["DU_SAMPLE_188472.7z"],"conn_id":"42474096","Token":"geetha.priya","proxy_name":"India","hash":"2a357570b"}

2020-04-14 07:34:46 INFO  app-proxy:151 - request : {"method":"registerSender","id":"reqFromProxy","jsonrpc":"2.0","params":{"origin":"India","hashName":"251e9a9742","Token":"nithya.d","fileList":["ATT_A515FXXU2ATD1.tar.md5"]}}

2020-04-14 08:48:02 INFO  app-proxy:53 - Params : {"file_list":["ATT_A515FXXU2ATD1.tar.md5"],"conn_id":"f10eec35","Token":"kumar.mari","proxy_name":"India","hash":"251e9a974222e8ba369d6b6c45628017"}

2020-04-14 01:24:53 INFO  app-proxy:151 - request : {"method":"registerSender","id":"reqFromProxy","jsonrpc":"2.0","params":{"origin":"India","hashName":"e061202","Token":"rajesh.raj","fileList":["OLLAY_HUM.tar.md5"]}}

我尝试使用以下脚本，

#!/bin/bash

file="application.log"
outputfile="output.log"
if [ -f "$outputfile" ]; then
echo "File exists & deleteing"
rm $outputfile
else
echo "File does not exist"
fi

while read line
do

if echo "$line" | grep 'file_list'; then
   echo $line >> "$outputfile"
elif echo "$line" | grep '"method":"registerSender"'; then
    echo $line >> "$outputfile"
else
    echo "no match";
fi
done < $file

但是我能够从file_list文件中过滤"method":"registerSender"和application.log条目，此外，我试图以以下图表格式提取上述日志数据。任何帮助表示赞赏。

Answer 1

您能不能尝试以下操作。

awk '
BEGIN{
  print "S.No Date Time file_list Token proxy_name"
}
match($0,/\{"file_list":\["[^"]*/){
  fileVal=substr($0,RSTART+15,RLENGTH-15)
}
match($0,/"Token":"[^"]*/){
  token_Val=substr($0,RSTART+9,RLENGTH-9)
}
match($0,/"proxy_name":"[^"]*/){
  proxyName=substr($0,RSTART+14,RLENGTH-14)
}
{
  split($1,arrayDate,"-")
  print ++count,arrayDate[1]"-"arrayDate[2]"-"arrayDate[3],$2,fileVal,token_Val,proxyName
}
'  Input_file | column -t

说明：添加以上详细说明。

awk '                                                     ##Starting awk program from here.
BEGIN{                                                    ##Starting BEGIN section from here.
  print "S.No Date Time file_list Token proxy_name"       ##Printing headers here.
}
match($0,/\{"file_list":\["[^"]*/){                       ##Using match function to match file_list":\[" till next " to get its value.
  fileVal=substr($0,RSTART+15,RLENGTH-15)                 ##saving matched value into variable fileVal here.
}
match($0,/"Token":"[^"]*/){                               ##Using match function to match "Token":"[ till next ".
  token_Val=substr($0,RSTART+9,RLENGTH-9)                 ##saving matched value into variable tolen_Val here.
}
match($0,/"proxy_name":"[^"]*/){                          ##using proxy_name":" till ".
  proxyName=substr($0,RSTART+14,RLENGTH-14)               ##saving matched value into variable proxyName here.
}
{
  split($1,arrayDate,"-")                                 ##Splitting 1st column into array with delimiter space here.
  print ++count,arrayDate[1]"-"arrayDate[2]"-"arrayDate[3],$2,fileVal,token_Val,proxyName   ##Printing all the values to get OP result here.
}
'  Input_file | column -t                                      ##Mentioning Input_file name and passing its value to column command to make output look better.