如何制作 filebeat 配置以从文件中收集特定行

Question

有一个oracle的文件日志aud，我需要收集整个块而不是整个文件。该块是从 ACTION 字段到 ACTION NUMBER

日志文件如下

Thu Nov  9 10:20:24 2023 +01:00
LENGTH : '373'
ACTION :[122] 'select 'export nls_nchar_characterset="'||value||'"' from nls_database_parameters where parameter='NLS_NCHAR_CHARACTERSET''
DATABASE USER:[1] '/'
PRIVILEGE :[6] 'SYSDBA'
CLIENT USER:[6] 'oracle'
CLIENT TERMINAL:[0] ''
STATUS:[1] '0'
DBID:[10] '3256053857'
SESSIONID:[10] '4294967295'
USERHOST:[12] 's01vl9926909'
CLIENT ADDRESS:[0] ''
ACTION NUMBER:[1] '3'

Thu Nov  9 10:20:24 2023 +01:00
LENGTH : '1575'
ACTION :[1323] 'select 'export db_patchset_new="'||max_ver||'"' from (
select length(regexp_replace(substr(description,0,decode(instr(description,'('),0,length(description),instr(description,'('))),'[^0.0-9]', '')) max_length,
       max(regexp_replace(substr(description,0,decode(instr(description,'('),0,length(description),instr(description,'('))),'[^0.0-9]', '')) max_ver
from   (select description from ( select da.description, da.action_time apply, nvl(dr.action_time, da.action_time-1) rollback
                                  from (select description, max(action_time) action_time from dba_registry_sqlpatch where action='APPLY'    group by description) da,
                                       (select description, max(action_time) action_time from dba_registry_sqlpatch where action='ROLLBACK' group by description) dr
                                  where da.description=dr.description(+)
                                )
        where apply > rollback
       )
where  (upper(description) like '%DATABASE%PATCH%' or upper(description) like '%DATABASE%RELEASE%') and upper(description) not like '%JAVA%' and upper(description) not like '%JVM%'
group by length(regexp_replace(substr(description,0,decode(instr(description,'('),0,length(description),instr(description,'('))),'[^0.0-9]', ''))
order by 1 desc
)
where rownum=1'
DATABASE USER:[1] '/'
PRIVILEGE :[6] 'SYSDBA'
CLIENT USER:[6] 'oracle'
CLIENT TERMINAL:[0] ''
STATUS:[1] '0'
DBID:[10] '3256053857'
SESSIONID:[10] '4294967295'
USERHOST:[12] 's01vl9926909'
CLIENT ADDRESS:[0] ''
ACTION NUMBER:[1] '3'

我使用 include_lines 模式来收集这些特定的行，但我的问题是，在“ACTION”字段中，有时存在包含长查询的操作，而我的conf仅收集第一行而不是整个查询。

这里的例子，我的事件没有收集整个查询，它停在单词 FROM

{"@timestamp":"2023-11-09T13:34:21.491Z","@metadata":{"beat":"filebeat","type":"_doc","version":"7.17.0"},"message":"ACTION:'select 'export db_patchset_new="'||max_ver||'"' from ('","fields":{"env":"staging"},"host":{"name":"s01vl9926909"},"event":{"timezone":"+01:00"},"log":{"offset":811,"file":{"path":"/apps/oracle/diag/rdbms/q08775kp1/Q08775KP10/audit/Q08775KP10_ora_29807_20231109143420100545459814.aud"}},"input":{"type":"log"},"ecs":{"version":"1.12.0"}

第一次我的文件 input_Oracle.yml 如下所示；它工作正常，但问题是 ACTION 字段

我的输入文件配置第一次工作正常

  enabled: true
  tags: ["linux-Oracle"]
  #ignore_older: 4h
  close_inactive: 30s

  paths:
   /apps/oracle/diag/rdbms/*/*/audit/*.aud

  include_lines: ['^LENGTH :', '^ACTION :', '^DATABASE USER:', '^PRIVILEGE :', '^CLIENT USER:', '^CLIENT TERMINAL:', '^STATUS:', '^DBID:', '^SESSIONID:', '^USERHOST:', '^CLIENT ADDRESS:', '^ACTION NUMBER:']

然后我尝试“多行消息”，但它对我不起作用

type: log
  enabled: true
  tags: ["linux-Oracle"]
  #ignore_older: 4h
  close_inactive: 30s

  paths:
    - /apps/oracle/diag/rdbms/*/*/audit/*.aud

  multiline:
    pattern: '^ACTION :'
    negate: true
    match: after

  include_lines: ['^LENGTH :', '^DATABASE USER:', '^PRIVILEGE :', '^CLIENT USER:', '^CLIENT TERMINAL:', '^STATUS:', '^DBID:', '^SESSIONID:', '^USERHOST:', '^CLIENT ADDRESS:', '^ACTION NUMBER:']`

有没有办法通过联合线做到这一点！！

Answer 1

Tldr；

您使用多行解析器确实是正确的。 但我认为你配置有点错误。

解决：

我相信这可以帮助你。

我正在匹配以大写单词开头的行。

filebeat.inputs:
- type: filestream
  id: srt
  paths:
    - /usr/share/filebeat/data.log
  include_lines: ['^LENGTH :', '^ACTION :', '^DATABASE USER:', '^PRIVILEGE :', '^CLIENT USER:', '^CLIENT TERMINAL:', '^STATUS:', '^DBID:', '^SESSIONID:', '^USERHOST:', '^CLIENT ADDRESS:', '^ACTION NUMBER:']
  parsers:
  - multiline:
      type: pattern
      pattern: '^[A-Z]+'
      negate: true
      match: after

output.console:
  pretty: true