FluentBit - Parsing from Path_Key

问题

Currently im adding the filepath using Path_Key. I am trying to grab multiple variables that exist in the Path_key.

/var/log/containers/**Application_Name**-**Application_Version**.log

Is it possible to extract these values from an existing field mapping?

回答1:

For extracting values for use in the Tag it is pretty straight forward, you would have an input like:

[INPUT]
    Name  tail
    Path  /var/log/containers/*-*.log
    Path_Key  filename
    Tag <appname>.<appversion>
    Tag_Regex  /(?<appname>[^-]+)-(?<appversion>[^.]+).log$

The Tag_Regex is used to set the <appname> and <appversion> variables that can be used to set the Tag.

As for setting these sort of values in a field in the log entry I couldn't find any "native" way to do it. However I was able to achieve something similar by using a Lua filter:

[INPUT]
    Name  tail
    Path  /var/log/containers/*-*.log
    Path_Key  filename

[FILTER]
    Name  lua
    Match  *
    script  helper.lua
    call  extract_app_fields

which invokes the the extract_app_fields function in the helper.lua file:

function extract_app_fields(tag, timestamp, record)
    retcode = 0
    filename = record['filename']
    
    if filename ~= nil then
        appname = filename('/([^-]+)-[^.]+\.log')
        appversion = filename('/[^-]+-([^.]+)\.log')
        
        if appname ~= nil then
            record['appname'] = appname
            retcode = 2
        end
        
        if appversion ~= nil then
            record['appversion'] = appversion
            retcode = 2
        end
    end
    
    return retcode, timestamp, record
end

The extract_app_fields function extracts the appname and appversion from the filename and updates the fields in the record if they can be determined.

Note: I'm very much a Lua novice, so there may be a better way to do this using Lua.

来源：https://stackoverflow.com/questions/62959018/fluentbit-parsing-from-path-key

标签

parsing

syslog

fluentd

fluent-bit