logstash-grok

问题 I need to capture two variables from this path using grok: /opt/data/app_log/server101.log server=needs to be anything after the last forward slash before the dot (in this case server101) index=needs to be the text between the last two forward slashes (in this case app_log) Any ideas how could do this in grok? grok { patterns_dir => ["/pattern"] match =>{path =>"%{WORD:dir1}\/%{WORD:dir2}\/%{WORD:index_name}\/%{WORD:server}\.%{WORD:file_type}"} match => {"message" => "%{TIMESTAMP_ISO8601

问题 I am trying to convert a timestamp field that is in the form $epoch.$microsec to $epoch_millis . Example: 1415311569.541062 --> 1415311569541 Logstash doesn't appear to have any means of multiplying numbers so ts * 1000 and casting to a long is out. Any ideas? 回答1: In your particular case you can indeed get away with turning the problem into a string manipulation problem, but you can also use the ruby filter: filter { ruby { # do some calculation code => "event['ts'] = (1000 * event['ts'].to

问题 I am new to regular expressions but I think people here may give me valuable inputs. I am using the logstash grok filter in which I can supply only regular expressions. I have a string like this /app/webpf04/sns882A/snsdomain/logs/access.log I want to use a regular expression to get the sns882A part from the string, which is the substring after the third "/", how can I do that? I am restricted to regex as grok only accepts regex. Is it possible to use regex for this? 回答1: Yes you can use

问题 I am trying to start with logstash and my application has following type of logs. Here 5 indicate 5 more lines will follow which are stats collected for different related things. These are basically application stats with each line indicating about one of the resource. Is there a way to properly parse it using logstash so that it can be use for Elastic search? [20170502 01:57:26.209 EDT (thread-name) package-name.classname#MethodName INFO] Some info line (5 stats): [fieldA: strvalue1| field2:

问题 In my Logstash shipper I want to filter out lines commented with the hash character: #This log row should be dropped. But one this should not. I was able to use grep filter, but as it is discouraged (going to be decommissioned), I'm trying to get a grok filter to do it instead. This filter is not working: grok { match => ["message", "^#.*"] drop_if_match => true } I also tried placing the regex in a custom pattern file, but didn't help. Any ideas? 回答1: Even simpler, if you're interested:

问题 I am using Logstash to parse postfix logs. I am mainly focused to get bounced email logs from postfix logs, and store it in database. In order to get logs, first I need to find ID generated by postfix corresponding to my message-id, and using that Id, I need to find status of an email. For following configuation, I am able to get the logs. grok { patterns_dir => "patterns" match => [ "message", "%{SYSLOGBASE} %{POSTFIXCLEANUP}", "message", "%{SYSLOGBASE} %{POSTFIXBOUNCE}" ] named_captures

问题 The requirement is to show the start 6 digits and last 4 digits and mask the remaining numbers of credit card in logstash. I applied gsub/mutate filter but the replacement string doesn't allow regex. Any other way this can be done in logstash? if [message] =~ '\d{16}' { mutate { gsub => ["message", "\d{6}\d{4}\d{4}", "\d{6}######\d{4}"] add_tag => "Masked CardNo" } } This code masks the credit card 3456902345871092 to \d{6}######\d{4} but it should be masked as 345690######1092. As an

问题 I have a bunch of log files that are named as 'XXXXXX_XX_yymmdd_hh:mm:ss.txt' - I need to include the date and time (separate fields) from the filename in fields that are added to Logstash. Can anyone help? Thanks 回答1: Use a grok filter to extract the date and time: filter { grok { match => [ "path", "^%{GREEDYDATA}/[^/]+_%{INT:date}_%{TIME:time}\.txt$" ] } } Depending on what goes instead of XXXXXX_XX you might prefer a stricter expression. Also, GREEDYDATA isn't very efficient. This might

问题 I am trying to adjust path name so that it no longer has the time stamp attached to the end. I am input many different logs so it would be impractical to write a conditional filter for every possible log. If possible I would just like to trim the last nine characters of the value. For example "random.log-20140827" would become "random.log" . 回答1: So if you know it's always going to be random.log-something -- if [path] =~ /random.log/ { mutate { replace => ["path", "random.log"] } } If you

问题 I am new to logstash and desparate to setup ELK for one of the usecase. I have found this question relevent to mine Why won't Logstash multiline merge lines based on grok'd field? If multiline filter do not merge lines on grok fields then how do I merge line 2 and 10 from the below log sample? Please help. Using grok patterns I have created a field 'id' which holds the value 715. Line1 - 5/08/06 00:10:35.348 [BaseAsyncApi] [qtp19303632-51]: INFO: [714] CMDC flowcxt=[55c2a5fbe4b0201c2be31e35]