logstash-grok

问题 I'm having issues with grok parsing. In ElasticSearch/Kibana the lines I match come up with the tag _grokparsefailure. Here is my logstash config : input { file { type => logfile path => ["/var/log/mylog.log"] } } filter { if [type] == "logfile" { mutate { gsub => ["message","\"","'"] } grok { match => { "message" => "L %{DATE} - %{TIME}: " } } } } output { elasticsearch { host => localhost port => 9300 } } lines/patterns I'm trying to match : L 08/02/2014 - 22:55:49: Log file closed : "

问题 Im trying to parse my apache2 error log and im having a bit of trouble.. It doesnt seem to be matching the filter. Im pretty sure the timestamp piece is wrong, but im not sure, and i cant really find any documentation to figure it out. Also, is there a way to get what is in fields.errmsg to me @message ? Log [Wed Jun 26 22:13:22 2013] [error] [client 10.10.10.100] PHP Fatal error: Uncaught exception '\Foo\Bar' Shipper Config input { file { 'path' => '/var/log/apache2/*-error.log' 'type' =>

问题 I've got a JSON of the format: { "SOURCE":"Source A", "Model":"ModelABC", "Qty":"3" } I'm trying to parse this JSON using logstash. Basically I want the logstash output to be a list of key:value pairs that I can analyze using kibana. I thought this could be done out of the box. From a lot of reading, I understand I must use the grok plugin (I am still not sure what the json plugin is for). But I am unable to get an event with all the fields. I get multiple events (one even for each attribute

问题 I am trying to grep only few fields from this output from logstash 1.repositories#create 2.\"repo\":\"username/reponame\" . please share your ideas to grep particular info from this outpput and assign this to another variable "message" => "<190>Nov 01 20:35:15 10-254-128-66 github_audit: {\"actor_ip\":\"192.168.1.1\",\"from\":\"repositories#create\",\"actor\":\"myuserid\",\"repo\":\"username/reponame\",\"action\":\"staff.repo_route\",\"created_at\":1516286634991,\"repo_id\":44743,\"actor_id\"

问题 I have Nginx as loadbalancer which is generating logs without year and second information in timestamp. One of those logs are 08-10 09:28 root ERROR Error connecting to CRS REST API : [Errno 111] Connection refused Error connecting to CRS REST API : [Errno 111] Connection refused The pattern for this is : (?m)%{MONTHNUM:monthNum}\-%{MONTHDAY:monthDay}\s*%{HOUR:hour}:%{MINUTE:minute}\s*%{WORD}\s*%{LOGLEVEL_CUSTOM:severity}\s*%{GREEDYDATA:messagePayload} While I understand that year information

问题 I'm trying to test the some custom log filter for logstash but somehow i'm not able to get it, I googled and looked over many examples but I am not able to create a one I want. Below is my log patterns: testhost-in2,19/01/11,06:34,04-mins,arnav,arnav 2427 0.1 0.0 58980 580 ? S 06:30 0:00 rm -rf /test/ehf/users/arnav-090119-184844,/dv/ehf/users/arnav-090119- testhost-in2,19/01/11,06:40,09-mins,arnav,arnav 2427 0.1 0.0 58980 580 ? S 06:30 0:00 rm -rf /dv/ehf/users/arnav-090119-184844,/dv/ehf

问题 I want to make a filter in Logstash(version 2.4) with different matches in the same grok. I would like to add different tags depending on the match. Basically, I receive three different message pattern: "##MAGIC##%message" "##REAL##%message" "%message" I am trying to do is: grok { match => {"message" => "##MAGIC##%{GREEDYDATA:magic_message}"} match => {"message" => "##REAL##%{GREEDYDATA:real_message}"} match => {"message" => "%{GREEDYDATA:basic_message}"} if [magic_message]{ overwrite => [

问题 I am using ELK(filebeat,logstash,elasticsearch,kibana) to make log management. In one log file, I have three kinds of format. I am using In one format, I have date+parameters+json+stacktrace. This kind of format text has multiple lines. In second format, it is just date+requestMethod(Get or post)+some text. It is in one line. In third format, it has date+ modular Name:(In this case, it is paymentAdmin)+json I suppose I could use logStash to handle these three kinds of format by if, else if

问题 Let me first say that I have gone through as many examples on here as I could that still do not work. I am not sure if it's because of the complicated nature of the JSON in the log file or not. I am looking to take the example log entry, have Logstash read it in, and send the JSON as JSON to ElasticSearch. Here is what the (shortened) example looks: [0m[0m16:02:08,685 INFO [org.jboss.as.server] (ServerService Thread Pool -- 28) JBAS018559: { "appName": "SomeApp", "freeMemReqStartBytes":

问题 I'm using Grok & Logstash to send access logs from Nginx to Elastic search. I'm giving Logstash all my access logs (with a wildcard, works well) and I would like to get the filename ( some part of it, to be exact ) and use it as a field. My config is as follows : input { file { path => "/var/log/nginx/*.access.log" type => "nginx_access" } } filter { if [type] == "nginx_access" { grok { match => { "message" => "%{COMBINEDAPACHELOG}" } match => { "path" => "%{GREEDYDATA}/%{GREEDYDATA:app}