logstash-grok

问题 Hi I am trying to send a json file with multiple objects to elasticsearch with the logstash so I can display the data using kibana. I have researched this extensively and simply cannot understand how to make the data formatted correctly to be used in kibana. I have tried to use different filters such as: json, date, and grok The issue is probably how I'm going about using these filters as I can't understand it's setup all to well. Here is a sample line of the input json file: {"time":"2015-09

问题 I am trying to configure logstash to manage my various log sources, one of which is Mongrel2. The format used by Mongrel2 is tnetstring , where a log message will take the form 86:9:localhost,12:192.168.33.1,5:57089#10:1411396297#3:GET,1:/,8:HTTP/1.1,3:200#6:145978#] I want to write my own grok patterns to extract certain fields from the above format. I started by testing my regex on the above message here, the regex is ^(?:[^:]*\:){2}([^,]*) this matches localhost . When I use the same regex

问题 I'm new in the elasticsearch community and I would like your help on something I'm struggeling with. My goal is to send huge quantity of log files to Elasticsearch using Filebeat. In order to do that I need to parse data using ingest nodes with Grok pattern processor. Without doing that, all my logs are not exploitable as each like fall in the same "message" field. Unfortunately I have some issues with the grok regex and I can't find the problem as It's the first time I work with that. My

问题 In the cluster that I am working on there are two main indexes, let's say indexA and indexB but these two indexes are indexed each day so normaly I have indexA-{+YYYY.MM.dd} and indexB-{+YYYY.MM.dd} . What I want is to have one alias that gathers indexA-{+YYYY.MM.dd} and indexB-{+YYYY.MM.dd} together and named alias-{+YYYY.MM.dd} . Does anyone know how to gather two indexes in one alias with logstash ? Thank you in advance 回答1: As far as I know, there's no way to do it with logstash directly.

问题 I'm creating a test suite for grok filter. Some logs are correctly enriched by logstash but not the rspec test. To test this I launched an instance of logstash with stdin/stdout and json for input and output. Here is the sample log (nginx access): 10.7.0.78 - - [14/Jan/2016:16:39:36 +0000] "GET /v1/swagger.json HTTP/1.1" 200 3720 "-" "python-requests/2.8.1" Logstash config: input { stdin { codec => "json" } } output { stdout { codec => "json" } } filter { if [file] =~ "nginx" { grok { match =

问题 I have log lines loaded in ElasticSearch which has the data scattered in multiple events, say event_id is in event (line) number 5 and event_action is available in event number 88, further event_port information is available in event number 455. How can i extract this data so that my output looks like following. For this case multiline codec will not work. { event_id: 1223 event_action: "socket_open" event_port: 76654 } Currently I have the log files persisted so i can get the file path from

问题 I am current trying to do a JavaScript post to Logstash by using a tcp input. JavaScript Post xhr = new XMLHttpRequest(); var url = "http://localhost:5043"; xhr.open("POST", url, true); xhr.setRequestHeader("Content-type", "application/json"); var data = JSON.stringify({"test" : hello}); xhr.send(data); Logstash config file input { tcp { port => 5043 } } filter{ } output { stdout { codec => rubydebug } } Output in console { "message" => "OPTIONS / HTTP/1.1\r", "@version" => "1", "@timestamp"

问题 I am using LogStash which accepts data from a log file, which has different types of logs. I tried this: filter { grok { match => { "message" => "%{WORD:tag} %{WORD:message} %{WORD:value} } } But it doesn't work. 回答1: I am using the grok filter to check if the log line is of one format. If the grok filter cannot parse the log line (such as with the json lines), _grokparsefailure will be added to the tags. You can then use this tag to differentiate between the two log type. filter { grok {

问题 I have below example in single line of text: valuationType=RiskAnalysis, commandType=SpreadRA, pricing_date=20161230 01:00:00.000, priority=51, CamelFileLastModified=1483346829000, CamelFileParent=/home/tisuat52/mount/tis/shared, message_size=239450, solstis_set_name=OFFICIAL, CamelFileRelativePath=TIS_RISKONE_SpreadRA_CREDITASIACNH_OFF_CreditGamma_Ido_RA_2016-12-30_1483138799000_Input.bin, command_status=OK, commandName=CREDITASIACNH_OFF_CreditGamma_Ido_RA, calculator_timestamp=20170102 04

问题 Can anybody help me writing the grok pattern for the following log line 07-Aug-2017|00:35:08,748 DEBUG [hostname] [Some WebApp Name] [6.9] [127.0.0.1] [1277] I am not able to find a way to accomodate '[' & ']' in the grok patterns. Any help will be appreciated. 回答1: This should match your pattern: %{MONTHDAY}-%{MONTH}-%{YEAR}\|%{TIME} %{LOGLEVEL} \[%{WORD} ] \[%{DATA}] \[%{NUMBER}] \[%{IP}] \[%{NUMBER}] As you can see squared bracket are escaped with backslashes like this: \[ and \] You might