elastic-stack

I have a sort of ELK stack, with fluentd instead of logstash, running as a DaemonSet on a Kubernetes cluster and sending all logs from all containers, in logstash format, to an Elasticsearch server. Out of the many containers running on the Kubernetes cluster some are nginx containers which output logs of the following format: 121.29.251.188 - [16/Feb/2017:09:31:35 +0000] host="subdomain.site.com" req="GET /data/schedule/update?date=2017-03-01&type=monthly&blocked=0 HTTP/1.1" status=200 body_bytes=4433 referer="https://subdomain.site.com/schedule/2589959/edit?location=23092&return=monthly"

we are building an opensource application which needs elasticsearch security feature. i am trying to find if the security feature is free for elastic search. elastic search website says Xpack is open now. Not sure if it is really opensource. Could someone please share your experience? This blog post explained some of the reasons why Elastic "opened" their XPack code. "Open" here simply means that they merged their private XPack repositories into the open ones. One of the reasons that the blog post above doesn't mention is that this move was mostly motivated to facilitate tedious engineering

Currently I am working on an application where I need to create documents from particular data from a file at specific location. I have set up logstash pipeline configuration. Here is what it looks like currently: input{ file{ path => "D:\ELK_Info\logstashInput.log" start_position => "beginning" } } #Possible IF condition here in the filter output { #Possible IF condition here http { url => "http://localhost:9200/<index_name>/<type_name>" http_method => "post" format => "json" } } I want to provide IF condition in output before calling API. The condition should be like, "If data from input

I currently have architecture with filebeat as the log shipper, which sends logs to log stash indexer instance and then to managed elastic search in AWS. Due to persistent TCP connections, I cannot load balance using AWS ELB multiple log stash indexer instances since filebeats always picks on of the instances and sends it there. So I decided to use redis. Now seeing how difficult it is to scale redis and make it highly available compontent in ELK stack I want to ask what is even the point of redis. I read a million times it acts as a buffer, but if filebeats stops sending logs to logstash if

I'm using only kibana to search ElasticSearch and i have several fields that can only take a few values (worst case, servername, 30 different values). I do understand what analyze do to bigger, more complex fields like this , but the small and simple ones i fail to understand the advance/disadvantage of anaylyzed/not_analyzed fields. So what are the benefits of using analyzed and not_analyzed for a "limited set of values" field (example. servername: server[0-9]* , no special characters to break)? What kind of search types will i lose in kibana? Will i gain any search speed or disk space?

I am having trouble using sprintf to reference the event fields in the elasticsearch output plugin and I'm not sure why. Below is the event received from Filebeat and sent to Elasticsearch after filtering is complete: { "beat" => { "hostname" => "ca86fed16953", "name" => "ca86fed16953", "version" => "6.5.1" }, "@timestamp" => 2018-12-02T05:13:21.879Z, "host" => { "name" => "ca86fed16953" }, "tags" => [ [0] "beats_input_codec_plain_applied", [1] "_grokparsefailure" ], "fields" => { "env" => "DEV" }, "source" => "/usr/share/filebeat/dockerlogs/logstash_DEV.log", "@version" => "1", "prospector" =