Kibana linking two independent events

Question

I have ELK configured for collecting data offline, the log files look something like this :

Info 2015-08-15 09:33:37,522 User 3 connected
Info 2015-08-15 10:         


        

Answer 1

One of the downsides of Elasticsearch is that it isn't a relational database - so cross referencing is much more limited. There's a nice blog post about it: Managing Relations inside Elasticsearch

But the long and short of it is - there isn't a way to query this sort of thing directly. Each event is a discrete document within your index, and there isn't any sort of cross reference.

So you have to do it the hard way. At the simplest level - that's query all the connect events, query all the disconnect events, and correlate them yourself with a scripting language.

You can make this slightly easier by pre-filtering your logs with a grok filter to add fields to your database.

if [type] == "syslog" and [message] =~ /connected/ {
    grok { 
        match => [ "message", "User %{POSINT:userid} %{WORD:conn}" ]
    }
}

Which will add a userid and conn field (containing "connected" or "disconnected").

But you will still have to manually correlate the queries with a database fetch using your favourite scripting language (so could just do the 'search and filter' in the script).