I'm trying to get a tile map visualization working in Kibana 4.0.1.
I can see the geoip.location data in the Discover section but when I go to Visualize it as a Tile Map, I select Geo Coordinates bucket type, then aggregation type GeoHash, then when I go to Field it's blank.
- Kibana 4.0.1
- Logstash 1.4.2-1-2c0f5a1
- Elasticsearch 1.4.4
- Running on Debian 7 64bit
Here's my logstash config:
input { udp { port => 5514 type => cisco } } filter { grok { match => { "message" => "\%{NUMBER:seq}: \*%{SYSLOGTIMESTAMP:date}: \%SEC-6-IPACCESSLOGP: list Internet\-In\-%{WORD:acl_ver} denied %{WORD:protocol} %{IP:src}\(%{NUMBER:sport}\) \-\> %{IP:dest}\(%{NUMBER:dport}\), %{INT:hitcnt}" } add_tag => ["grok_match", "cisco_acl_message"] remove_field => [ "message" ] } geoip { source => "src" target => "geoip" add_tag => ["geoip"] database => "/etc/logstash/GeoLiteCity.dat" } } output { elasticsearch { host => localhost index => [ "firewall-%{+YYYY.MM.DD}" ] } }And this is an example of a log record:
{ "_index": "firewall-2015.04.105", "_type": "cisco", "_id": "dJhGF6RtQuGXtlBTRCu2mQ", "_score": null, "_source": { "@version": "1", "@timestamp": "2015-04-15T21:06:08.357Z", "type": "cisco", "host": "172.17.10.1", "num": "190", "seq": "1872", "date": "Apr 15 21:08:05.878", "acl_ver": "20150223", "protocol": "tcp", "src": "94.102.51.96", "sport": "26820", "dest": "12.34.56.78", "dport": "5900", "hitcnt": "1", "tags": [ "grok_match", "cisco_acl_message", "geoip", "_grokparsefailure", "geoip" ], "geoip": { "ip": "94.102.51.96", "country_code2": "NL", "country_code3": "NLD", "country_name": "Netherlands", "continent_code": "EU", "region_name": "07", "city_name": "Amsterdam", "postal_code": "1000", "latitude": 52.349999999999994, "longitude": 4.916699999999992, "timezone": "Europe/Amsterdam", "real_region_name": "Noord-Holland", "location": [ 4.916699999999992, 52.349999999999994 ], "coordinates": [ 4.916699999999992, 52.349999999999994 ] } }, "fields": { "@timestamp": [ 1429131968357 ] }, "sort": [ 1429131968357 ] }Any ideas what I'm missing?
ANSWER
After Alain pointed me in the right direction I started looking at field mappings. Here's how I went about it:
First I checked the geoip field type (my index is called firewall*)
curl http://localhost:9200/firewall*/_mapping/cisco/field/geoip.location?prettyThis came back with:
{ "firewall-2015.04.107" : { "mappings" : { "cisco" : { "geoip.location" : { "full_name" : "geoip.location", "mapping":{"location":{"type":"float"}} } } } } }The float type location is why I can't add a tile map visualization. I need to change this to geo_point.
After loads of digging I found one way to change the mapping of the location type to geo_point was to use an output template. I copied the default elasticsearch_templte.json file from
/opt/logstash/lib/logstash/outputs/elasticsearch/elasticsearch-template.jsonto
/etc/logstash/templates/elasticsearch-firewall.jsonEdit it and change the template from logstash* to firewall* (or whatever your index name pattern is) "template" : "firewall*",
Edit the logstash config file and change the output to this:
output { elasticsearch { host => localhost index => [ "firewall-%{+YYYY.MM.DD}" ] template => "/etc/logstash/templates/elasticsearch-firewall.json" template_name => "firewall" } }Delete the firewall* index.
WARNING: This will delete all of your existing searchable data! If this data is important you'll need to find a way to dynamically change your field type without deleting the index. I've seen mention of this in google searches so I know it's possible.
curl -XDELETE http://localhost:9200/firewall*I restarted logstash and elasticsearch after this.
Once I started receiving logs again I checked the mapping again
curl http://localhost:9200/firewall*/_mapping/cisco/field/geoip.location?pretty { "firewall-2015.04.107" : { "mappings" : { "cisco" : { "geoip.location" : { "full_name" : "geoip.location", "mapping":{"location":{"type":"geo_point"}} } } } } }See how the location type is now geo_point :-)
Now I can add a Tile Map visualization.