Kibana stopped working and now server not getting ready although kibana.service starts up nicely

问题

Without any major system update of my Ubuntu (4.4.0-142-generic #168-Ubuntu SMP), Kibana 7.2.0 stopped working. I am still able to start the service with sudo systemctl start kibana.service and the corresponding status looks fine. There is only a warning and no error, this does not seem to be the issue:

# sudo systemctl status kibana.service 
● kibana.service - Kibana
   Loaded: loaded (/etc/systemd/system/kibana.service; enabled; vendor preset: enabled)
   Active: active (running) since Wed 2019-07-10 09:43:49 CEST; 22min ago
 Main PID: 14856 (node)
    Tasks: 21
   Memory: 583.2M
      CPU: 1min 30.067s
   CGroup: /system.slice/kibana.service
           └─14856 /usr/share/kibana/bin/../node/bin/node --no-warnings --max-http-header-size=65536 /usr/share/kibana/bin/../src/cli -c /etc/kibana/kibana.yml

Jul 10 09:56:36 srv003 kibana[14856]: {"type":"log","@timestamp":"2019-07-10T07:56:36Z","tags":["warning","task_manager"],"pid":14856,"message":"The task maps_telemetry \"Maps-maps_telemetry\" is not cancellable."}

Nevertheless, when I visit http://srv003:5601/ on my client machine, I keep seeing only (even after waiting 20 minutes):

Kibana server is not ready yet

On the server srv003 itself, I see

me@srv003:# curl -XGET http://localhost:5601/status -I
curl: (7) Failed to connect to localhost port 5601: Connection refused

This is a strange since Kibana seems to be really listening at that port and the firewall is disabled for testing purposes:

root@srv003# sudo lsof -nP -i | grep 5601
node      14856        kibana   18u  IPv4 115911041      0t0  TCP 10.0.0.72:5601 (LISTEN)
root@srv003# sudo ufw status verbose
Status: inactive

There is nothing suspicious in the log of kibana.service either:

root@srv003:/var/log# journalctl -u kibana.service | grep -A 99 "Jul 10 10:09:14"
Jul 10 10:09:14 srv003 systemd[1]: Started Kibana.
Jul 10 10:09:38 srv003 kibana[14856]: {"type":"log","@timestamp":"2019-07-10T08:09:38Z","tags":["warning","task_manager"],"pid":14856,"message":"The task maps_telemetry \"Maps-maps_telemetry\" is not cancellable."}

My Elasticsearch is still up and running. There is nothing interesting in the corresponding log files about Kibana:

root@srv003:/var/log# cat elasticsearch/elasticsearch.log  |grep kibana
[2019-07-10T09:46:25,158][INFO ][o.e.c.m.MetaDataIndexTemplateService] [srv003] adding template [.kibana_task_manager] for index patterns [.kibana_task_manager]
[2019-07-10T09:47:32,955][INFO ][o.e.c.m.MetaDataCreateIndexService] [srv003] [.monitoring-kibana-7-2019.07.10] creating index, cause [auto(bulk api)], templates [.monitoring-kibana], shards [1]/[0], mappings [_doc]

Now I am running a bit out of options, and I hope somebody can give me another hint.

Edit: I do not have any Kibana plugins installed.

Consulted sources:

How to fix "Kibana server is not ready yet" error when using AKS
Kibana service is running but can not access via browser to console
Why won't Kibana Node server start up?
https://discuss.elastic.co/t/failed-to-start-kibana-7-0-1/180259/3 - most promising thread, but nobody ever answered
https://discuss.elastic.co/t/kibana-server-is-not-ready-yet-issue-after-upgrade-to-6-5-0/157021
https://discuss.elastic.co/t/kibana-server-not-ready/162075

回答1:

It looks like if Kibana enters the described undefined state, a simple reboot of the computer is necessary. This is of course not acceptable for a (virtual or physical) machine where other services are running.

来源：https://stackoverflow.com/questions/56966684/kibana-stopped-working-and-now-server-not-getting-ready-although-kibana-service

标签

ElasticSearch

service

kibana

elastic-stack