Monitoring

The stack

The log is manage by a ELK stack. All the main components (aka Elastic, Logstash and Kibana) are installed on the OLS1 server. They run as services, then to start, stop or restart, the command is to use "sudo service elasticsearch|logstash|kibana start|stop|restart".

The log are transmitted to logstash through either collectd (for system metrics of all the servers) or filebeat (for service log).

Knwon Issue:

WIth the among of data that is managed, the elasticsearch storage could be full rapidly. To avoid it, in OLS1 server a tool (curator) is istalled to remove the old logs. The script to remove the logs that have more than 10 days old, is in /curator-script. To run it : curator /curator-script/delete_indices