Monitoring

Monitorimg

Collectd

Collectd is simple data collector, use plugins to collect data and output plugins to send tada to another tool (heka in our confuration)
Collectd is collecting following metrics (compute node, simple cluster):

Metrics

Please see plugin details on collectd man page: https://collectd.org/documentation/manpages/collectd.conf.5.shtml#

• cpu (CPU usage)
• df (disk usage/free size)
• disk (disk usage/IOPS)
• interface (interfece usage/bytes sent and received )
• load (Linux LA)
• memory (memory usage)
• processes (detailed monitoring of collect and hekad)
• swap (swap usage)]

Output

Collectd saves all data in rrd files and sends it to heka using write_http plugin )https://collectd.org/documentation/manpages/collectd.conf.5.shtml#plugin_write_http). It sends data in JSON format to local hekad (BTW Why do we use local heka on each node?)
Plugin configuration:

<LoadPlugin write_http>
  Globals false
</LoadPlugin>

<Plugin "write_http">
  <URL "http://127.0.0.1:8325">
    Format "JSON"
    StoreRates true
  </URL>
</Plugin>

Hekad is listen on 127.0.0.1:8325

# netstat -ntpl | grep 8325
tcp        0      0 127.0.0.1:8325          0.0.0.0:*               LISTEN      15368/hekad

Debug

It is possible to debug data tranfering from collectd to hekad. e.g. you can use tcpflow or you favorite tool to dump http traffic
Run dumping tool:

• heka is listen on port 8325, taken from write_http config
• lo interface is loopback, heka is listen on 127.0.0.1, so it is easy to find interface
  • # ip ro get 127.0.0.1 local 127.0.0.1 dev lo src 127.0.0.1 cache <local>
  • dev lo is device you need.
• # tcpflow -i lo port 8325
• Example of output:
  # cat 127.000.000.001.45848-127.000.000.001.08325 | head -8 POST / HTTP/1.1 User-Agent: collectd/5.4.0.git Host: 127.0.0.1:8325 Accept: */* Content-Type: application/json Content-Length: 4064 [{"values":[2160],"dstypes":["gauge"],"dsnames":["value"],"time":1453203196.259,"interval":10.000,"host":"node-7","plugin":"processes","plugin_instance":"collectd","type":"ps_stacksize","type_instance":""},{"values":[0,1999.74],"dstypes":["derive","derive"],"dsnames": ... skip ...

Config Files

All config files are in /etc/collectd/
/etc/collectd/conf.d stores plugin configuration files

# ls -lsa /etc/collectd/conf.d/
4 -rw-r----- 1 root root  169 Jan 18 16:38 05-logfile.conf
4 -rw-r----- 1 root root   71 Jan 18 16:38 10-cpu.conf
4 -rw-r----- 1 root root  289 Jan 18 16:38 10-df.conf
4 -rw-r----- 1 root root  145 Jan 18 16:38 10-disk.conf
4 -rw-r----- 1 root root  189 Jan 18 16:38 10-interface.conf
4 -rw-r----- 1 root root   72 Jan 18 16:38 10-load.conf
4 -rw-r----- 1 root root   74 Jan 18 16:38 10-memory.conf
4 -rw-r----- 1 root root   77 Jan 18 16:38 10-processes.conf
4 -rw-r----- 1 root root  138 Jan 18 16:38 10-swap.conf
4 -rw-r----- 1 root root   73 Jan 18 16:38 10-users.conf
4 -rw-r----- 1 root root  189 Jan 18 16:38 10-write_http.conf
4 -rw-r----- 1 root root   66 Jan 18 16:38 processes-config.conf

On controller there are more metrics:

d# ls -1
05-logfile.conf
10-apache.conf
10-cpu.conf
10-dbi.conf
10-df.conf
10-disk.conf
10-interface.conf
10-load.conf
10-match_regex.conf
10-memcached.conf
10-memory.conf
10-mysql.conf
10-processes.conf
10-swap.conf
10-target_notification.conf
10-users.conf
10-write_http.conf
99-chain-PostCache.conf
dbi_cinder_services.conf
dbi_mysql_status.conf
dbi_neutron_agents.conf
dbi_nova_services.conf
mysql-nova.conf
openstack.conf
processes-config.conf

Heka

Heka is an open source stream processing software system developed by Mozilla. Heka is a “Swiss Army Knife” type tool for data processing, useful for a wide variety of different tasks, such as:

• Loading and parsing log files from a file system.
• Accepting statsd type metrics data for aggregation and forwarding to upstream time series data stores such as graphite or InfluxDB.
• Launching external processes to gather operational data from the local system.
• Performing real time analysis, graphing, and anomaly detection on any data flowing through the Heka pipeline.
• Shipping data from one location to another via the use of an external transport (such as AMQP) or directly (via TCP).
• Delivering processed data to one or more persistent data stores.

Inputs

There are 2 types of input plugins used in heka

• HttpListenInput
  • 127.0.0.1:8325; collectd_decoder
• LogstreamerInput
  • /var/log/libvirt; libvirt_decoder
  • file_match = '(?P<Service>nova|cinder|keystone|glance|heat|neutron|murano)-all\.log$', openstack_decoder
  • "/var/log/dashboard\.log$'; decoder = "openstack_decoder"; splitter = "TokenSplitter"
  • file_match = '(?P<Service>nova|cinder|keystone|glance|heat|neutron|murano)-all\.log$'; differentiator = [ 'openstack.', 'Service' ]; decoder = "openstack_decoder"; splitter = "openstack_splitter"
  • file_match = '(?P<Service>ovs\-vswitchd|ovsdb\-server)\.log$';differentiator = [ 'Service' ];decoder = "ovs_decoder";splitter = "TokenSplitter"
  • file_match = '(?P<Service>daemon\.log|cron\.log|haproxy\.log|kern\.log|auth\.log|syslog|messages|debug)';differentiator = [ 'system.', 'Service' ];decoder = "system_decoder"

Splitters

Splitter details: https://hekad.readthedocs.org/en/v0.10.0/config/splitters/index.html
There are only one custom splitter:

[openstack_splitter]
type = "RegexSplitter"
delimiter = '(<[0-9]+>)'
delimiter_eol = false