Heka

Heka

Heka is an open source stream processing software system developed by Mozilla. Heka is a “Swiss Army Knife” type tool for data processing, useful for a wide variety of different tasks, such as:

• Loading and parsing log files from a file system.
• Accepting statsd type metrics data for aggregation and forwarding to upstream time series data stores such as graphite or InfluxDB.
• Launching external processes to gather operational data from the local system.
• Performing real time analysis, graphing, and anomaly detection on any data flowing through the Heka pipeline.
• Shipping data from one location to another via the use of an external transport (such as AMQP) or directly (via TCP).
• Delivering processed data to one or more persistent data stores.

Inputs

There are 2 types of input plugins used in heka

• HttpListenInput
  • 127.0.0.1:8325; collectd_decoder
• LogstreamerInput
  • /var/log/libvirt; libvirt_decoder
  • file_match = '(?P<Service>nova|cinder|keystone|glance|heat|neutron|murano)-all\.log$', openstack_decoder
  • "/var/log/dashboard\.log$'; decoder = "openstack_decoder"; splitter = "TokenSplitter"
  • file_match = '(?P<Service>nova|cinder|keystone|glance|heat|neutron|murano)-all\.log$'; differentiator = [ 'openstack.', 'Service' ]; decoder = "openstack_decoder"; splitter = "openstack_splitter"
  • file_match = '(?P<Service>ovs\-vswitchd|ovsdb\-server)\.log$';differentiator = [ 'Service' ];decoder = "ovs_decoder";splitter = "TokenSplitter"
  • file_match = '(?P<Service>daemon\.log|cron\.log|haproxy\.log|kern\.log|auth\.log|syslog|messages|debug)';differentiator = [ 'system.', 'Service' ];decoder = "system_decoder"

Splitters

Splitter details: https://hekad.readthedocs.org/en/v0.10.0/config/splitters/index.html
There are only one custom splitter:

[openstack_splitter]
type = "RegexSplitter"
delimiter = '(<[0-9]+>)'
delimiter_eol = false

Decoders

decoder-collectd.toml
decoder-libvirt.toml
decoder-openstack.toml
decoder-ovs.toml
decoder-system.toml

Heka Debugging

[RstEncoder]

[output_file]
type = "FileOutput"
#message_matcher = "Fields[aggregator] == NIL && Type == 'heka.sandbox.afd_node_metric'"
message_matcher = "Fields[aggregator] == NIL"
path = "/var/log/heka-debug.log"
perm = "666"
flush_count = 100
flush_operator = "OR"
#encoder = "nagios_afd_nodes_encoder_debug"
encoder = "RstEncoder"