Heka
Heka
Heka is an open source stream processing software system developed by Mozilla. Heka is a “Swiss Army Knife” type tool for data processing, useful for a wide variety of different tasks, such as:
- Loading and parsing log files from a file system.
- Accepting statsd type metrics data for aggregation and forwarding to upstream time series data stores such as graphite or InfluxDB.
- Launching external processes to gather operational data from the local system.
- Performing real time analysis, graphing, and anomaly detection on any data flowing through the Heka pipeline.
- Shipping data from one location to another via the use of an external transport (such as AMQP) or directly (via TCP).
- Delivering processed data to one or more persistent data stores.
Configuration overview
All LMA heka config files are located in /etc/lma_collector folder. e.g. on controller there are follwing confguration files:
amqp-openstack_error.toml amqp-openstack_info.toml amqp-openstack_warn.toml decoder-collectd.toml decoder-http-check.toml decoder-keystone_7_0.toml decoder-keystone_wsgi.toml decoder-mysql.toml decoder-notification.toml decoder-openstack.toml decoder-ovs.toml decoder-pacemaker.toml decoder-rabbitmq.toml decoder-swift.toml decoder-system.toml encoder-elasticsearch.toml encoder-influxdb.toml encoder-nagios_afd_nodes_debug.toml encoder-nagios_afd_nodes.toml encoder-nagios_gse_global_clusters.toml encoder-nagios_gse_node_clusters.toml filter-afd_api_backends.toml filter-afd_api_endpoints.toml filter-afd_node_controller_cpu.toml filter-afd_node_controller_log-fs.toml filter-afd_node_controller_root-fs.toml filter-afd_node_mysql-nodes_mysql-fs.toml filter-afd_service_apache_worker.toml filter-afd_service_cinder-api_http_errors.toml filter-afd_service_glance-api_http_errors.toml filter-afd_service_heat-api_http_errors.toml filter-afd_service_keystone-admin-api_http_errors.toml filter-afd_service_keystone-public-api_http_errors.toml filter-afd_service_mysql_node-status.toml filter-afd_service_neutron-api_http_errors.toml filter-afd_service_nova-api_http_errors.toml filter-afd_service_rabbitmq_disk.toml filter-afd_service_rabbitmq_memory.toml filter-afd_service_rabbitmq_queue.toml filter-afd_service_swift-api_http_errors.toml filter-afd_workers.toml filter-gse_global.toml filter-gse_node.toml filter-gse_service.toml filter-heka_monitoring.toml filter-http_metrics.toml filter-influxdb_accumulator.toml filter-influxdb_annotation.toml filter-instance_state.toml filter-resource_creation_time.toml filter-service_heartbeat.toml global.toml httplisten-collectd.toml httplisten-http-check.toml input-aggregator.toml logstreamer-keystone_7_0.toml logstreamer-keystone_wsgi.toml logstreamer-mysql.toml logstreamer-openstack_7_0.toml logstreamer-openstack_dashboard.toml logstreamer-ovs.toml logstreamer-pacemaker.toml logstreamer-rabbitmq.toml logstreamer-swift.toml logstreamer-system.toml multidecoder-aggregator.toml output-aggregator.toml output-dashboard.toml output-elasticsearch.toml output-influxdb.toml output-nagios_afd_nodes.toml output-nagios_gse_global_clusters.toml output-nagios_gse_node_clusters.toml scribbler-aggregator_flag.toml splitter-openstack.toml splitter-rabbitmq.toml
Heka's configuration files can be divided into follwing groups:
- Inputs
- Splitters
- Decoders
- Filters
- Encoders
- Outputs
Inputs
On controller there are following inputs groups:
AMQPInput
AMQP input (https://hekad.readthedocs.org/en/v0.10.0/config/inputs/amqp.html)
There are followinf AMQP inputs:
- amqp-openstack_error.toml
- amqp-openstack_info.toml
- amqp-openstack_warn.toml
All AMQP inputs looks like:
[openstack_error_amqp] type = "AMQPInput" url = "amqp://nova:nova_password@192.168.0.2:5673/" exchange = "nova" exchange_type = "topic" exchange_durability = false exchange_auto_delete = false queue_auto_delete = false queue = "lma_notifications.error" routing_key = "lma_notifications.error" decoder = "notification_decoder" splitter = "NullSplitter" can_exit = true
The only difference between AMQP inputs are queue and routing_key parameter:
queue = "lma_notifications.info" routing_key = "lma_notifications.info"
All AMQP inputs use one decoder to decode AMQP messages: notification_decoder, configuration can be found in decoder-notification.toml file.
LMA plugin configures openstack services to use 'lma_notifications' as notification_topics, e.g :
# cat /etc/nova/nova.conf | grep lma notification_topics=lma_notifications
so heka is enable to get messages from queue and decode it.
Also, it is possible to see rabbitmq messages using trace plugin, for details please see: http://wiki.sirmax.noname.com.ua/index.php/Rabbitmq_trace#RabbitMQ_log_messages
HttpListenInput=
httplisten-collectd.toml httplisten-http-check.toml
TcpInput
input-aggregator.toml
LogstreamerInput=
logstreamer-keystone_7_0.toml logstreamer-keystone_wsgi.toml logstreamer-mysql.toml logstreamer-openstack_7_0.toml logstreamer-openstack_dashboard.toml logstreamer-ovs.toml logstreamer-pacemaker.toml logstreamer-rabbitmq.toml logstreamer-swift.toml logstreamer-system.toml
Splitters
Splitter details: https://hekad.readthedocs.org/en/v0.10.0/config/splitters/index.html
There are only one custom splitter:
[openstack_splitter] type = "RegexSplitter" delimiter = '(<[0-9]+>)' delimiter_eol = false
Decoders
decoder-collectd.toml decoder-libvirt.toml decoder-openstack.toml decoder-ovs.toml decoder-system.toml
Heka Debugging
[RstEncoder] [output_file] type = "FileOutput" #message_matcher = "Fields[aggregator] == NIL && Type == 'heka.sandbox.afd_node_metric'" message_matcher = "Fields[aggregator] == NIL" path = "/var/log/heka-debug.log" perm = "666" flush_count = 100 flush_operator = "OR" #encoder = "nagios_afd_nodes_encoder_debug" encoder = "RstEncoder"
