Heka Decoders

Heka Decoders

Decoders parse the contents of the inputs to extract data from the text format and map them onto a Heka message schema. List of all available decoders: https://hekad.readthedocs.org/en/v0.10.0/config/decoders/index.html
On controller we have the follwing decoders configured:

• decoder-collectd.toml
• decoder-http-check.toml
• decoder-keystone_7_0.toml
• decoder-keystone_wsgi.toml
• decoder-mysql.toml
• decoder-notification.toml
• decoder-openstack.toml
• decoder-ovs.toml
• decoder-pacemaker.toml
• decoder-rabbitmq.toml
• decoder-swift.toml
• decoder-system.toml

All custom decoders decoders are SandboxDecoder

SandboxDecoder simple example

Sandbox documentation: https://hekad.readthedocs.org/en/v0.10.0/sandbox/index.html
Sandbox decoder is complex part of Heka, so for better understanding there it is possible to create simple input and simple decoder. Idea is

• use the same input source as was used in collectd (read data from file "/var/log/collectd_in_data")
• use lua-based decoder to decode input data

Input

The best way to poll file is Heka. FilePollingInput. Configuration is pretty simple, this input plugin read data from file every ticker_interval. FilePollingInput can be used to get data from /proc file system, e.g. from /proc/loadavg. One more example is application which rewrites 'stats' files.
E.g. we have the following configuration:

[test_input]
type = "FilePollingInput"
ticker_interval = 1
file_path = "/var/log/collectd_in_data"
decoder = "test_decoder"

Decoder

Decoder in general are getting data from input plugins and convert it into Heka internal format.

[test_decoder]
type = "SandboxDecoder"
filename = "/usr/share/lma_collector/decoders/test_decoder.lua"

Decoder code is not so complex: <syntaxhighlight lang="lua">require "string"

local msg = {

   Type = "test_decoder",
   Payload = nil,
   Fields = {}

}

local count = 0

function process_message ()

   count = count + 1
   local data = read_message("Payload")
   msg.Fields.test_decoderData  = data
   msg.Fields.test_decoderCount = count
   msg.Fields.test_decoderDataToNumber  = tonumber(data)
   inject_message(msg)
   return 0

end </syntaxhighlight>

This decoder reads input data and creates message contains 2 fields:

• test_decoderDataToNumber - data from input message converted from string into number
• test_decoderCount - count of decoded messages since heka start
• test_decoder - data "as is"

Outut example in debug log:

:Timestamp: 2016-01-30 22:28:04 +0000 UTC
:Type: test_decoder
:Hostname: node-6
:Pid: 0
:Uuid: 199a7eea-7020-43a1-a593-a6781da1335f
:Logger: test_input
:Payload:
:EnvVersion:
:Severity: 7
:Fields:
    | name:"test_decoderData" type:double value:8.88999888e+08
    | name:"test_decoderData" type:string value:"888999888"
    | name:"test_decoderCount" type:double value:34

More details related to Lua Sandbox : https://hekad.readthedocs.org/en/v0.10.0/sandbox/index.html
Please pay your attention on restrictions: e.g. print and io functions are not available in Lua Sandbox