Heka Decoders

Heka Decoders

Decoders parse the contents of the inputs to extract data from the text format and map them onto a Heka message schema. List of all available decoders is here
On controller we have the follwing decoders configured:

• decoder-collectd.toml
• decoder-http-check.toml
• decoder-keystone_7_0.toml
• decoder-keystone_wsgi.toml
• decoder-mysql.toml
• decoder-notification.toml
• decoder-openstack.toml
• decoder-ovs.toml
• decoder-pacemaker.toml
• decoder-rabbitmq.toml
• decoder-swift.toml
• decoder-system.toml

All custom decoders decoders are SandboxDecoder

SandboxDecoder simple example

Sandbox documentation
Sandbox decoder is complex part of Heka, so for better understanding there it is possible to create simple input and simple decoder. Idea is

• use the same input source as was used in collectd (read data from file "/var/log/collectd_in_data")
• use lua-based decoder to decode input data

Input

The best way to poll file is Heka. FilePollingInput. Configuration is pretty simple, this input plugin read data from file every ticker_interval. FilePollingInput can be used to get data from /proc file system, e.g. from /proc/loadavg. One more example is application which rewrites 'stats' files.
E.g. we have the following configuration:

[test_input]
type = "FilePollingInput"
ticker_interval = 1
file_path = "/var/log/collectd_in_data"
decoder = "test_decoder"

Decoder

Decoder in general are getting data from input plugins and convert it into Heka internal format.

[test_decoder]
type = "SandboxDecoder"
filename = "/usr/share/lma_collector/decoders/test_decoder.lua"

Decoder code is not so complex: <syntaxhighlight lang="lua">require "string"

local msg = {

   Type = "test_decoder",
   Payload = nil,
   Fields = {}

}

local count = 0

function process_message ()

   count = count + 1
   local data = read_message("Payload")
   msg.Fields.test_decoderData  = data
   msg.Fields.test_decoderCount = count
   msg.Fields.test_decoderDataToNumber  = tonumber(data)
   inject_message(msg)
   return 0

end </syntaxhighlight>

This decoder reads input data and creates message contains 2 fields:

• test_decoderDataToNumber - data from input message converted from string into number
• test_decoderCount - count of decoded messages since heka start
• test_decoder - data "as is"

Outut example in debug log:

:Timestamp: 2016-01-30 22:28:04 +0000 UTC
:Type: test_decoder
:Hostname: node-6
:Pid: 0
:Uuid: 199a7eea-7020-43a1-a593-a6781da1335f
:Logger: test_input
:Payload:
:EnvVersion:
:Severity: 7
:Fields:
    | name:"test_decoderData" type:double value:8.88999888e+08
    | name:"test_decoderData" type:string value:"888999888"
    | name:"test_decoderCount" type:double value:34

More details related to Lua Sandbox
Please pay your attention on restrictions: e.g. print and io functions are not available in Lua Sandbox