Filebeat

Filebeat

Это заготовка заметки про Filebeat

Уcтановка

из той же репы что и Elasticsearch

wget -qO - https://artifacts.elastic.co/GPG-KEY-elasticsearch | sudo apt-key add -
echo "deb https://artifacts.elastic.co/packages/7.x/apt stable main" | sudo tee -a /etc/apt/sources.list.d/elastic-7.x.list
sudo apt-get update
sudo apt-get install filebeat

Ссылки

• http://vasilisc.com/filebeat-elk
• https://www.elastic.co/guide/en/beats/filebeat/current/configuring-howto-filebeat.html

Примеры конфигов

Autodiscovery

Приложение на руби пишет логи вида

2021-09-02T10:31:26.646Z 7 TID-8n4j WARN: /gems/ruby/2.7.0/gems/sidekiq-5.2.9/lib/sidekiq/processor.rb:274:in `block in constantize'
/gems/ruby/2.7.0/gems/sidekiq-5.2.9/lib/sidekiq/processor.rb:271:in `each'
/gems/ruby/2.7.0/gems/sidekiq-5.2.9/lib/sidekiq/processor.rb:271:in `inject'
<пропущено много строе>
/gems/ruby/2.7.0/gems/sidekiq-5.2.9/lib/sidekiq/util.rb:25:in `block in safe_thread'
/gems/ruby/2.7.0/gems/logging-2.3.0/lib/logging/diagnostic_context.rb:474:in `block in create_with_logging_context'
2021-09-02T10:31:31.906Z 7 TID-8n6f AiReviews::Tags::SyncStatusJob JID-3e0d1a6527c0ce06ddc34b14 INFO: start

Для того что бы записать его в лог не построчно и не указывать путь к логу используется конструкция:

filebeat.autodiscover:
  providers:
    - type: docker
      templates:
        - condition:
            regexp:
              docker.container.image: "server"
          config:
            - type: container
              paths:
                - /var/lib/docker/containers/${data.docker.container.id}/*.log
              multiline.pattern: '^202[0-9]-'
              multiline.negate: true
              multiline.match: after
              fields:
                type: "ruby_application_logs"
                docker_autodiscovery: "true"
                source_hostname: "testbyvlada.arturhaunt.com"
              fields_under_root: true

Multileine

filebeat.inputs:
- type: log
  enabled: true
  paths:
    - /var/www/deploy/backend/shared/log/staging.log
  exclude_files: ['\.gz$']
  fields:
    nginx_logs: "false"
    ruby_application_logs: "true"
    type: "ruby_application_logs"
    source_hostname: "testbyvlada.arturhaunt.com"
  fields_under_root: true
  multiline:
    pattern: '[A-Z], \[[0-9]{4}'
    negate: true
    match: after