See www.zabbix.com for the official Zabbix site.

Tagging logstash

From Zabbix.org
Jump to: navigation, search

Efficiently ship Logstash events to Zabbix and augment events inside Logstash with Zabbix context

Challenge and goal

While it is possible to submit Logstash events to Zabbix with the zabbix output, this involves a number of challenges:

  • Find the proper technical Zabbix hostname to submit the event for (unless it is clear by convention)
  • Match an item key which has to exist
  • It's inefficient: Forks zabbix_sender on every invocation -- This is no longer true with later versions

Due to the fact that marker queries can be used in Kibana to show Zabbix events or comments with a Zabbix context, it's worthwhile to try tagging every single event with Zabbix host groups and similar information. This information can then be used for access control to Kibana too, using something like the elasticsearch-proxy, if you are still using Kibana 3. If you are using LDAP for authentication, the same user name can be used in Zabbix and Kibana and consequently grant the same permissions as in Zabbix. Additional filters may be added, of course!

  1. Events arrive in Logstash through any supported means
  2. The event is augmented with Zabbix information, originally obtained through the API
  3. The augmented event is stored in Elasticsearch and selected data of selected events is submitted to Zabbix, eventually using the sender protocol

Concept

This concept is based on the ideas of don_e and untergeek. A daemon shall be implemented to circumvent excessive API access and forking. It uses the Zabbix API to regularly obtain configuration information (users, usergroups, hosts, host groups, inventory, ...). It is connected to Logstash through the zeromq filter. Events keep coming in, are enriched and returned. The daemon has an additional socket that Logstash connects to through zeromq output. The Zabbix sender protocol is used to eventually submit the data.

If re-queing can be implemented, Zabbix trapper discovery rules could be used to create items and triggers on demand and then re-submit the data, once the item is available. -- I abandoned this concept for a simpler solution:

An API script is querying hosts and their host groups. It transforms the structure to suitable YAML, which is then used in Logstash's translate filter. This filter is capable of reloading dictionaries at regular intervals. Thus you can add the YAML generating script as a cron job and set translate to reload it once in a while.

#!/usr/bin/python
# -*- coding: utf-8 -*-
from zabbix_api import ZabbixAPI
import yaml

server="https://example.com"
username="Admin"
password="zabbix"
zapi = ZabbixAPI(server=server, path="/zabbix", log_level=0, timeout=100)
zapi.login(username, password)

hosts = zapi.host.get({"output":["host"], "selectGroups": True})
result = dict()

for host in hosts:
        result[host["host"]] = list()
        for group in host["groups"]:
                result[host["host"]].append(group["name"])

print yaml.dump(result)


...
  translate {
    field => "@source_host"
    dictionary_path => "/etc/logstash/dicts/zabbix_hosts_hostgroups"
    fallback => "unknown"
    destination => "zbx"   
  }
...

The fallback options comes with the added advantage of being able to detect messages being sent for a hostname that is not monitored in Zabbix.