See www.zabbix.com for the official Zabbix site.

Docs/comment for logstash

From Zabbix.org
Jump to: navigation, search

Sending comments to Logstash from within the Zabbix frontend

Proposed use

Annotations can be invaluable when you try to interpret a graph. Zabbix currently does not offer such a feature. The closest you get are comments on events, but that requires events, is bound to acknowledging them event and the comments are not shown on graphs.

If you are running Logstash yet, you may have come across markers yet: chart_markers.png The design goal is to make such comments visible on graphs. However, they are just regular Logstash events, so you can use them in every way Elasticsearch or Kibana allow.

This hack is intended to supply comments that can not be gathered from existing logs or can be submitted otherwise, for instance as Zabbix events, by a configuration management tool or a ticket system. The added value is the context of Zabbix hosts and host groups affected by what the comment says.

Example comments

  • Increased NIC hardware buffer size
  • Customer reports problem with an application
  • Deployed new version of application
  • Adjusted firewall rules for better performance
  • Killed some daemon manually due to excessive memory use
  • Advertisement campaign started
  • Performance benchmark begins

Implementation

Zabbix

Under Monitoring/Comments

A user can pick from a list of hosts and hostgroups he has access to -- much like in Configuration/Maintenance. A timestamped comment can be sent to a Logstash JSON input via TCP, containing the timestamped comment, as well as arrays of host groups and hosts the comment is "tagged" with. It also contains the Zabbix user's last name.

First tab of the comment form, showcasing the calendar widget
Tab showing the host and hostgroup selection

Logstash & Kibana

A TCP JSON input and a date filter are used to process the incoming data. The data can be used within all Kibana widgets. The design use case are annotations, along with Zabbix events, shipped by other means already.

A Zabbix event ("Name change") alongside a comment ("Apache update")
Tabular view of a comment; hostgroups and hosts are arrays

Alternative implementations

  • Implement the sender protocol to submit the information and cause Zabbix events

How to install it

Download the patch and apply it by running the following from your frontend directory:

 patch -p1 < name_of_the.patch

Edit comments.php and change logserver.example.com to your Logstash host and adjust the port, if necessary.

Configure the following input and filter for Logstash:

input {
  tcp { # Zabbix deployment messages
    codec => "json"
    port => 5555
    type => zabbix_deployment
  }
}

filter { # Convert timestamp
  if [type] == "zabbix_deployment" {
    date {
      match => [ "timestamp", "UNIX" ]
      remove_field => "timestamp"
    }
    mutate {
      add_field => [ "annotation", "%{user}: %{comment} -- Groups: %{hostgroups}, Hosts: %{hosts}" ]
    }
  }
}

Add a marker query like the following to your graphs and select "annotation" as the tooltip field:

 type:zabbix_deployment AND (hostgroups.raw:"whatever group" OR hosts.raw:(this* OR that))

Weaknesses

  • Comments are not visible in Zabbix yet

Chances

  • Integration with graphs: Click on a graph and get a pre-filled form
  • Remove the tab