See www.zabbix.com for the official Zabbix site.

Docs/specs/ZBXNEXT-2442-more-info

From Zabbix.org
Jump to: navigation, search

Additional considerations not included in the specification

IPC

Option #1. Custom protocol over stream (Unix sockets, named FIFOs, TCP): workers request new work when done with the current alert, queue manager finds work.

  • TCP: might need encryption even though on the same machine.
  • TCP: slow - especially for large alerts.
  • TCP: easy to implement - blocking API is already in place.
  • Unix sockets or named FIFOs not currently used in Zabbix, new API needed.
  • Unix sockets provide basic security mechanism with file permissions.

Option #2. SysV shared memory. Queue manager adds alerts from DB, updates alerts in DB, workers take locks and work with alerts from memory.

  • Extensive locking in case of many workers. Still negligible compared to network latencies?
  • Extensive copying when resizing or de-fragmenting the shared memory segment.
    • Can we make the size of the alert in memory static to ease replacement implementation?

Option #3. 3rd party MQ architecture.

  • Neither can be easily adopted in Zabbix, not in scope of this ZBXNEXT.
    • Dependencies on complex brokers, (many) heavyweight libraries.
    • Many are not written in C.
    • Licensing problems.
    • Portability problems - searching for "X on AIX/Solaris/HP-UX" reveals many issues.
    • Few have reached maturity, adopting them for LTS is thus risky from functional PoV.
  • The most suitable for our needs is nanomsg.
    • Only beta releases available for now.
    • No commercial entity behind it - could become abandon-ware quickly.
    • MIT/X11 license.

Option #4. IPC message queues.

  • SysV queues supported on all platforms however the default limits differ significantly.
    • Not used elsewhere in Zabbix.
    • With large amount of data - dangerous with slow readers (that might be blocking in alert sending or DB update cycle).
  • POSIX queues
    • The defaults on Linux are quite limiting for our use case. Maximum message size: 8K, Maximum number of messages in queue (per-system): 10!.
      • "/proc/sys/fs/mqueue/msg_max"
      • "/proc/sys/fs/mqueue/msgsize_max"
    • Not supported on older systems.

Shared Memory

For shared memory model the size of the alert structure should be static to avoid extensive copying and reference validation when inserting/updating/deleting alerts or resizing shared memory segment (de-fragmenting).

All of the relevant fields from "alerts" table and "media_type" table have fixed maximum lengths. The exceptions is the "message" field in the "alerts" table (type TEXT).

  • Maximum size of the "TEXT" SQL type: up to 2GB.
  • Maximum size in "dbschema.c": 2K or 64K (depends on the DB).
    • The length is enforced in "add_message_alert()" --> "zbx_db_insert_add_values()" --> "zbx_db_insert_add_values_dyn()".

Alternative design #1: fork() worker process to handle each alert

Alerter forks a new worker process for sending each alert (including retries). Although the UML looks complex this is the simplest approach to the problem.

Alerter-fork-mainloop.png

Advantages

  • A "slow" media type will not block delivery of other alerts because each alert is handled in isolation.
  • Throughput is increased but is subject to the per media type rate control.
  • Hang or crash of the worker process doesn't impact alerts for other media types.
  • Rate control can be trivially implemented in the main alerter process, no need for shared memory or other synchronization schemes.

Disadvantages

  • Memory limitations - each worker consumes the same amount of memory as main alerter.
  • This solution doesn't integrate well in the existing Zabbix process infrastructure (self monitoring, etc.).
  • fork() is slow (but not as slow as waiting on network events).

Open questions

  • Performance characteristics of this solution.
  • What to do if the worker hangs or crashes.

Alternative design #2: per media type alerters

Proof-of-concept of this solution is already implemented in ZBX-9654.

Another conceptually simple idea is to have separate alerter processes per each media type. The main alerter process starts new media-type handlers when new media types are added. Media type handlers quit if the corresponding media type was disabled or deleted.

Alerter-mediatypes-controller-mainloop.png

Alerter-mediatypes-mainloop.png

Advantages

  • Each media type handled separately, do not influence other media types.
  • If media type handler crashes the main alerter process starts a replacement handler.

Disadvantages

  • Throughput is improved by a little bit if many media types used. If only one media type is used, then throughput remains the same.
  • This solution doesn't integrate well in the existing Zabbix process infrastructure.

Alternative design #3: StartAlerters=n in server configuration

This naive solution is implemented in ZBXNEXT-2442 branch (r55633, r55635).

A trivial solution - start configurable amount of alerters during server's start-up. Each alerter handles different set of alerts based on alertid. The sets never intersect.

Since there is no controlling process (alert scheduler) the principal difficulty with this solution is safe, efficient alert distribution between processes.

Even with sending rate control mechanism in place some alerters could be idling while alerts of faster media types are stuck behind alerts of slow media types in other alerters.

Alerter-startalerters-mainloop.png

Advantages

  • Allows user to balance risks (throughput vs. media type independence).
  • Integrates well into existing Zabbix process infrastructure.

Disadvantages

  • Synchronization between alerter processes is needed to keep track of sending rate per media types.
  • Alert ordering is a problem: in many situations it makes no sense to try to close a ticket if the ticket is not created yet.
  • Alert delivery for sequential media types is very difficult (token passing or locking of some sort is needed).

Alternative design #4:

1. Separate set of alerters for each media (e-mail, SMS, Jabber, etc.).

Implemented in ZBXNEXT-2442 branch (r55772).

The number of alerters per media can be configured as per needs basis. For example, if Ez Texting is not used then alerter for this media can be disabled in server's configuration. If there is only one GSM modem then only one SMS alerter should be started while there can still be many e-mail and jabber alerters.

2. Synchronization of the currently handled tuples between alerters of particular media

To support per-recipient ordered delivery we need the alerters to handle distinct tuples (media, media type (specific server or script), recipient) at any given time.

For holding currently handled tuples we can use either SysV shared memory or DB itself. Using the DB has some advantages: it would be more elegant and require less code changes. It would also allow for easier troubleshooting however this solution requires changes to DB schema and thus DB upgrade. With shared memory we would have to implement something like "not exists" SQL statement.

Alerter-StartTypeAlerters-mainloop.png

In the DB we have to add the respective fields to the media_type table as well last_sent field.

Advantages

  • The alarm selection can be done with one "select" statement if the currently handled tuples are saved in a separate table.

Risks

  • Excessive waiting on locks?
    • Unlikely since DB access is still much faster than network delays.
    • The upper configuration limit of alerters per media is 1000 to be consistent with other Zabbix processes.
  • Load on DB will be increased, too much overhead in maintaining possibly thousands of DB connections. This is in fact significant risk.