See www.zabbix.com for the official Zabbix site.

Docs/howto/high availability

From Zabbix.org
Jump to: navigation, search

High Availability setup for Zabbix 1.8 - 2.4

General information/overview

Zabbix server runs on one node at a time

In this howto we will set up high availability Zabbix using Zabbix appliance. Instructions here apply to OpenSUSE 11.3, upon which Zabbix appliance 1.8.4 is based. The procedure has been tested with Zabbix 2.0, 2.2 and 2.4 as well.

Instructions will assume vmdk images - you are free to use any of the other provided image formats.

The expected end result is having two Zabbix instances, using a shared database. Zabbix servers work in an active-passive mode, where Zabbix server is ever running on a single node only. Additionally, both high availability nodes share a virtual IP which is grouped together with Zabbix server (that is, Zabbix server follows the IP).

OpenAIS/Corosync is used for low level availability checking and Pacemaker ensures services are properly switched from one node to another.

Note that these instructions are not Zabbix specific when it concerns the high availability setup - any daemon should be able to be set up this way.

Acknowledgements

Thanks to the sites and people who helped to piece things together.

Prerequisites

  • A virtualisation environment, unless setting up on separate physical systems.
  • A separate MySQL database on a different system from those two. You may use another Zabbix appliance system. This howto will assume such a setup.
  • Two IP addresses for Zabbix systems, another IP for failover. It is assumed that failover nodes will acquire IP addresses over DHCP.
  • Failover node systems should have access to internet as we will be installing additional packages.

Initial system setup

We'll start by downloading Zabbix appliance and setting up two instances of it.

Download the appliance from Zabbix download page and verify that we have it.

$ ls
Zabbix_x86.i686-1.8.4.vmx.tar.gz

Let's create two copies of virtual machines and name them appropriately so that we can easily distinguish them.

$ appliancename=Zabbix_x86.i686-1.8.4
$ applianceversion=${appliancename#*-}
$ tar -zxvf $appliancename.vmx.tar.gz
$ mv ${appliancename%%.*}-$applianceversion zabbix1
$ cp -r zabbix1 zabbix2
$ for i in zabbix1/* zabbix2/*; do mv $i ${i%/*}/${i%/*}.${i##*.}; done
$ for i in 1 2; do sed -i "s/$appliancename/zabbix$i/" zabbix$i/zabbix$i.vmx; done

Now start up the virtual machines.

For access to these virtual machines use account root with password zabbix. For more detail on the appliance, see it's documentation.

From now on, the systems will be referred to like this:

  • zabbix1 - first failover system
  • zabbix2 - second failover system
  • zabbixdb - system with the Zabbix database
  • zabbix - failover IP

You should either replace system names with IPs, corresponding to your environment, or populate entries in /etc/hosts.

You might want to benefit from SSH public key authentication, so optionally allow access without password from your workstation:

$ for i in 1 2; do ssh -l root zabbix$i "mkdir ~/.ssh"; scp ~/.ssh/id_dsa.pub root@zabbix$i:.ssh/authorized_keys2; done

Note that you will be asked for password 4 times.

For convenience, we can also set hostnames for both of these systems to zabbix1 and zabbix2:

$ for i in 1 2; do ssh -l root zabbix$i "echo \"zabbix$i\" > /etc/HOSTNAME; hostname zabbix$i"; done
Note: Hostnames of cluster nodes must be different otherwise they will not be distinguished as different ones. If you add several nodes to the cluster with the same hostname, fix the hostnames, restart OpenAIS (rcopenais restart) and remove the incorrect node from the cluster (crm node delete <incorrect node>).

Readying Zabbix for failover

Now we should prepare Zabbix for failover configuration.

First, we will stop Zabbix server, because it will be managed by the high availability configuration.

$ for i in 1 2; do ssh -l root zabbix$i "chkconfig -d zabbix_server; rczabbix_server stop"; done

Database

Now let's set up database. We will use a database name zabbix_failover. On zabbixdb, execute:

 $ mysql
 Welcome to the MySQL monitor.  Commands end with ; or \g.
 Your MySQL connection id is 79913
 Server version: 5.1.46 SUSE MySQL RPM
 
 Copyright (c) 2000, 2010, Oracle and/or its affiliates. All rights reserved.
 This software comes with ABSOLUTELY NO WARRANTY. This is free software,
 and you are welcome to modify and redistribute it under the GPL v2 license
 
 Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.
 mysql> create database zabbix_failover character set utf8;
 Query OK, 1 row affected (0.00 sec)
 mysql> grant all privileges on zabbix_failover.* to zabbix@'%' identified by 'someslightlybetterpasswordshouldbeusedhere';
 Query OK, 0 rows affected (0.09 sec)

You will want to choose a better password.

If using Zabbix appliance for the database server, stop Zabbix server on that system, it wouldn't serve a useful purpose for us.

Now let's create a copy of the database from zabbix1 system (the following commands again can be executed from your workstation):

$ ssh -l root zabbix1 "mysqldump zabbix | bzip2 -9 > zabbix_appliance_db.bz2"

Which we then move over to the system, hosting Zabbix database and restore there (we first copy it to the local system to avoid dealing with host keys and authentication from failover systems to the database system):

$ scp root@zabbix1:zabbix_appliance_db.bz2 /tmp
$ scp /tmp/zabbix_appliance_db.bz2 root@zabbixdb:
$ ssh -l root zabbixdb "bzcat zabbix_appliance_db.bz2 | mysql zabbix_failover"

Now we don't need the database running on the failover systems anymore, so we can stop it to save some resources. We'll also clear out crontab for zabbix user so that no Zabbix health checking scripts are used:

$ for i in 1 2; do ssh -l root zabbix$i "chkconfig -d mysql; rcmysql stop; crontab -r -u zabbix"; done

Make sure that MySQL port is open on the database system. If you are using Zabbix appliance for that as well, execute on your workstation:

 $ ssh -l root zabbixdb "SuSEfirewall2 open EXT TCP mysql; SuSEfirewall2 stop; SuSEfirewall2 start"

Zabbix frontend

Modify these values in /usr/share/zabbix/conf/zabbix.conf.php on both failover nodes. Make sure to use the password you configured for Zabbix user before:

$DB["SERVER"]        = 'zabbixdb';
$DB["DATABASE"]      = 'zabbix_failover';
$DB["USER"]          = 'zabbix';
$DB["PASSWORD"]      = 'someslightlybetterpasswordshouldbeusedhere';
$ZBX_SERVER_NAME     = 'Zabbix 1.8 Failover';

Optionally, you may also change the address that Zabbix frontend will try to find Zabbix server on. While in our scenario failover frontend will always have Zabbix server running on the local system, you might want to be able to access both frontends individually on their corresponding IP addresses, in which case they should connect to the failover Zabbix server always.

$ZBX_SERVER          = 'failover';

That's it, frontend is ready for the failover.

Zabbix server

Edit /etc/zabbix/zabbix_server.conf on both failover nodes and modify these values. Again, use the password you configured for Zabbix user before:

DBHost=zabbixdb
DBName=zabbix_failover
DBUser=zabbix
DBPassword=someslightlybetterpasswordshouldbeusedhere

Zabbix agent

Edit /etc/zabbix/zabbix_agentd.conf on both failover nodes and modify Server parameter to be:

Server=zabbix,zabbix1,zabbix2

Either replace these hostnames with the IP addresses you use, or configure them in /etc/hosts for both failover nodes.

We are using failover system for active checks and allowing incoming connections from failover IP and both node individual addresses as well to make debugging easier later (for example, using zabbix_get will work from either node).

On zabbix1, set Hostname parameter to be:

Hostname=zabbix1

On zabbix2, set Hostname parameter to be:

Hostname=zabbix2

Restart Zabbix agent on both failover nodes. For that, you may execute from your workstation:

$ for i in 1 2; do ssh -l root zabbix$i "rczabbix_agentd restart"; done

Firewall settings

We should also allow incoming connections to Zabbix server and agent ports - both for passive and active items to work. On your workstation, execute:

$ for i in 1 2; do ssh -l root zabbix$i "SuSEfirewall2 open EXT TCP zabbix-agent zabbix-trapper; SuSEfirewall2 stop; SuSEfirewall2 start"; done

OpenSUSE already has Zabbix entries in /etc/services so we can use them instead of numeric ports.

Monitoring some systems

Completely optional - but let's also set some monitoring of systems. As a minimum, we could monitor both failover nodes and the database system. That way we will be able to see that whichever nodes are up are being monitored, as well as database being constantly monitored.

Access any one of the frontends (as they both connect to the same db currently, it does not matter which one you use).

Monitoring both nodes

Let's start by monitoring both failover nodes. Go to Configuration -> Hosts and edit the default Zabbix server. Change it to be Zabbix server 1. Update its IP address, or use host name, if you entered these names in /etc/hosts files for both nodes. If using host name, don't forget to select DNS name in the Connect to dropdown.

Zserver1 properties.png

As this server isn't running MySQL anymore, mark checkbox next to Template_App_MySQL entry in the Linked templates block, then click Unlink and clear.

Zserver templates.png

When done, click Save. Now let's monitor the other node as well - click on Zabbix server 1, then Clone and modify the values for the second node. When done, save that one as well.

Monitoring database server

Do the same for the database system as well - if it's running on Zabbix appliance, it should provide data for all the same items just fine. You should unlink Template_Zabbix_Server from this host, and most likely Template_App_Apache as well. When doing so, make sure to use Unlink and clear button.

Set proper Server and Hostname parameters in /etc/zabbix/zabbix_agentd.conf, then restart Zabbix agent.

Failover setup

For failover, we will use Pacemaker and OpenAIS/Corosync.

Preparing the failover basics

To install all required software packages, execute on your workstation:

$ for i in 1 2; do ssh -l root zabbix$i "zypper --non-interactive in pacemaker-mgmt crmsh"; done
Note: On older appliance/OpenSUSE versions package crmsh is not available/needed.

Now we should set up Corosync configuration.

$ for i in 1 2; do ssh -l root zabbix$i "cp /etc/corosync/corosync.conf.example /etc/corosync/corosync.conf"; done

Edit /etc/corosync/corosync.conf on both nodes and set bindnetaddr parameter to your network base address. So if your IP address is 10.2.3.4 and netmask is 255.0.0.0, bindnetaddr should be set to 10.0.0.0. If the IP address is 192.168.5.13 and netmask is 255.255.255.0, bindnetaddr should be 192.168.5.0. See OpenAIS documentation for more information.

 bindnetaddr:    <network base address>

Now, on the first failover node run:

 # corosync-keygen

This command will generate a key. For that, additional entropy is used, and on a remote system the easiest way to generate that will be to run find / - you may do that in another SSH session. According to Wikipedia entry,

'The Linux kernel generates entropy from keyboard timings, mouse movements, and IDE timings and makes the random character data available to other operating system processes through the special files /dev/random and /dev/urandom.'

Once they key is generated, copy it from the first failover system to the second one by executing on the first system:

# scp /etc/corosync/authkey zabbix2:/etc/corosync

Then secure the key - you may execute from your workstation:

$ ssh -l root zabbix2 "chown root:root /etc/corosync/authkey; chmod 400 /etc/corosync/authkey"

Then modify /etc/corosync/corosync.conf on both nodes. On zabbix1, set nodeid to be:

nodeid: 1

On zabbix2, set nodeid to be:

nodeid: 2

Before continuing, we should allow communication for Corosync in the firewall. For that, execute on your workstation:

$ for i in 1 2; do ssh -l root zabbix$i "SuSEfirewall2 open EXT UDP 5405; SuSEfirewall2 stop; SuSEfirewall2 start"; done

Now we are ready to start the software that will ensure high availability. On your workstation, execute:

$ for i in 1 2; do ssh -l root zabbix$i "chkconfig -a openais; rcopenais start"; done

With OpenAIS/Corosync daemons running, we can check whether both nodes can communicate properly. On either of them, execute:

 # crm configure show

The output should look like this:

node zabbix1
node zabbix2
property $id="cib-bootstrap-options" \
        dc-version="1.1.2-8b9ec9ccc5060457ac761dce1de719af86895b10" \
        cluster-infrastructure="openais" \
        expected-quorum-votes="2"

Most importantly, you should see two nodes. If that is not the case, revisit previous steps and ensure they are all completed successfully.

Setting up actual failover

Now is the time for real failover setup to happen.

On either node, run:

 # crm configure

In this prompt, execute the following commands (make sure to replace failover_IP_address with actual IP you want to use for the failover):

Note: If you accidentally run the commands without setting correct IP address, correct it and run crm resource cleanup failover-ip.
crm(live)configure# primitive failover-ip ocf:heartbeat:IPaddr params ip=<failover_IP_address> op monitor interval=2s
crm(live)configure# primitive failover-zabbix_server lsb::zabbix_server op monitor interval=5s
crm(live)configure# group zabbix_server-cluster failover-ip failover-zabbix_server
crm(live)configure# property stonith-enabled=false
crm(live)configure# property default-resource-stickiness="100"
crm(live)configure# verify
crm(live)configure# exit
There are changes pending. Do you want to commit them? y
bye


Let's look at what that all means.

  1. First we set the IP address that should be used for the failover. We are monitoring for the availability of it every 2 seconds.
  2. Second we monitor whether an LSB service zabbbix_server is running. We check that every 5 seconds.
  3. Then we group them in a group called zabbix_server-cluster - this ensures they move from one node to another together, which is what we want - otherwise IP can belong to one system, but Zabbix server service will run on another.
  4. We disable STONITH (cutely named "Shoot The Other Node In The Head") functionality, because otherwise crm will complain a lot.
  5. We also set default stickiness to 100. A service can be made to prefer some nodes over others, so if it finds a more preferred node alive, it will move to that node. We would want the Zabbix server service to move around as little as possible, so after failover and initial node coming back we want it to keep on running on whichever node it happens to be running. As we have two nodes only, any value over 0 will do. This parameter accepts either integer, or string "infinity" - if default stickiness is set to infinity, service will never migrate from one node to another, unless a node dies (which also prevents forced moving from working). See Pacemaker documentation for more information.

Once we have this setup configured, let's verify whether it has been saved properly:

# crm configure show
node zabbix1
node zabbix2
primitive failover-ip ocf:heartbeat:IPaddr \
        params ip="<failover_IP_address>" \
        op monitor interval="2s"
primitive failover-zabbix_server lsb:zabbix_server \
        op monitor interval="5s"
group zabbix_server-cluster failover-ip failover-zabbix_server
property $id="cib-bootstrap-options" \
        dc-version="1.1.2-8b9ec9ccc5060457ac761dce1de719af86895b10" \
        cluster-infrastructure="openais" \
        expected-quorum-votes="2" \
        stonith-enabled="false" \
        default-resource-stickiness="100"

That seems to be so. But how do we make it all work now? Let's try to find the status of our new-built cluster:

Cluster started working by bringing Zabbix server up on node 2
# crm_mon --one-shot
============
Last updated: Thu Jan  6 16:57:27 2011
Stack: openais
Current DC: zabbix1 - partition with quorum
Version: 1.1.2-8b9ec9ccc5060457ac761dce1de719af86895b10
2 Nodes configured, 2 expected votes
1 Resources configured.
============

Online: [ zabbix2 zabbix1 ]

 Resource Group: zabbix_server-cluster
     failover-ip        (ocf::heartbeat:IPaddr):        Started zabbix2
     failover-zabbix_server     (lsb:zabbix_server):    Started zabbix2

Wonderful! It looks like it is already working, and both Zabbix server and virtual IP are started on zabbix2 node in this case. You can doublecheck that by checking whether zabbix_server process is running on the second node and whether the failover IP is assigned to any interface in the output of ifconfig (usually on eth0:0 interface).

Running crm_mon without --one-shot parameter will continuously show you cluster status.

Testing failover

With the cluster seemingly configured successfully, let's test whether it works. Shutdown the node that currently is running our services (zabbix2 in this case, but verify that in crm_mon output). On the active node, run:

 # shutdown -h now

Once the node has shut down, let's see what's happening - on the live node, execute:

# crm_mon --one-shot
============
Last updated: Thu Jan  6 17:13:05 2011
Stack: openais
Current DC: zabbix1 - partition WITHOUT quorum
Version: 1.1.2-8b9ec9ccc5060457ac761dce1de719af86895b10
2 Nodes configured, 2 expected votes
1 Resources configured.
============

Online: [ zabbix1 ]
OFFLINE: [ zabbix2 ]

Great, it sees that zabbix1 is online, zabbix2 offline... but wait, we don't see out resources running anywhere. And the text "partition WITHOUT quorum" seems suspicious. Right - by default, at least two active nodes are needed to have quorum and agree which node will be running services. That makes our two node cluster useless, though. Actual formula used to calculate quorum is "total_nodes - 1 < 2 * active_nodes". Again, see Pacemaker documentation for more information.

Luckily, this is something we can change. For that, on the live node execute:

 # crm configure property no-quorum-policy="ignore"

This tells the cluster to ignore quorum, and if a node sees other node missing, it takes over - which is what we want to achieve. Now we can check whether that helped any:

# crm_mon --one-shot
============
Last updated: Thu Jan  6 17:17:52 2011
Stack: openais
Current DC: zabbix1 - partition WITHOUT quorum
Version: 1.1.2-8b9ec9ccc5060457ac761dce1de719af86895b10
2 Nodes configured, 2 expected votes
1 Resources configured.
============

Online: [ zabbix1 ]
OFFLINE: [ zabbix2 ]

 Resource Group: zabbix_server-cluster
     failover-ip        (ocf::heartbeat:IPaddr):        Started zabbix1
     failover-zabbix_server     (lsb:zabbix_server):    Started zabbix1

While the other node is still down and there is still no quorum, both virtual IP and Zabbix server are running on the remaining node. You can again verify that by looking for zabbix_server process and IP address in the output of ifconfig.

Successful failover

Let's try to test the failover process once more. Start up the node that we previously shut down. Once it has started, check the cluster status again:

# crm_mon --one-shot
============
Last updated: Thu Jan  6 17:21:22 2011
Stack: openais
Current DC: zabbix1 - partition with quorum
Version: 1.1.2-8b9ec9ccc5060457ac761dce1de719af86895b10
2 Nodes configured, 2 expected votes
1 Resources configured.
============

Online: [ zabbix2 zabbix1 ]

 Resource Group: zabbix_server-cluster
     failover-ip        (ocf::heartbeat:IPaddr):        Started zabbix1
     failover-zabbix_server     (lsb:zabbix_server):    Started zabbix1

Both nodes are up now, they have (not needed by now) quorum, but Zabbix server and virtual IP still reside on zabbix1. The reason is the default stickyness that we set at 100 - services will not migrate from node to node unless absolutely necessary. Let's make it necessary...

On the node that currently has our services running, execute:

 # shutdown -h now

Wait for a short while and check cluster status on the remaining node:

# crm_mon --one-shot
============
Last updated: Thu Jan  6 17:23:13 2011
Stack: openais
Current DC: zabbix2 - partition WITHOUT quorum
Version: 1.1.2-8b9ec9ccc5060457ac761dce1de719af86895b10
2 Nodes configured, 2 expected votes
1 Resources configured.
============

Online: [ zabbix2 ]
OFFLINE: [ zabbix1 ]

 Resource Group: zabbix_server-cluster
     failover-ip        (ocf::heartbeat:IPaddr):        Started zabbix2
     failover-zabbix_server     (lsb:zabbix_server):    Started zabbix2

Wonderful, failover happens as expected, and both IP and Zabbix server process migrated to the other node. Of course, Zabbix frontend continues to be accessible to users as well, even though it moves from one node to another. Great thing is that users won't even have to re-login, because the session status stays available in the same database.

Manually modifying node preference

Moving a specific resource

While resource moving from one node to another automatically is a great thing, sometimes you might want to force a resource to run on a specific node - maybe there's a planned maintenance for a node, maybe you want to free up some resources in a node. To force a service move from one node to another, you may use crm_resource -M command.

If our Zabbix service is running on node zabbix2, but we would like to move it on zabbix1, we would execute:

 # crm_resource -M -r zabbix_server-cluster -H zabbix1

Note that we are specifying service group name, not individual Zabbix server or failover IP services.

This command modifies the score for that specific service on that specific node, setting it to "infinity" - and that is higher than the default stickiness of 100. That is an important reason why we used a numeric default stickiness instead of setting it to "infinity". If we make such a change, cluster configuration is modified like this:

# crm configure show
node zabbix1
node zabbix2
primitive failover-ip ocf:heartbeat:IPaddr \
        params ip="<failover_ip>" \
        op monitor interval="2s"
primitive failover-zabbix_server lsb:zabbix_server \
        op monitor interval="5s"
group zabbix_server-cluster failover-ip failover-zabbix_server
location cli-prefer-zabbix_server-cluster zabbix_server-cluster \
        rule $id="cli-prefer-rule-zabbix_server-cluster" inf: #uname eq zabbix1
property $id="cib-bootstrap-options" \
        dc-version="1.1.2-8b9ec9ccc5060457ac761dce1de719af86895b10" \
        cluster-infrastructure="openais" \
        expected-quorum-votes="2" \
        stonith-enabled="false" \
        default-resource-stickiness="100" \
        no-quorum-policy="ignore"

With service running on zabbix1 node now, we could reboot it. If we do so, as expected, virtual IP and Zabbix server migrate to zabbix2. But hey, after zabbix1 comes back up, our service group migrates to zabbix1 again. We usually don't want that, why did it move again? Because of that preference being set to "infinity", of course. So after we have finished with your maintenance, we should unset this preference rule by executing:

 # crm_resource -U -r zabbix_server-cluster

Disabling a specific node

Alternatively, we can make a specific node not used for any services, if we are running several of them. To do that, execute on the node you want to stop hosting any resources:

 # crm_standby --attr-value on
# crm configure show
node zabbix1 \
        attributes standby="on"

(rest of the output skipped)

It is also possible to disable another, specific node by specifying it's name:

 # crm_standby --node=zabbix1 --attr-value on

To allow that node to host resources again, execute:

 # crm_standby --node=zabbix1 -D
 Deleted nodes attribute: id=nodes-zabbix1-standby name=standby

If running this command on that specific node, node name can be omitted.

See Pacemaker documentation for more information.

Monitoring cluster status

There are lots of things one can monitor on a cluster - clustering services, individual failover services, failover IP...

We could monitor something simple - which node is currently running Zabbix server. To do that in a hackish way, we could use crm_mon as a Zabbix external check.

Making the item work

Note: All steps here must be performed on both nodes.

First we should allow the zabbix user to execute crm_mon command. Let's modify /etc/sudoers. Somewhere in the file, add:

 zabbix ALL=(ALL) NOPASSWD: /usr/sbin/crm_mon --one-shot

That's still not enough - our user zabbix does not have a valid shell set, so we should change that as well:

 # chsh -s /bin/bash zabbix

Let's modify /etc/zabbix/zabbix_server.conf and change the ExternalScripts directive:

ExternalScripts=/var/lib/zabbix/externalscripts
Note: The following step should only be performed on the node which is currently running the server

As we changed Zabbix server configuration file, we must restart it:

 # rczabbix_server restart
Note: All further steps must be performed on both nodes again.

Then let's create the directory:

 # mkdir /var/lib/zabbix/externalscripts

In that directory, we create a file called cluster.node:

 # touch cluster.node
 # chmod 700 cluster.node
 # chown zabbix cluster.node

Then, populate the file with the following contents:

#!/bin/bash
crmout=$(sudo /usr/sbin/crm_mon --one-shot | grep $2 | sed 's/.*Started zabbix//')
echo ${crmout:-0}

This script has some pretty limiting assumptions - it checks cluster status for service that is passed as the first and only parameter, and assumes that all nodes are named zabbix<node_number>. Then it just returns the node number that is running our service group, if any. If it can't be found, 0 is returned. It could easily return node name by changing sed expression to s/.*Started //. Note that it also assumes a service only runs on a single node at a time.

Setting up the item

Now let's create the actual Zabbix configuration. In Zabbix frontend (either on any of the nodes explicitly or on the failover interface), navigate to Configuration -> Hosts. This item will be cluster-wide, so let's add a new host which will gather this information. Click on Create Host and enter "Zabbix cluster" in the Name field

Clusterhost.png

When done, click Save. Click on Items for the host we just created and click Create Item. Modify the following fields:

  • Description: enter "Cluster node running Zabbix server"
  • Type: choose "External check"
  • Key: enter cluster.node[failover-zabbix_server]
  • Update interval: change to "60"
  • Keep history: change to "7"
  • New application: enter "Zabbix cluster"

Final item details should look like this:

Clusternode item.png

When done, click Save.

Visiting Monitoring -> Latest data, we should be able to see this item soon enough and it should show us which node is currently running Zabbix server.

Setting up triggers

Now we may create a triggers that would check this item. Just above the item list there's a navigation bar - click on Triggers in it.

Hostnavbar.png

In the trigger list, click on Create Trigger.

So what trigger could we create? Maybe one that checks whether Zabbix service is running? That should be easy - we should just check whether the current node is 0, right? But that wouldn't be very useful, as there would be no Zabbix server to record this status... Of course, this would be a useful check if you have any other service clustered. For Zabbix, though, we could monitor whether it has failed over from one node to another.

Make these changes:

  • Name: enter "Zabbix failover happened"
  • Expression: enter {Zabbix cluster:cluster.node[failover-zabbix_server].diff()(0)}=1
  • Severity: choose Average

Final result should look like this:

Clusternode trigger.png

If so, click Save.

So in the end the result of the item should display which node Zabbix server is currently running on:

Current node.png

And if failover would happen, a trigger would activate (that we could also send a message upon) as visible in Monitoring -> Triggers:

Failover happened.png

We would get a frontend message as well:

Frontend notification about failover.png

It is also possible to get a visual overview of how the service was transferring between nodes by looking at the graph - for example, here quite a lot of node transfers have happened during the last hour:

Current node graph.png

Of course, this is a very primitive monitoring just to illustrate the possibilities. There are lots of ways to improve it further - for example, we could denote in the trigger name the node to which migration happened by changing trigger name to Zabbix failover happened to node {ITEM.LASTVALUE}.

Conclusion

It works.

Note: When Zabbix server starts up, it schedules items to be checked at any time starting from now till whatever is the item interval. This is done in a deterministic fashion - if an item with update interval of half an hour is scheduled to be checked 15 minutes from now, restarting a server after a minute from now will reschedule that item at the exact same time. The benefit for us in the failover setup is that this will ensure items being checked at the very same time even after Zabbix server is stopped on one system and started on another.

Further steps

With everything working, there's of course much more that could still be done.

Using only nodes that can reach router

There are many considerations that you may make Pacemaker take into consideration when deciding where to run some service. That includes load on nodes and availability of other resources. One might be running Zabbix server only on node that can reach central router (instructions from Pacemaker example documentation).

Note: This example is not verified.
crm configure primitive pingd ocf:pacemaker:pingd \ 
                      params host_list=<router_ip> multiplier=100 \
                      op monitor interval=15s timeout=5s
crm configure clone pingdclone pingd meta globally-unique=false

This tells the cluster to only run the group on a node with a working network connection to the default gateway.

crm(pingd)configure# location my_web_cluster_on_connected_node my_web_cluster \
 rule -inf: not_defined pingd or pingd lte 0

Using GUI client

If using more of the cluster functionality, one might want to try out a GUI client as well. On OpenSUSE, it is available in the package pacemaker-mgmt-client .

LCMC offers a novel visualisation of the cluster infrastructure.

Shoot The Other Node In The Head

STONITH (Shoot The Other Node In The Head) - the feature we disabled - allows to force full removal of a node in a cluster to ensure that it will not pollute the data in any way. This would be useful in our Zabbix cluster, as two servers running at the same time would surely end up with incorrect data being inserted in the database, and possibly even bigger problems. For that, usually an independent device that can cut power to nodes that should be "shot in the head" is used.

Troubleshooting

On some systems, the IP failover may not succeed, and in the logs you may find an error message "Cannot use default route w/o netmask".

This problems seems to be caused by a Pacemaker resource agent bug, and only appears on some systems, as it seems to depend on the order in which routes are reported. You can test this by running in shell:

 OCF_RESKEY_ip=<failover_IP_address> /usr/lib/heartbeat/findif

If this fails with the same error message, you can work around the problem by explicitly setting the netmask in the primitive specification by adding cidr_netmask="24" (or whatever is the correct netmask for your address), for example:

 crm(live)configure# primitive failover-ip ocf:heartbeat:IPaddr params ip=<failover_IP_address> cidr_netmask="24" op monitor interval=2s

This issue is not present on OpenSUSE 11.4, but is present on OpenSUSE 12.1.

Thanks to rasto and andreask on Freenode/#linux-ha for helping with this issue.

Tips

Default editor and pager for crm may be changed by executing:

 crm options editor editor_command
 crm options pager pager_command