See www.zabbix.com for the official Zabbix site.

Docs/specs/ZBXNEXT-3071

From Zabbix.org
Jump to: navigation, search

History cache optimization

ZBXNEXT-3071

Status: v1.2

Owner: Andris Zeila

Summary

Currently history cache works fast when the stored items / values ratio is high. But the lower the ratio is, the more history slots must be iterated to pick the values to process. Normally it is not a problem, but in specific cases when a small subset of items is flooding server with values it can cause the processing to slow down and in worst cases even practically stop.

Specification

History cache will be split into two parts - history cache and history index. Both parts will have their own shared memory segment.

.-----------------------------------------.         .-----------------------------------------------.
|              History Cache              |         |                 History Index                 |
|                                         |         |                                               |
|                                         |         |    history_items (hashset)                    |
|      .-------------------------.        |         |    .-------------------------.                |
|      |      zbx_hc_data_t      |<------------.    |    |      zbx_hc_item_t      |<-----------.   |
|      |-------------------------|        |    |    |    |-------------------------|--.         |   |
|      | ...                     |        |    |    |    | zbx_uint64_t  itemid    |  |<------. |   |
|      |                         |        |    '---------| zbx_hc_data_t *tail     |--|--.    | |   |
|  .---| zbx_hc_data_t   *next;  |        |    .---------| zbx_hc_data_t *head     |  |  |<-. | |   |
|  |   '-------------------------'        |    |    |    '-------------------------'  |--|  | | |   |
|  |                                      |    |    |       | zbx_hc_data_t *head     |  |  | | |   |
|  |                                      |    |    |       '-------------------------'  |  | | |   |
|  |   .-------------------------.        |    |    |          | zbx_hc_data_t *head     |  | | |   |
|  '-->|      zbx_hc_data_t      |        |    |    |          '-------------------------'  | | |   |
|      |-------------------------|        |    |    |                                       | | |   |
|      | ...                     |        |    |    |                                       | | |   |
|      |                         |        |    |    |                                       | | |   |
|  .---| zbx_hc_data_t   *next;  |        |    |    |                                       | | |   |
|  |   '-------------------------'        |    |    |                                       | | |   |
|  v                                      |    |    |      history_queue (binary heap)      | | |   |
| ...     ...     ...     ...             |    |    |      .------------------------.       | | |   |
|                                         |    |    |      | elem1                  |-------' | |   |
|  |   .-------------------------.        |    |    |      | elem2                  |---------' |   |
|  '-->|      zbx_hc_data_t      |<------------'    |      | elem3                  |-----------'   |
|      |-------------------------|        |         |      | ...                    |               |
|      | ...                     |        |         |      '------------------------'               |
|      |                         |        |         |                                               |
|      | zbx_hc_data_t   *next;  |        |         |                                               |
|      '-------------------------'        |         |                                               |
|                                         |         |                                               |
|                                         |         |                                               |
'-----------------------------------------'         '-----------------------------------------------'

History Cache

The item values are stored in history cache as a linked list of zbx_hc_data_t structures for each item. Floating and unsigned values are stored directly in zbx_hc_data_t structure. The string and log values are allocated in history cache and the zbx_hc_data_t structure contains references to the allocated values.

The history cache size is configured with configuration file parameter HistoryCacheSize. The history cache size determines the maximum number of values that can be stored in history cache.

The history cache memory allocator must handle out of memory situations without exiting with low memory error.

History Index

The history index provides access to item values stored in history cache. The history index consists of two parts - items hashset and queue binary heap. Items hashset holds references to the oldest (tail) and newest (head) values for all items in history cache. It's used when adding item values to history cache. Queue determines the processing order of items (it will be chronological order unless item priorities are introduced) and is used when picking items from history cache for processing.

The history index size is configured with configuration file parameter HistoryIndexCacheSize. The history index size depends on the number of items monitored by server/proxy.

History index memory allocator will exit with low memory error in out of memory situation.

Adding values

  • get item by itemid from history items hashset, add new item if none found.
  • allocate zbx_hc_data_t structure and string/log values if required in history cache.
  • add the allocated structure at the head of item value list.
  • add item to history queue if it was just created.

Picking items for processing

  • pop the first N elements from history queue. This prevents item from being processed by other history syncers.
  • lock triggers for picked items, put already locked items back in history queue.
  • process the oldest (at item's value list tail) values of picked items.
  • remove the oldest values.
  • if items still have more values - put them back in queue, otherwise remove them from history items hashset.

Details

A new configuration parameter HistoryIndexSize must be added.

HistoryTextCacheSize configuration parameter must be removed.

Internal items and triggers monitoring history text cache must be removed.

New internal items must be added:

  • zabbix[wcache,index,pfree]
  • zabbix[wcache,index,free]
  • zabbix[wcache,index,used]
  • zabbix[wcache,index,total]

Existing internal items to monitor history cache must be updated.

History cache monitoring must be replaced with history index monitoring in Zabbix server templates. The new item/trigger names will be:

  • Zabbix history index cache, % free
  • Less than 25% free in the history index cache

The access to the new history cache implementation must be made so it could be dropped in with least changes in current code.

Internal API


/******************************************************************************
 *                                                                            *
 * Function: hc_add_item_values                                               *
 *                                                                            *
 * Purpose: adds item values to the history cache                             *
 *                                                                            *
 * Parameters: item_values      - [IN] the item values to add                 *
 *             item_values_num  - [IN] the number of item values to add       *
 *                                                                            *
 * Comments: If the history cache is full this function will wait until       *
 *           history syncers processes values freeing enough space to store   *
 *           the new value.                                                   *
 *                                                                            *
 ******************************************************************************/
static void	hc_add_item_values(dc_item_value_t *item_values, int item_values_num);


/******************************************************************************
 *                                                                            *
 * Function: hc_pop_items                                                     *
 *                                                                            *
 * Purpose: pops the next batch of history values from cache for processing   *
 *                                                                            *
 * Parameters: history_items - [OUT] the locked history items                 *
 *                                                                            *
 * Comments: The history_items must be returned back to history cache with    *
 *           hc_push_history() function after they have been processed.       *
 *                                                                            *
 ******************************************************************************/
static void	hc_pop_items(zbx_vector_ptr_t *history_items);

/******************************************************************************
 *                                                                            *
 * Function: hc_get_item_values                                               *
 *                                                                            *
 * Purpose: gets item history values                                          *
 *                                                                            *
 * Parameters: history       - [OUT] the history valeus                       *
 *             history_items - [IN] the history items                         *
 *                                                                            *
 * Return value: the number of returned history values                        *
 *                                                                            *
 ******************************************************************************/
static int	hc_get_item_values(ZBX_DC_HISTORY *history, zbx_vector_ptr_t *history_items);

/******************************************************************************
 *                                                                            *
 * Function: hc_push_items                                                    *
 *                                                                            *
 * Purpose: push back the locked history items into history cache             *
 *                                                                            *
 * Parameters: history_items - [IN] the locked history items                  *
 *                                                                            *
 * Return value: timestamp of the next history item to sync                   *
 *                                                                            *
 * Comments: This function removes processed history data from history cache. *
 *           If there are no more data for this item, the the item itself is  *
 *           removed from history index.                                      *
 *                                                                            *
 ******************************************************************************/
static int	hc_push_items(zbx_vector_ptr_t *history_items);

/******************************************************************************
 *                                                                            *
 * Function: hc_update_history_queue                                          *
 *                                                                            *
 * Purpose: updates history queue by queuing missing history items            *
 *                                                                            *
 * Comments: This function is called before full history sync to ensure that  *
 *           all history items are queued and will be synced.                 *
 *                                                                            *
 ******************************************************************************/
static void	hc_update_history_queue();


hc_add_item_values() function will be used in dc_flush_history() to flush locally cached values to history cache. Basically it will replace current dc_flush_history() implementation.

hc_pop_items() function will be used in DCsync_history() to take the items from history index for processing.

hc_get_item_values() function will be used in DCsync_history() to populate ZBX_DC_HISTORY array with the history data of unlocked items returned from hc_pop_items() after locked (by triggers) items have been marked.

hc_push_items() function will be used in DCsync_history() to put back the items to history index.

hc_update_history_queue() function will be used in DCsync_history() before full sync to ensure all items are returned in history queue.

Documentation

  • What's new
  • Upgrade notes
  • Internal items
  • Server/proxy configuration

ChangeLog

  • v1.2 Updated item picking logic
  • v1.1 Added internal item and trigger names
  • v1.0 Initial release