[zabbix] out of memory 로 인한 무한 시작반복 에러

증상

• zabbix server 시작하면서 child process 하나가 죽음에 따라 zabbix server 가 시작하지 못하고 이를 반복하는 현상.

 

zabbix server log

3960401:20221221:102110.876 Starting Zabbix Server. Zabbix 5.4.7 (revision 84dc2ec5dc).
3960401:20221221:102110.877 ****** Enabled features ******
3960401:20221221:102110.877 SNMP monitoring:           YES
3960401:20221221:102110.877 IPMI monitoring:           YES
3960401:20221221:102110.877 Web monitoring:            YES
3960401:20221221:102110.877 VMware monitoring:         YES
3960401:20221221:102110.877 SMTP authentication:       YES
3960401:20221221:102110.877 ODBC:                      YES
3960401:20221221:102110.877 SSH support:               YES
3960401:20221221:102110.877 IPv6 support:              YES
3960401:20221221:102110.877 TLS support:               YES
3960401:20221221:102110.877 ******************************
3960401:20221221:102110.877 using configuration file: /etc/zabbix/zabbix_server.conf
3960401:20221221:102110.879 current database version (mandatory/optional): 05040000/05040000
3960401:20221221:102110.879 required mandatory version: 05040000
3960401:20221221:102110.882 server #0 started [main process]
3960402:20221221:102110.883 server #1 started [configuration syncer #1]
3960402:20221221:102110.996 __mem_malloc: skipped 0 asked 72 skip_min 18446744073709551615 skip_max 0
3960402:20221221:102110.996 [file:dbconfig.c,line:102] __zbx_mem_malloc(): out of memory (requested 72 bytes)
3960402:20221221:102110.996 [file:dbconfig.c,line:102] __zbx_mem_malloc(): please increase CacheSize configuration parameter
3960402:20221221:102110.996 === memory statistics for configuration cache ===
3960402:20221221:102110.996 free chunks of size     24 bytes:       74
3960402:20221221:102110.996 free chunks of size     32 bytes:       16
3960402:20221221:102110.996 free chunks of size     40 bytes:        3
3960402:20221221:102110.996 free chunks of size     48 bytes:        5
3960402:20221221:102110.996 free chunks of size     56 bytes:        5
3960402:20221221:102110.996 min chunk size:         24 bytes
3960402:20221221:102110.996 max chunk size:         56 bytes
3960402:20221221:102110.996 memory of total size 7245880 bytes fragmented into 71398 chunks
3960402:20221221:102110.996 of those,       2928 bytes are in      103 free chunks
3960402:20221221:102110.996 of those,    7242952 bytes are in    71295 used chunks
3960402:20221221:102110.996 of those,    1142352 bytes are used by allocation overhead
3960402:20221221:102110.996 ================================
3960402:20221221:102110.996 === Backtrace: ===
3960402:20221221:102110.997 13: /usr/sbin/zabbix_server: configuration syncer [syncing configuration](zbx_backtrace+0x3f) [0x55ca984db5ec]
3960402:20221221:102110.997 12: /usr/sbin/zabbix_server: configuration syncer [syncing configuration](__zbx_mem_malloc+0x180) [0x55ca984d6281]
3960402:20221221:102110.997 11: /usr/sbin/zabbix_server: configuration syncer [syncing configuration](+0x1a0671) [0x55ca98461671]
3960402:20221221:102110.997 10: /usr/sbin/zabbix_server: configuration syncer [syncing configuration](zbx_hashset_insert_ext+0x12c) [0x55ca984e1658]
3960402:20221221:102110.997 9: /usr/sbin/zabbix_server: configuration syncer [syncing configuration](zbx_hashset_insert+0x2d) [0x55ca984e152a]
3960402:20221221:102110.997 8: /usr/sbin/zabbix_server: configuration syncer [syncing configuration](+0x1ad335) [0x55ca9846e335]
3960402:20221221:102110.997 7: /usr/sbin/zabbix_server: configuration syncer [syncing configuration](DCsync_configuration+0xc37) [0x55ca9846fe80]
3960402:20221221:102110.997 6: /usr/sbin/zabbix_server: configuration syncer [syncing configuration](dbconfig_thread+0x122) [0x55ca98327998]
3960402:20221221:102110.997 5: /usr/sbin/zabbix_server: configuration syncer [syncing configuration](zbx_thread_start+0x37) [0x55ca984eb533]
3960402:20221221:102110.997 4: /usr/sbin/zabbix_server: configuration syncer [syncing configuration](MAIN_ZABBIX_ENTRY+0xb39) [0x55ca98315fcd]
3960402:20221221:102110.997 3: /usr/sbin/zabbix_server: configuration syncer [syncing configuration](daemon_start+0x2f5) [0x55ca984db216]
3960402:20221221:102110.997 2: /usr/sbin/zabbix_server: configuration syncer [syncing configuration](main+0x2f6) [0x55ca983153c0]
3960402:20221221:102110.997 1: /lib64/libc.so.6(__libc_start_main+0xf3) [0x7f0f26a9a493]
3960402:20221221:102110.997 0: /usr/sbin/zabbix_server: configuration syncer [syncing configuration](_start+0x2e) [0x55ca983142be]
3960401:20221221:102110.998 One child process died (PID:3960402,exitcode/signal:1). Exiting ...
3960401:20221221:102110.998 syncing trend data...
3960401:20221221:102110.998 syncing trend data done
3960401:20221221:102110.998 Zabbix Server stopped. Zabbix 5.4.7 (revision 84dc2ec5dc).

 

 

원인

• 추가하는 host 수가 많아 짐에 따라 사용하는 cache memory 가 부족하여 OOM(out of memory) 발생

 

3960402:20221221:102110.996 [file:dbconfig.c,line:102] __zbx_mem_malloc(): out of memory (requested 72 bytes)
3960402:20221221:102110.996 [file:dbconfig.c,line:102] __zbx_mem_malloc(): please increase CacheSize configuration parameter

 

 

 

조치

• zabbix_server.conf의 cachesize를 늘려준다.

CacheSize=1G