Zabbix large host delete or modify timeout

When modifying a host with many items or a template with many linked hosts, I was experiencing what I would call a timeout.  I selected the item or trigger I wished to modify or delete and then selected delete.  The result, after about 10 seconds of waiting, was to return a blank screen.  When I clicked on that host or template that I was modifying, the item I tried to delete was still there.

This became a huge problem when I was dealing with this issue.  I had a run away LLD trigger that continued to duplicate itself on all of my 150+ Windows hosts.  The underlying problem was that a disk space item that I created was using a lower case ‘c’ and the LLD rule to find drives was creating a trigger for an upper case ‘C’.  Fire and brimstone rained and I had 150+ Windows hosts with 120 ‘low disk space on c: drive’ triggers.

The duplication fix was easy, as explained in the above link, but I was still stuck with thousands of extra triggers that existed across my Windows hosts.

My attempt to delete the problem trigger so that it would be removed from all the hosts upon the next discovery period resulted in this ‘timeout’ issue where nothing seemed to happen.

I could ‘unlink and clear’ each host individually from the problem template but this would wipe out any history on that host, not to mention it would have taken forever to do this on 150+ hosts.  I was stuck.

Searches through the Zabbix bug list returned few possibilities.  I was starting to write up my own bug report when I checked a log that I should have checked from the beginning.  The Apache httpd log showed promise:

 PHP Fatal error:  Allowed memory size of 268435456 bytes exhausted

This was it!  Zabbix requires a minimum PHP memory_limit of 128M.  I set mine to 256M because… 256M is bigger than 128M…  I don’t know, seemed like it may be beneficial to increase this when I started.

Well, I guess my environment has grown to a point where 256M isn’t enough.  In fact, I had to up this to 2G before the error went away and a was finally able to delete that trigger.

The change is dynamic for PHP but you will need to restart httpd before Zabbix picks it up.

So, short story, long…  Zabbix was starved for memory to complete this delete task and upping the PHP memory_limit in php.ini fixed the problem.

I have since dropped the memory_limit setting to 512M but I think keeping it at 2G would have been safe.  If PHP didn’t need to use that amount of memory it wouldn’t have horded it, I don’t think.

Let me know if this has happened to you.

Later.


2 Comments on “Zabbix large host delete or modify timeout”

  1. Shekar says:

    Hi,
    Just curious, how did it fixed for you. My memory limit in php.ini is set to 512M. I have a trigger written for switches ( snmp based ) to check the operational status, but i am not able to delete it. I am not even able to delete any items on that template, not able to delete any discovery items ( ifName – snmp ) and item prototypes also. Zabbix simply shows my a blank page. Any pointers really appreciated.

    Thanks

    • Chris says:

      Your memory limit setting totally depends on your system; just keep giving it more memory in php.ini until it finally works. I had to bump my memory limit in php.ini to 2G before I could delete my large template. You can always set it back to something lower once you complete the stuck task.


Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s