Zabbix trigger math

I have a number of more complex or compound triggers in my Zabbix environment.  I regularly create trigger expressions using OR ( | ) to test for multiple conditions.  It makes me feel better that I can watch for a particular value in a result as well as whether no data has been received by the Zabbix server and alert accordingly.

I recently had a request to monitor the number of results in a log file.  If this results number dropped by, say, 10 percent since the last check then the trigger should fire an alert.  I wasn’t quite sure how to accomplish this so I visited the Zabbix Triggers page once again.  Usually, I reference the functions section but the expression operators section caught my eye this time.

I knew that I was going to have to do some trigger math to solve my problem but I wasn’t even sure that it could be done.  I questioned the Mighty Zabbix and should have known better.  Mixed in with the well-known OR, AND, less than, and greater than is the ability to use addition, subtraction, multiplication, and division in your trigger expressions.  Exactly what I needed.

The trigger ended up being this simple.  We all learned this type of arithmetic in grade school.

({Host_1:item_A.last(0)}/{Host_1:item_A.prev(0)})<0.90

This trigger basically says, take the current result (last) divided by the last result (prev) and if it is less than .90 (a 10%+ change since the last result) the trigger is true.

This is exactly what I was looking for.  I added an OR that checked for no data over the last 10 minutes for additional monitoring and I was all set.

| {Host_1:item_A.nodata(600)}=1

This trigger math will come in handy in the future for sure.  I may even look through some of my triggers to see which ones may be improved by different logic.

Give this trigger arithmetic a try when you have a chance.  I’m sure you’ll find it as helpful as I did.

 


Get Splunk data into Zabbix

Splunk is a very powerful tool for managing log files.  Zabbix is a very powerful tool for monitoring items and alerting upon conditions.  It is possible to get Splunk data into Zabbix for sending alerts when a condition is met.

Here’s how I did it.  This assumes you have a working Splunk service and a working Zabbix service.  Details of configuring these two services is available elsewhere.

To summarize, I use the splunk command line client to run the desired search against my main Splunk server.  That data is parsed and sent to Zabbix using zabbix_sender.  Once the data is in Zabbix, alerts are configured using items, triggers, etc as any other Zabbix item.

Splunk is installed on my Zabbix server and configured so the forward-server points at my main Splunk server.

Test that you can successfully retrieve data from your Splunk server

/opt/splunk/bin/splunk search "<searchname>" -auth <username>:<password> "-uri https://<splunkserver>:8089"
  •  <searchname> is whatever you are searching for.  This is the same as what you would type into the search field in the Splunk GUI.
  • <username>:<password> is the credentials used to access your main Splunk server
  • <splunkserver> – IP or hostname of your Splunk server

Once you successfully retrieve data from the Splunk server using the command line Splunk client, parse your data into a format that can easily be input to zabbix_sender.

 sed 's/ /n/g;s/=/ /g;s/"//g' | sed 's/^/- /g'

Using sed, I first replaced the space delimiter with a new line, removed the “=” and then removed the double quotes around the result.  Finally, I prepended a ‘-‘ to each line.  This is used by zabbix_sender to represent the hostname field.  This may or may not work for you exactly; you are ultimately looking to pass these results via standard input to zabbix_sender in the following format:

<hostname> <key> <value>

<hostname> will be provided by ‘-‘ in our results which will use the -s argument to zabbix_sender.

This Zabbix_sender example shows how arguments are passed:

zabbix_sender -c <zabbix_agentd.conf> -s <host as registered in Zabbix> -i -

Put it all together and you get something like this:

/opt/splunk/bin/splunk search "sourcetype=foobar earliest=-5m" -auth admin:changeme -uri https://splunk.company.net:8089 | sed 's/ /n/g;s/=/ /g;s/"//g' | sed 's/^/- /g' | /usr/local/bin/zabbix_sender -vv -c /etc/zabbix/zabbix_agentd.conf -s splunkHost -i -

Remove the -vv from zabbix_sender once you test it out.  I have cronned this to run every 10 minutes and created items in Zabbix for each key that is returned by Splunk.

Create your items on the Zabbix server with the ‘Zabbix Trapper’ type.  Once the items in Zabbix are receiving data from your cronned task, you can create triggers, graphs, etc. Just like any other items in Zabbix.

Let me know if this works for you or if you have done the same thing in a different way.


Zabbix – modify last value column width

A minor annoyance of Zabbix is that the Last Value column is limited to 20 characters wide in the dashboard.

You see this a lot when you are capturing interface names from network devices because they tend to be quite long.

Here’s a shot of the default column width and some interface names.  Hard to read aren’t they?

zab.column.width.before

last value column width – before

Modifying the width of this column has to be done in the source of the dashboard.  This sounds more difficult that it is.  In fact, you only have to modify 2 lines of code and save the changes.  No service restarts, no recompiling.

From your Zabbix server, use your favorite editor to open /var/www/html/zabbix/include/items.inc.php.  Change the path to point to wherever your frontend files are if they are not in the same place as mine.

If you are running version 1.8.12, you should jump down to lines 1410 and 1411 in the above file and change the 20 to a 50.  If you are running version 2.0.0, the values to change are on lines 898 and 899.  Save the file after you make your changes and then refresh your dashboard screen to see the change.

Here’s what the lines look like after I made my change from 20 to 50.

                 if (zbx_strlen($lastvalue) > 50) {
                         $lastvalue = zbx_substr($lastvalue, 0, 50).' ...';

You could go higher than 50 but, to me, that seems to be a good balance of showing enough of the value without taking up the entire screen.

Here is the result of the change

zab.column.width.after

There it is.  Let me know if this helped you out.


Zabbix 2.0 low level discovery – network interfaces

I recently posted how I set up the low level discovery for file systems.  Today I’ll show you how to set up network interface discovery and soon SNMP.

Zabbix 2.0 makes it easy to add network devices, file systems, and SNMP devices for monitoring.  They call their new method low-level discovery and I will show you how I set it up so you can quickly start monitoring your environment.

The new discovery rules are created in the Zabbix UI under Configuration, Templates, Discovery.  I have created network interface discovery rules in the Windows template and the Linux template.

Network Interface Discovery Rules

There is no difference in how I set up this discovery rule between Windows and Linux.  I simply made the same rules under each template.  Here are the details for setting up the low level network interface discovery for the Windows and Linux templates.

Name: Windows Network Discovery
Type: Zabbix agent (active)
Key: net.if.discovery
Update interval: 7200
Keep lost resource period: 0
Filter macro: {#IFNAME}
Regexp: @Network-discovery

After reading some comments on the Zabbix forums, it looks like using the global regular expressions helps a lot with the Windows network interface discovery.  Without the regexp you will get a lot of results on your typical Windows box.  Linux has a similar issue but usually the only ‘extra’ interface that is found is the LO, so it’s not as big of a deal.  Either way, this regular expression works for both Windows and Linux.

Set your global regular expressions under Administration, General, Regular Expressions.  I’ve created mine to not match to loopback interfaces and other generic interfaces.  Here’s a screenshot showing my config.

win-ldd-net-regexp

Network Interface Prototypes

Once the discovery entry is made, you should create item prototypes for the items you are interested in.  The prototype will be applied to each of the discovered network interfaces.

I created 2 prototype items for the low level network interface discovery for both Windows and Linux.

  • Incoming Traffic
  • Outgoing Traffic

Each item prototype is pretty much the same, just a change for the name and key.

Incoming traffic

Name: Incoming - $1
Type:Zabbix agent (active)
Key: net.if.in[{#IFNAME}]
Type: Numeric (unsigned)
Units: bps
Use custom multiplier: 8
Update interval: 60
Keep history: 7
Keep trends: 365
Store value: Delta (speed per second)
Applications: Network

outgoing traffic

Name: Outgoing - $1
Type:Zabbix agent (active)
Key: net.if.out[{#IFNAME}]
Type: Numeric (unsigned)
Units: bps
Use custom multiplier: 8
Update interval: 60
Keep history: 7
Keep trends: 365
Store value: Delta (speed per second)
Applications: Network

I didn’t set up any trigger prototypes for the network interfaces but I did create a graph prototype.

This shows a graph prototype for Windows, but just like the items the graph prototype for Linux is exactly the same.

lld-net_graph_discovery

That’s about it.  The addition of low-level discovery in Zabbix is fantastic and I’m sure you will immediately see how helpful it can be once you get it set up in Zabbix 2.0.

Let me know what you think about this new feature.  I’ll post information about SNMP discovery shortly.