Get Splunk data into Zabbix

Splunk is a very powerful tool for managing log files.  Zabbix is a very powerful tool for monitoring items and alerting upon conditions.  It is possible to get Splunk data into Zabbix for sending alerts when a condition is met.

Here’s how I did it.  This assumes you have a working Splunk service and a working Zabbix service.  Details of configuring these two services is available elsewhere.

To summarize, I use the splunk command line client to run the desired search against my main Splunk server.  That data is parsed and sent to Zabbix using zabbix_sender.  Once the data is in Zabbix, alerts are configured using items, triggers, etc as any other Zabbix item.

Splunk is installed on my Zabbix server and configured so the forward-server points at my main Splunk server.

Test that you can successfully retrieve data from your Splunk server

/opt/splunk/bin/splunk search "<searchname>" -auth <username>:<password> "-uri https://<splunkserver>:8089"
  •  <searchname> is whatever you are searching for.  This is the same as what you would type into the search field in the Splunk GUI.
  • <username>:<password> is the credentials used to access your main Splunk server
  • <splunkserver> – IP or hostname of your Splunk server

Once you successfully retrieve data from the Splunk server using the command line Splunk client, parse your data into a format that can easily be input to zabbix_sender.

 sed 's/ /n/g;s/=/ /g;s/"//g' | sed 's/^/- /g'

Using sed, I first replaced the space delimiter with a new line, removed the “=” and then removed the double quotes around the result.  Finally, I prepended a ‘-‘ to each line.  This is used by zabbix_sender to represent the hostname field.  This may or may not work for you exactly; you are ultimately looking to pass these results via standard input to zabbix_sender in the following format:

<hostname> <key> <value>

<hostname> will be provided by ‘-‘ in our results which will use the -s argument to zabbix_sender.

This Zabbix_sender example shows how arguments are passed:

zabbix_sender -c <zabbix_agentd.conf> -s <host as registered in Zabbix> -i -

Put it all together and you get something like this:

/opt/splunk/bin/splunk search "sourcetype=foobar earliest=-5m" -auth admin:changeme -uri https://splunk.company.net:8089 | sed 's/ /n/g;s/=/ /g;s/"//g' | sed 's/^/- /g' | /usr/local/bin/zabbix_sender -vv -c /etc/zabbix/zabbix_agentd.conf -s splunkHost -i -

Remove the -vv from zabbix_sender once you test it out.  I have cronned this to run every 10 minutes and created items in Zabbix for each key that is returned by Splunk.

Create your items on the Zabbix server with the ‘Zabbix Trapper’ type.  Once the items in Zabbix are receiving data from your cronned task, you can create triggers, graphs, etc. Just like any other items in Zabbix.

Let me know if this works for you or if you have done the same thing in a different way.

Advertisements

2 Comments on “Get Splunk data into Zabbix”

  1. kaba says:

    hi good article
    but just want to know where we use zabbix_sender? on the server zabbix or hote remote
    because en confuge. i just begin to use zabbix so i want to monitor the data base of mysql of the host remote without install zabbix-agent on my host remote. some body tell me to use zabbix_sender but i don’t know how to use zabbix_sender to do it . there is not alof article who talk about it. so i need your help 🙂 if you have an idea about it

    sorry i can’t speak english very good 🙂
    regard


Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google+ photo

You are commenting using your Google+ account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

w

Connecting to %s