Get Splunk data into ZabbixPosted: June 15, 2012
Splunk is a very powerful tool for managing log files. Zabbix is a very powerful tool for monitoring items and alerting upon conditions. It is possible to get Splunk data into Zabbix for sending alerts when a condition is met.
Here’s how I did it. This assumes you have a working Splunk service and a working Zabbix service. Details of configuring these two services is available elsewhere.
To summarize, I use the splunk command line client to run the desired search against my main Splunk server. That data is parsed and sent to Zabbix using zabbix_sender. Once the data is in Zabbix, alerts are configured using items, triggers, etc as any other Zabbix item.
Splunk is installed on my Zabbix server and configured so the forward-server points at my main Splunk server.
Test that you can successfully retrieve data from your Splunk server
/opt/splunk/bin/splunk search "<searchname>" -auth <username>:<password> "-uri https://<splunkserver>:8089"
- <searchname> is whatever you are searching for. This is the same as what you would type into the search field in the Splunk GUI.
- <username>:<password> is the credentials used to access your main Splunk server
- <splunkserver> – IP or hostname of your Splunk server
Once you successfully retrieve data from the Splunk server using the command line Splunk client, parse your data into a format that can easily be input to zabbix_sender.
sed 's/ /n/g;s/=/ /g;s/"//g' | sed 's/^/- /g'
Using sed, I first replaced the space delimiter with a new line, removed the “=” and then removed the double quotes around the result. Finally, I prepended a ‘-‘ to each line. This is used by zabbix_sender to represent the hostname field. This may or may not work for you exactly; you are ultimately looking to pass these results via standard input to zabbix_sender in the following format:
<hostname> <key> <value>
<hostname> will be provided by ‘-‘ in our results which will use the -s argument to zabbix_sender.
This Zabbix_sender example shows how arguments are passed:
zabbix_sender -c <zabbix_agentd.conf> -s <host as registered in Zabbix> -i -
Put it all together and you get something like this:
/opt/splunk/bin/splunk search "sourcetype=foobar earliest=-5m" -auth admin:changeme -uri https://splunk.company.net:8089 | sed 's/ /n/g;s/=/ /g;s/"//g' | sed 's/^/- /g' | /usr/local/bin/zabbix_sender -vv -c /etc/zabbix/zabbix_agentd.conf -s splunkHost -i -
Remove the -vv from zabbix_sender once you test it out. I have cronned this to run every 10 minutes and created items in Zabbix for each key that is returned by Splunk.
Create your items on the Zabbix server with the ‘Zabbix Trapper’ type. Once the items in Zabbix are receiving data from your cronned task, you can create triggers, graphs, etc. Just like any other items in Zabbix.
Let me know if this works for you or if you have done the same thing in a different way.