Using AWS SES for Zabbix email notifications

I’ve been using Amazon Simple Email Service as the SMTP server for my home lab notifications.  I don’t have to worry about setting up my own SMTP server and there is a minimal cost to this configuration.  My SES account is set for ‘sandbox’ access where there is a limit to daily email of 200.  I sent an email to Amazon Support and they upped the quota to 1000.  The pricing schedule says that it’s $.10 per 1000 email per day.  I send about 20 per month, so I see a $.01 charge every couple of months.  Good deal to me.

To get this going you’ll have to set up your SES account on AWS.  I won’t go into detail, but would do a post if someone wants more detail.

Once your account is set up, you’ll need to note the SMTP credentials from SES homepage, then under Email Sending you’ll click on SMTP Settings.

You’ll need the following to set up a media type in Zabbix:

  • SMTP Username
  • SMTP Password
  • Servername
  • port

Once you have these credentials, jump over to the Zabbix dashboard and click Administration, then Media types.  Click Create New Media Type and your new type a name ( IE: Email – AWS SES).  Here’s what I entered in the media type to get this working:

Type - EmailSMTP
server - servername from SES SMTP Settings page
SMTP server port - 587
SMTP helo - servername from SES SMTP Settings page
SMTP email - an email that's been verified by SES
Connection Security - STARTTLS
Authentication - Normal Password
Username - SES SMTP Username
Password - SES SMTP Password
Enabled - checked

Click add to save your config and let’s move on to configuring the new media type for a user.

Under Administration then Users, select the user who will be receiving trigger notifications and click the Media tab for that user.  Add a new Media type of Email – AWS SES then enter an email address in Send to.  Modify When active and Use if severity if desired.  Make sure it’s Enabled, then click Add.

Click Update and move on to Configuration, Actions.

Make sure the Event Source is set to TriggersCreate Action if you don’t already have one.  Enter a descriptive name and add Conditions to the Action.  See below for some good starting conditions.  These will send an email if a trigger is at least a severity of High.  It will also not send email when a host is in maintenance status.  Change these to your needs.

Once you finish the Action tab, move to the Operations tab.  Here you decide how the email message will be formatted and how the email will be sent.

I configure my actions to send an email at some interval (1hr in this example) until the solution is resolved or acknowledged.  This way no one can say they didn’t see the alert…

Click the Recovery Operation tab and add an entry under Operations.  Sam as you did on the Operations tab, click New, then

Operations type - Send Message
Send to Users - <select a user>
Send only to  - All or Email - AWS SES

Click add to save your operation details entries.

Once you’ve added the Operations and Recovery Operations steps, you should save your changes by clicking Add

Now, it’s time to test out your new email alerts.

An easy way to test this is to stop a zabbix-agent service or otherwise activate a trigger that has a severity of at least High.  If this is a lab test box, one trick I use regularly is to create an item that watches a file then set a trigger to alert when the file is missing.

Item
Name - Trigger Test
Type - Zabbix Agent (active)
Key - vfs.file.exists[/tmp/zabbix-trigger-test-.txt]
Type of Information - Numeric (unsigned)
Data Type - Decimal

Trigger
Name - Trigger Test
Severity - Disaster
Expression - {hostname001:vfs.file.exists[/tmp/zabbix-trigger-test-.txt].last()}<>1

Now just create a file called /tmp/zabbix-trigger-test.txt on your host.  When you want to test the trigger, simply rename/delete the file and the trigger will activate.  Add the file back and the trigger goes to RESOLVED state.

Troubleshooting the email send happens in Monitoring, Problems.  Show Recent Problems and in the Actions column you will either see Done or FailuresDone is good and everything is working as expected.  Failures will show an error message to help track down the issue.  Click on Failures and hover over the ‘i’ to see what the error message is.  You are on your own for figuring out what the error means.  It’s most likely a wrong port or connection security on your media type, but you’ll have to track it down.

This write-up, in general, will work with any SMTP server you’d like to connect to.  I hope this helps someone get their email alerts working in Zabbix.

 

Advertisements

AWS S3 on CentOS/RHEL for off-site backups

AWS S3 has been around for a long time now.  Yet, I am just now getting around to using it for an off-site backup location.

Here’s my plan.  Backup to local RAID disk on my backup server, then make an off-site copy up to S3.  In my mind, this covers the 3-2-1 backup rule.  Here’s how I see it broken down.  Reach out if you think my thought process is off.

3 copies: The first copy is your primary data, second copy is the local backup on your backup server, and the 3rd copy is what gets put into S3.

2 media types: This is where I might be off.  One is the local backup and the second is S3.  I question this a little because I’ve seen out there on the internet where some are talking about different physical media types, but I think that is overly redundant as long as you ensure that your off-site backup is secure.  IE: first is hard drive so second can’t be hard drive.   What do you think?

1 off-site copy:  The copy out to S3.

This seems like a pretty solid backup policy.

How to set it up

yum install gcc libstdc++-devel gcc-c++ curl-devel libxml2-devel openssl-devel mailcap automake fuse-devel fuse-libs git libcurl-devel libxml2-devel make
git clone https://github.com/s3fs-fuse/s3fs-fuse.git
cd s3fs-fuse/
./autogen.sh
./configure
make && make install
ln -s /usr/local/bin/s3fs /usr/bin/s3fs
  • Once you have fuse and s3fs installed, create a bucket in S3, and record credentials for user with access to bucket. s3fs will use /etc/passwd-s3fs for credential storage. Please enter your bucket credentials in /etc/passwd-s3fs as follows:
accessKeyId:secretAccessKey
  • If you have multiple buckets that will be mounted to this machine, add the credentials in /etc/passwd-s3fs as follows:
bucketName:accessKeyId:secretAccessKey
  • create a directory for mounting the s3 bucket
mkdir -p /mnt/s3fs-bucketname
  • manually mount the bucket into the mount point
s3fs -o use_cache=/tmp/cache bucketname /mnt/s3fs-bucketname

The -f switch is helpful to run the process in the foreground to troubleshoot mounting.

  • Once you confirm the mount is successful, you can enter the mount attributes in /etc/fstab so it mounts at startup.
s3fs#bucketname /mnt/s3fs-bucketname fuse allow_other,use_cache=/tmp/cache 0 0


Set up your backup client to put an extra copy in your /mnt/s3fs-bucketname directory.  If you were really paranoid about data loss, you could always age your data in S3 to send it to Glacier at a certain time.  I need to run with this for a little while and see what works best for my use case.  Let me know if this works for you.

 

 


export hostgroups to xml via API with python

According to the Zabbix docs, the only way to export hostgroups is through the API.  My exposure to the Zabbix API is limited, but I knew there were coding giants out there whose shoulders I could stand on.

I would like to give credit to someone directly, but the code I found had no author listed.  Here’s the link to the original on the zabbix.org wiki site for reference.

https://www.zabbix.org/wiki/Python_script_to_export_all_Templates_to_individual_XML_files

The code, as is, works great for exporting templates, but I needed to make some changes to get it to export hostgroups.  Luckily, the API reference pages on the Zabbix website are very helpful.

I’ll leave it up to you to diff the 2 versions to see exactly what changed, but for the basic summary, modify a couple parameters and a couple object properties and the script can used to export many other things.

See the API reference pages for the hostgroup method details.  https://www.zabbix.com/documentation/2.4/manual/api/reference/hostgroup/get

Here’s what I ended up with and it works great!  This will export all the hostgroups into separate xml files and put them into the ./hostgroups directory.

 

#!/usr/bin/python

#
 # pip install py-zabbix
 #
 # source: https://www.zabbix.org/wiki/Python_script_to_export_all_Templates_to_individual_XML_files
 #
 # usage: python zabbix_export_hostgroups_bulk.py --url https://<zabbix server name>/zabbix --user <api user> --password <user passwd>
 #

import argparse
 import logging
 import time
 import os
 import json
 import xml.dom.minidom
 from zabbix.api import ZabbixAPI
 from sys import exit
 from datetime import datetime

parser = argparse.ArgumentParser(description='This is a simple tool to export zabbix hostgroups')
 parser.add_argument('--hostgroups', help='Name of specific hostgroup to export',default='All')
 parser.add_argument('--out-dir', help='Directory to output hostgroups to.',default='./hostgroups')
 parser.add_argument('--debug', help='Enable debug mode, this will show you all the json-rpc calls and responses', action="store_true")
 parser.add_argument('--url', help='URL to the zabbix server (example: https://monitor.example.com/zabbix)',required = True)
 parser.add_argument('--user', help='The zabbix api user',required = True)
 parser.add_argument('--password', help='The zabbix api password',required = True)
 args = parser.parse_args()

if args.debug:
 logging.basicConfig(level = logging.DEBUG, format='%(asctime)s %(message)s', datefmt='%m/%d/%Y %I:%M:%S %p')
 logger = logging.getLogger(__name__)

def main():
 global args
 global parser

if None == args.url :
 print "Error: Missing --url\n\n"
 exit(2)

if None == args.user :
 print "Error: Missing --user\n\n"
 exit(3)

if None == args.password :
 print "Error: Missing --password\n\n"
 exit(4)

if False == os.path.isdir(args.out_dir):
 os.mkdir(args.out_dir)

zm = ZabbixHostgroups( args.url, args.user, args.password )

zm.exportHostgroups(args)

class ZabbixHostgroups:

def __init__(self,_url,_user,_password):
 self.zapi = ZabbixAPI(url=_url, user=_user, password=_password)

def exportHostgroups(self,args):
 request_args = {
 "output": "extend"
 }

if args.hostgroups != 'All':
 request_args.filter = {
 "name": [args.hostgroups]
 }

result = self.zapi.do_request('hostgroup.get',request_args)
 if not result['result']:
 print "No matching name found for '{}'".format(hostname)
 exit(-3)

if result['result']:
 for t in result['result']:
 dest = args.out_dir+'/'+t['name']+'.xml'
 self.exportTemplate(t['groupid'],dest)

def exportTemplate(self,tid,oput):

print "groupid:",tid," output:",oput
 args = {
 "options": {
 "hostgroups": [tid]
 },
 "format": "xml"
 }

result = self.zapi.do_request('configuration.export',args)
 hostgroup = xml.dom.minidom.parseString(result['result'].encode('utf-8'))
 date = hostgroup.getElementsByTagName("date")[0]
 # We are backing these up to git, steralize date so it doesn't appear to change
 # each time we export the hostgroups
 date.firstChild.replaceWholeText('2016-01-01T01:01:01Z')
 f = open(oput, 'w+')
 f.write(hostgroup.toprettyxml().encode('utf-8'))
 f.close()

if __name__ == '__main__':
 main()

 


Install latest Ruby version on CentOS/RHEL with RVM

While trying to install rack and passenger on a CentOS 6.8 box I ran into errors…

ERROR:  Error installing rack:
        rack requires Ruby version >= 2.2.2.
ERROR:  Error installing passenger:
        rake requires Ruby version >= 1.9.3.

Seems newer versions of rack and passenger are looking for more recent versions of Ruby than what is installed via CentOS/RHEL RPMs.  I haven’t needed to upgrade Ruby beyond the version that the installed RPMs provide, so I needed to research a bit to get past this road block.

I found a utility called RVM (Ruby Version Manager) that can be used quite easily to upgrade Ruby to pretty much any version you need.  I chose to install the latest stable version of Ruby.  RVM also allows you to have multiple versions of Ruby installed on your system and quickly switch between them.

It’s pretty easy to use.  Here’s what I did:

  • Install some required RPMs
yum -y install gcc-c++ patch readline readline-devel zlib zlib-devel libyaml-devel libffi-devel openssl-devel make bzip2 autoconf automake libtool bison iconv-devel sqlite-devel
  • Download and install the latest stable version of RVM
curl -sSL get.rvm.io | bash -s stable
  • Set up environment for Ruby
source /etc/profile.d/rvm.sh
rvm install 2.3.1
  • Set default version once installation completes
rvm use 2.3.1 --default
  • Finally, check that the version is correct
ruby --version

It’s that easy! Once I updated Ruby to the latest version I was able to successfully install both Rack and Passenger. Problem solved!

Thanks to the references I used to get this working:

RVM: Ruby Version Manager

How to Install Ruby 2.1.8 on CentOS & RHEL using RVM

Install Puppet Server CentOS 6.5/6.4


chroot sftp with OpenSSH

Overview

This describes configuring OpenBSD server specifically, but the sshd_config settings should work on any distro.

The result will be users with sftp only privileges where upon login they will be jailed into a directory and only have write access to a subdirectory.

Requirements

A recent version of OpenBSD or some other Linux variant running openssh-server

/etc/ssh/sshd_config

Add the following to your /etc/ssh/sshd_config file:

# override default of no subsystems
#Subsystem      sftp    /usr/libexec/sftp-server

# sftp configuration
Subsystem       sftp    internal-sftp

  Match Group sftponly
    ChrootDirectory %h
    ForceCommand internal-sftp
    X11Forwarding no
    AllowTCPForwarding no
    PasswordAuthentication yes

jail directory and user configuration

A quick and dirty bash script to configure the user directories.


 #!/bin/bash
 SFTPUSER=<username>
 SFTPDIR=/sftp_jail
 useradd -d $SFTPDIR/$SFTPUSER -s /sbin/nologin -g sftponly $SFTPUSER
 mkdir -p $SFTPDIR/$SFTPUSER/upload
 chown root:sftponly $SFTPDIR
 chmod 700 $SFTPDIR
 chown root:sftponly $SFTPDIR/$SFTPUSER
 chmod 750 $SFTPDIR/$SFTPUSER
 chown $SFTPUSER:nobody $SFTPDIR/$SFTPUSER/upload
 chmod 700 $SFTPDIR/$SFTPUSER/upload 

Notes

  • User will not be allowed to write to their home directory, but they will be allowed to write to the ‘upload’ subdirectory.
  • Users will have read-only access to their home directory.
  • Restart the sshd server after making any changes to /etc/ssh/sshd_config

Puppet agent on Windows 7

Puppet is very particular about the Ruby version on Windows.  While 2.2 and 2.3 versions of Ruby are available, puppet only runs without complaint on Ruby 2.1 on my Windows 7 box.

As of May 2016, I installed ruby 2.1.8 and puppet-agent 3.8.7.  I also had to install some gems to make puppet-agent happy.

 gem install win32-security win32-dir require win32-process top win32-service

Here are the links to downloads for puppet-agent and ruby:

https://downloads.puppetlabs.com/windows/

http://rubyinstaller.org/downloads/

No issues if the right version of ruby and the right gems.


Using ntpstat to check NTPD status with Zabbix

The standard way of checking a service in Zabbix checks that the service is running, but I wanted to know not only that the NTPD service was running but that the time was synchronized.  ntpstat is a great utility that does both, checks that the ntpd service is running and then tells you whether the server is synchronized.   ntpstat will report the synchronization state of the NTP daemon running on the local machine.  ntpstat returns 0 if clock is synchronized.  ntpstat returns 1 if clock is  not  synchronized.  ntpstat returns 2 if clock state is unknown, for example if ntpd can’t be contacted.

I created a Zabbix item to use ntpstat.  Here are the 2 ways I have used this new check:

The first way to use ntpstat with Zabbix is to simply create an item using the system.run function.

Name - ntpstat status
Type - Zabbix agent (active)
Key  - system.run[ntpstat &> /dev/null ; echo $?]
Type of Information - Text

Ensure EnableRemoteCommands=1 is set in your zabbix_agentd.conf file for this to work.

The second way to create the item is to use custom user parameters.  This requires a file modification on the monitored instance, so if you have a lot of instances to monitor or do not have a good way to automate this file modification, you may want to stick with option 1

I like creating new userparameter files for custom parameters.

/etc/zabbix/zabbix_agend.d/userparameter_custom_linux.conf
UserParameter=custom.net.ntpstat,ntpstat &> /dev/null ; echo $?

Then create an item similar to above but with a change to the key

Name - ntpstat status
Type - Zabbix agent (active)
Key  - custom.net.ntpstat
Type of Information - Numeric (unsigned)
Data Type - Decimal

Once your custom userparameter file is placed you’ll need to restart the zabbix agent. The last step with either item creation option is to create a trigger that alerts when the returned value is not 0.

I like this check much better than my original one that just alerted when the ntpd service was down.  Now I get alerted before time synchronization issues become an issue for the applications.

This was tested on both CentOS 6.7 and CentOS 7.1, but this should work on your Linux distro of choice as long as you have ntpstat installed.

Hope this helps