Monitoring status of AWS EC2 Snapshots within Nagios

I recently wrote a script to automate the creation of snapshots for EBS volumes for Amazon EC2 instances (https://deangrant.wordpress.com/2013/08/06/aws-create-ec2-snapshot-based-on-metadata-tag-value/).

Following on from this I wanted to report the status of snapshots completed and return this status to Nagios. This was to be achieved by comparing the number of EBS volumes that contained a specific metadata tag value to the number of snapshots created on a particular day.

As per usual this script was to be written in Windows Powershell and importing the Powershell for Amazon Web Services ((http://aws.amazon.com/powershell/) snap-in to the current powershell session.

If (-not (Get-Module AWSPowershell -ErrorAction SilentlyContinue))
{ 
Import-Module "C:\Program Files(x86)\AWS Tools\Powershell\AWSPowershell\AWSPowershell.psd1" > $null
}

Once the snap-in has been imported we will need to set our AWS Credentials and AWS Region for this session:

Set-AWSCredentials -AccessKey XXXXXXXXXXXXXXXXXXXXXX -SecretKey XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX
Set-DefaultAWSRegion eu-west-1

As part of the script I will be required to output the Date string in two different formats one for the snapshot description and one for the date passed in the string for Nagios status information.

$Date = (Get-Date).toString ('ddMMyyyy')
$StatusDate = (Get-Date).toString ('dd/MM/yyyy')

Now we are required to compare the number of EBS volumes which match the metadata tag value and the number of snapshots created. Firstly, we will create the filter for the EBS volumes which in this instance was to return all volumes with the¬†metadata tag value ‚ÄėEBS Snapshot: Yes‚Äô and use the Get-EC2Volume cmdlet to return this volume and store within a variable.

Filter = (New-Object Amazon.EC2.Model.Filter).WithName("tag:EBS Snapshot").WithValue("Yes")
$Volumes = Get-EC2Volume -Filter $Filter

Now we have returned all our EBS volumes, we need to return all the snapshots created on the current date. This information is stored within the description of the snapshot upon creation in the format ‘EBS Snapshot created on ddMMyyyy’. To return all snapshots with this filter the Get-EC2Snapshot cmdlet is used to return all snapshots containing the filter.

$Snapshots = Get-EC2Snapshot | Where-Object ($_.Description -like ("EBS Snapshot Created on " + $Date + "*")}

Now it is time to compare the counts of variables returned, also in this case I am only creating a warning threshold only to generate my return codes.

$Warning = $Volumes.Count -5
If ($Snapshots.Count -eq $Volume.Count) {$returncode = 0} 
ElseIf ($Snapshots.Count -lt $Volume.Count and $Snapshots.Count -gt $Warning) {$returncode = 1}
ElseIf ($Snapshots.Count -lt $Warning) {$returncode = 2}

Now all that is left, is to exit the script and return the exit code to Nagios. However, before we do so I want to return Status Information as well to provide the number of snapshots performed on a certain date and the number of snapshots compared to actual EBS volumes.

"Total number of EBS Snapshots performed on " + $StatusDate + ": " + $SnapshotCount + "/" + $Volumes.Count 
exit $returncode

Below, is an example of a formatted Status Information message generated:

Total Number of EBS Snapshots performed on 12/09/2013: 137/137

There are one or two issues with the script, if a EBS volume is created during the day and no snapshot has been performed this will report that there are more volumes than snapshots, therefore if six EBS volumes were created this would then turn a warning. This can be negated by running the external script within the service command less frequently, in my case I run this once per day.

While the script was created to be executed as an external script within Nagios, this can be run standalone from Windows Powershell. If your are looking to add external scripts to Nagios such as this one see the below link for more information;

https://deangrant.wordpress.com/2013/09/12/creating-and-running-external-scripts-within-nagios-xi/

The full Windows Powershell script can be downloaded from the below link:

https://app.box.com/s/jm88wcrtosfc7xcisbn7

Auto-signing client certificates on Puppet Master Server

I am currently using Puppet to automate infrastructure within a cloud offering, Amazon Web Services (AWS) for those interested!. 

As part of deployment of new instances, I require the client certificate to be automatically signed by the Puppet master server to immediately begin the configuration. 

This can be achieved by allowing the master server to automatically sign all clients (*) certificates, as below:

$ cat /etc/puppet/autosign.conf 
*

However, this can be perceived to be a security risk as any machine could connect without authorisation and request manifest files which may contain sensitive information. I suppose one other option could be use a filter based on a naming convention (in the below example, the DNS name is used as a match) used internally as below, but again the solution saves time but does not remove the security risk.

$ cat /etc/puppet/autosign.conf 
*.dean.local

For those using AWS, I found a great article at http://www.codelord.net/2010/12/19/using-puppet-to-automatically-configure-new-ec2-instances where using the EC2 API tools to get the DNS name of the deployed instance and adds this as a trusted client to /etc/puppet/autosign.conf. This then can be run as a CRON job to automatically sign the client certificates on the master server. 

 

Eastern Time zone format and creating tasks in Ylastic

After signing up to the Ylastic service (ylastic.com) which provides a unified interface to manage your Amazon Web Services cloud environment, I started to struggle with the simple task of creating scheduled tasks using CRON expressions.

From the initial configuration, it would appear that Ylastic does not support using special characters other than the asterisk. Now that was not the issue I was running into, once the scheduled task was created it would appear to be invoked as configured using the CRON expression.

After a little bit of head scratching the Ylastic service only supports the Eastern Time zone, and does not take into consideration the time zone you configure for your account. So when creating your scheduled tasks you will need to configure scheduled times in Eastern Time zone format, so in my case I got used to adjusting the time zone (-05:00) when creating my tasks.