Using Amazon CloudWatch Command Line Tool to record metrics

Introduction

Amazon CloudWatch provides seamless integration for monitoring AWS resources like EC2 instances, RDS instances, EBS volumes, etc based on CPU utilization, data transfer and disk usage.

There are two types of monitoring: basic and detailed. Basic Monitoring reports at a five-minute frequency. Detailed Monitoring reports at a one-minute frequency while aggregating by AMI ID and instance type.

Monitoring data is retained for two weeks, even if your instance is terminated.

Below are the resources to metrics mapping: (For example, CloudWatch tracks request count and latency of Elastic Load Balancer)

ELB - request count, latency
EBS - read/write latency
RDS - freeable memory, available storage space
SQS - number of messages sent and received

You can also send custom metrics to CloudWatch by using the Put API.

You can view your stats in the AWS Management Console.

For overall status of all AWS services, check AWS Service Health Dashboard.


Setting up Amazon CloudWatch Command Line Tool

Spin up a EC2 instance. (Skip this if you are using your home computer).

Begin by downloading the CloudWatch CLI Tool.

mkdir /opt/tools/aws
cd /opt/tools/aws
wget http://ec2-downloads.s3.amazonaws.com/CloudWatch-2010-08-01.zip

Install zip and unzip the package.

sudo apt-get install zip
unzip CloudWatch-2010-08-01.zip 

Check if you have JAVA installed

java -version

If not, read Install Java OpenJDK 7 on Amazon EC2 Ubuntu.

Set AWS_CLOUDWATCH_HOME path in ~/.bashrc

export AWS_CLOUDWATCH_HOME=/opt/tools/aws/CloudWatch-1.0.13.4
export PATH=$PATH:$AWS_CLOUDWATCH_HOME/bin

Enter your AWS Access Key ID and Secret Access Key in the file $AWS_CLOUDWATCH_HOME/bin/credential-file-path.template. You can find your credentials in the AWS Management Console.

AWSAccessKeyId=
AWSSecretKey=

chmod 600 credential-file-path.template

Rename credential-file-path.template to something else (ex. aws_credentials)

Move this file to somewhere else. You may be using this in some other service. For instance, move to /opt/tools/aws.

Add this entry to ~/.bashrc

export AWS_CREDENTIAL_FILE=/opt/tools/aws/aws_credentials

Update ~/.bashrc.

source ~/.bashrc

Test the tool:

mon-cmd

You should see the following:

Command Name                       Description
------------                       -----------
help
mon-delete-alarms                  Delete alarms
mon-describe-alarm-history         Describe alarm history
mon-describe-alarms                Describe alarms fully.
mon-describe-alarms-for-metric     Describe all alarms associated with a single metric
mon-disable-alarm-actions          Disable all actions for a given alarm
mon-enable-alarm-actions           Enable all actions for a given alarm
mon-get-stats                      Get metric statistics
mon-list-metrics                   List user's metrics
mon-put-data                       Put metric data
mon-put-metric-alarm               Create a new alarm or update an existing one
mon-set-alarm-state                Manually set the state of an alarm
version                            Prints the version of the CLI tool and the API.

Publish data points to CloudWatch

CloudWatch allows you to publish data points via PUT requests. CloudWatch only works with data that's in UTC timestamp and within the past two weeks (Only data within two weeks would be retained).

In this example, you will feed CloudWatch with some custom data points.

Execute the following data sets. But substitute the date below to be within a few hours before.
C

Set A (4 data points):

mon-put-data -m RequestLatency -n "Test001" -t 2013-04-16T20:30:00Z -v 87 -u Milliseconds
mon-put-data -m RequestLatency -n "Test001" -t 2013-04-16T20:30:00Z -v 51 -u Milliseconds
mon-put-data -m RequestLatency -n "Test001" -t 2013-04-16T20:30:00Z -v 125 -u Milliseconds
mon-put-data -m RequestLatency -n "Test001" -t 2013-04-16T20:30:00Z -v 235 -u Milliseconds

Set B (Instead of sending individual data points, send sum, min, max and sample count):

mon-put-data -m RequestLatency -n "Test001" -t 2013-04-16T21:30:00Z -s "Sum=577,Minimum=65,Maximum=189,SampleCount=5" -u Milliseconds 

Set C:

mon-put-data -m RequestLatency -n "Test001" -s "Sum=806,Minimum=47,Maximum=328,SampleCount=6" -u Milliseconds

The above are data within latency within three hours. Just think of them as some data points.

Let's get the data summary:

mon-get-stats -m RequestLatency -n "Test001" -s "Average" --start-time 2013-04-16T19:30:00Z --headers

Results:

Time                 Average             Unit
2013-04-16 20:30:00  124.5               Milliseconds
2013-04-16 21:30:00  115.4               Milliseconds
2013-04-16 22:29:00  134.33333333333334  Milliseconds

You can also see the Visual Representation in the AWS Management Console.

Login to CloudWatch.

Click on Metrics in the Left Panel. Select Test001 in the "Viewing" Dropdown box.

You can also create alarms based on this metric.

This concludes the tutorial. If you are interested in more advanced tools, check out this post - Using Munin to monitor EC2 instances on Amazon.