Using Munin to monitor EC2 instances on Amazon

After playing around with CloudWatch, I find the interface very confusing to use. The biggest problem  is EC2 instance are described by AMI image ID rather than my pre-defined machine tag name (Let me know in the comments below if you can figure out how).

I was looking at a few monitoring tools (Nagis, Cacti, Munin, Zabbix) and decided to try to out Munin. The biggest motivator for me is that Instagram is also using Munin.

Let's begin by spinning an Ubuntu instance as the Munin master.

Installing Munin Master and Munin Nodes:

Install munin and munin-node

apt-get install munin
apt-get install munin-node

Install apache (for viewing reports from the Web)

apt-get install apache2

For all the instances you want to monitor, install Munin Node.

apt-get install munin-node

For these node instances, we will edit munin-node.conf

vi /etc/munin/munin-node.conf

Change the host_name. Name this to be something descriptive so you will know what this machine is. The master node will report using this name.

host_name {api1.monetize24hours.com}

Change allow from

allow ^127\.0\.0\.1$

to

allow ^.*$

This is saying allow all internal IPs to connect to. Since AWS elastic address changes all the time, it's better to set it to allow all. Do NOT set it to the instance's external address else you will be charged for data transfer. Make sure all the machines are behind a firewall.

Restart the Munin node.

/etc/init.d/munin-node restart

Repeat the settings above for all the Munin nodes.

Now in the Master Munin node, edit vi /etc/munin/munin.conf. Search for

[localhost.localdomain]
    address 127.0.0.1
    use_node_name yes

Change it to

[api1.monetize24hours.com]
    address ip-00-000-000-000.ec2.internal

This value must match the host name you defined in the Munin node above. The address is the ec2 private address of the Munin node. This is how Munin master will aggregate and report the data.


Showing Data on Webpages

Make sure the Munin master can connect to your Munin nodes.

telnet {private_ec2_address} 4949

Port 4949 is used for Munin internodes communication

If it doesn't connect, add port 4949 for the Munin node's security group.

You can find the Munin master's security group name by clicking on the Security Group and checking the Details tab. If looks something like "sg-e0000000".

Now edit /etc/munin/munin.conf to tweak the log and graph generation directories.

bdir   /var/lib/munin
htmldir /var/www/munin
logdir /var/log/munin
rundir  /var/run/munin

Change the above directories. Create them if they don't exist. Make sure you set the appropriate permissions for the directories.

Wait for 5 to 10 minutes. The Perl cron will gather data.

Access the graphs by

{public_ec2_address}/munin

You will want to secure the webpages so no one else can access them. Either secure them by ip or username and password.

We will use .htaccess in the following example.

htpasswd -c /etc/apache2/.munin_htpasswd admin

Create /var/www/munin/.htaccess, and put the following:

AuthUserFile /etc/apache2/.munin_htpasswd
AuthGroupFile /dev/null
AuthName EnterPassword
AuthType Basic

Edit /etc/apache2/sites-available/default.

Change AllowOverride None to AllowOverride All

Restart apache.

service apache2 restart