AWS DevOps & Developer Productivity Blog

Monitor Website Latency using CloudWatch Logs CLI Plugin

With CloudWatch Logs, you can monitor your systems and applications in near real-time using your existing log files in just a few quick steps.  The CloudWatch Logs console and agent can also help you quickly generate CloudWatch Metrics, CloudWatch Alarms, and invoke CloudWatch Actions based on log events.

In this post we’ll demonstrate how to monitor website latency using CloudWatch Logs, a small python script and a powerful command line tool that plugs into the AWS CLI in three steps. To follow along, you’ll need to have a Linux server with Pip and  Python version 2.6 or greater installed.

1. Installing the CloudWatch Logs AWS CLI Plugin

To get started, first, you need to install the CloudWatch Logs AWS CLI plugin on a server or Amazon Elastic Compute Cloud (EC2) instance with Python by running the following commands:

sudo pip install –upgrade
       awscli-cwlogs
 
aws configure set plugins.cwlogs cwlogs

Follow the prompts to configure the CLI, which ask you for your AWS access key as well as the default region.  You can also use an IAM Role if you do not want to use the AWS access key.  More information can be found in the AWS CLI documentation and the IAM documentation .       

2. Setup the Website Latency Monitoring Script

We’re going to use a short Python script that we’ll call latency.py. This tool does a basic GET request on a website and afterwards records the response latency to a log.  If any errors are found in a JSON document, it prints that document to a standard output:

import urllib2
import time
import json
import sys
request = urllib2.Request(‘http://aws.amazon.com/cloudwatch’)
while True:
       start = time.time()
       result = {}
       try:
              response = urllib2.urlopen(request)
              result[‘code’] = response.code
       except urllib2.HTTPError as e: 
              result[‘code’] = e.code
              result[‘reason’] = e.reason
       result[‘elapsed’] = time.time()-start
       print json.dumps(result)
       sys.stdout.flush()
       time.sleep(1)
 
3. Posting Log Data to CloudWatch Logs

In the final step, we’ll send the log data from the Python script to CloudWatch Logs.  We’ll need to choose a Log Group name (“latencies” in this example) and a log stream (“cloudwatch”.)  :

python latency.py | aws logs push –log-group-name latencies –log-stream-name cloudwatch

The push plugin sends your entries every five seconds to be stored in the CloudWatch logs log group “latencies” in the log stream “cloudwatch”

Now in a different terminal, let’s do the pull command:

 aws logs pull –follow –log-group latencies –log-stream-name cloudwatch

Every few seconds you’ll see log events streaming to your command line. You can also pipe this output through grep and script it in your tool chain. Another powerful use of the pull command is to allow your DevOps to monitor logs from your infrastructure without actually logging into the remote hosts.

To learn more about concepts in this post, you can click on these links to learn more about CloudWatch Logs, CloudWatch Metrics, CloudWatch Alarms, and CloudWatch Actions.