Processing Messages in AWS with Auto Scaling and SQS Message Queues
AWS Training Architect II in Content
In this hands-on lab, we will cover how an Auto Scaling group of EC2 instances can be used to process messages in an SQS queue based on queue size. This design uses the AWS Job Observer Pattern. In this pattern, the queue is monitored for increases and an Auto Scaling group scales accordingly. Amazon CloudWatch is essentially the observer. By setting an alarm on a metric for messages available in the queue, the CloudWatch alarm can trigger an Auto Scaling group to scale out when the message queue size exceeds the alarm value. We will configure this architecture and be able to test it by sending messages to the SQS queue.
Processing Messages in AWS with Auto Scaling and SQS Message Queues
We're going to see how an Auto Scaling Group of EC2 instances can be used to process messages, based on queue size, in an SQS queue. In this AWS Job Observer Pattern, an Auto Scaling Group scales up or down in response to the queue's size. By setting an alarm on a metric for messages available in the queue, CloudWatch can trigger an Auto Scaling Group scale out when the message queue size exceeds the alarm value. We'll configure this architecture and test it by sending messages to the SQS queue.
Creating the SQS Message Queue
In the AWS Console, under Application Integration, we'll find the Simple Queue Service (SQS). Click on it, then Get started now. We'll give our queue a real spiffy name, like linuxacademy, and choose Standard Queue underneath that. Now, down at the bottom right of the screen, click Configure Queue. We're not changing anything, so click Create Queue.
Now we've got to go to Cloudwatch and configure the alarm, but we've got to wait a few minutes before Cloudwatch can see this queue we've just created. Give it 5-10 minutes and come back.
Creating a CloudWatch Alarm to Monitor the SQS Queue
In a new browser tab, get into the AWS console, in Management Tools, and select CloudWatch. Once we're in there, click on Metrics (in the left-hand menu) and find the SQS section. Click on SQS, then Queue Metrics. We'll see some for our
linuxacademy queue. Click on Alarms, then Create Alarm, and we'll see a layout similar to what we were just looking at. In the SQS Metrics section, click on Queue Metrics. The one we're concerned with here is ApproximateNumberOfMessagesVisible. Check the associated box, then click the Next button in the lower left.
On this next screen, we'll name our alarm sqs-messages-in-queue, and give it that for a description too. In the Whenever area below, we'll set the is: to >= and 40. Set for: to 1 (so that it reads "for 1 out of 1 datapoints").
Below this, in the Actions section, we'll click the Delete link. Then, over on the right we'll change the interval to 1 Minute. We can leave everything else alone, and click Create Alarm down at the bottom of the screen.
Don't mind the INSUFFICIENT_DATA warning, it will say OK in a little bit.
Let's step back here and head to our EC2 Management Console. We need our instance's public IP address. Fire up a terminal, and log into our instance with SSH. Use the credentials provided on the lab overview page.
Once we're in, we'll become
root right away and install some software:
[linuxacademy@host]$ su - ... [root@host]$ pip install awscli==1.6.6 [root@host]$ aws configure
This will start the configuration of the software. We can breeze through most of these. Just make sure we set
us-east-1 as a Default region name.
Now we can send a test message. First, let's grab the URL of our SQS queue from the SQS Management Console, then run this command:
[root@host]$ aws sqs send-message --queue-url https://&ltURL_FROM_SQS_CONSOLE> --message-body linuxacademytest
Back in our Management Console, if we hit our browser's refresh button, we should see the message we sent sitting in there.
Set up the Auto Scaling Group
Let's head back over to the EC2 Management Console, and select Launch Configurations in the left-hand menu.
On this screen, click Create Launch Configuration. Select the Amazon Linux AMI (not the Amazon Linux 2 AMI) here, and on the next screen leave the t2.micro line checked.
In the Create Launch Configuration details page, set the Name value to sqs-pool-workers. That's all we need to do here, so click Skip Storage.
The defaults in the security group section are fine, so click Review, and on the review page, click Create launch configuration.
In the key pair pop-up, select Proceed without a key pair, check the I acknowledge... box, and click Create launch configuration.
On this next screen, click the Create an Auto Scaling group using this launch configuration button.
We'll land in the Create Auto Scaling Group screen. Give our group a Group name of sqs-pool-asg, leave Group size at 1 instance, and pick our vpc in the Network dropdown. We'll get a new dropdown where we can add our two subnets (one each in
us-east-1b). We can leave the Advanced Details section alone, and click Next: Configure scaling policies.
On the next page, select Use scaling policies to adjust the capacity of this group, and set the policy to scale between 1 and 4 instances.
Now let's click the Scale the Auto Scaling group using step or simple scaling policies link, and in the Increase Group Size section, set the following values:
- Execute policy when: Select our sqs-messages-in-queue alarm.
- Take the action: Add 1 instances when 40 <= ApproximateNumberOfMessagesVisible < +infinity
- In the Decrease Group Size section, click the X button. Normally we wouldn't do this, but we're just testing for scaling up here, and don't need to bother with the scaling down aspect of things.
Now we can click Review, and then on the next screen we can click Create Auto Scaling Group. Once it's created, rather than Close, let's click on View your Auto Scaling groups.
We should have a second EC2 instance spinning up, and we can check by looking in the EC2 Dashboard, under Instances.
Using the AWS CLI to Test the Configuration of the SQS Queue
Back in our terminal, lets write a script that will let us test whether or not things are working as planned. Use whatever editor suits your fancy to make the file
sqs.sh, and let's put these commands in it:
#! /bin/bash for i in $(seq 1 60) do aws sqs send-message --queue-url https://&ltURL_FROM_SQS_CONSOLE>/linuxacademy --message-body linuxacademytest done
aws command, you may notice, is the same one we ran earlier when we sent the first message.
Make this script executable, and run it:
[root@host]$ chmod u+x sqs.sh [root@host]$ ./sqs.sh
This will start sending messages, sixty of them, to our queue.
Now, back in the SQS Management Console, we can refresh and see 61 messages sitting there now. And now, if we head back into the CloudWatch Management Console, click on our alarm, and see the alarm details down in the lower half of the screen. It may take a couple minutes, but refreshing the page will eventually show our 61 messages here. This should trigger auto scaling.
Back in the EC2 Management Console, we'll see (in Auto Scaling Groups, down in the left-hand menu) that there's an instance "warming up." Eventually, if we get in and look at our actual EC2 instances, we'll see a second one spinning up.
We set up auto scaling based on messages in the SQS queue, and it's working well. We're done. Congratulations!