Trigger a Batch Process using Amazon SQS

Hands-On Lab

 

Photo of Mark Richman

Mark Richman

AWS Training Architect II in Content

Length

01:00:00

Difficulty

Intermediate

In this hands-on learning activity, you will trigger a batch process task using SQS. This learning activity uses images on the format to showcase the batch processing architecture. Objects uploaded to the input S3 bucket trigger an event that sends object details to the SQS queue. The ECS task deploys a Docker container that reads from that queue, parses the message containing the object name, and then downloads the object. Once transformed, it will upload the objects to the S3 output bucket.

What are Hands-On Labs?

Hands-On Labs are scenario-based learning environments where learners can practice without consequences. Don't compromise a system or waste money on expensive downloads. Practice real-world skills without the real-world risk, no assembly required.

Trigger a Batch Process using Amazon SQS

Introduction

In this hands-on lab, we will trigger a batch process task using SQS. This lab uses images on the format to showcase the batch processing architecture.

Solution

Log in to the live AWS environment using the credentials provided. Make sure you're in the N. Virginia (us-east-1) region throughout the lab.

Create the S3 Event Trigger for the SQS Queue

  1. Navigate to S3.
  2. Click to open the input bucket.
  3. In the Properties tab, click the Events card.
  4. Click + Add notification, and set the following values:
    • Name: ObjectCreate
    • Events: ObjectCreate (All)
    • Suffix: .jpg
    • Send to: SQS Queue
    • SQS: SQSBatchQueue
  5. Click Save.

Create the ECS Service

  1. Navigate to ECS.
  2. Click Clusters in the left-hand menu.
  3. Click to open the listed cluster.
  4. In the Services tab, click Create, and set the following values:
    • Launch type: EC2
    • Service name: batch
    • Number of tasks: 1
    • Leave the rest of the defaults
  5. Click Next Step.
  6. We don't need a load balancer, so click Next step.
  7. For Service Auto Scaling, select Configure Service Auto Scaling to adjust your service's desired count.
  8. Set the following values:
    • Minimum number of tasks: 1
    • Desired number of tasks: 1
    • Maximum number of tasks 5
    • IAM role for Service Auto Scaling:
  9. In the Automatic task scaling policies section, set the following values:
    • Scaling policy type: Step scaling
    • Scale out:
      • Policy name: ScaleOutPolicy
      • Execute policy when: Use an existing Alarm, SQSQueueDepth
      • Scaling action: Add 1 tasks when 5
    • Scale in:
      • Policy name: ScaleInPolicy
      • Execute policy when: Use an existing Alarm, SQSQueueDepth
      • Scaling action: Remove 1 tasks when 5
  10. Click Next step.
  11. Review, and then click Create Service.
  12. When your service status is finished updating, click View Service.
  13. Review your input, and click Create Service.
  14. Click View Service.
  15. Refresh the table to see the Last status become RUNNING.

Trigger the Batch Job

  1. Navigate to S3.
  2. Open the input bucket, and click Upload.
  3. Upload a .jpg file from your computer.
  4. Back in the S3 page, open the output bucket.
  5. There should be two folders: resized and thumbs.

Conclusion

Congratulations on completing this hands-on lab!