Skip to main content

Troubleshooting a Detection, Alerting, and Response Workflow in AWS

Hands-On Lab

 

Photo of

Training Architect

Length

01:00:00

Difficulty

Intermediate

Welcome to this hands-on AWS Learning Activity for Troubleshooting a Detection, Alerting, and Response Workflow. This activity provides you with the opportunity to get hands-on experience solving a real-world scenario where an EC2 instance is experiencing multiple failed logins, and we want to take the instance offline in response to this event automatically. Resources for this activity are on Github.

What are Hands-On Labs?

Hands-On Labs are scenario-based learning environments where learners can practice without consequences. Don't compromise a system or waste money on expensive downloads. Practice real-world skills without the real-world risk, no assembly required.

Troubleshooting a Detection, Alerting, and Response Workflow in AWS

Introduction

Welcome to this hands-on AWS Learning Activity for Troubleshooting a Detection, Alerting, and Response Workflow.

This activity provides you with the opportunity to get hands-on experience solving a real-world scenario where an EC2 instance is experiencing multiple failed logins, and we want to take the instance offline in response to this event automatically.

Resources for this activity are on Github.

Logging In

Log in to the live AWS environment using the credentials provided. Make sure you're in the N. Virginia (us-east-1) region throughout the lab.

Verify Alarm Settings on the Metric Filter

Complete the following:

  1. Navigate to CloudWatch.
  2. Select Logs.
  3. Click on 1 filter in the Metric Filters column for the log group in there.
  4. Click the pencil button next to the InvalidSSHLoginAlarm alarm.
  5. Scroll down to Conditions and change the threshold value from 222 to 2.
  6. Click Update alarm.

An Alarm Should Trigger an Email Notification

Complete the following:

  1. Navigate to SNS.
  2. Click on Topics, then choose the AlarmNotificationTopic on the next screen.
  3. Click on Create subscription.
  4. In the next screen, the Topic ARN is already populated. Choose Email from the Protocol dropdown, and enter an email address in the Endpoint box. Click on Create subscription again to finish it.
  5. Now confirm the subscription (by clicking the link in the email you get).

An Alarm Should Trigger the Lambda Function

SNS

  1. Navigate to SNS.
  2. Click on Topics, then choose the AlarmNotificationTopic on the next screen.
  3. Click on Create subscription.
  4. In the next screen, the Topic ARN is already populated. Choose AWS Lambda from the Protocol dropdown, and select our Lambda function in the Endpoint dropdown. Leave Version of alias set to default. Click on Create subscription again to finish it.
  5. Now confirm the subscription (by clicking the link in the email you get)

Test the Alarm

Open up a terminal and try to get into the EC2 instance with SSH. Enter wrong passwords so that they register as failed login attempts.

Trust But Verify

After at least two login failures, things on the AWS side should have kicked into action. Wait a little bit (30-45 seconds) and then we can check things out. We should have an email alerting us to the failures. Now navigate to Logs in Cloudwatch again. There are two. One is the log that was already there, but there's a new one now, which is our Lambda function's log. Opening that should reveal that the EC2 instance was shut down.

We should also see an alarm in red.

Now navigate to EC2, and we'll see that we have zero running instances. Click on Instances and we'll see the only one we have set up, and that it is in a stopped state.

Conclusion

What we've done here is to shut down an EC2 instance after two failed SSH login attempts. We did it using an alarm and a Lambda function. Congratulations!