Custom Logging Using CloudWatch and CloudWatch Logs

Hands-On Lab

 

Photo of Adrian Cantrill

Adrian Cantrill

Training Architect

Length

00:30:00

Difficulty

Beginner

In this hands-on lab, we will configure custom CloudWatch logging using the CloudWatch agent and CloudWatch alarms. We will cover the configuration of the CloudWatch monitoring agent to monitor the occurrence of SSH attempts to our EC2 instance and create an alarm for frequent invalid authentication attempts. We will also configure monitoring for our Memory Used percentage in CloudWatch. This is only one small example of CloudWatch's potential. We could easily monitor our application logs and fire Lambda functions or SNS topics based upon certain metric activity.

What are Hands-On Labs?

Hands-On Labs are scenario-based learning environments where learners can practice without consequences. Don't compromise a system or waste money on expensive downloads. Practice real-world skills without the real-world risk, no assembly required.

Custom Logging Using CloudWatch and CloudWatch Logs

Introduction

In this hands-on lab, we will configure custom CloudWatch logging using the CloudWatch agent and CloudWatch alarms. We will cover the configuration of the CloudWatch monitoring agent to monitor the occurrence of SSH attempts to our EC2 instance and create an alarm for frequent invalid authentication attempts. We will also configure monitoring for our Memory Used percentage in CloudWatch.

Solution

Log in to the live environment with the cloud_user credentials provided.

Make sure you are using the N. Virginia (us-east-1) region throughout the lab.

To generate invalid user authentication attempts for the /var/log/secure log stream, you can try to authenticate to your EC2 instance as a user that does not exist.

Create an IAM Role and Attach the "CloudWatchAgentServerPolicy"

  1. Navigate to IAM.
  2. Select Roles on the left-hand menu, and then click Create role.
  3. With AWS service selected, choose EC2 from the list.
  4. Click Next: Permissions.
  5. In the search/filter box, filter for "CloudWatchAgentServerPolicy" and then select it.
  6. Click Next: Tags.
  7. Leave the tags as-is, and click Next: Review.
  8. Provide a role name of "CloudWatchAgentServerRole".
  9. Click Create role.

Associate the Role with the EC2 Instance

  1. Navigate to EC2.
  2. Select the existing running EC2 instance, and click Actions > Instance Settings > Attach/Replace IAM Role.
  3. Set the IAM role to our "CloudWatchAgentServerRole", and click Apply.

Install and Configure the CloudWatch Agent

  1. With the instance still connected, click Connect and copy the instance's public DNS.

  2. Open a terminal session, and log in to the instance via SSH.

  3. Download the CloudWatch agent package:

    wget https://s3.amazonaws.com/amazoncloudwatch-agent/amazon_linux/amd64/latest/amazon-cloudwatch-agent.rpm
  4. Install the CloudWatch agent:

    sudo rpm -U ./amazon-cloudwatch-agent.rpm
  5. Execute the CloudWatch agent wizard:

    sudo /opt/aws/amazon-cloudwatch-agent/bin/amazon-cloudwatch-agent-config-wizard
  6. There will be a series of questions. Answer as follows:

    On which OS are you planning to use the agent?
    1. linux
    
    Are you using EC2 on On-Premises hosts?
    1. EC2
    
    Which user are you planning to run the agent?
    1. root
    
    Do you want to turn on StatsD daemon?
    2. no
    
    Do you want to monitor metrics from CollectD?
    2. no
    
    Do you want to monitor any host metrics? e.g. CPU, memory, etc.
    1. yes
    
    Do you want to monitor cpu metrics per core? Additional CloudWatch charges may apply.
    2. no
    
    Do you want to add ec2 dimensions (ImageID, InstanceID, InstanceType, AutoScalingGroupName) into all of your metrics if the info is available?
    2. no
    
    Would you like to collect your metrics at high resolution (sub-minute resolution)? This enables sub-minute resolution for all metrics, but you can customize for specific metrics in the output json file.
    2. 10s
    
    Which default metrics config do you want?
    1. Basic
    
    Are you satisfied with the above config? Note: it can be manually customized after the wizard completes to add additional times.
    1. yes
    
    Do you have any existing CloudWatch Log Agent configuration file to import for migration?
    2. no
    
    Do you want to monitor any log files?
    1. yes
  7. For Log file path, type /var/log/secure.

  8. For Log group name, type sshlogs.

  9. For Log stream name, hit Enter to use the default of [{instance_id}]. NOTE: If you enter a typo anywhere in the wizard process, you will need to re-run the wizard or edit the JSON output file to resolve any typo-related issues.

  10. When it asks, "Do you want to specify any additional log files to monitor?", enter 2 (no).

  11. When it asks, "Do you want to store the config in the SSM parameter store?", enter 2 (no).

  12. Start the CloudWatch agent:

    sudo /opt/aws/amazon-cloudwatch-agent/bin/amazon-cloudwatch-agent-ctl -a fetch-config -m ec2 -c file:/opt/aws/amazon-cloudwatch-agent/bin/config.json -s
  13. Give it a few minutes for the logs to be completely exported to CloudWatch.

  14. Navigate to CloudWatch in the AWS console to verify the SSH logs and custom memory metrics are making their way to CloudWatch.

  15. Click Metrics in the left-hand menu.

  16. In the Custom Namespaces section, click CWAgent.

  17. In the Metrics section, click host to view the metrics.

Configure Metric Filter for SSH Logs

  1. Click Logs in the left-hand menu.
  2. Select the circle next to our sshlogs log group, and click Create Metric Filter.
  3. In the Filter Pattern field, enter "[month, day, time, ip, connid, status = Invalid, ...]".
  4. Click Test Pattern.
  5. Click Assign Metric.
  6. In the Metric Details section, set the following values:
    • Metric Namespace: Custom SSH Metrics
    • Metric Name: sshInvalid
    • Metric Value: 1
    • Default Value: 0
  7. Click Create Filter. It may take a few minutes to appear.
  8. Click Metrics in the left-hand menu.
  9. Click the All metrics tab.
  10. In the Custom Namespaces section, click Custom SSH Metrics.
  11. In the Metrics section, click Metrics with no dimensions.
  12. Select sshInvalid.
  13. Click the Graphed metrics tab.
  14. Change the Statistic to Sum.
  15. Click Logs in the left-hand menu.
  16. Click sshlogs.
  17. Click the listed log stream.
  18. Scroll through to locate invalid users and failed passwords for invalid users.

Conclusion

Congratulations on successfully completing this hands-on lab!