Cloud with DJ

Background

AWS Cloud Cost Optimization is all about managing and reducing your cloud expenses while ensuring your applications run smoothly. As a DevOps and Cloud Engineer, one of your tasks could be implementing cost optimization in AWS environment.

By managing your storage wisely, deleting old EBS snapshots you don’t need, moving rarely accessed data to cheaper storage options, and compressing your data to save space, you practice storage optimization reducing your cloud bill.

Project Outline

Set up a Lambda function to help you save on storage costs by finding and deleting EBS snapshots that aren’t connected to any active EC2 instances.

Use a Lambda function to help you manage storage costs by cleaning up the old EBS snapshots. It first gathers all snapshots owned by the account and checks for any active EC2 instances, both running and stopped. Then, it goes through each snapshot to see if its associated volume is no longer in use by any active instance. If it finds any unused snapshots, it deletes them to free up storage space.

Prerequisite

• Access to AWS account.
• Knowledge of Lambda, EBS and EC2
• Familiarity with Python and Boto3 library

Step 1- Create an EC2 instance

One of our goals is to verify the existence of an EC2 instance. Therefore, we need to create one. Check out this repository where I describe in detail how to create a EC2 instance.

Step 2- Verify EBS Volume

• Go to the “Storage” section of your newly created EC2 instance. This will bring up the details page for the volume, where you can see information like its size, state, and type.
• You’ll notice that this volume was automatically created when you set up the instance and is used as the root volume for your EC2 instance.

Step 3- Create a snapshot of the volume attached to the instance

• In the left panel of your EC2 dashboard, go to snapshots, and click on create snapshot.

• Select Volume as resource type.
• Choose the volume ID you verified previously.
• Add a name in the description.
• Next, click on create snapshot.

Step 4- Create a Lambda function

• As stated above, using a Lambda function will help manage storage costs by cleaning up the old EBS snapshots.
• Go to lambda dashboard.
• Click on create a function.

• Select Author from scratch.
• Enter the function name.
• Select python 3.12 (or newer version).
• Then click on create function.

• Copy the code for the lambda function from this repository: click here.
• Go to the code source of your lambda function. Click on lambda.py
• Replace the current code

• By the code from the repository

• Click Deploy, then Test.
• A new window configure test event will open.
• Click Save.

• Now go to the configuration tab. Click Edit to extend the default execution time of the function from 3 seconds to 10 seconds due to the larger code size, as 3 seconds are insufficient.

• Stay in the configuration tab. Click on Permissions, then the role. I will grant permission to allow Lambda to describe the ebs volumes.

• Create a new policy for this role. Go to IAM dashboard.
• Click create policy.
• Select EC2 as service. In the search bar, enter snapshot. Then select Delesnapshot and DescribeSnapshot.

• For resources, select All.
• Click Next.
• Name the new policy. Then click create policy.

• Go back to the IAM Console with the details of the IAM role associated with your Lambda function.
• Scroll down to the “Permissions” section of the IAM role details page.
• Click on the “Attach policy” button, and attach the new policy you created.

Step 4- Execute Lambda and Verify result

• Go to Lambda dashboard and execute the code.
• You should get an error message.

• If you look at the lambda python code, you should see that the Lambda function now includes the description of EC2 instances and volumes, so please update the policy accordingly.
• Edit the previous policy or add a new policy (with DescribeVolumes and DescribeInstances permission).

• We have all the policies attached to EBS. It’s time to test our function again

• There are no errors and our code is ready to be used.

• Keep in mind that our goal is to have our Lambda function help us save on storage costs by finding and deleting EBS snapshots that aren’t connected to any active EC2 instances.
• For that reason, delete the instance, which will also remove the attached volume, and then rerun the Lambda function.

• As intended, Snapshot is deleted when there is no existing EC2.

Voila! You’ve implemented Cost Optimization in your AWS environment. You used a Lambda function and boto3 library that delete EBS snapshots that aren’t connected to any active EC2 instances.