Streamlining AWS Cost Visibility: Automated Email Reports with Lambda and SES
In today’s cloud-centric world, managing costs in AWS is crucial for optimizing resources and maintaining budgetary control. One effective way to stay on top of your AWS spending is through automated cost monitoring and reporting. In this blog post, we’ll walk you through the process of setting up a cost-monitoring and reporting system using AWS Lambda, SES (Simple Email Service), and EventBridge (formerly known as CloudWatch Events).
In many cases, there are members of senior leadership or finance decision-makers who don’t have access to AWS accounts and therefore depend on individuals or additional custom processes to share billing information. This task becomes specifically complicated when there is a complex account organization structure in place.
In such cases, you can email cost reports periodically and automatically to these groups or individuals using AWS Lambda. In this blog post, you’ll learn how to send automated emails for AWS billing usage and consumption drifts from previous days.
Introduction
Monitoring AWS costs manually can be time-consuming and prone to oversight. By automating the process, you can receive regular cost reports directly in your email inbox, enabling you to track spending trends, identify cost anomalies, and take proactive measures to optimize resource usage.
AWS provides the Cost Explorer API to enable you to programmatically query data for the cost and usage of AWS services. This solution uses a Lambda function to query aggregated data from the API, format that data, and send it to a defined list of recipients.
- Amazon EventBridge (Amazon CloudWatch Events) is configured to cue the Lambda function at a specific time.
- The function uses the AWS Cost Explorer API to fetch the cost details for each account.
- The Lambda function calculates the change in cost over time and formats the information to be sent in an email.
- The formatted information is passed to Amazon Simple Email Service (Amazon SES).
- The report is emailed to the recipients configured in the environment variables of the function.
Prerequisites
Before we dive into the setup process, make sure you have the following prerequisites in place:
- An AWS account with appropriate permissions to create Lambda functions, IAM roles, SES configurations, and EventBridge rules.
- Basic knowledge of AWS services like Lambda, SES, and EventBridge.
- Familiarity with Python programming for writing Lambda functions.
Step 1: Creating the Lambda Function
Instructions:
- Navigate to the AWS Lambda console.
- Click on “Create function”.
- Choose “Author from scratch”.
- Provide a name, select Python as the runtime, and choose an existing execution role with the necessary permissions.
- Click on “Create function”.
- Copy the provided Lambda function code and paste it into the Lambda editor.
- Save the Lambda function.
Note: The AWS Cost Explorer API lets you directly access the interactive, ad-hoc query engine that powers AWS Cost Explorer. Each request will incur a cost of $0.01
AWS Lambada functon Code :
This code will send a report of the last week for resources that are tagged with “Key”: “Cost-billing” and “Values”: [“Platform”].
# Lambda function code to fetch AWS cost data and send email report
# Import necessary libraries
import json
import boto3
import datetime
from botocore.exceptions import ClientError
import pandas as pd
def lambda_handler(event, context):
# AWS clients
billing_client = boto3.client('ce')
ses_client = boto3.client('ses', region_name='us-east-1')
# Getting dates (yyyy-MM-dd) and converting to string
today = datetime.date.today()
one_week_ago = today - datetime.timedelta(days=7) # 1 week ago
str_today = str(today)
str_one_week_ago = str(one_week_ago)
# Get total cost for the previous week
response_total = billing_client.get_cost_and_usage(
TimePeriod={
'Start': str_one_week_ago,
'End': str_today,
},
Granularity='DAILY',
Metrics=['UnblendedCost'],
Filter={
"Tags": {
"Key": "Cost-billing",
"Values": ["Platform"]
}
}
)
total_cost_weekly = response_total["ResultsByTime"][0]['Total']['UnblendedCost']['Amount']
total_cost_weekly = float(total_cost_weekly)
total_cost_weekly = round(total_cost_weekly, 3)
total_cost_weekly = '$' + str(total_cost_weekly)
# Get detailed billing for individual resources
response_detail = billing_client.get_cost_and_usage(
TimePeriod={
'Start': str_one_week_ago,
'End': str_today,
},
Granularity='DAILY',
Metrics=['UnblendedCost'],
GroupBy=[
{
'Type': 'DIMENSION',
'Key': 'SERVICE'
},
{
'Type': 'DIMENSION',
'Key': 'USAGE_TYPE'
}
]
)
resources = {'Service': [], 'Usage Type': [], 'Cost': []}
for result in response_detail['ResultsByTime'][0]['Groups']:
group_key = result['Keys']
service = group_key[0]
usage_type = group_key[1]
cost = result['Metrics']['UnblendedCost']['Amount']
cost = float(cost)
cost = round(cost, 3)
if cost > 0:
cost = '$' + str(cost)
resources['Service'].append(service)
resources['Usage Type'].append(usage_type)
resources['Cost'].append(cost)
df = pd.DataFrame(resources)
html_table_weekly = df.to_html(index=False)
message_weekly = f'Total Cost of AWS all account for Platform the last week (from {str_one_week_ago} to {str_today}) was'
html_weekly = f"""
<html>
<head>
<style>
body {{
font-family: Arial, sans-serif;
color: white;
background-color: black;
}}
h2 {{
color: white;
font-size: 25px;
text-align: center;
}}
h1 {{
color: #333333;
font-size: 40px;
text-align: center;
background-color: yellow;
}}
p {{
color: white;
font-size: 30px;
line-height: 1.5;
margin-bottom: 20px;
text-align: center;
}}
p1 {{
font-size: 10px;
text-align: center;
margin-left: auto;
margin-right: auto;
}}
table {{
width: 100%;
border-collapse: collapse;
}}
th, td {{
border: 1px solid white;
padding: 8px;
text-align: left;
}}
th {{
background-color: #333333;
color: white;
}}
tr:nth-child(even) {{
background-color: #555555;
}}
</style>
</head>
<body>
<p> ----- AWS All Account report for Team Platform (from {str_one_week_ago} to {str_today}) ----- </p>
<h2> {message_weekly} </h2>
<h1> <strong> <em> {total_cost_weekly} </em></strong> </h1>
<h3> ----- The detailed cost by service can be found in table below ----- </p>
<table>
{html_table_weekly}
</table>
<p1></p1>
</body>
</html>
"""
# Log email content
print("Email Content:")
print(html_weekly)
# Email sending
recipients = ['email@domain.com','email2@domain.com']
for recipient in recipients:
message = {
'Subject': {'Data': 'AWS all account cost report'},
'Body': {'Html': {'Data': html_weekly}}
}
try:
response = ses_client.send_email(
Source='email@domain.com',
Destination={'ToAddresses': [recipient]},
Message=message
)
print(f"Email sent successfully to {recipient}. Message ID: {response['MessageId']}")
print("Response:")
print(response)
except ClientError as e:
print("Error sending email to {}: {}".format(recipient, e))Here is a visual representation of how the email will appear when it is sent to your designated email addresses.
In this case, you are creating an AWS Lambda function that utilizes the Cost Explorer API and Amazon Simple Email Service (SES) to retrieve and email your AWS cost data. The Lambda function will be written in Python 3.9 and will need the appropriate permissions to access the Cost Explorer API and SES. It is important to carefully select an IAM role with the necessary permissions, rather than using the highly-permissive “AdministratorAccess” policy.
Step 2: Setting up AWS SES
Before you can use SES to send emails, you will need to verify the email address that you want to use. This ensures that you have permission to send emails from that address and helps to protect the integrity of the service. To verify an email address, simply visit the Amazon SES dashboard and follow the prompts to complete the verification process.
Steps:
- Log in to the SES console.
- Under Configuration, choose Verified Identities.
- Choose Create Identity.
- Choose the identity type Email Address, fill out the Email address field with the sender email, and choose Create Identify.
- Repeat this step for all receiver emails.
The email IDs included will receive an email for confirmation. Once confirmed, the status shows as verified in the Verified Identities tab of the SES console. The verified email IDs will start receiving the email with the cost reports.
Step 3: Configuring AWS EventBridge
Running the function Periodically
To automate the execution of your AWS Lambda function regularly, you can use the Amazon EventBridge service to create a scheduled event. EventBridge supports two types of schedules: cron-based and rate-based. The cron-based schedule offers more flexibility, but the rate-based schedule is simpler to set up and may be more suitable for certain use cases.
In this case, I have chosen to use a rate-based schedule to trigger Lambda function. With this type of schedule, I can specify the frequency at which the event should occur, such as once per day or once per week. This allows you to easily run your function on a predetermined schedule, without the need to manually invoke it each time.
The scheduled pattern that I used is shown below.
To complete the process of setting up your scheduled Lambda function, you will need to specify the target for the event. In this case, you will select the “Lambda” API as the target and then choose your Lambda function from the dropdown menu. This will associate your function with the scheduled event so that it will be executed according to the schedule you have configured.
Once you have selected the target and chosen your Lambda function, you can proceed to create the schedule. This will activate the event and your function will begin executing according to the schedule you have specified. By using EventBridge and a scheduled event, you can automate the execution of your Lambda function and ensure that your AWS cost and usage data is regularly retrieved and processed.
To wrap up, it is important to regularly monitor and manage your AWS resources and costs to avoid unexpected expenses and optimize your use of the platform. By using tools like the AWS Cost Explorer, cost allocation tags, and the AWS Budgets service, you can gain a better understanding of your resource usage and costs and identify opportunities to reduce expenses.
Instructions:
- Navigate to the AWS EventBridge console.
- Create a new rule with a schedule expression to trigger the Lambda function weekly.
- Add the Lambda function as the target for the EventBridge rule.
Conclusion
Automating AWS cost monitoring and reporting is essential for maintaining control over your AWS spending and optimizing resource utilization. By leveraging AWS Lambda, SES, and EventBridge, you can set up a robust system that delivers regular cost reports directly to your inbox, allowing you to stay informed and proactive in managing your AWS costs.
In this blog post, we’ve provided a step-by-step guide along with code snippets and instructions to help you set up your own automated cost monitoring and reporting system in AWS. With this system in place, you can gain valuable insights into your AWS spending and take timely actions to optimize costs and maximize efficiency.
Please do suggest if any changes or corrections are required. You can reach me for any queries.
Creator :
If you like this blog and find it helpful then please do share and do not forget to clap and follow me on Medium & Linkedin here.
