Python SDK for AWS — Boto3

Hi Everyone,

In this article , I want to write about Boto3 and how super useful it is , using a case study.

Suppose you have a task to download a file from S3 bucket in Account A and upload it to another S3 bucket on a different Account B on AWS. There are 2 ways to do it, the Hard way and the Smart way.

The Hard Way -

Login to AWS Account A and download the file to your Local. Next log out of Account A and login to Account B on AWS . We upload the file that was download and voila , its done. But what if this has to be done on a daily basis? Lets assume this process takes 10 mins everyday. So on a week , you are wasting about an hour of your time. So definitely , lets AUTOMATE !

The Smart Way -

We write a script that tells the AWS to do the exact same task for us! But wait, how do we communicate with the AWS account via Scripts?

Boto3

Boto3 is the Python SDK for AWS. So what it essentially means is that , the Boto3 package from Python contains APIs that helps to communicate with AWS.

Fun Fact : Boto was named after the fresh water dolphin native to the Amazon river.

Prerequisites:

• Install boto3 package using the command

pip install boto3

• Make sure the current folder has a hidden directory ‘.aws ‘ . This directory must have 2 files — config and credentials, to store info about AWS accounts.

and that’s it!

Basic Components of Boto3

• A Session will be created , for the operations to take place. Unlike the usual process, we do not have to close the session. Typically, the profile information is passed to create a session. Keyword “Session”
• Client/Resource — Both of these objects gets the job done. Resource can only be used for limited services of AWS. Client gives the response in JSON(dictionary) format and hence a bit more complicated. Keyword “client”/”resource”

Example File using Client Object

I had a use case where I wanted to download the latest file from an S3 bucket in Account A and upload it to an S3 bucket in Account B.

Lets see how we can achieve this.

Firstly, we need to check if boto3 is installed in our system. This can be verified using the command

boto3 --version

Next , we check for .aws folder. Ensure that the profile credentials are configured on credentials file , and configurations (like region and profile name ) is configured inside config file. Both these files should be present inside .aws folder.

Next , the very interesting Scripting part , s3_transfer.py :

import boto3

#Creating a session
session = boto3.Session(profile_name='<name>')
s3_source = session.client('s3')

#fetching latest file from source S3 Bucket using a lambda 
get_last_modified = lambda obj: int(obj['LastModified'].strftime(%s))
objs= s3_source.list_objects_v2(Bucket='<source bucket>')['Contents']

#fetching filename 
last_added_file = [obj['Key'] for obj in sorted(objs, key=get_last_modified)][0]

#downloading file from S3
s3_source.download_file('<source bucket>',<filename> last_added_file,<path in s3>
print ("Downloaded the file to local")

#creating a new session and uploading the file
session2 = boto3.Session(profile_name='<name>')
s3_dest = session2.client('s3')
with open('<filename',"rb") as f:
  s3_dest.upload_fileobj(f, <destination bucket> , filename)

Once the script is ready, you run the file using the simple command

python3 s3_transfer.py

And there you have it! Automation has saved much of your time!!

Next part will include my favorite part about Paginators :)