Boto3 Concepts

Hi everyone,

In the previous article, we saw the basics of Boto3. Now exploring more important and may I say, luxury concepts of Boto3 :D

Waiters

Why do we need waiters ?

Suppose we have a service that we not just want to use but want to know the status of it , like if its in progress/completed. So, some of the AWS Resources have an object called Waiters. These waiters have various methods that can help us identify if the particular task is completed, by waiting for the task to wait until its completed. This helps to move onto next tasks in a synchronous matter.

What if it didn’t exist ?

Imagine we need to deploy EC2 instances and add into the Auto Scaling Group . We could write a script to first bring up an instance and add them to the ASG group. During the execution, after we run the command, it would obviously take some bring up the instances. But , as these commands run one after the another, it would not wait for the other command to finish and move to the command that we defined to add instances to the ASG. But, since the instances are just being bought up, it would result into an error . That’s the kind of impact, that this command can make.

Demonstrating Waiter concept

Here is a sample code to create a snapshot from an EC2 and wait until it has been created .

import boto3 

client = boto3.client('ec2', region_name='eu-west-1')
snapshot = client.create_snapshot(VolumeId='vol-xxxx')

snapshot_id = snapshot['SnapshotId']
# Create a reusable Waiter and pass the waiter operation
snapshot_waiter = client.get_waiter('snapshot_completed')

"""
 Pass the configurations for the Waiter object ( using wait() )
 created above so that the script knows who it needs to keep a check on :)
"""
snapshot_waiter.wait(
  SnapshotIds=[snapshot_id],
  WaiterConfig={
          'Delay': 123,
          'MaxAttempts': 123
      }
)

Here, we create a snapshot using the client object. The ‘create_snapshot’ is a method to create snapshot from the Volume ID .

• ‘get_waiter’ is a method that creates a waiter object for client object. Here , the name of the created waiter is ‘snapshot_waiter’ , which is a custom name that I have provided.
• A waiter object can have many methods under it. This particular waiter method ‘snapshot_completed’ waits until the snapshot is created.
• Here, the ‘wait’ method is associated with the custom waiter object that I have created and passes some parameters and configurations associated with it.
• WaiterConfig is A dictionary that provides parameters to control waiting behavior. Its an optional argument to have a controlled execution time.

Notes:

• Its good to implement this under try/catch block for Error Handling.
• There are N number of methods for waiters and hence its always best to refer the Documentation ,given you have a good understanding of the Skeleton of the Script.

Paginators

Why Paginators?

Let me tell you a use-case that we had to implement for Paginators. We had a CloudWatch Log Group and Log Streams. When we view the page from AWS Console, remember how we scroll down to view more logs? Well, to do it through API calls, Paginators come into picture :) As simple as that, but the impact it has is huge!

Just like Waiter objects, Paginator objects also needs to be created and can be given custom names, and can be passed with Arguments for the same.

• ‘get_paginator’ is a method that creates a paginator object for client object. Here , the name of the created paginator is ‘page_iterator’, which is a custom name that I have provided.

Here’s a sample code that iterates through the pages in the S3 bucket objects.

import boto3

client = boto3.client('s3', region_name='us-west-2')

# Create a reusable Paginator and pass the paginator operation
paginator = client.get_paginator('list_objects')


operation_parameters = {'Bucket': 'my-bucket',
                        'Prefix': 'foo/baz'}

# Create a PageIterator from the Paginator
page_iterator = paginator.paginate(**operation_parameters)

for page in page_iterator:
    print(page['Contents'])

Just like waiters, these are also available, only to a few resources in AWS.

Meta Objects

Why Meta Objects ?

Like I discussed in my previous article , there are 2 main types of object creation ie. either Client or Resources.

A major drawback of Resource object is that it may not have all the functionalities of a Client Object.

• The Resource version of the code is much simpler, more compact, and has more capability. It does pagination for you and it exposes properties (Hight level Abstraction)
• The Client has more methods for operating onto AWS Resources. It also prints output as dictionary(Low Level Abstraction)

So , as you can see, each has its benefits based on the use case.

So, there might be cases when you want to convert Resource object to Client , since Client provides a lower level abstraction and there are methods which might not be available for Resource.

Here is an example for it. Suppose we want to turn a Resource object to client:

import boto3 
# Creating Session
client = boto3.session.Session(profile='xxx')

# Creating a Resource Object
resource = session.resource('ec2')

#Converting a Resource to Client using meta 
client= resource.meta.client.<client method>

Now, this goes without saying, but , the other way wouldn’t make sense since the Client provides the Low Level Abstraction , and it has methods to perform more operations , compared to the Resource.

That’s about it for now. You now must have a strong basics on Boto3. We’ve looked into the greener side of coding. But there could be chances that , maybe, an S3 bucket that you’re trying to fetch files from, doesn’t exist . So how do we handle such cases? You’ll get to know about it very soon!

References:

1. Client vs Resource — https://www.learnaws.org/2021/02/24/boto3-resource-client