Automate AWS Infrastructure with Boto 3

 When I first joined a Software Design team, I realized there were a lot of simple AWS infrastructure changes that took up a large chunk of our engineering team’s time. I didn’t want to spend my valuable coding time on these manual, yet essential, tasks so I set out on a mission to automate them. Since I had wanted to build my Python scripting skills anyway, I discovered a way to solve two problems at once — using the software development kit Boto 3 to automate my simple, manual AWS tasks using Python.

Boto 3 is your new best friend when it comes to creating Python scripts for AWS resources. According to the documentation, “Boto is the Amazon Web Services (AWS) SDK for Python. It enables Python developers to create, configure, and manage AWS services, such as EC2 and S3. Boto provides an easy to use, object-oriented API, as well as low-level access to AWS services.”

Before working with Boto 3 it is important to correctly download and configure it. Check out their instructions on how to set your credentials.

Boto 3 makes it easy to get all the information you need from your AWS resources as well as to modify them. To get started with Boto 3, first create a Python file which will act as the script and import the SDK.

import boto3

You can simply use it to create a “client” with the AWS resource you would like to extract information from or modify. Here I am creating a client for EC2 instances.

ec2_client = boto3.client('ec2')

You can then use this client to perform any of the methods listed in the EC2 service section of the Boto 3 documentation. This documentation is super helpful in listing all of the resources you can use, their various methods, and how exactly to configure these methods. Feel free to explore!

To get the most out of learning about AWS scripting using Boto 3, I figured I would walk you through three different use cases (each in a separate article) to show why you may want to use Boto 3, and how to do so. In this, the first installment, we’re going to automate adding tags to instances.

Writing the Script

Imagine you have hundreds of instances in your AWS account and you want to add a tag so that they will only run during certain times of the day. However, you don’t want to spend time manually adding the tag to every instance because that would take hours. Luckily, there’s an easy solution!

You can use Boto 3 to get the names of all of the instances in your account and then add the tag to each one.

First, create the EC2 client.

ec2_client = boto3.client('ec2')

Next, we need to find the method that will help us find the names of our instances. Unfortunately, there is no method that simply lists the names of your instances. The closest thing to getting what we want is describe_instances().

response = ec2_client.describe_instances()

This will return all kinds of information about our instances. What we are really looking for is simply the name, so we will have to sift through all of the data this returns us.

But, after looking through all of the data, we still can’t seem to find the instance name… That’s because describe_instances() doesn’t give us the name! We actually want to use the InstanceId.

But first we need to find the list that we actually want to parse through using a for-loop. Each item should represent an instance. In order to do that we need to see where all of the instance descriptions are located in the response.

instances = response['Reservations']

There we go! Now we can loop through each instance in the list of instance descriptions that we’ve collected.

Notice that it prints a bunch of key metrics that we don’t really care about in the context of this problem. Let’s narrow it down to print the instance_id.

for instance in instances:
    print(instances['Instances'][0]['InstanceId']

Now that we know how to find the instance_id for each instance in our account, let’s see how we would add a tag to an instance.

After looking through all of the methods available for EC2s it looks like create_tags would make the most sense for what we want to do. This method, according to the Boto 3 documentation, “adds or overwrites the specified tags for the specified Amazon EC2 resource or resources.” Notice you can add a completely new tag. Or, if you have one that already exists but has a wrong value, you can change that as well.

This method takes in a list of resources, which means we need to save the instance ids that we are collecting in our for-loop.

instance_ids = []for instance in instances:
    instance_ids.append(instance['Instances'][0]['InstanceId'])

This is better! Since the create_tags method takes in a list of resource ids, we created an empty list and appended the instance id to the list in each iteration of the for-loop. Now we can use the list instance_ids as the value for ‘Resources’ in the create_tags method.

tag_creation = ec2_client.create_tags(
    Resources = 
       instance_ids,
    Tags = [
       {
          'Key': 'operatingHours',
          'Value': 'start=(M-S,0);stop=(M-F,8);tz=et'
       }
    ]
)

This will place a tag called operatingHours on all of the instances. If you want to apply this tag to a small subset of instances, then you can use the filter option with the describe_instances method you used earlier.

response = ec2_client.describe_instances(
            Filters = [
                {
                      'Name': 'group-name',
                      'Values': [
                          'security-test',
                      ]
                },
            ]
)

Here is an example of this. You can filter by security group name using the ‘group-name’ filter. This command would look for all instances that have the security group ‘security-test’.

Viola! Your operating hour tags are now added to all of your instances.

Check out the next installment in our series where we’ll cover automating AWS health checks.