Boto: Dynamically get aws_access_key_id and aws_secret_access_key in Python code from config?

Question

I have my aws_access_key_id and aws_secret_access_key stored in ~/.boto and was wondering if there was a way for me to retrieve these values in my python code using Boto as I need to insert them in to my SQL statement to to copy a CSV file from S3.

Answer 1

This should work:

import boto
access_key = boto.config.get_value('Credentials', 'aws_access_key_id')
secret_key = boto.config.get_value('Credentials', 'aws_secret_access_key')

Answer 2

Here's a helper that will look in ~/.aws/credentials if boto.config doesn't work. I didn't look into it in great detail, but it kind of appears that Boto 2 doesn't look in ~/.aws/credentials.

def get_aws_credentials():
    # I think this will look in ~/.boto ([Credentials] section)
    aws_access_key_id = boto.config.get_value("Credentials", 'aws_access_key_id')
    aws_secret_access_key = boto.config.get_value("Credentials", 'aws_secret_access_key')

    # I don't think Boto 2 looks in ~/.aws/credentials, so we look
    if aws_access_key_id is None or aws_secret_access_key is None:
        with open(os.path.expanduser("~/.aws/credentials")) as f:
            for line in f:
                try:
                    key, val = line.strip().split('=')
                    if key == 'aws_access_key_id':
                        aws_access_key_id = val
                    elif key == 'aws_secret_access_key':
                        aws_secret_access_key = val
                except ValueError:
                    pass
        return aws_access_key_id, aws_secret_access_key

Answer 3

Because the aws credentials & boto files both use the .ini format, you can parse them with ConfigParser. Here's an example of parsing the ~/.aws/credentials file (this is python 2, but should be easy enough to port to python 3):

from os.path import expanduser
import ConfigParser

def read_credentials_from_config_section(section_name):
    # parsing ~/.aws/credentials but it's just as easy to parse ~/.boto
    aws_credentials_path = os.path.join(expanduser("~"), '.aws', 'credentials')
    c = ConfigParser.ConfigParser()
    c.read(aws_credentials_path)
    return c.get(section_name, 'aws_access_key_id'), c.get(section_name, 'aws_secret_access_key')

Use via:

k, s = read_credentials_from_config_section('default')

If you want to use the ~/.boto file, modify the above code to read the ~/.boto file, and adjust for its naming conventions -- the code is very similar.

An alternative way to read the ~/.aws/credentials file (assuming you have awscli installed) is to shell out to the aws cli and let it deal with the details. This is a lot slower, though (takes ~1.5s to run on my machine, which is unacceptable for a lot of use cases).

import subprocess
print subprocess.check_output(['aws', 'configure', 'get', 'aws_access_key_id', '--profile', aws_profile_name])
print subprocess.check_output(['aws', 'configure', 'get', 'aws_secret_access_key', '--profile', aws_profile_name])