How to use python ElasticSearch package with ECS Instance Role instead of passing credentials?

问题

I've built an app that queries ElasticSearch at various times. Now that I'm moving the app to be hosted on a ECS, I want to switch my queries to be based on the Instance Role permissions rather than credentials that I've been passing as environment variables.

Based on the docs, if I pass no credentials to Boto3, I ought to get use my Instance Role. But in order to send my query using the ElasticSearch package in python, I have to pass credentials. Is there a way to do this using Instance Roles?

The example code here gave me hope that I would just pass in the `boto3.Session().get_credentials() function to retrieve the Instance Role credentials. But it turns out, as explained by my aws admin, that Instance Roles do not actually have credentials (aws_key, aws_secret_key). As a result my code seems to generate an AWS key which is not valid on the cloud and returns only null values for my queries.

I've been told that the only alternative is to use the ES Boto3 client and not pass credentials...but using ES with this client is not nearly as friendly as using the ElasticSearch python package interface.

Is there a way to make it work where the ElasticSearch package could query the ES without passing credentials and only relying on the Instance Role permissions?

Here is my code so far:

if os.environ.get('LOCAL_DEV')==1:
  AWS_KEY = os.environ['AWS_ACCESS_KEY']
  AWS_SECRET_KEY = os.environ['AWS_SECRET_KEY']
else:
  credentials = boto3.Session().get_credentials() #fails to use instance role
  AWS_KEY=credentials.access_key
  AWS_SECRET_KEY=credentials.secret_key

# It pulls my environment variables when running locally, and when running on the cluster, I need it to use the Instance Role
  
#### below is how I initialize and query with the ElasticSearch package


import elasticsearch as es
from requests_aws4auth import AWS4Auth
from constants import AWS_KEY, AWS_SECRET_KEY

awsauth = AWS4Auth(AWS_KEY, AWS_SECRET_KEY, 'us-west-2', 'es') 
#requires a credential input
host ='search-xxx.xxx.xxxxxxx.xxxx.amazonaws.com'
e_s = es.Elasticsearch(
            hosts=[{'host': host, 'port': 443}],
            http_auth=awsauth,
            use_ssl=True,
            verify_certs=True,
            connection_class=es.RequestsHttpConnection
            )

   res = e_s.search(index='foo', body='bar')
    res = res['hits']['hits']

Any help on this would be very appreciated

回答1:

What I ended up doing was pulling temporary credentials using boto3 that were based on the the InstanceRole permissions. Since these are temporary credentials, they need to be rotated. However, it worked! I was stumped until I discovered the need to use a temporary token in the AWS4Auth command

temp_access=boto3.Session(region_name='foo-region').get_credentials().access_key
temp_secret=boto3.Session(region_name='foo-region').get_credentials().secret_key
temp_token=boto3.Session(region_name='foo-region').get_credentials().token

########
from requests_aws4auth import AWS4Auth
awsauth=AWS4Auth(temp_access, temp_secret, 'us-west-2', 'es', session_token=temp_token)

es.Elasticsearch(hosts=[{'host':host, 'port':443}],
                 http_auth=awsauth, 
                 use_ssl=True, 
                 verify_certs=True, 
                 connection_class=es.RequestsHttpConnection)

This article was very helpful

来源：https://stackoverflow.com/questions/62646560/how-to-use-python-elasticsearch-package-with-ecs-instance-role-instead-of-passin

标签

python

amazon-web-services

ElasticSearch

boto3

amazon-ecs