I am posting a json to AWS elasticsearch,using a java lambda function.
public Object handleRequest(DynamodbEvent dynamodbEvent, Context context) { //code to general the json document AmazonDynamoDBClient amazonDynamoDBClient = new AmazonDynamoDBClient(); List<DynamodbEvent.DynamodbStreamRecord> dynamodbStreamRecordlist = dynamodbEvent.getRecords(); if (!dynamodbStreamRecordlist.isEmpty()) { DynamodbEvent.DynamodbStreamRecord record = dynamodbStreamRecordlist.get(0); if(record.getEventSource().equalsIgnoreCase("aws:dynamodb")) tableName = getTableNameFromARN(record.getEventSourceARN()); } LaneAnnotation laneAnnotation = new LaneAnnotation(); ScanRequest scanRequest = new ScanRequest().withTableName(tableName); ScanResult result = amazonDynamoDBClient.scan(scanRequest); List<Lines> linesFinalList = new ArrayList<Lines>(); if(result != null) { for (Map<String, AttributeValue> item : result.getItems()) { //code for looping through the table items and generating a json object for the elastic search model } //Code to post the json below - RestTemplate restTemplate = new RestTemplate(); SimpleClientHttpRequestFactory clientHttpRequestFactory = (SimpleClientHttpRequestFactory)restTemplate.getRequestFactory(); clientHttpRequestFactory.setConnectTimeout(10000); clientHttpRequestFactory.setReadTimeout(10000); HttpEntity<String> entity = new HttpEntity<String>(<json goes here>, headers); try{ restTemplate.exchange(endpoint, HttpMethod.POST, entity, String.class); }catch(Exception e){ e.printStackTrace(); } } However, I see the following error when I test my AWS lambda function -
org.springframework.web.client.HttpClientErrorException: 403 Forbidden at org.springframework.web.client.DefaultResponseErrorHandler.handleError(DefaultResponseErrorHandler.java:91) at org.springframework.web.client.RestTemplate.handleResponse(RestTemplate.java:700) at org.springframework.web.client.RestTemplate.doExecute(RestTemplate.java:653) at org.springframework.web.client.RestTemplate.execute(RestTemplate.java:613) at org.springframework.web.client.RestTemplate.exchange(RestTemplate.java:531) at com.here.aws.LambdaApplication.handleRequest(LambdaApplication.java:166) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at lambdainternal.EventHandlerLoader$PojoMethodRequestHandler.handleRequest(EventHandlerLoader.java:456) at lambdainternal.EventHandlerLoader$PojoHandlerAsStreamHandler.handleRequest(EventHandlerLoader.java:375) at lambdainternal.EventHandlerLoader$2.call(EventHandlerLoader.java:1139) at lambdainternal.AWSLambda.startRuntime(AWSLambda.java:285) at lambdainternal.AWSLambda.<clinit>(AWSLambda.java:57) at java.lang.Class.forName0(Native Method) at java.lang.Class.forName(Class.java:348) at lambdainternal.LambdaRTEntry.main(LambdaRTEntry.java:94) I even modified the access policy and added my IP address. Have others faced this too? How did you resolve it?> Any help will be appreciated.
EDIT1: I am now trying to incorporate signing of the request as is mentioned here - https://aws.amazon.com/blogs/security/how-to-control-access-to-your-amazon-elasticsearch-service-domain/
Will report back if it goes well.
EDIT2:
Here's the second way of sending a request that I tried referring to the link above-
@Override public Object handleRequest(DynamodbEvent dynamodbEvent, Context context) { AmazonDynamoDBClient amazonDynamoDBClient = new AmazonDynamoDBClient(); List<DynamodbEvent.DynamodbStreamRecord> dynamodbStreamRecordlist = dynamodbEvent.getRecords(); if (!dynamodbStreamRecordlist.isEmpty()) { DynamodbEvent.DynamodbStreamRecord record = dynamodbStreamRecordlist.get(0); if(record.getEventSource().equalsIgnoreCase("aws:dynamodb")) tableName = getTableNameFromARN(record.getEventSourceARN()); } LaneAnnotation laneAnnotation = new LaneAnnotation(); ScanRequest scanRequest = new ScanRequest().withTableName(tableName); ScanResult result = amazonDynamoDBClient.scan(scanRequest); List<Lines> linesFinalList = new ArrayList<Lines>(); if(result != null) { for (Map<String, AttributeValue> item : result.getItems()) { //Generate the json object that needs to be sent in the request } HttpHeaders headers = new HttpHeaders(); headers.setContentType(MediaType.APPLICATION_JSON_UTF8); Request<?> request = new DefaultRequest<Void>(SERVICE_NAME); request.setContent(new ByteArrayInputStream(elasticSearchModel.toString().getBytes())); request.setEndpoint(URI.create(endpoint)); request.setHttpMethod(HttpMethodName.POST); AWS4Signer signer = new AWS4Signer(); signer.setServiceName(SERVICE_NAME); signer.setRegionName(Regions.US_EAST_1.getName()); AWSCredentialsProvider credsProvider = new DefaultAWSCredentialsProviderChain(); AWSCredentials creds = credsProvider.getCredentials(); // Sign request with supplied creds signer.sign(request, creds); log.info("Request signed"); ExecutionContext executionContext = new ExecutionContext(true); ClientConfiguration clientConfiguration = new ClientConfiguration(); AmazonHttpClient client = new AmazonHttpClient(clientConfiguration); MyHttpResponseHandler<Void> responseHandler = new MyHttpResponseHandler<Void>(); MyErrorHandler errorHandler = new MyErrorHandler(); Response<Void> response = client.execute(request, responseHandler, errorHandler, executionContext); return dynamodbEvent; } However, I get the following error -
Check the signature you provided. Check your AWS Secret Access Key and signing method. Consult the service documentation for details. The Canonical String for this request should have been 'GET / host:somehostname-XXXXXXXXXXXXXXXX.us-east-1.es.amazonaws.com x-amz-date:20170130T105736Z x-amz-security-token:FQoDYXdzEG4aDJJ4ryjXXXXXXXXXXXXXXXX/auMHooYENY6YXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX host;x-amz-date;x-amz-security-token e3b0c4429XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' The String-to-Sign should have been 'AWS4-HMAC-SHA256 20170130T105736Z 20170130/us-east-1/es/aws4_request 9a5b4c92ec121c333f8cdXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' "}" 10:57:36.818 [main] DEBUG org.apache.http.headers - http-outgoing-1 << HTTP/1.1 403 Forbidden