My worker tier returns 400 Error, CloudWatch Error?

Question

I have a worker tier Elastic Beanstalk Application connected to a SQS. Over, and over and over again I keep getting HTTP 400 Error returned in my log, but I don't get any error messages at all in the log.

[14/Apr/2014:18:03:26 +0000] "POST /customer-registered HTTP/1.1" 400 192 "-" "aws-sqsd"

I dont get any errors at error_log, the only error I can find in my log is the following, which is located at /var/log/aws-sqsd/default.log:

2014-04-14T18:02:58Z error: AWS::CloudWatch::Errors::AccessDenied: User: arn:aws:sts::809571490243:assumed-role/aws-elasticbeanstalk-ec2-role/i-a00fffe2 is not authorized to perform: cloudwatch:PutMetricData

If I go to my IAM User (I only have 1) I have the following policy:

{
   "Version": "2012-10-17",
   "Statement": [
   {
     "Effect": "Allow",
     "Action": "*",
     "Resource": "*"
   }
]
}

I'm not sure if they refer to this user in the error message when it says "elasticbeanstalk-ec2-role", but I cant find any other user profiles. I'm also unsure if this is even the error that is causing any issues.

Here is the code I'm trying to run: http://pastebin.com/Tsnnht8b . Nothing special. It works fine running on local machine.

My queue messages goes like {domain:"http://www.example.com"}

Any ideas?

EDIT: I was able to solve the CloudWatch:Errors:AccessDenied by adding a Role to the ec2. But that did not solve the problem, I still get HTTP 400 on all requests even if I simplify my code as much as I can, removing all extra modules except Flask and just trying to print "success" and return 200.

EDIT2: I have been able to get it down to that the issue seem to have something to do with json = request.get_json() or request.json. As soon as I use this line, I get the following information in my log:

[Tue Apr 15 06:01:12 2014] [notice] caught SIGTERM, shutting down
[Tue Apr 15 06:01:13 2014] [notice] suEXEC mechanism enabled (wrapper: /usr/sbin/suexec)
[Tue Apr 15 06:01:13 2014] [notice] Digest: generating secret for digest authentication ...
[Tue Apr 15 06:01:13 2014] [notice] Digest: done
[Tue Apr 15 06:01:13 2014] [notice] Apache/2.2.25 (Unix) DAV/2 mod_wsgi/3.4 Python/2.7.5 configured -- resuming normal operations

This even happens if I'm not even using the information that is recieved in the request. For example if I do the following:

domain = request.json
domain = "http://staticdomain.com"
...

I still get the error. So it seems to be involved in the request of the POST json that the SQS sends to my EC2.

Answer 1

I "solved" this by changing the SQS Settings to send the messages in text/plain instead of application/json. Then I recieve the data by the following:

domain = base64.b64decode(request.data) 

I use b64decode because the messages are base64 encoded. But point being that when sending the SQS Message in plain text, you fetch it by request.data.

It's not a fix to the json problem. I was not able to locate why json was causing these issues, but in my case it does not matter if I send it in json or text because it's just a single string that is being send. Hopefully this helps someone.

Answer 2

The following policy should give your EBS instance access to your SQS instance:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "QueueAccess",
            "Action": [
                "sqs:ChangeMessageVisibility",
                "sqs:DeleteMessage",
                "sqs:ReceiveMessage"
            ],
            "Effect": "Allow",
            "Resource": "*"
        },
        {
            "Sid": "MetricsAccess",
            "Action": [
                "cloudwatch:PutMetricData"
            ],
            "Effect": "Allow",
            "Resource": "*"
        }
    ]
}