ELB to backend server using HTTPS with self-signed certificate

Question

I am looking to setup an ELB that uses HTTPS to communicate with backend servers. I am trying to setup a proof of concept using a single backend server, but can't seem to get the ELB to communicate with server. I am almost certain this is a certificate issue since any setup without SSL works perfectly.

How can I set this up? I have tried various suggestions from multiple answers and blog posts, but no luck.

What I am doing now is setting a self-signed certificate using the following commands (from AWS ELB -> Backend Server over HTTPS with Self-Signed Certificate):

$ openssl genrsa \
  -out /path/to/ssl.key 2048
$ openssl req \
  -sha256 \
  -new \
  -key /path/to/ssl.key \
  -out /path/to/ssl.csr
$ openssl x509 \
  -req \
  -days 365 \
  -in /path/to/ssl.csr \
  -signkey /path/to/ssl.key \
  -out /path/to/ssl.crt

I have tried multiple domain names, when signing, and I can curl using them:

curl https://[Public DNS, or private DNS or IP used to create the SSL crt]/status --cacert /path/to/ssl.crt

Is there a domain/IP/DNS entry I should use here? I feel pretty good that curl works at least.

Currently my nginx config (in a site-enabled file) looks like this:

server {
    listen 443 ssl;
    listen [::]:443 ssl;

    ssl_certificate     /path/to/ssl.crt;
    ssl_certificate_key /path/to/ssl.key;

    server_name <dummy value of "_" or name used to make SSL certs>;
    client_max_body_size 20M;
    access_log  /var/log/nginx/access.log;
    error_log  /var/log/nginx/error.log;

    location / {
        proxy_pass         http://127.0.0.1:8000/;
        proxy_redirect     off;

        proxy_set_header   Host             $host;
        proxy_set_header   X-Real-IP        $remote_addr;
        proxy_set_header   X-Forwarded-For  $proxy_add_x_forwarded_for;
    }
}

This works with the curl command above.

I have tried classic and application load balancers. With classic, I have tried adding the ssl.crt contents - I do not see a similar option with the application load balancers, though I would like to use them if possible because they can forward HTTP->HTTPS really easily. Either way, neither the classic or application load balancer is communicating with the server.

Any suggestions for what is missing? Or how to determine what is missing?

Answer 1

This was close, just a few small steps missing. I got this working with an ALB ELB.

First, I used a script similar to the one described here: https://myopswork.com/how-to-do-end-to-end-encryption-of-data-in-transit-b-w-aws-alb-and-ec2-3b7fd917cddd

#!/bin/bash

DIR=$(dirname $0)

domain=$(uname -n)
echo "Generating SSL for $domain"
commonname="$domain"
country="US"
state="California"
locality="LA"
organization="My Inc."
organizationalunit="Org"
email="my@email.com"

# Optional
password=dummypassword

echo "Generating key request for $domain"

mkdir -p /etc/ssl/private
chmod 700 /etc/ssl/private
mkdir -p /etc/ssl/certs

# Generate a key
openssl genrsa -des3 -passout pass:$password -out /etc/ssl/private/$domain.key 2048 -noout

# Remove passphrase from the key. Comment the line out to keep the passphrase
echo "Removing passphrase from key"
openssl rsa -in /etc/ssl/private/$domain.key -passin pass:$password -out /etc/ssl/private/$domain.key

# Create the request
echo "Creating CSR"
openssl req -new -key /etc/ssl/private/$domain.key -out /etc/ssl/private/$domain.csr -passin pass:$password \
    -subj "/C=$country/ST=$state/L=$locality/O=$organization/OU=$organizationalunit/CN=$commonname/emailAddress=$email"

# Create the cert
openssl x509 -req -days 365 -in /etc/ssl/private/$domain.csr -signkey /etc/ssl/private/$domain.key -out /etc/ssl/certs/$domain.crt

# Setup nginx config
sed "s/{{hostname}}/${domain}/" < $DIR/template.conf > /etc/nginx/sites-available/site.conf
ln -sf /etc/nginx/sites-available/site.conf /etc/nginx/sites-enabled/site.conf

The template looked something like this:

server {
    # listen 80 #uncomment to also listen on port 80 - useful for debugging
    listen 443 ssl;
    listen [::]:443 ssl;
    server_name {{hostname}};

    ssl_certificate /etc/ssl/certs/{{hostname}}.crt;
    ssl_certificate_key /etc/ssl/private/{{hostname}}.key;
    ssl_protocols TLSv1 TLSv1.1 TLSv1.2;
    ssl_prefer_server_ciphers on;
    ssl_ciphers "EECDH+AESGCM:EDH+AESGCM:AES256+EECDH:AES256+EDH";

    add_header Strict-Transport-Security "max-age=63072000; includeSubdomains";
    add_header X-Frame-Options sameorigin;
    add_header X-Content-Type-Options nosniff;

    location / {
        ...
    }
}

The domain looked something like ip-172-10-11-12.

To debug everything I ran something like the following - this is from memory so it may have details off. I began by making sure I could curl the server locally by hitting nginx:

curl https://ip-172-10-11-12/healthcheck --cacert /etc/ssl/certs/ip-172-10-11-12.crt

Then I got the ELB address, and make sure I could curl against that. I had to go on a machine that could access the ELB machine. Note that due to security rules, the ELB was not pinagable, but was curl-able. I believe I tested this 2 ways. First, I tried:

curl https://elb-address/healthcheck --insecure

Then I added ip-172-10-11-12 to the /etc/hosts file and tried:

curl https://ip-172-10-11-12/healthcheck --cacert /cert/file/copied/onto/machine

Once I got that working, the ALB ELB started working. I had to check firewall rules, AWS security groups, etc. before this last call worked. But when it did work, the ELB started seeing the server.

I also had 1 final insight while debugging this: If the ELB is being accessed from the public internet, the ELB must only have public subnets, and the public subnets should be in the same availability zone as target machines