How to correctly use Nginx as reverse proxy for multiple Apache Docker containers with SSL?

问题

Given the following docker containers:

• an nginx service that runs an unmodified official nginx:latest image
  • container name: proxy
• two applications running in separate containers based on modified official php:7.4.1-apache images
  • container names: app1 and app2
• all containers proxy, app1, and app2 are in the same Docker-created network with automatic DNS resolution

With the following example domain names:

• local.web.test
• local1.web.test
• local2.web.test

I want to achieve the following behavior:

• serve local.web.test from nginx as the default server block
• configure nginx to proxy requests from local1.web.test and local2.web.test to app1 and app2, respectively, both listening on port 80
• configure nginx to serve all three domain names using a wildcard SSL certificate

I experience two problems:

1. I notice the following error in the nginx logs:
   • 2020/06/28 20:00:59 [crit] 27#27: *36 SSL_read() failed (SSL: error:14191044:SSL routines:tls1_enc:internal error) while waiting for request, client: 172.19.0.1, server: 0.0.0.0:443
2. the mod_rpaf seems not to work properly (i.e., the ip address in the apache access logs is of the nginx server [e.g., 172.19.0.2] instead of the ip of the client that issues the request
   • 172.19.0.2 - - [28/Jun/2020:20:05:05 +0000] "GET /favicon.ico HTTP/1.0" 404 457 "http://local1.web.test/" "Mozilla/5.0 (Windows NTndows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/83.0.4103.116 Safari/537.36
   • the output of phpinfo() for Apache Environment shows that:
     • HTTP_X_REAL_IP lists the client ip
     • SERVER_ADDR lists the app1 container ip (e.g., 172.19.0.4)
     • REMOTE_ADDR shows the proxy container ip (e.g., 172.19.0.2) instead of the client ip

---

To make this reproducible, this is how everything is set up. I tried this on my Windows machine so there are two preliminary steps.

1. Preliminary steps

   a. in my C:\Windows\System32\drivers\etc\hosts file I added the following:

   127.0.0.1 local.web.test
   127.0.0.1 local1.web.test
   127.0.0.1 local2.web.test
   

   b. I generated a self-signed SSL certificate with the Common Name set to *.local.test via

   openssl req -x509 -sha256 -nodes -newkey rsa:2048 -days 365 -keyout localhost.key -out localhost.crt
   

2. The proxy service setup

   a. the nginx.yml for the docker-compose:

   version: "3.8"
   
   services:
       nginx:
           image: nginx:latest
           container_name: proxy
           ports:
               - "80:80"
               - "443:443"
           volumes:
               - ./nginx:/etc/nginx/conf.d
               - ./certs:/etc/ssl/nginx
               - ./static/local.web.test:/usr/share/nginx/html
           networks:
               - proxy
   networks:
       proxy:
           driver: bridge
   

   b. within ./nginx that is bind mounted at /etc/nginx/conf.d there is a file default.conf that contains:

   server {
       listen 80 default_server;
       server_name local.web.test;
   
       location / {
           return 301 https://$host$request_uri;
       }
   }
   
   server {
       listen 443 ssl;
       server_name local.web.test;
   
       location / {
           root   /usr/share/nginx/html;
           index  index.html index.htm;
       }
   
       ssl_certificate /etc/ssl/nginx/localhost.crt;
       ssl_certificate_key /etc/ssl/nginx/localhost.key;
   }
   

   c. the ./certs:/etc/ssl/nginx bind mounts the folder containing the self-signed certificate and key

   d. the ./static/local.web.test:/usr/share/nginx/html makes available a file index.html that contains

   <h1>local.web.test</h1>
   

3. The app1 and app2 services setup

   a. the apache.yml for the docker-compose:

   version: "3.8"
   
   services:
       app1:
           build:
               context: .
               dockerfile: apache.dockerfile
           image: app1
           container_name: app1
           volumes:
               - ./static/local1.web.test:/var/www/html
           networks:
               - exp_proxy
   
       app2:
           build:
               context: .
               dockerfile: apache.dockerfile
           image: app2
           container_name: app2
           volumes:
               - ./static/local2.web.test:/var/www/html
           networks:
               - exp_proxy
   
   networks:
       # Note: the network is named `exp_proxy` because the root directory is `exp`.
       exp_proxy:
           external: true
   

   b. the apache.dockerfile image looks like this:

   # Base image.
   FROM php:7.4.1-apache
   
   # Install dependencies.
   RUN apt-get update && apt-get install -y curl nano wget unzip build-essential apache2-dev
   
   # Clear cache.
   RUN apt-get clean && rm -rf /var/lib/apt/lists/*
   
   # Change working directory,
   WORKDIR /root
   
   # Fetch mod_rpaf.
   RUN wget https://github.com/gnif/mod_rpaf/archive/stable.zip
   
   # Unzip.
   RUN unzip stable.zip
   
   # Change working directory,
   WORKDIR /root/mod_rpaf-stable
   
   # Compile and install.
   RUN make && make install
   
   # Register the module for load.
   RUN echo "LoadModule rpaf_module /usr/lib/apache2/modules/mod_rpaf.so" > /etc/apache2/mods-available/rpaf.load
   
   # Copy the configuration for mod_rpaf.
   COPY ./apache/mods/rpaf.conf /etc/apache2/mods-available/rpaf.conf
   
   # Enable the module.
   RUN a2enmod rpaf
   
   # Set working directory.
   WORKDIR /var/www/html
   

   c. the file ./apache/mods/rpaf.conf copied contains:

   <IfModule mod_rpaf.c>
       RPAF_Enable             On
       RPAF_Header             X-Real-Ip
       RPAF_ProxyIPs           127.0.0.1
       RPAF_SetHostName        On
       RPAF_SetHTTPS           On
       RPAF_SetPort            On
   </IfModule>
   

   d. the ./static/local1.web.test:/var/www/html bind mounts an index.php file containing:

   <h1>local1.web.test</h1>
   <?php phpinfo(); ?>
   

   the same goes for ./static/local2.web.test:/var/www/html

   e. the 000-default.conf virtual hosts in app1 and app2 are not modified:

   <VirtualHost *:80>
       ServerAdmin webmaster@localhost
       DocumentRoot /var/www/html
       ErrorLog ${APACHE_LOG_DIR}/error.log
       CustomLog ${APACHE_LOG_DIR}/access.log combined
   </VirtualHost>
   

4. Starting the setup

   a. start the proxy server

   docker-compose -f nginx.yml up -d --build
   

   b. start the app1 and app2 services

   docker-compose -f apache.yml up -d --build
   

   c. check containers to see if mod_rpaf is enabled

   docker-compose -f apache.yml exec app1 apachectl -t -D DUMP_MODULES 
   

   d. add two files in ./nginx that will be available on the proxy container at /etc/nginx/conf.d

   • local1.web.test.conf containing

   upstream app-1 {
       server app1;
   }
   
   server {
       listen 80;
       server_name local1.web.test;
   
       location / {
           return 301 https://$host$request_uri;
       }
   }
   
   server {
       listen 443 ssl;
       server_name local1.web.test;
   
       location / {
           proxy_pass http://app-1;
           proxy_set_header Host $host;
           proxy_set_header X-Real-IP $remote_addr;
           proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
           proxy_set_header X-Forwarded-Proto $scheme;
       }
   
       ssl_certificate /etc/ssl/nginx/localhost.crt;
       ssl_certificate_key /etc/ssl/nginx/localhost.key;
   }
   

   • the second file is local2.web.test.conf with a similar setup (i.e., number 1 is replaced with 2)

   e. check the config and restart the proxy container (or reload the nginx server)

   docker-compose -f nginx.yml exec proxy nginx -t 
   docker-compose -f nginx.yml exec proxy service nginx reload
   

The issues:

1. when I run docker logs proxy -f I notice the SSL internal error mentioned above: SSL_read() failed
   • someone faced a similar error (http2: SSL read failed while sending req in nginx) but in that case, the message more specifically points to the certificate authority
2. if I run docker logs app1 -f and visit https://local1.web.test, the ip in the GET request matches the ip of the proxy container (i.e., 172.19.0.2) and not that of the remote client
   • I suspect the cuprit is this RPAF_ProxyIPs 127.0.0.1, but I can't manually fix the ip because I don't know what ip the container will get in the exp_proxy network
   • also I can't use the hostname because RPAF_ProxyIPs expects an ip
   • docker inspect proxy shows "IPAddress": "172.19.0.2"
   • docker inspect app1 shows "IPAddress": "172.19.0.4"

I can't seem to understand what goes wrong and would appreciate your help.

来源：https://stackoverflow.com/questions/62647604/how-to-correctly-use-nginx-as-reverse-proxy-for-multiple-apache-docker-container