The compose templates provided by the SSDT do not publish ports for the applications outside of the docker host. Each ITC must decide how they are going to be mapping the Tomcat port from each application (port 8080) to make it available to external web browsers.

This article explains how to use nginx-proxy to create a reverse proxy which automatically updates as containers are started and stopped. Note this is just one option for the reverse proxy.

The goal of this article is to

start with a basic reverse proxy
add SSL encryption with a signed (or self-signed) certificate
Optionally, use LetsEncrypt.org to automatically generate signed certificates

Tutorial

Below are steps to configure nginx-proxy on your docker host. The tutorial assumes the following:

Docker hostdocker-01.example.comsub-domain for virtual hostsdemo.example.com

Create a Reverse Proxy on port 80

Wildcard DNS: Create DNS records in the domain to point to the docker host.
```
*.demo.example.com   IN    CNAME docker-01.example.com
```
This wildcard will be used for all virtual hosts on this machine. If you intend to also use other subdomains for this host, you can add hostnames in other domains. For example, if “sampletown” has their own domain, then you could also add these RR’s to sampletown’s domain:
```
usas                 IN CNAME docker-01.example.com
usps                 IN CNAME docker-01.example.com
```
Because USAS and USPS run in separate containers they each must have their own virtual domain.
Create a directory on the docker host, e.g. /data/proxy/ and create the following docker-compose.yml:
Some newer versions of nginx may contribute to some network issues. This has not been verified, but to pull an older version use:

image: jwilder/nginx-proxy:0.9.1

image: jwilder/nginx-proxy:0.9.1-alpine

Different information about the images can be seen here: https://hub.docker.com/r/jwilder/nginx-proxy go to the tags column to see the versions
```
version: "3"
services:
  proxy:
     image: jwilder/nginx-proxy:alpine
     restart: always
     volumes:
       - /var/run/docker.sock:/tmp/docker.sock:ro
       - ./certs:/etc/nginx/certs:ro
       - ./vhost.d:/etc/nginx/vhost.d
       - ./html:/usr/share/nginx/html
       - ./conf.d:/etc/nginx/conf.d
     environment:
       - DEFAULT_HOST=demo.example.com
     ports:
       - "443:443"
       - "80:80"
```
The volume definitions are not strictly necessary at this point. However, adding them here is harmless and will make the subsequent instructions easier.
Run the proxy:
```
> docker-compose up -d
```
You should now be able to visit: http://sampletown.demo.example.com/. However, you’ll receive a “503 Service Temporarily Unavailable” because nginx-proxy does not know how to discover our service containers.
In your district’s compose project, add (or modify) docker-compose.override.yml file to define VIRTUAL_HOST and VIRTUAL_PORT environments for each service:
```
usasapp:
  environment:
    - VIRTUAL_HOST=sampletown-usas.demo.example.com
    - VIRTUAL_PORT=8080
uspsapp:
  environment:
    - VIRTUAL_HOST=sampletown-usps.demo.example.com
    - VIRTUAL_PORT=8080
```
The nginx-proxy container will monitor docker events. Each time a container starts or stops, which has a VIRTUAL_HOST variable, it will create a new nginx configuration which reverse proxies port 80 for the virtual domain to 8080 of the container.
Rebuild the district’s containers with:
```
docker-compose up -d
```
After the apps start, you should be able to reach the applications at: http://sampletown-usas.demo.example.com and http://sampletown-usps.demo.example.com.
Repeat the previous two steps for each school district.
If the host is not working, you can check the nginx-proxy log files with:
```
cd /data/proxy
docker-compose logs
```

HTTPS proxy on port 443

If you wish to use LetsEncrypt.org for automated certificate signing, then skip this section.

Now that we have a reverse proxy, we can secure the port using HTTPS. In this example, we are creating a wildcard certificate to match the wildcard DNS entry. In this example, the “Common Name” is “*.demo.example.com".

A wildcard certificate only covers one level of subdomains. For example, you cannot use *.example.com as a wildcard certificate for sampletown-usas.demo.example.com because, in this case there are two subdomain levels. The wildcard certificate needs to be *.demo.example.com.

Create a certificate and CSR in the proxy’s ./certs directory (this volume was mounted in the proxy’s docker-compose.yml file above).

data/proxy# mkdir -p certs
data/proxy# cd certs
data/proxy/certs# # Create a private key:
data/proxy/certs# openssl genrsa -out demo.example.com.key 2048
data/proxy/certs# # Create a CSR from the new key:
data/proxy/certs# openssl req -new -sha256 -key demo.example.com.key -out demo.example.com.csr
-----
Country Name (2 letter code) [AU]:US
State or Province Name (full name) [Some-State]:Ohio
Locality Name (eg, city) []:Archbold
Organization Name (eg, company) [Internet Widgits Pty Ltd]:Your Organization name
Organizational Unit Name (eg, section) []:Your OU
Common Name (e.g. server FQDN or YOUR name) []:*.demo.example.com.
Email Address []:[email protected]
Please enter the following 'extra' attributes
to be sent with your certificate request
A challenge password []:
An optional company name []:

Send the CSR to your favorite signing authority, or self sign it:

 data/proxy/certs# openssl x509 -req -sha256 -days 3650 -in  demo.example.com.csr -signkey  demo.example.com.key -out  demo.example.com.crt

Configure nginx to listen on port 443. Add port mapping to the proxy’s docker-compose.yml file:

proxy:
   image: jwilder/nginx-proxy
   restart: always
   volumes:
     - /var/run/docker.sock:/tmp/docker.sock:ro
     - ./certs:/etc/nginx/certs:ro
     - ./vhost.d:/etc/nginx/vhost.d
     - ./html:/usr/share/nginx/html
   environment:
     - DEFAULT_HOST=demo.example.com
   ports:
     - "80:80"
     - "443:443"

Recreate the proxy container with:
```
docker-compose up -d 
```

This exposes port 443 for SSL. We are leaving port 80 exposed because the nginx-proxy will automatically redirect port 80 to 443. Now we can access our application at: https://sampletown.demo.example.com/. If the cert is self-signed, you’ll get a browser warning. It will go away when/if you have the certificate signed.

After receiving the signed certificate from the signing authority, replace the .crt file created above with the signed certificate. In the example above, the self-signed certificate is named demo.example.com.crt. That file should be replaced with the file from the signing authority. Note: The names of the certificate files are important. The certificate file name must be must match the domain name it applies to. Again, from the above example, the wildcard domain name is *.demo.example.com so the certificate and keys must be named demo.example.com.crt and demo.example.com.key.

By convention, nginx-proxy will use the domain name to find the most specific certificate first and then drop prefixes until it finds a match. In this case, it will look for sampletown.demo.example.com.crt and then demo.example.com.crt . This allows you to have different signed certificates for different domain names on the same proxy.

Automatic Signed Certificates with LetsEncrypt.org

LetsEncrypt is a service which issues free automated certificates. Before using the service, you should review the https://letsencrypt.org/about/ and current rate limits.

LetsEncrypt.org is a service which automates the process of creating, signing, installing and renewing Domain Validation (DV) Certificates. These are certificates provide the lowest level of host verification but do ensure encrypted traffic to the users browser.

The steps below show how to configure an extra container to automatically create and install certificates using jrcs/letsencrypt-nginx-proxy-companion. This is a non-intrusive way to add letsencrypt to an existing proxy configuration.

Pulling new images

If the certificate renewal appears to stop working, make sure you have the most current version of the image(s) and restart the application.

docker pull jwilder/nginx-proxy:alpine
docker pull jrcs/letsencrypt-nginx-proxy-companion

docker-compose up -d

Requirements:

You must expose port 80 and 443 of your docker host to the outside via your firewall. That is, the docker host must have a public IP address and be accessible on both port 80 and 443 to the outside. DNS entries must exist in the global DNS for the virtual host(s) which point to the docker host’s IP address. When your host makes a certificate request, LetsEncrypts service will callback to your host for verification. If the remote service can not reach your host, then they can not verify your control of the domain name and the signing request will fail.

Steps to enable LetsEncrypt

First, if you haven’t already exposed port 80 and 443 on the nginx-proxy. The proxy’s docker-compose.yml should look like this:

version: "3"
services:
  proxy:
     image: jwilder/nginx-proxy:alpine
     restart: always
     volumes:
       - /var/run/docker.sock:/tmp/docker.sock:ro
       - ./certs:/etc/nginx/certs:ro
       - ./vhost.d:/etc/nginx/vhost.d
       - ./html:/usr/share/nginx/html
       - ./conf.d:/etc/nginx/conf.d
     environment:
       - DEFAULT_HOST=demo2.ssdt.io
     ports:
       - "443:443"
       - "80:80"
     labels:
       - "com.github.jrcs.letsencrypt_nginx_proxy_companion.nginx_proxy=true"
  le:
    image: jrcs/letsencrypt-nginx-proxy-companion
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock:ro
      - ./certs:/etc/nginx/certs:rw
      - ./vhost.d:/etc/nginx/vhost.d:rw
      - ./html:/usr/share/nginx/html:rw
    environment:
      - NGINX_PROXY_CONTAINER=proxy_proxy_1

Notice that the companion shares the volumes with the nginx-proxy container. The companion container will monitor docker events and automatically create certificates and place them in the correct directories for the primary nginx server.

Recreate the proxy with
```
docker-compose up -d
```
For each district you want to have a certificate created for, edit the docker-compose.override.yml file to create the LETSENCRYPT_HOST and LETSENCRYPT_EMAIL variables, like:
```
usasapp:
  environments:
    - VIRTUAL_HOST=usas.sampletown.org
    - VIRTUAL_PORT=8080
    - LETSENCRYPT_HOST=usas.sampletown.org
    - [email protected]
```
The companion image will create, sign and install a certificate for any service with the LETSENCRYPT_HOST variable. In most cases, the VIRTUAL_HOST and LETSENCRYPT_HOST will be set to the same value.
Restart the district’s services with:
```
docker-compose up -d
```

Because nginx-proxy and the companion container use separate environment variables, you can use a traditionally signed certificate for some hosts (see previous section) and letsencrypt certificates for others. For example, you might use a wildcard certificate for “*.demo.example.com” hosts and a letsencrypt certificate for a district’s “vanity domain” (usas.sampletown.org and usps.sampletown.org).

Updating NGINX Proxy

If you want to update the version of NGINX, possibly due to a security advisory, the following steps are suggested

Determine the current image being used and check the NGINX version

Get image from docker-compose file

Go to the directory containing the docker-compose files for the proxy. In this example, it is /data/proxy

cd /data/proxy
cat docker-compose.yml | grep 'image'

Example Result
 image: jwilder/nginx-proxy
 image: jrcs/letsencrypt-nginx-proxy-companion

Check the nginx version of that image

 Get the image ID:
 docker image ls | grep 'nginx'
 Result
 jwilder/nginx-proxy latest fcb5a96e19c1 20 months ago 161MB

Check that image to see what the ngixn version is using the ID from the previous command
 docker image inspect fcb5a96e19c1 | grep 'NGINX' <-- this uses the image id from the docker image ls command
 Result:
 docker image inspect fcb5a96e19c1 | grep 'NGINX'
 "NGINX_VERSION=1.17.6", 
 "NGINX_VERSION=1.17.6",

Note this shows a compromised version

Pull new image and check version

Note the version to pull depends on the version of nginx proxy being used. The image to pull will match the image name found in the docker-compose file

docker pull jwilder/nginx-proxy:latest
 latest: Pulling from jwilder/nginx-proxy
 07aded7c29c6: Pull complete
 bbe0b7acc89c: Pull complete
 44ac32b0bba8: Pull complete
 91d6e3e593db: Pull complete
 8700267f2376: Pull complete
 4ce73aa6e9b0: Pull complete
 6e90715ff75c: Pull complete
 8291a33eed7e: Pull complete
 6ddfe39af3a9: Pull complete
 fa40559ef5bc: Pull complete
 0db4d49e6f76: Pull complete
 062b69f5d6b6: Pull complete
 Digest: sha256:0eba2c7c08c6dd26cdd8209223ce11ea39906a6b677adc1e076401a637dd1043
 Status: Downloaded newer image for jwilder/nginx-proxy:latest

Repeat these steps to verify the version - note the old image is still there with no tag; that is normal

docker image ls | grep 'nginx' <-- to get the image id and make sure a newer one is downloaded
jwilder/nginx-proxy latest 2ffbdd7281bf 4 days ago 149MB
jwilder/nginx-proxy <none> fcb5a96e19c1 20 months ago 161MB

Inspect the new image to get the NGINX version:
docker image inspect 2ffbdd7281bf | grep 'NGINX'"NGINX_VERSION=1.21.3",

Activate it in your container

docker-compose down
docker-compose up -d
docker-compose ps <-- to make sure it came up

Make sure the container shows the new NGINX version

Note that the container name includes the name of the directory where the proxy files are placed. In this example, the directory has /proxy, so we can grep on this.

Find the container ID
docker ps  | grep 'proxy' <- to get the container ID

e27b852c849a jwilder/nginx-proxy "/app/docker-entrypo…" About a minute ago Up About a minute 0.0.0.0:80->80/tcp, 0.0.0.0:443->443/tcp proxy_proxy_1

docker inspect e27b852c849a | grep 'NGINX' <-- using the container ID
"NGINX_VERSION=1.21.3",

Issues with Data Collector after upgrading

If you have issues with the data collector/SIF agents after upgrading NGINX, where there are SSL handshake errors, one may see ” Received fatal alert: handshake_failure”; you may have to do the following:

In each districts .env file, add this line:

SSL_POLICY=Mozilla-Old

And restart the instance, and then niginx.

Explanation:

New versions of NGINX don’t all support the TLS version required by the Data Collector. In the past, we recommended SSL_Policy=Mozilla-Intermediate to resolve that issue. Now, however, higher versions of nginx need the SSL_Policy=Mozilla-Old in order to allow the older version of TLS. Note this is Case Sensitive.

Using with Inventory

See Inventory Installation and Migration Guide for more details on setting up inventory instances

Sometimes there are issues with connecting/reconnecting with the inventory apps, especially on reload of the data. The applications will start correctly, but connecting will result in a 502 or 503 error. This due to the networks not reconnecting properly. The steps to determine this are:

Make sure nginx is up and running

Issue a docker inspect command for the nginx container and see if the inventory instances are connected. If the inventory containers are connected manually, but they aren’t configured in the yml file, the docker networks will not be reconnected if the container is destroyed and recreated.

Sometimes the ones that are getting a 502 error may need the network disconnected before reconnected.

Miscellaneous tips and tidbits:

Issues with LetsEncrypt

ACMEv1/ACMEv2 error

LetsEncrypt no longer supports ACMEv1 for certificate management. If your site stops automatically renewing/generating certificates, this may appear in the logs:

docker-compose logs --tail=50 le
...
le_1 | 2020-01-09 11:48:43,395:INFO:simp_le:1382: Generating new account key
le_1 | ACME server returned an error: urn:acme:error:unauthorized :: The client lacks sufficient authorization ::
Account creation on ACMEv1 is disabled. Please upgrade your ACME client to a version that supports ACMEv2 / RFC 8555.
See https://community.letsencrypt.org/t/end-of-life-plan-for-acmev1/88430 for details.

To resolve this, make sure you have the latest images. See https://github.com/JrCs/docker-letsencrypt-nginx-proxy-companion

To check if it successful, go to the certs/accounts directory. You will see something like this if it is using ACMEv2 – in this case our proxy docker-compose.yml is in /data/proxy:

/data/proxy/certs/accounts# ls
acme-v02.api.letsencrypt.org

Checking site configuration

Check http/2 and ALPN configuration: https://tools.keycdn.com/http2-test
Check SSL, Chain, and Security: https://www.ssllabs.com/ssltest/

Useful Commands:

Force certificate renewal for all:

docker exec name_of_lets_encrypt_container/app/force_renew
Example:
 docker exec proxy_le_1 /app/force_renew

Get certificate status – note in this case, I did a force renew on 3/30, so 90 days is 6/28.

docker exec name_of_lets_encrypt_container/app/cert_status
Example:
 docker exec proxy_le_1 /app/cert_status
##### Certificate status #####
/etc/nginx/certs/files.ssdt.io/fullchain.pem: no corresponding chain.pem file, unable to verify certificate
Certificate was issued by Let's Encrypt Authority X3
Certificate is valid until Jun 28 13:11:46 2020 GMT
...