Dockerized Caddy proxying to https within Docker network

[dnk8n (Dean Kayton)]

I have a web app (Scala Play framework 2.5 based) that I would like to dockerize, but that insists on https and will redirect any http requests to https. I would like to use Caddy (also Dockerized) without changing the app (for now). Are there any documents or tutorials on how to achieve this? Most of what I have seen relies on the underlying app to receive http requests.

I have been reading the following link, which describe setting up https on a dev environment. Could they potentially be applicable for setting up https on the internal Docker network?

Please note. I don’t think the double https is necessary for the long run. I am just trying to get a workaround in place for now and over time we can phase out the unnecessary work the app is currently doing. On another project, I have a Scala Play framework 2.7 based web app working correctly with the following configs:

Caddyfile:

app.***.com {
    proxy / ***-play-app:9000 {
        transparent
    }
    log /var/log/caddy/app.***.com-access.log
    errors /var/log/caddy/app.***.com-error.log
}

And docker-compose.yml file:

version: '3.7'

services:

  ***-play-app:
    container_name: ***-play-app
    hostname: ***-play-app
    restart: unless-stopped
    image: openjdk:8-jre
    volumes:
      - /var/www/${APPLICATION_FQDN}/artifacts/${APPLICATION_SNAPSHOT_DIR}:/var/www/${APPLICATION_FQDN}
      - /var/www/${APPLICATION_FQDN}/server/conf/aws.conf:/.../*.conf
    networks:
      - ***
    working_dir: /var/www/${APPLICATION_FQDN}
    expose:
      - 9000
    command:
      - /var/www/${APPLICATION_FQDN}/bin/server
      - -Ddb.default.url=${APPLICATION_DB_ENGINE}://${APPLICATION_DB_USER}:${APPLICATION_DB_PASSWORD}@${APPLICATION_DB_SERVER}/${APPLICATION_DB_NAME}
      - -Dconfig.file=/.../*.conf
    environment:
      - APPLICATION_SECRET
      - APPLICATION_DB_ENGINE
      - APPLICATION_DB_USER
      - APPLICATION_DB_PASSWORD
      - APPLICATION_DB_SERVER
      - APPLICATION_DB_NAME
      - APPLICATION_FQDN

  ***-caddy:
    container_name: ***-caddy
    hostname: ***-caddy
    restart: unless-stopped
    image: abiosoft/caddy:1.0.1
    depends_on:
      - ***-play-app
    volumes:
      - /var/www/${APPLICATION_FQDN}/caddy/Caddyfile:/etc/Caddyfile
      - /var/www/${APPLICATION_FQDN}/artifacts/caddy/.caddy:/root/.caddy
      - /var/www/${APPLICATION_FQDN}/artifacts/caddy/logs:/var/log/caddy
    environment:
      ACME_AGREE: 'true'
    networks:
      - ***
    ports:
      - 80:80
      - 443:443

networks:

  ***:

and the *.conf file:

include "application"

http.port=9000
pidfile.path=/dev/null

play.http.secret.key=changeme
play.http.secret.key=${?APPLICATION_SECRET}

play.filters.hosts {
  allowed = [${?APPLICATION_FQDN}]
}

db.default.migration.auto=true
db.default.username=${?APPLICATION_DB_USER}
db.default.password=${?APPLICATION_DB_PASSWORD}
quillDb.dataSource.serverName=${?APPLICATION_DB_SERVER}
quillDb.dataSource.databaseName=${?APPLICATION_DB_NAME}

[Whitestrake (Matthew Fay)]

Hi @dnk8n, have you tried simply proxying to the app’s HTTPS endpoint?

If the HTTPS certificate provided by the app is not trusted, you can tell Caddy not to verify it using the insecure_skip_verify subdirective in your proxy. Note that this is, as mentioned, insecure - but not more so than using HTTP.

https://caddyserver.com/docs/proxy

[dnk8n (Dean Kayton)]

I did try that. But the app itself screams about no certificates, etc. I will try again and paste the error messages here.

[Whitestrake (Matthew Fay)]

The app screams about certificates? Does it need Caddy to provide a client certificate or something?

[dnk8n (Dean Kayton)]

To be honest I am not sure about that, I am annoyingly abstracted away from how this app works. I am comfortable in Python, Scala not so much. I recently joined the team and their devs are not very good at getting back to me.

But one of their devs said that they could probably make a code change to disable auto https redirecting.

The behavior of the app is that if it senses an http request, it redirects to https. And for some reason in Play 2.5 this was done in code because there was not a simple config flag in 2.5 (play.filters.https.redirectEnabled = false works in Play 2.6+, see link)

That link should give a clearer idea of what is taking place.

I don’t know how to get an internal application to successfully respond to https on the internal Docker network.

[Whitestrake (Matthew Fay)]

If Caddy’s going straight to HTTPS, you shouldn’t need to worry about the redirect. You’ll need to sort out what the app needs to be able to talk over HTTPS.

What do you get when you run wget --server-response --spider ***-play-app:9000 from within the Caddy container? Should give us some insight into exactly what is going wrong.

[dnk8n (Dean Kayton)]

@Whitestrake

wget --server-response --spider ***-play-app:9000

Thanks very much for getting back so quickly. Spinning it up now to check.

[dnk8n (Dean Kayton)]

I executed the command you gave me, and this is what I got @Whitestrake

Connecting to ***-play-app:9000 (172.18.0.3:9000)
  HTTP/1.1 301 Moved Permanently
  Location: https://***-play-app:9000/
Connecting to ***-play-app:9000 (172.18.0.3:9000)
ssl_client: ***-play-app: handshake failed: error:1408F10B:SSL routines:ssl3_get_record:wrong version number
wget: error getting response: Connection reset by peer

[Whitestrake (Matthew Fay)]

OK, here’s your problem.

Your app is redirecting HTTP visitors on port 9000… to HTTPS on port 9000.

Obviously the app can’t listen for HTTP and HTTPS on the same port. So when the client reconnects via HTTPS on port 9000, and gets a HTTP response, we see this issue.

Your app will need to be updated. It’s unworkable in its current state. It must either provide a real HTTPS listener, or it needs to accept HTTP, but right now it’s doing neither.

[dnk8n (Dean Kayton)]

Even when I start both an http port (9000) and an https port (9443), it still tries to redirect to https://***play-app:9000.

When I do wget --server-response --spider https://dotdr-play-app:9443

I get:

Connecting to dotdr-play-app:9443 (172.20.0.3:9443)
ssl_client: dotdr-play-app: certificate verification failed: self signed certificate
wget: error getting response: Connection reset by peer

Thinking I can maybe get the app to accept self signed certificate

[dnk8n (Dean Kayton)]

Now adding your original suggestion, it works slightly better. Although now I have to try understand why the application is redirecting to the main marketing page!

wget --server-response --spider --no-check-certificate https://***-play-app:9000

Connecting to ***-play-app:9000 (172.20.0.3:9000)
  HTTP/1.1 303 See Other
  Location: http://***.co
Connecting to ***.co (*.*.*.*)

[dnk8n (Dean Kayton)]

The Caddyfile now looks like:

***.***.co.za {
    proxy / https://***-play-app:9000 {
        transparent
        insecure_skip_verify
    }
    log /var/log/caddy/access.log
    errors /var/log/caddy/error.log
}

The app now has https over 9000 and http disabled.

[Whitestrake (Matthew Fay)]

That looks good! Let us know if you run into further problems.

[dnk8n (Dean Kayton)]

@Whitestrake

The only way I could work out how to get things working without resorting to self-signed certs for https and thereby being able to omit Caddy flag, insecure_skip_verify…

I only went through these extra steps going down the wrong rabbit hole… in the end it was simpler and more automated just to use insecure_skip_verify (because security at this layer is somewhat redundant, if I understand correctly))

First pass the app some extra ssl related flags it wanted:

      - -Dplay.server.https.keyStore.path=/etc/letsencrypt/live/***.co.za/keyStore.jks
      - -Dplay.server.https.keyStore.password=***
      - -Djdk.tls.ephemeralDHKeySize=2048
      - -Djdk.tls.rejectClientInitiatedRenegotiation=true

To get it in the format, keyStore.jks I followed - How to configure a Play application to use Let’s Encrypt certificate?

So this meant I needed to create a wildcard cert on the host using certbot and mount the volume /etc/letsencrypt/live/***.co.za:/etc/letsencrypt/live/***.co.za into the play docker application. This way it can validate https requests. I also needed to port forward 9000 to the host (and create a security group rule for AWS so that only the instance itself can access port 9000).

The final caddy file looks like (any way to avoid this duplication btw? Are snippets the way?):

demo.***.co.za {
    proxy / https://demo.***.co.za:9000 {
        transparent
    }
    log /var/log/caddy/access.demo.log
    errors /var/log/caddy/error.demo.log
}

app.***.co.za {
    proxy / https://app.***.co.za:9000 {
        transparent
    }
    log /var/log/caddy/access.app.log
    errors /var/log/caddy/error.app.log
}

So the traffic route is:

• user browses to link http://demo..co.za or https://demo..co.za OR http://app..co.za or https://app..co.za
• The DNS ‘A’ record of both demo..co.za and app..co.za point to the same host
• this routes from host to forwarded (host ↔ caddy docker container) port 80 or 443 (the only ports open to the public)
• this makes its way to dockerized caddy (see conf above)
• Caddy proxies the request to https://demo.***.co.za:9000 (port 9000 only open to this instance, not the public)
• Because of transparent directive, the application can derive demo or app from the host headers (which according to my understanding will remain the same no matter where caddy routes the request, is that right?)
• Depending on app or demo, the corresponding web app is displayed

Note: I believe the reason I cannot route straight to the play app on the internal docker network is that I have no way of proving ownership of ‘***-play-app’. Therefore I have to go via external DNS and back to the instance through the domain name I have installed certs for.

Maybe there is a simpler way. But by the time I find out I won’t need it… because we will allow the app to receive http requests. Then the caddy conf would look more similar to the original one posted in this discussion.

[Whitestrake (Matthew Fay)]

@dnk8n

I only went through these extra steps going down the wrong rabbit hole… in the end it was simpler and more automated just to use insecure_skip_verify (because security at this layer is somewhat redundant, if I understand correctly))

Yeah, I’d agree with that assessment.

Unless your threat model consists of malicious software inside your Docker networks, sniffing or impersonating traffic between Docker-run services, that somehow can’t access the TLS assets used inside your Docker containers, self-signed certificates are A-OK.

@dnk8n

Note: I believe the reason I cannot route straight to the play app on the internal docker network is that I have no way of proving ownership of ‘***-play-app’. Therefore I have to go via external DNS and back to the instance through the domain name I have installed certs for.

Yep - can’t do it without distributing your own CA, an arduous process at best, possible to automate but unwieldy.

Unfortunate this goes external, but at least it’s HTTPS secured if it does, haha.

@dnk8n

Maybe there is a simpler way. But by the time I find out I won’t need it… because we will allow the app to receive http requests.

Probably just as secure as unverified HTTPS when we’re talking about the inside of a controlled network. It’ll simplify things a lot.