HTTPS in Backend, Caddy as Proxy ends in Errror 502

Help
1

1. Caddy version (caddy version):

Dockerfile (for Proxy and Webserver):

FROM caddy:2.2.1-builder-alpine AS builder

RUN xcaddy build \
    --with github.com/ueffel/caddy-brotli

FROM caddy:2.2.1

COPY --from=builder /usr/bin/caddy /usr/bin/caddy

2. How I run Caddy:

target layout:
Internet --https--> Router with destination NAT --https--> Dockerhost --https--> Caddy Proxy --https--> Caddy Webservers

functional layout:
```Internet --https–> Router with destination NAT --https–> Dockerhost --https–> Caddy Proxy --http–> Caddy Webservers``

For Let’s Encrypt I use the HTTP Challenge at the moment. Planned is do use DNS Challenge with CloudNS, as soon as it is available for Caddy v2.

a. System environment:

OS: OpenSUSE Leap 15.2
Docker: 19.03.11

b. Command:

docker-compose up -d

c. Service/unit/compose file:

Docker-Compose Files:
Loadbalancer/Proxy:

version: "3.7"

services:
  caddy:
    build: ./caddy
    restart: unless-stopped
    networks:
      medinastation_proxy:
    ports:
      - "80:80"
      - "443:443"
      - "443:443/udp"
    environment:
      - TZ=Europe/Zurich
    volumes:
      - /home/ei8ht/ch.starterpage/medinastation/caddy/Caddyfile:/etc/caddy/Caddyfile
      - /home/ei8ht/ch.starterpage/medinastation/caddy/site:/srv
      - /home/ei8ht/ch.starterpage/medinastation/caddy/data:/data
      - /home/ei8ht/ch.starterpage/medinastation/caddy/config:/config
      - /home/ei8ht/ch.starterpage/medinastation/caddy/log:/var/log

networks:
  medinastation_proxy:
    external: true

Webserver:

version: "3.7"
services:
  nifaweb:
    build: ./caddy
    container_name: nifaweb
    restart: unless-stopped
    environment:
      - TZ=Europe/Zurich
    volumes:
      - /home/ei8ht/ch.einmalmitprofis/nightfall/www/caddy/Caddyfile:/etc/caddy/Caddyfile
      - /home/ei8ht/ch.starterpage/medinastation/caddy/data:/data
      - /home/ei8ht/ch.einmalmitprofis/nightfall/www/caddy/config:/config
      - /home/ei8ht/ch.einmalmitprofis/nightfall/www/caddy/log/:/var/log
    networks:
      medinastation_proxy:
        ipv6_address: 2001:1620:58a:cafe::3333
      nifawebback:

networks:
  medinastation_proxy:
    external: true
  nifawebback:

d. My complete Caddyfile or JSON config:

Caddyfile Loadbalancer/Proxy:

{
  experimental_http3  
  #acme_ca https://acme-staging-v02.api.letsencrypt.org/directory
  email mymail@redacted.me
}

dev.nightfall.einmalmitprofis.ch {
  encode br gzip
  reverse_proxy https://nifaweb
  log {
    output file /var/log/nifa.log {
      roll_size 1gb
      roll_keep 5
      roll_keep_for 720h
    }
  }
}

Caddyfile Webserver:

{
  experimental_http3
#  acme_ca https://acme-staging-v02.api.letsencrypt.org/directory
  email mymail@redacted.me
}

dev.nightfall.einmalmitprofis.ch {
  encode br gzip
  respond "Hello World on nifaweb"
  log {
    output file /var/log/nifa.log
  }

3. The problem I’m having:

As long as i set the reverse proxy to http://nifaweb and start the webserver in http only, everything works as expected. At the moment I activate https in the “backend”, I get an http error 502. It doesn’t matter if I try to connect with http2 or http3.

4. Error messages and/or full log output:

http2 error:

{"level":"error","ts":1605811536.2540739,"logger":"http.log.access.log1","msg":"handled request","request":{"remote_addr":"10.10.1.151:49152","proto":"HTTP/2.0","method":"GET","host":"dev.nightfall.einmalmitprofis.ch","uri":"/","headers":{"Sec-Fetch-Site":["none"],"Sec-Fetch-Dest":["document"],"Accept-Language":["de-DE,de;q=0.9,en-US;q=0.8,en;q=0.7"],"Cache-Control":["max-age=0"],"User-Agent":["Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/86.0.4240.183 Safari/537.36"],"Accept":["text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.9"],"Sec-Fetch-Mode":["navigate"],"Sec-Fetch-User":["?1"],"Accept-Encoding":["gzip, deflate, br"],"Upgrade-Insecure-Requests":["1"]},"tls":{"resumed":false,"version":772,"cipher_suite":4865,"proto":"h2","proto_mutual":true,"server_name":"dev.nightfall.einmalmitprofis.ch"}},"common_log":"10.10.1.151 - - [19/Nov/2020:18:45:36 +0000] \"GET / HTTP/2.0\" 502 0","duration":0.301805448,"size":0,"status":502,"resp_headers":{"Server":["Caddy"],"Alt-Svc":["h3-29=\":443\"; ma=2592000"]}}

http3:

{"level":"error","ts":1605811555.4709983,"logger":"http.log.access.log1","msg":"handled request","request":{"remote_addr":"10.10.1.151:48045","proto":"HTTP/3","method":"GET","host":"dev.nightfall.einmalmitprofis.ch","uri":"/","headers":{"Upgrade-Insecure-Requests":["1"],"User-Agent":["Mozilla/5.0 (X11; Linux x86_64; rv:82.0) Gecko/20100101 Firefox/82.0"],"Accept":["text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8"],"Accept-Language":["de,en-US;q=0.7,en;q=0.3"],"Accept-Encoding":["gzip, deflate, br"],"Dnt":["1"],"Alt-Used":["dev.nightfall.einmalmitprofis.ch"]},"tls":{"resumed":false,"version":0,"cipher_suite":0,"proto":"","proto_mutual":false,"server_name":""}},"common_log":"10.10.1.151 - - [19/Nov/2020:18:45:55 +0000] \"GET / HTTP/3\" 502 0","duration":0.302131777,"size":0,"status":502,"resp_headers":{"Server":["Caddy"],"Alt-Svc":["h3-29=\":443\"; ma=2592000"]}

http2 from remote (Mobile):

{"level":"error","ts":1605811847.390072,"logger":"http.log.error.log1","msg":"remote error: tls: internal error","request":{"remote_addr":"178.197.229.8:61253","proto":"HTTP/2.0","method":"GET","host":"dev.nightfall.einmalmitprofis.ch","uri":"/","headers":{"User-Agent":["Mozilla/5.0 (Android 9; Mobile; rv:82.0) Gecko/82.0 Firefox/82.0"],"Accept":["text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8"],"Accept-Language":["de-DE"],"Accept-Encoding":["gzip, deflate, br"],"Dnt":["1"],"Upgrade-Insecure-Requests":["1"],"Te":["trailers"]},"tls":{"resumed":false,"version":772,"cipher_suite":4865,"proto":"h2","proto_mutual":true,"server_name":"dev.nightfall.einmalmitprofis.ch"}},"duration":0.30183744,"status":502,"err_id":"24dmkste1","err_trace":"reverseproxy.(*Handler).ServeHTTP (reverseproxy.go:441)"}

Curl from the Dockerhost (set dev.nightfall.einmalmitprofis.ch in the hostfile to 127.0.0.1):

ei8ht@tycho:~/ch.starterpage/medinastation> curl -IL --http2 https://dev.nightfall.einmalmitprofis.ch
HTTP/2 502 
alt-svc: h3-29=":443"; ma=2592000
server: Caddy
date: Thu, 19 Nov 2020 19:11:59 GMT

5. What I already tried:

As I’m a newbie to Caddy I don’t have much experience. Also I’m not to familiar with proxies when it comes to layer 7 protocols. I read the documentation several times, serached on the net. Most Infos I find about this is for Caddy v1. I’m sure to just miss one thing for https/tls connections to the backend but I can’t find it.

6. Links to relevant resources:

2

There’s a lot to unpack here.

If you’re Caddy in Docker, inside the same network, there’s generally no reason to proxy from one container to another over HTTPS. At this point, the connection is being made inside a private network, so it’s already safe.

Your nifaweb container isn’t publicly acessible on ports 80 and 443 (because your other Caddy container is), so it would never be able to get Let’s Encrypt certificates. The ACME protocol requires the server to be accessible to solve either the HTTP or ALPN challenges:

There are ways you could make HTTPS work though, but I wouldn’t recommend them here, because they’re relatively complex to set up and maintain.

Gist of it is that Caddy ships with an acme_server directive which can allow a Caddy instance to act as an ACME server, so it could issue certificates for other ACME clients (i.e. other Caddy instances). This is typically called mTLS (mutual TLS). But it also requires configuring all other Caddy instances to trust the ACME server one by trusting the CA certificate the first one generates, and the client ones would need to be configured to use that one as the ACME server.

3

Many thanks for your explanation. Now I’m a bit confused. Isn’t it stated, that several caddy servers act as one, as long as they use the same storage (Automatic HTTPS — Caddy Documentation)? That’s also the reason why I need the DNS Challenge. However, LEGO can’t handle custom timeouts and CloudNS is to slow actual.
If so, how can I achieve this? The idea behind is, that the reverse proxy does all the let’s Encrypt stuff and the other webservers (there will be more than just one when the setup is ready) use this certificates.
If this isn’t possible, is caddy capable to act as a L4 Loadbalancer with http3 support?

Why do I want https in the backend:

  • There are customer sites with logins and I don’t want the customers (or the customers customer) credentials in cleartext on my machine. Not even in the backend.
  • why http2/3 in the front when the rest runs with the old and slow http 1.1?
1 Like
4

Yes that’s true

I don’t see where you configured the DNS challenge in your config, and you haven’t installed a plugin that would allow you to use the DNS challenge.

I’m not sure what you mean by this.

Yeah, the caddy-l4 plugin can do this:

HTTP 1.1 isn’t actually slow in server to server scenarios like proxying. Connections are kept open and reused, so the overhead is not very high.

Are you running untrusted code on that machine? Cause if so, it’s already game-over.

Once the first Caddy instance handles the request, the connection is all within the docker network, so the only way something could intercept it is if it’s running as root on the host machine (or as a user with access to the docker network), pretty sure.

1 Like
5

Dear Francis,
again many thanks for your answer.

My apololgies. I forgot to mention that I set up a Test with Caddy and the deprecated LEGO Plugin before. I gave up after several days of try and error, purged everything and started over with HTTP Challenge. Instead of the Brotli Plugin was the LEGO in the Dockerfile. I have to use LEGO as I use CloudNS as DNS provider, at least till mid 2022, and there is no plugin for the new DNS Challenge Module.

When I tried to use the DNS challenge with the LEGO Plugin I had a timeout problem. I see that the update on CloudNS is running, the SOA serial counts up. But because the DNS is still replicating (it takes around 5 minutes). The API call from LEGO to CloudNS is working therefore. LEGO tries to solve the DNS Challenge somewhat about 30 (or 60, not sure and I don’t have the logs anymore) seconds after the start. But because the update takes longer, it is unable to solve the DNS challenge and cancels the process. With the cancellation, the TXT entry is deleted. After some increasing cooldown period LEGO tries and fails again. When I set the timout env variable for LEGO and CloudNS, nothing happens. The problem is similar to this one: Caddy2 with DNS challenge and Namecheap in Docker - #11 by francislavoie
In the thread Timeout problems using dns-01 challenge @matt stated that there is a problem with env variables (last post).
But as this is a complete different Problem I would prefer to open another thread for it if needed. Maybe I find enough motivation to refresh my, very very rusted, programming skills and learn Go :).

Will look at it! Thanks for this information!

Thanks for this advice. As said in the entrypost, I’m not familiar with proxying. Just ask me plain networkstuff up to L3, including dynamic routing and some MPLS stuff ;). I will give it a try. And when I can convince one customer to not use http2 push anymore (even google has anounced the end http2 push support in Chrome), this should work fine.

What do you understand under untrusted code? I’m planning to run at least:

  • PiHole
  • at least 2 Wordpress Sites, one being migrated to static pages with HUGO (or similar) somwhen in 2021
  • 1 EQDKP-Plus (PHP based CMS and Raidplanner)
  • 1 Nextcloud or Owncloud instance

on this machine, all separated by docker.

6

This topic was automatically closed after 30 days. New replies are no longer allowed.