Caddy reverse_proxy behind caddy reverse_proxy?

1. Caddy version:

system 1:
v2.6.4

system 2:
v2.6.4

2. How I installed, and run Caddy:

system 1:
Dockerfile based on caddy:latest and installing cloudflare plugin

system 2:
caddy:alpine

a. System environment:

system 1:
Ubuntu 20.04.5 LTS
Docker version 20.10.21, build baeda1f

system 2:
Debian GNU/Linux 10
Docker version 20.10.21, build baeda1f

b. Command:

see docker-compose below

c. Service/unit/compose file:

docker 1

version: "3.7"

services:
  caddy:
    build: .
    restart: unless-stopped
    ports:
      - "80:80"
      - "443:443"
      - "443:443/udp"
    volumes:
      - $PWD/caddy/Caddyfile:/etc/caddy/Caddyfile
      - $PWD/caddy/site:/srv
      - $PWD/caddy/data:/data
      - $PWD/caddy/config:/config
    environment:
      - CF_API_TOKEN

docker 2

version: '3'

services:
  caddy:
    image: caddy/caddy:alpine
    restart: unless-stopped
    container_name: proxy
    volumes:
    - ./caddy/Caddyfile:/etc/caddy/Caddyfile:ro
    - ./caddy/data:/data
    ports:
    - 443:443
    - 80:80
    environment:
    - LOG_FILE=/data/logs/caddy.log
    - DOMAIN=dl.l.bradford.la

... other services ...

d. My complete Caddy config:

caddy 1

(defaults) {
  tls {
    dns cloudflare {env.CF_API_TOKEN}
  }
}

dl.l.bradford.la {
  reverse_proxy https://172.16.1.50:443 {
    transport http {
      tls_insecure_skip_verify
    }
  }
  import defaults
}

tv.l.bradford.la {
  reverse_proxy 172.16.1.51:8096

  import defaults
}
~

caddy 2:

(defaults) {
  log {
        output file {$LOG_FILE} {
        roll_size 50MiB # https://caddyserver.com/docs/caddyfile/directives/log#log
        roll_keep 5 # https://caddyserver.com/docs/caddyfile/directives/log#log
    }
    level DEBUG
  }
  tls internal
}

{$DOMAIN}:443 {
  import defaults

  handle /nzbget/ {
    reverse_proxy http://nzbget:6789
  }
}

3. The problem I’m having:

I am trying something a bit weird. I have 2 systems (1 and 2). System 1 uses cloudflare to generate valid certs for my internal LAN (dl.l.bradford.la and tv.l.bradford.la). tv.l.bradford.la proxies great, no issues.

dl.l.bradford.la has issues acting as a reverse proxy to another caddy reverse proxy (system 2). This second reverse_proxy will ideally act as a reverse proxy for several docker services as subdirectories of dl.l.bradford.la. It doesn’t seem to work properly, getting hung up on tls. I added the transport http tls_insecure_skip_verify to hopefully help that, but it still is giving an error 502.

4. Error messages and/or full log output:

Navigating to https://dl.l.bradford.la/nzbget/ gives the error message in caddy on system 1 (no log output on system 2)

{
    "level": "error",
    "ts": 1676929066.575315,
    "logger": "http.log.error",
    "msg": "remote error: tls: internal error",
    "request": {
        "remote_ip": "172.16.10.2",
        "remote_port": "57293",
        "proto": "HTTP/3.0",
        "method": "GET",
        "host": "dl.l.bradford.la",
        "uri": "/nzbget/",
        "headers": {
            "Accept-Language": [
                "en-US,en;q=0.9,fr;q=0.8"
            ],
            "Dnt": [
                "1"
            ],
            "Accept": [
                "text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.7"
            ],
            "Sec-Fetch-Site": [
                "cross-site"
            ],
            "Sec-Fetch-User": [
                "?1"
            ],
            "Sec-Ch-Ua-Platform": [
                "\"Windows\""
            ],
            "User-Agent": [
                "Mozilla/5.0 (Windows NT 6.3; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/97.0.4692.99 Safari/537.36"
            ],
            "Sec-Fetch-Mode": [
                "navigate"
            ],
            "Sec-Ch-Ua": [
                "\"Chromium\";v=\"110\", \"Not A(Brand\";v=\"24\", \"Microsoft Edge\";v=\"110\""
            ],
            "Upgrade-Insecure-Requests": [
                "1"
            ],
            "Sec-Fetch-Dest": [
                "document"
            ],
            "Accept-Encoding": [
                "gzip, deflate, br"
            ],
            "Cache-Control": [
                "max-age=0"
            ],
            "Sec-Ch-Ua-Mobile": [
                "?0"
            ]
        },
        "tls": {
            "resumed": true,
            "version": 772,
            "cipher_suite": 4865,
            "proto": "h3",
            "server_name": "dl.l.bradford.la"
        }
    },
    "duration": 0.031568152,
    "status": 502,
    "err_id": "25rs1cvej",
    "err_trace": "reverseproxy.statusError (reverseproxy.go:1299)"
}

5. What I already tried:

Google-fu and refining my caddy files to try and eliminate any other possibility. It seems it is finally hung up on the tls connection between caddy 1 and caddy 2.

6. Links to relevant resources:

Don’t use caddy/caddy. Use caddy instead. The caddy/caddy image is our CI target, and it should not be used outside of our internal use.

Path matching is exact in Caddy. That will only handle exactly /nzbget/ and nothing else. You should use /nzbget/* instead.

I’d recommend using subdomains instead of subpaths. See this article:

Turn on the debug global option on both servers, to see more detailed logs.

I think what’s happening is SNI gets set to 172.16.1.50 instead of dl.l.bradford.la which means the upstream doesn’t know the correct certificate to choose.

Why are you proxying over HTTPS? If this is in your local network, then there’s probably no problem with proxying over HTTP instead. The only concern is if there’s untrusted devices in your network that could intercept the traffic. But that’s probably unlikely.

2 Likes

Thanks for the catch.

I read that article before. I want to try subdirectories for two reasons -

  1. My domain names are getting a bit ridiculous at this point, if I keep this structure anyway
  2. I don’t want to have DNS entries for each of the many many services I would have running behind dl.

I understand some of my services will not like it, but I want to give it a shot this way first.

I thought that Caddy would prefer a HTTPS connection.

With debugging turned on, original config, these are my error messages on system 1, no log messages on system 2:

{
    "level": "debug",
    "ts": 1677082676.526556,
    "logger": "events",
    "msg": "event",
    "name": "tls_get_certificate",
    "id": "2011b7a9-fcd7-484d-84a3-2f1d0ab4d8c7",
    "origin": "tls",
    "data": {
        "client_hello": {
            "CipherSuites": [
                4865,
                4866,
                4866,
                4867
            ],
            "ServerName": "dl.l.bradford.la",
            "SupportedCurves": [
                29,
                23,
                24
            ],
            "SupportedPoints": null,
            "SignatureSchemes": [
                1027,
                2052,
                1025,
                1283,
                2053,
                1281,
                2054,
                1537,
                513
            ],
            "SupportedProtos": [
                "h3"
            ],
            "SupportedVersions": [
                772
            ],
            "Conn": {}
        }
    }
}
{
    "level": "debug",
    "ts": 1677082676.5273,
    "logger": "tls.handshake",
    "msg": "choosing certificate",
    "identifier": "dl.l.bradford.la",
    "num_choices": 1
}
{
    "level": "debug",
    "ts": 1677082676.527804,
    "logger": "tls.handshake",
    "msg": "default certificate selection results",
    "identifier": "dl.l.bradford.la",
    "subjects": [
        "dl.l.bradford.la"
    ],
    "managed": true,
    "issuer_key": "acme-v02.api.letsencrypt.org-directory",
    "hash": "1ae13787c9b4ea79c861529899a2b2d4d3db7d459605f7838e566cc4c1430325"
}
{
    "level": "debug",
    "ts": 1677082676.5278308,
    "logger": "tls.handshake",
    "msg": "matched certificate in cache",
    "remote_ip": "172.16.10.2",
    "remote_port": "53698",
    "subjects": [
        "dl.l.bradford.la"
    ],
    "managed": true,
    "expiration": 1684621269,
    "hash": "1ae13787c9b4ea79c861529899a2b2d4d3db7d459605f7838e566cc4c1430325"
}
{
    "level": "debug",
    "ts": 1677082676.53279,
    "logger": "http.handlers.reverse_proxy",
    "msg": "selected upstream",
    "dial": "172.16.1.50:443",
    "total_upstreams": 1
}
{
    "level": "debug",
    "ts": 1677082676.5392623,
    "logger": "http.handlers.reverse_proxy",
    "msg": "upstream roundtrip",
    "upstream": "172.16.1.50:443",
    "duration": 0.006414977,
    "request": {
        "remote_ip": "172.16.10.2",
        "remote_port": "53698",
        "proto": "HTTP/3.0",
        "method": "GET",
        "host": "dl.l.bradford.la",
        "uri": "/nzbget/",
        "headers": {
            "Sec-Fetch-User": [
                "?1"
            ],
            "Sec-Ch-Ua-Mobile": [
                "?0"
            ],
            "Sec-Ch-Ua-Platform": [
                "\"Windows\""
            ],
            "Sec-Fetch-Dest": [
                "document"
            ],
            "Upgrade-Insecure-Requests": [
                "1"
            ],
            "Sec-Fetch-Mode": [
                "navigate"
            ],
            "X-Forwarded-Proto": [
                "https"
            ],
            "Accept": [
                "text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.7"
            ],
            "Accept-Encoding": [
                "gzip, deflate, br"
            ],
            "Dnt": [
                "1"
            ],
            "Sec-Fetch-Site": [
                "cross-site"
            ],
            "X-Forwarded-Host": [
                "dl.l.bradford.la"
            ],
            "User-Agent": [
                "Mozilla/5.0 (Windows NT 6.3; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/97.0.4692.99 Safari/537.36"
            ],
            "Sec-Ch-Ua": [
                "\"Chromium\";v=\"110\", \"Not A(Brand\";v=\"24\", \"Microsoft Edge\";v=\"110\""
            ],
            "Accept-Language": [
                "en-US,en;q=0.9,fr;q=0.8"
            ],
            "X-Forwarded-For": [
                "172.16.10.2"
            ],
            "Cache-Control": [
                "max-age=0"
            ]
        },
        "tls": {
            "resumed": false,
            "version": 772,
            "cipher_suite": 4865,
            "proto": "h3",
            "server_name": "dl.l.bradford.la"
        }
    },
    "error": "remote error: tls: internal error"
}
{
    "level": "error",
    "ts": 1677082676.5404313,
    "logger": "http.log.error",
    "msg": "remote error: tls: internal error",
    "request": {
        "remote_ip": "172.16.10.2",
        "remote_port": "53698",
        "proto": "HTTP/3.0",
        "method": "GET",
        "host": "dl.l.bradford.la",
        "uri": "/nzbget/",
        "headers": {
            "Sec-Ch-Ua-Mobile": [
                "?0"
            ],
            "Sec-Ch-Ua-Platform": [
                "\"Windows\""
            ],
            "Sec-Fetch-Dest": [
                "document"
            ],
            "Sec-Ch-Ua": [
                "\"Chromium\";v=\"110\", \"Not A(Brand\";v=\"24\", \"Microsoft Edge\";v=\"110\""
            ],
            "Upgrade-Insecure-Requests": [
                "1"
            ],
            "Sec-Fetch-Mode": [
                "navigate"
            ],
            "Accept-Encoding": [
                "gzip, deflate, br"
            ],
            "Accept-Language": [
                "en-US,en;q=0.9,fr;q=0.8"
            ],
            "Dnt": [
                "1"
            ],
            "Sec-Fetch-Site": [
                "cross-site"
            ],
            "Cache-Control": [
                "max-age=0"
            ],
            "User-Agent": [
                "Mozilla/5.0 (Windows NT 6.3; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/97.0.4692.99 Safari/537.36"
            ],
            "Accept": [
                "text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.7"
            ],
            "Sec-Fetch-User": [
                "?1"
            ]
        },
        "tls": {
            "resumed": false,
            "version": 772,
            "cipher_suite": 4865,
            "proto": "h3",
            "server_name": "dl.l.bradford.la"
        }
    },
    "duration": 0.009241442,
    "status": 502,
    "err_id": "660f4pu19",
    "err_trace": "reverseproxy.statusError (reverseproxy.go:1299)"
}

When I change system 1 config to remove the https and port 443 bit, and system 2 to remove the 443 port on the service block, I get different errors, tls: first record does not look like a TLS handshake, as if system 1 is trying to force a ssl connection anyway:

System 1 errors, no log messages on system 2:

{
    "level": "debug",
    "ts": 1677083100.662735,
    "logger": "events",
    "msg": "event",
    "name": "tls_get_certificate",
    "id": "cbb7539a-2209-4d8d-8e7b-e78249382044",
    "origin": "tls",
    "data": {
        "client_hello": {
            "CipherSuites": [
                47802,
                4865,
                4866,
                4866,
                4867,
                49195,
                49199,
                49196,
                49200,
                52393,
                52392,
                49171,
                49172,
                156,
                157,
                47,
                53
            ],
            "ServerName": "dl.l.bradford.la",
            "SupportedCurves": [
                64250,
                29,
                23,
                24
            ],
            "SupportedPoints": "AA==",
            "SignatureSchemes": [
                1027,
                2052,
                1025,
                1283,
                2053,
                1281,
                2054,
                1537
            ],
            "SupportedProtos": [
                "h2",
                "http/1.1"
            ],
            "SupportedVersions": [
                43690,
                772,
                771
            ],
            "Conn": {}
        }
    }
}
{
    "level": "debug",
    "ts": 1677083100.6627333,
    "logger": "events",
    "msg": "event",
    "name": "tls_get_certificate",
    "id": "99fd5706-3a8b-43a8-895d-2b86ac9dc789",
    "origin": "tls",
    "data": {
        "client_hello": {
            "CipherSuites": [
                4865,
                4866,
                4866,
                4867
            ],
            "ServerName": "dl.l.bradford.la",
            "SupportedCurves": [
                29,
                23,
                24
            ],
            "SupportedPoints": null,
            "SignatureSchemes": [
                1027,
                2052,
                1025,
                1283,
                2053,
                1281,
                2054,
                1537,
                513
            ],
            "SupportedProtos": [
                "h3"
            ],
            "SupportedVersions": [
                772
            ],
            "Conn": {}
        }
    }
}
{
    "level": "debug",
    "ts": 1677083100.6651504,
    "logger": "tls.handshake",
    "msg": "choosing certificate",
    "identifier": "dl.l.bradford.la",
    "num_choices": 1
}
{
    "level": "debug",
    "ts": 1677083100.6652548,
    "logger": "tls.handshake",
    "msg": "choosing certificate",
    "identifier": "dl.l.bradford.la",
    "num_choices": 1
}
{
    "level": "debug",
    "ts": 1677083100.665747,
    "logger": "tls.handshake",
    "msg": "default certificate selection results",
    "identifier": "dl.l.bradford.la",
    "subjects": [
        "dl.l.bradford.la"
    ],
    "managed": true,
    "issuer_key": "acme-v02.api.letsencrypt.org-directory",
    "hash": "1ae13787c9b4ea79c861529899a2b2d4d3db7d459605f7838e566cc4c1430325"
}
{
    "level": "debug",
    "ts": 1677083100.6657655,
    "logger": "tls.handshake",
    "msg": "matched certificate in cache",
    "remote_ip": "172.16.10.2",
    "remote_port": "17596",
    "subjects": [
        "dl.l.bradford.la"
    ],
    "managed": true,
    "expiration": 1684621269,
    "hash": "1ae13787c9b4ea79c861529899a2b2d4d3db7d459605f7838e566cc4c1430325"
}
{
    "level": "debug",
    "ts": 1677083100.6657474,
    "logger": "tls.handshake",
    "msg": "default certificate selection results",
    "identifier": "dl.l.bradford.la",
    "subjects": [
        "dl.l.bradford.la"
    ],
    "managed": true,
    "issuer_key": "acme-v02.api.letsencrypt.org-directory",
    "hash": "1ae13787c9b4ea79c861529899a2b2d4d3db7d459605f7838e566cc4c1430325"
}
{
    "level": "debug",
    "ts": 1677083100.6658442,
    "logger": "tls.handshake",
    "msg": "matched certificate in cache",
    "remote_ip": "172.16.10.2",
    "remote_port": "60610",
    "subjects": [
        "dl.l.bradford.la"
    ],
    "managed": true,
    "expiration": 1684621269,
    "hash": "1ae13787c9b4ea79c861529899a2b2d4d3db7d459605f7838e566cc4c1430325"
}
{
    "level": "debug",
    "ts": 1677083100.6741283,
    "logger": "http.handlers.reverse_proxy",
    "msg": "selected upstream",
    "dial": "172.16.1.50:80",
    "total_upstreams": 1
}
{
    "level": "debug",
    "ts": 1677083100.6944005,
    "logger": "http.handlers.reverse_proxy",
    "msg": "upstream roundtrip",
    "upstream": "172.16.1.50:80",
    "duration": 0.020239333,
    "request": {
        "remote_ip": "172.16.10.2",
        "remote_port": "17596",
        "proto": "HTTP/2.0",
        "method": "GET",
        "host": "dl.l.bradford.la",
        "uri": "/nzbget/",
        "headers": {
            "Accept-Language": [
                "en-US,en;q=0.9,fr;q=0.8"
            ],
            "Cache-Control": [
                "max-age=0"
            ],
            "Sec-Fetch-Site": [
                "cross-site"
            ],
            "X-Forwarded-Proto": [
                "https"
            ],
            "User-Agent": [
                "Mozilla/5.0 (Windows NT 6.3; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/97.0.4692.99 Safari/537.36"
            ],
            "Sec-Ch-Ua-Platform": [
                "\"Windows\""
            ],
            "Accept": [
                "text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.7"
            ],
            "X-Forwarded-For": [
                "172.16.10.2"
            ],
            "X-Forwarded-Host": [
                "dl.l.bradford.la"
            ],
            "Sec-Ch-Ua-Mobile": [
                "?0"
            ],
            "Accept-Encoding": [
                "gzip, deflate, br"
            ],
            "Dnt": [
                "1"
            ],
            "Upgrade-Insecure-Requests": [
                "1"
            ],
            "Sec-Fetch-User": [
                "?1"
            ],
            "Sec-Ch-Ua": [
                "\"Chromium\";v=\"110\", \"Not A(Brand\";v=\"24\", \"Microsoft Edge\";v=\"110\""
            ],
            "Sec-Fetch-Dest": [
                "document"
            ],
            "Sec-Fetch-Mode": [
                "navigate"
            ]
        },
        "tls": {
            "resumed": false,
            "version": 772,
            "cipher_suite": 4865,
            "proto": "h2",
            "server_name": "dl.l.bradford.la"
        }
    },
    "error": "tls: first record does not look like a TLS handshake"
}
{
    "level": "error",
    "ts": 1677083100.6959682,
    "logger": "http.log.error",
    "msg": "tls: first record does not look like a TLS handshake",
    "request": {
        "remote_ip": "172.16.10.2",
        "remote_port": "17596",
        "proto": "HTTP/2.0",
        "method": "GET",
        "host": "dl.l.bradford.la",
        "uri": "/nzbget/",
        "headers": {
            "Dnt": [
                "1"
            ],
            "Accept-Language": [
                "en-US,en;q=0.9,fr;q=0.8"
            ],
            "Sec-Ch-Ua-Platform": [
                "\"Windows\""
            ],
            "Upgrade-Insecure-Requests": [
                "1"
            ],
            "Accept": [
                "text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.7"
            ],
            "Sec-Fetch-Site": [
                "cross-site"
            ],
            "Sec-Fetch-User": [
                "?1"
            ],
            "Sec-Fetch-Dest": [
                "document"
            ],
            "Cache-Control": [
                "max-age=0"
            ],
            "User-Agent": [
                "Mozilla/5.0 (Windows NT 6.3; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/97.0.4692.99 Safari/537.36"
            ],
            "Sec-Ch-Ua-Mobile": [
                "?0"
            ],
            "Sec-Fetch-Mode": [
                "navigate"
            ],
            "Accept-Encoding": [
                "gzip, deflate, br"
            ],
            "Sec-Ch-Ua": [
                "\"Chromium\";v=\"110\", \"Not A(Brand\";v=\"24\", \"Microsoft Edge\";v=\"110\""
            ]
        },
        "tls": {
            "resumed": false,
            "version": 772,
            "cipher_suite": 4865,
            "proto": "h2",
            "server_name": "dl.l.bradford.la"
        }
    },
    "duration": 0.024515995,
    "status": 502,
    "err_id": "kw3c4uxjn",
    "err_trace": "reverseproxy.statusError (reverseproxy.go:1299)"
}

Finally, I commented out the transport http block in system 1’s Caddyfile, and when I tried to access https://dl.l.bradford.la/nzbget/ I get a flood of these messages in system 1 (with the selected upstream and upstream roundtrip messages repeating forever, seems the browser keeps retrying since it’s a 308?):

{
    "level": "debug",
    "ts": 1677083493.5872257,
    "logger": "events",
    "msg": "event",
    "name": "tls_get_certificate",
    "id": "510e19a3-140a-4f6f-9e3a-4adcc9407272",
    "origin": "tls",
    "data": {
        "client_hello": {
            "CipherSuites": [
                4865,
                4866,
                4866,
                4867
            ],
            "ServerName": "dl.l.bradford.la",
            "SupportedCurves": [
                29,
                23,
                24
            ],
            "SupportedPoints": null,
            "SignatureSchemes": [
                1027,
                2052,
                1025,
                1283,
                2053,
                1281,
                2054,
                1537,
                513
            ],
            "SupportedProtos": [
                "h3"
            ],
            "SupportedVersions": [
                772
            ],
            "Conn": {}
        }
    }
}
{
    "level": "debug",
    "ts": 1677083493.5896192,
    "logger": "tls.handshake",
    "msg": "choosing certificate",
    "identifier": "dl.l.bradford.la",
    "num_choices": 1
}
{
    "level": "debug",
    "ts": 1677083493.5903869,
    "logger": "tls.handshake",
    "msg": "default certificate selection results",
    "identifier": "dl.l.bradford.la",
    "subjects": [
        "dl.l.bradford.la"
    ],
    "managed": true,
    "issuer_key": "acme-v02.api.letsencrypt.org-directory",
    "hash": "1ae13787c9b4ea79c861529899a2b2d4d3db7d459605f7838e566cc4c1430325"
}
{
    "level": "debug",
    "ts": 1677083493.5903957,
    "logger": "tls.handshake",
    "msg": "matched certificate in cache",
    "remote_ip": "172.16.10.2",
    "remote_port": "54380",
    "subjects": [
        "dl.l.bradford.la"
    ],
    "managed": true,
    "expiration": 1684621269,
    "hash": "1ae13787c9b4ea79c861529899a2b2d4d3db7d459605f7838e566cc4c1430325"
}
{
    "level": "debug",
    "ts": 1677083493.5930116,
    "logger": "events",
    "msg": "event",
    "name": "tls_get_certificate",
    "id": "44dbbd46-6f65-45a5-a8b0-09d83eed6536",
    "origin": "tls",
    "data": {
        "client_hello": {
            "CipherSuites": [
                2570,
                4865,
                4866,
                4866,
                4867,
                49195,
                49199,
                49196,
                49200,
                52393,
                52392,
                49171,
                49172,
                156,
                157,
                47,
                53
            ],
            "ServerName": "dl.l.bradford.la",
            "SupportedCurves": [
                27242,
                29,
                23,
                24
            ],
            "SupportedPoints": "AA==",
            "SignatureSchemes": [
                1027,
                2052,
                1025,
                1283,
                2053,
                1281,
                2054,
                1537
            ],
            "SupportedProtos": [
                "h2",
                "http/1.1"
            ],
            "SupportedVersions": [
                23130,
                772,
                771
            ],
            "Conn": {}
        }
    }
}
{
    "level": "debug",
    "ts": 1677083493.5930424,
    "logger": "tls.handshake",
    "msg": "choosing certificate",
    "identifier": "dl.l.bradford.la",
    "num_choices": 1
}
{
    "level": "debug",
    "ts": 1677083493.5930474,
    "logger": "tls.handshake",
    "msg": "default certificate selection results",
    "identifier": "dl.l.bradford.la",
    "subjects": [
        "dl.l.bradford.la"
    ],
    "managed": true,
    "issuer_key": "acme-v02.api.letsencrypt.org-directory",
    "hash": "1ae13787c9b4ea79c861529899a2b2d4d3db7d459605f7838e566cc4c1430325"
}
{
    "level": "debug",
    "ts": 1677083493.5930507,
    "logger": "tls.handshake",
    "msg": "matched certificate in cache",
    "remote_ip": "172.16.10.2",
    "remote_port": "17629",
    "subjects": [
        "dl.l.bradford.la"
    ],
    "managed": true,
    "expiration": 1684621269,
    "hash": "1ae13787c9b4ea79c861529899a2b2d4d3db7d459605f7838e566cc4c1430325"
}
{
    "level": "debug",
    "ts": 1677083493.5969803,
    "logger": "http.handlers.reverse_proxy",
    "msg": "selected upstream",
    "dial": "172.16.1.50:80",
    "total_upstreams": 1
}
{
    "level": "debug",
    "ts": 1677083493.6109798,
    "logger": "http.handlers.reverse_proxy",
    "msg": "upstream roundtrip",
    "upstream": "172.16.1.50:80",
    "duration": 0.013964488,
    "request": {
        "remote_ip": "172.16.10.2",
        "remote_port": "54380",
        "proto": "HTTP/3.0",
        "method": "GET",
        "host": "dl.l.bradford.la",
        "uri": "/nzbget/",
        "headers": {
            "Sec-Fetch-Dest": [
                "document"
            ],
            "Sec-Ch-Ua-Platform": [
                "\"Windows\""
            ],
            "Dnt": [
                "1"
            ],
            "Sec-Ch-Ua-Mobile": [
                "?0"
            ],
            "Sec-Fetch-Mode": [
                "navigate"
            ],
            "User-Agent": [
                "Mozilla/5.0 (Windows NT 6.3; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/97.0.4692.99 Safari/537.36"
            ],
            "Accept": [
                "text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.7"
            ],
            "Sec-Fetch-User": [
                "?1"
            ],
            "Accept-Language": [
                "en-US,en;q=0.9,fr;q=0.8"
            ],
            "Sec-Ch-Ua": [
                "\"Chromium\";v=\"110\", \"Not A(Brand\";v=\"24\", \"Microsoft Edge\";v=\"110\""
            ],
            "Upgrade-Insecure-Requests": [
                "1"
            ],
            "Sec-Fetch-Site": [
                "none"
            ],
            "X-Forwarded-For": [
                "172.16.10.2"
            ],
            "X-Forwarded-Host": [
                "dl.l.bradford.la"
            ],
            "Cache-Control": [
                "max-age=0"
            ],
            "Accept-Encoding": [
                "gzip, deflate, br"
            ],
            "X-Forwarded-Proto": [
                "https"
            ]
        },
        "tls": {
            "resumed": false,
            "version": 772,
            "cipher_suite": 4865,
            "proto": "h3",
            "server_name": "dl.l.bradford.la"
        }
    },
    "headers": {
        "Date": [
            "Wed, 22 Feb 2023 16:31:33 GMT"
        ],
        "Content-Length": [
            "0"
        ],
        "Location": [
            "https://dl.l.bradford.la/nzbget/"
        ],
        "Server": [
            "Caddy"
        ]
    },
    "status": 308
}

At the same time, system 2 gets these error messages on repeat as well:

{
    "level": "info",
    "ts": 1677083597.9435456,
    "logger": "http.log.access.log0",
    "msg": "handled request",
    "request": {
        "remote_ip": "172.16.1.71",
        "remote_port": "50020",
        "proto": "HTTP/1.1",
        "method": "GET",
        "host": "dl.l.bradford.la",
        "uri": "/nzbget/",
        "headers": {
            "User-Agent": [
                "Mozilla/5.0 (Windows NT 6.3; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/97.0.4692.99 Safari/537.36"
            ],
            "Sec-Fetch-Dest": [
                "document"
            ],
            "Sec-Fetch-Mode": [
                "navigate"
            ],
            "X-Forwarded-Host": [
                "dl.l.bradford.la"
            ],
            "Accept-Encoding": [
                "gzip, deflate, br"
            ],
            "Accept-Language": [
                "en-US,en;q=0.9,fr;q=0.8"
            ],
            "Cache-Control": [
                "max-age=0"
            ],
            "Sec-Ch-Ua": [
                "\"Chromium\";v=\"110\", \"Not A(Brand\";v=\"24\", \"Microsoft Edge\";v=\"110\""
            ],
            "Upgrade-Insecure-Requests": [
                "1"
            ],
            "X-Forwarded-For": [
                "172.16.10.2"
            ],
            "Accept": [
                "text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.7"
            ],
            "Sec-Ch-Ua-Mobile": [
                "?0"
            ],
            "Sec-Ch-Ua-Platform": [
                "\"Windows\""
            ],
            "Sec-Fetch-Site": [
                "none"
            ],
            "Sec-Fetch-User": [
                "?1"
            ],
            "X-Forwarded-Proto": [
                "https"
            ],
            "Dnt": [
                "1"
            ]
        }
    },
    "user_id": "",
    "duration": 0.0000122,
    "size": 0,
    "status": 308,
    "resp_headers": {
        "Location": [
            "https://dl.l.bradford.la/nzbget/"
        ],
        "Content-Type": [],
        "Server": [
            "Caddy"
        ],
        "Connection": [
            "close"
        ]
    }
}

It looks like system 2 caddy is redirecting me to the same URL, which is why the browser in this case has a TOO MANY REDIRECTS error.

I’m stumped and probably missing something fundamental.

The important part is that the incoming connection to Caddy is encrypted, over the public internet. Once it’s inside of your network, then the risk is drastically lower.

This looks wrong. It seems like you had the proxy configured to do a TLS but to port 80, which is mismatched.

If you make an HTTP request to a server that’s expecting HTTPS, then it’ll serve an HTTP->HTTPS redirect. So you’d have to change your 2nd server to only listen on port 80 with :80 or prefix the domain with http:// (depending on your routing needs).

Thanks for your responses @francislavoie. I changed my server 1 Caddyfile to

{
  debug
}

:80 {
  handle /nzbget/* {
    reverse_proxy http://nzbget:6789
  }
}

and server 2 Caddyfile to

{
  debug
}

(defaults) {
  tls {
    dns cloudflare {env.CF_API_TOKEN}
  }
}

dl.l.example.com {
  reverse_proxy http://172.16.1.50:80 {
#    transport http {
#      tls_insecure_skip_verify
#    }
  }
  import defaults
}

tv.l.example.com {
  reverse_proxy 172.16.1.51:8096

  import defaults
}

That seems to have fixed it, I’m able to access my services in subdirectories. Thanks!

I still want to force TLS between service 1 and service 2. I thought it would just take uncommenting the tls_insecure_skip_verify and setting the destination to https in system 1, while changing :80 to :443 in system 2, but I get 502 errors again, such as these on system 2:

"no matching certificates and no custom selection logic"
"all external certificate managers yielded no certificates and no errors"
"no certificate matching TLS ClientHello"
"http: TLS handshake error from 172.16.1.71:60680: no certificate available for '172.18.0.6'"

172.18.0.6 is the internal docker IP for system 2 caddy. Looks like it’s not automatically generating the certs like I would expect tls internal to do on system 2.

This comment makes me think I have the right idea about tls internal, but it seems it doesn’t work between two caddies?

The tricky thing to get right is SNI and host matching. If those aren’t right, then the TLS connection between them will still fail.

Using tls_insecure_skip_verify means you’re turning off all security so it’s effectively the same as using HTTP. The data’s encrypted, but it can be trivially man-in-the-middle attacked. And using TLS adds overhead (increased latency) so there’s essentially no benefit to it unless you do establish trust. And that involves copying the root CA cert from the upstream server to your front one and pointing the transport config to trust certificates signed by that root.

Looking closer at the logs, I’m seeing that "server_name":"" and "sni":"". So it doesn’t look like any SNI is being passed from system 1, which is starting to make sense.

Reading this got me to tls_server_name which got me to explicitly pass on the SNI to system 2. All good! TLS proxying between system 1 and system 2 is working. Now on to getting proper certs.

server 1 Caddyfile:

{
  debug
}

(defaults) {
  tls {
    dns cloudflare {env.CF_API_TOKEN}
  }
}

tv.l.example.com {
  reverse_proxy 172.16.1.51:8096

  import defaults
}

dl.l.example.com {
  reverse_proxy https://172.16.1.50 {
    transport http {
      tls_server_name {host}
      tls
      tls_insecure_skip_verify
    }

  }
  import defaults
}

New question:

Is there any way to use the same Lets Encrypt cert (that system 1 negotiates automatically) for the connection between system 1 and 2, since the SNI is the same? Any way to share these certs between caddy instances that’s in-built?

If not, I will probably use tls_dns_cloudflare on server 2 to simplify valid cert management.

You can share storage between the two instances. See Automatic HTTPS — Caddy Documentation

1 Like

Ultimately, I moved from subdirectories (it was a PITA to get each service to properly behave) to subdomains. I use a wildcard subdomain site block and the cloudflare dns provider works great and no more reverse proxying to another reverse proxy. Each system takes care of its own certs.

Thanks for all your help, I learned a lot.

1 Like

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.