Reverse_proxy to another reverse_proxy over https isn't working

1. The problem I’m having:

I’m trying to setup a Caddy server to reverse proxy a remote https server (which is also a Caddy server).

If i setup caddy to reverse proxy a remote http server on that same system (ie not behind the remote caddy) it works. For context, just running python -m http.server for the test. Once i change it to hit the remote Caddy i get the “white screen” issue. i get a 200 response with headers and no body.

What i’m trying to do is (i would assume) fairly simple:
User connects to https://project-app.testdom.com - Caddy proxies https://project.testdom.com.

What does work - setting the reverse_proxy to http://project.testdom.com:8000

2. Error messages and/or full log output:

wget from inside the originating Caddy docker container to https://project.testdom.com works file.
The destination Caddy docker also has no problem pinging project-app.testdom.com), so i’m fairly sure i’ve ruled out any possible network or routing issues but i’m happy to run additional verifications.

Source server IP is 10.0.2.180, which is also where i ran the below curl from.

curl output:

*   Trying 10.0.2.180:443...
* Connected to project-app.testdom.com (10.0.2.180) port 443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
*  CAfile: /etc/ssl/certs/ca-certificates.crt
*  CApath: /etc/ssl/certs
* TLSv1.0 (OUT), TLS header, Certificate Status (22):
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* TLSv1.2 (IN), TLS header, Certificate Status (22):
* TLSv1.3 (IN), TLS handshake, Server hello (2):
* TLSv1.2 (IN), TLS header, Finished (20):
* TLSv1.2 (IN), TLS header, Supplemental data (23):
* TLSv1.3 (IN), TLS handshake, Encrypted Extensions (8):
* TLSv1.2 (IN), TLS header, Supplemental data (23):
* TLSv1.3 (IN), TLS handshake, Certificate (11):
* TLSv1.2 (IN), TLS header, Supplemental data (23):
* TLSv1.3 (IN), TLS handshake, CERT verify (15):
* TLSv1.2 (IN), TLS header, Supplemental data (23):
* TLSv1.3 (IN), TLS handshake, Finished (20):
* TLSv1.2 (OUT), TLS header, Finished (20):
* TLSv1.3 (OUT), TLS change cipher, Change cipher spec (1):
* TLSv1.2 (OUT), TLS header, Supplemental data (23):
* TLSv1.3 (OUT), TLS handshake, Finished (20):
* SSL connection using TLSv1.3 / TLS_AES_128_GCM_SHA256
* ALPN, server accepted to use h2
* Server certificate:
*  subject: CN=*.testdom.com
*  start date: Jun 20 00:00:00 2023 GMT
*  expire date: Jul 20 23:59:59 2024 GMT
*  subjectAltName: host "project-app.testdom.com" matched cert's "*.testdom.com"
*  issuer: C=GB; ST=Greater Manchester; L=Salford; O=Sectigo Limited; CN=Sectigo RSA Domain Validation Secure Server CA
*  SSL certificate verify ok.
* Using HTTP2, server supports multiplexing
* Connection state changed (HTTP/2 confirmed)
* Copying HTTP/2 data in stream buffer to connection buffer after upgrade: len=0
* TLSv1.2 (OUT), TLS header, Supplemental data (23):
* TLSv1.2 (OUT), TLS header, Supplemental data (23):
* TLSv1.2 (OUT), TLS header, Supplemental data (23):
* Using Stream ID: 1 (easy handle 0x5601ae80aeb0)
* TLSv1.2 (OUT), TLS header, Supplemental data (23):
> GET / HTTP/2
> Host: project-app.testdom.com
> user-agent: curl/7.81.0
> accept: */*
>
* TLSv1.2 (IN), TLS header, Supplemental data (23):
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
* TLSv1.2 (IN), TLS header, Supplemental data (23):
* Connection state changed (MAX_CONCURRENT_STREAMS == 250)!
* TLSv1.2 (OUT), TLS header, Supplemental data (23):
* TLSv1.2 (IN), TLS header, Supplemental data (23):
* TLSv1.2 (IN), TLS header, Supplemental data (23):
* TLSv1.2 (IN), TLS header, Supplemental data (23):
< HTTP/2 200
< alt-svc: h3=":443"; ma=2592000
< date: Thu, 08 Feb 2024 03:35:32 GMT
< server: Caddy
< server: Caddy
< content-length: 0
<
* Connection #0 to host project-app.testdom.com left intact

Logs from the originating Caddy (project-app.testdom.com)

{"level":"info","ts":1707360930.5410893,"msg":"using provided configuration","config_file":"/etc/caddy/Caddyfile","config_adapter":"caddyfile"}
{"level":"warn","ts":1707360930.5430615,"logger":"caddyfile","msg":"Unnecessary header_up X-Forwarded-For: the reverse proxy's default behavior is to pass headers to the upstream"}
{"level":"warn","ts":1707360930.5431561,"logger":"caddyfile","msg":"Unnecessary header_up X-Forwarded-Proto: the reverse proxy's default behavior is to pass headers to the upstream"}
{"level":"warn","ts":1707360930.5463045,"msg":"Caddyfile input is not formatted; run 'caddy fmt --overwrite' to fix inconsistencies","adapter":"caddyfile","file":"/etc/caddy/Caddyfile","line":61}
{"level":"info","ts":1707360930.5476136,"logger":"admin","msg":"admin endpoint started","address":"localhost:2019","enforce_origin":false,"origins":["//localhost:2019","//[::1]:2019","//127.0.0.1:2019"]}
{"level":"info","ts":1707360930.5481033,"logger":"tls.cache.maintenance","msg":"started background certificate maintenance","cache":"0xc0005a9180"}
{"level":"debug","ts":1707360930.5485675,"logger":"events","msg":"event","name":"cached_unmanaged_cert","id":"09e52156-601b-4ef9-9620-7835dddd3d1f","origin":"tls","data":{"sans":["*.testdom.com","testdom.com"]}}
{"level":"debug","ts":1707360930.5485957,"logger":"tls.cache","msg":"added certificate to cache","subjects":["*.testdom.com","testdom.com"],"expiration":1721520000,"managed":false,"issuer_key":"","hash":"584290dbd4acc536535658d78c5e2cb038bc0beddee28578e7ab8f458412c145","cache_size":1,"cache_capacity":10000}
{"level":"info","ts":1707360930.5486772,"logger":"http.auto_https","msg":"skipping automatic certificate management because one or more matching certificates are already loaded","domain":"project-app.testdom.com","server_name":"srv0"}
{"level":"info","ts":1707360930.5486884,"logger":"http.auto_https","msg":"skipping automatic certificate management because one or more matching certificates are already loaded","domain":"wiki-app.testdom.com","server_name":"srv0"}
{"level":"info","ts":1707360930.5486922,"logger":"http.auto_https","msg":"skipping automatic certificate management because one or more matching certificates are already loaded","domain":"login.testdom.com","server_name":"srv0"}
{"level":"info","ts":1707360930.5486958,"logger":"http.auto_https","msg":"skipping automatic certificate management because one or more matching certificates are already loaded","domain":"apps.testdom.com","server_name":"srv0"}
{"level":"info","ts":1707360930.5486987,"logger":"http.auto_https","msg":"enabling automatic HTTP->HTTPS redirects","server_name":"srv0"}
{"level":"debug","ts":1707360930.5487416,"logger":"http.auto_https","msg":"adjusted config","tls":{"automation":{"policies":[{}]}},"http":{"servers":{"remaining_auto_https_redirects":{"listen":[":80"],"routes":[{},{}],"logs":{"logger_names":{"apps.testdom.com":"log1","login.testdom.com":"log0","project-app.testdom.com":"log3","wiki-app.testdom.com":"log2"}}},"srv0":{"listen":[":443"],"routes":[{"handle":[{"handler":"subroute","routes":[{"handle":[{"handler":"reverse_proxy","headers":{"request":{"set":{"Host":["{http.request.host}"],"X-Forwarded-Port":["{server_port}"],"X-Forwarded-Ssl":["on"],"X-Real-Ip":["{http.request.remote}"],"X-Url-Scheme":["{http.request.scheme}"]}}},"transport":{"compression":false,"protocol":"http","tls":{"insecure_skip_verify":true}},"upstreams":[{"dial":"project.testdom.com:443"}]}]}]}],"terminal":true},{"handle":[{"handler":"subroute","routes":[{"handle":[{"handler":"headers","response":{"set":{"Server":["Test-Server/1.1"]}}},{"handler":"reverse_proxy","headers":{"request":{"set":{"Host":["{http.request.host}"],"X-Forwarded-For":["{http.request.remote}"],"X-Forwarded-Port":["{server_port}"],"X-Forwarded-Proto":["{http.request.scheme}"],"X-Real-Ip":["{http.request.remote}"]}}},"transport":{"protocol":"http","tls":{"insecure_skip_verify":true}},"upstreams":[{"dial":"wiki.testdom.com:443"}]}]}]}],"terminal":true},{"handle":[{"handler":"subroute","routes":[{"handle":[{"handler":"headers","response":{"set":{"Server":["Test-Server/1.1"]}}},{"handler":"reverse_proxy","upstreams":[{"dial":"testsrv2:5180"}]}]}]}],"terminal":true},{"handle":[{"handler":"subroute","routes":[{"handle":[{"handler":"headers","response":{"set":{"Server":["Test-Server/1.1"]}}},{"handler":"reverse_proxy","upstreams":[{"dial":"testsrv:9520"}]}]}]}],"terminal":true}],"tls_connection_policies":[{"match":{"sni":["project-app.testdom.com"]},"certificate_selection":{"any_tag":["cert0"]},"protocol_min":"tls1.3"},{"match":{"sni":["wiki-app.testdom.com"]},"certificate_selection":{"any_tag":["cert0"]},"protocol_min":"tls1.3"},{"match":{"sni":["login.testdom.com"]},"certificate_selection":{"any_tag":["cert0"]},"protocol_min":"tls1.3"},{"match":{"sni":["apps.testdom.com"]},"certificate_selection":{"any_tag":["cert0"]},"protocol_min":"tls1.3"},{}],"automatic_https":{},"logs":{"logger_names":{"apps.testdom.com":"log1","login.testdom.com":"log0","project-app.testdom.com":"log3","wiki-app.testdom.com":"log2"}}}}}}
{"level":"info","ts":1707360930.5493941,"logger":"http","msg":"enabling HTTP/3 listener","addr":":443"}
{"level":"info","ts":1707360930.549538,"msg":"failed to sufficiently increase receive buffer size (was: 208 kiB, wanted: 2048 kiB, got: 416 kiB). See https://github.com/quic-go/quic-go/wiki/UDP-Buffer-Sizes for details."}
{"level":"debug","ts":1707360930.549634,"logger":"http","msg":"starting server loop","address":"[::]:443","tls":true,"http3":true}
{"level":"info","ts":1707360930.549642,"logger":"http.log","msg":"server running","name":"srv0","protocols":["h1","h2","h3"]}
{"level":"debug","ts":1707360930.5496693,"logger":"http","msg":"starting server loop","address":"[::]:80","tls":false,"http3":false}
{"level":"info","ts":1707360930.5496728,"logger":"http.log","msg":"server running","name":"remaining_auto_https_redirects","protocols":["h1","h2","h3"]}
{"level":"info","ts":1707360930.5498781,"msg":"autosaved config (load with --resume flag)","file":"/config/caddy/autosave.json"}
{"level":"info","ts":1707360930.5498872,"msg":"serving initial configuration"}
{"level":"warn","ts":1707360930.5512843,"logger":"tls","msg":"storage cleaning happened too recently; skipping for now","storage":"FileStorage:/data/caddy","instance":"b6bffde1-c654-424a-961e-de6dad85de56","try_again":1707447330.551282,"try_again_in":86399.999999285}
{"level":"info","ts":1707360930.551461,"logger":"tls","msg":"finished cleaning storage units"}
{"level":"debug","ts":1707360957.3604174,"logger":"events","msg":"event","name":"tls_get_certificate","id":"2a4da2c9-a00a-4ca3-bcd5-0714777825d0","origin":"tls","data":{"client_hello":{"CipherSuites":[4866,4867,4865,49196,49200,159,52393,52392,52394,49195,49199,158,49188,49192,107,49187,49191,103,49162,49172,57,49161,49171,51,157,156,61,60,53,47,255],"ServerName":"project-app.testdom.com","SupportedCurves":[29,23,30,25,24,256,257,258,259,260],"SupportedPoints":"AAEC","SignatureSchemes":[1027,1283,1539,2055,2056,2057,2058,2059,2052,2053,2054,1025,1281,1537,771,769,770,1026,1282,1538],"SupportedProtos":["h2","http/1.1"],"SupportedVersions":[772,771],"RemoteAddr":{"IP":"10.0.2.180","Port":45730,"Zone":""},"LocalAddr":{"IP":"172.18.0.5","Port":443,"Zone":""}}}}
{"level":"debug","ts":1707360957.3610153,"logger":"tls.handshake","msg":"no matching certificate; will choose from all certificates","identifier":"project-app.testdom.com"}
{"level":"debug","ts":1707360957.3610215,"logger":"tls.handshake","msg":"choosing certificate","identifier":"project-app.testdom.com","num_choices":1}
{"level":"debug","ts":1707360957.3610375,"logger":"tls.handshake","msg":"custom certificate selection results","identifier":"project-app.testdom.com","subjects":["*.testdom.com","testdom.com"],"managed":false,"issuer_key":"","hash":"584290dbd4acc536535658d78c5e2cb038bc0beddee28578e7ab8f458412c145"}
{"level":"debug","ts":1707360957.361057,"logger":"tls.handshake","msg":"matched certificate in cache","remote_ip":"10.0.2.180","remote_port":"45730","subjects":["*.testdom.com","testdom.com"],"managed":false,"expiration":1721520000,"hash":"584290dbd4acc536535658d78c5e2cb038bc0beddee28578e7ab8f458412c145"}
{"level":"debug","ts":1707360957.3663607,"logger":"http.handlers.reverse_proxy","msg":"selected upstream","dial":"project.testdom.com:443","total_upstreams":1}
{"level":"debug","ts":1707360957.3740268,"logger":"http.handlers.reverse_proxy","msg":"upstream roundtrip","upstream":"project.testdom.com:443","duration":0.007593216,"request":{"remote_ip":"10.0.2.180","remote_port":"45730","client_ip":"10.0.2.180","proto":"HTTP/2.0","method":"GET","host":"project-app.testdom.com","uri":"/","headers":{"User-Agent":["curl/7.81.0"],"X-Forwarded-For":["10.0.2.180"],"X-Forwarded-Proto":["https"],"X-Forwarded-Host":["project-app.testdom.com"],"X-Real-Ip":["10.0.2.180:45730"],"X-Url-Scheme":["https"],"Accept":["*/*"],"X-Forwarded-Port":["{server_port}"],"X-Forwarded-Ssl":["on"]},"tls":{"resumed":false,"version":772,"cipher_suite":4865,"proto":"h2","server_name":"project-app.testdom.com"}},"headers":{"Alt-Svc":["h3=\":443\"; ma=2592000"],"Server":["Caddy"],"Content-Length":["0"],"Date":["Thu, 08 Feb 2024 02:55:57 GMT"]},"status":200}

Logs from the destination Caddy (project.testdom.com)

{"level":"info","ts":1707360936.4139159,"msg":"using provided configuration","config_file":"/etc/caddy/Caddyfile","config_adapter":"caddyfile"}
{"level":"warn","ts":1707360936.4152844,"msg":"Caddyfile input is not formatted; run 'caddy fmt --overwrite' to fix inconsistencies","adapter":"caddyfile","file":"/etc/caddy/Caddyfile","line":1}
{"level":"info","ts":1707360936.416333,"logger":"admin","msg":"admin endpoint started","address":"localhost:2019","enforce_origin":false,"origins":["//localhost:2019","//[::1]:2019","//127.0.0.1:2019"]}
{"level":"debug","ts":1707360936.417219,"logger":"events","msg":"event","name":"cached_unmanaged_cert","id":"1c64f88c-f298-41e5-a011-76210f4bfe28","origin":"tls","data":{"sans":["*.testdom.com","testdom.com"]}}
{"level":"debug","ts":1707360936.417244,"logger":"tls.cache","msg":"added certificate to cache","subjects":["*.testdom.com","testdom.com"],"expiration":1721520000,"managed":false,"issuer_key":"","hash":"584290dbd4acc536535658d78c5e2cb038bc0beddee28578e7ab8f458412c145","cache_size":1,"cache_capacity":10000}
{"level":"warn","ts":1707360936.4173205,"logger":"http.auto_https","msg":"server is listening only on the HTTP port, so no automatic HTTPS will be applied to this server","server_name":"srv1","http_port":80}
{"level":"info","ts":1707360936.4174397,"logger":"http.auto_https","msg":"skipping automatic certificate management because one or more matching certificates are already loaded","domain":"project.testdom.com","server_name":"srv0"}
{"level":"info","ts":1707360936.417451,"logger":"http.auto_https","msg":"enabling automatic HTTP->HTTPS redirects","server_name":"srv0"}
{"level":"debug","ts":1707360936.4174774,"logger":"http.auto_https","msg":"adjusted config","tls":{"automation":{"policies":[{}]}},"http":{"servers":{"srv0":{"listen":[":443"],"routes":[{"handle":[{"handler":"subroute","routes":[{"handle":[{"handler":"headers","response":{"set":{"Server":["Caddy"]}}},{"handler":"reverse_proxy","upstreams":[{"dial":"project:80"}]}]}]}],"terminal":true}],"tls_connection_policies":[{"match":{"sni":["project.testdom.com"]},"certificate_selection":{"any_tag":["cert0"]},"protocol_min":"tls1.3"},{}],"automatic_https":{},"logs":{"logger_names":{"project.testdom.com":"log0"}}},"srv1":{"listen":[":80"],"routes":[{"handle":[{"handler":"subroute","routes":[{"handle":[{"handler":"static_response","headers":{"Location":["https://project.testdom.com{http.request.uri}"]},"status_code":301}]}]}],"terminal":true},{}],"automatic_https":{"disable":true}}}}}
{"level":"info","ts":1707360936.4176705,"logger":"tls.cache.maintenance","msg":"started background certificate maintenance","cache":"0xc0004a9000"}
{"level":"info","ts":1707360936.4180548,"logger":"http","msg":"enabling HTTP/3 listener","addr":":443"}
{"level":"info","ts":1707360936.418153,"msg":"failed to sufficiently increase receive buffer size (was: 208 kiB, wanted: 2048 kiB, got: 416 kiB). See https://github.com/quic-go/quic-go/wiki/UDP-Buffer-Sizes for details."}
{"level":"debug","ts":1707360936.4182482,"logger":"http","msg":"starting server loop","address":"[::]:443","tls":true,"http3":true}
{"level":"info","ts":1707360936.4182575,"logger":"http.log","msg":"server running","name":"srv0","protocols":["h1","h2","h3"]}
{"level":"debug","ts":1707360936.4183028,"logger":"http","msg":"starting server loop","address":"[::]:80","tls":false,"http3":false}
{"level":"info","ts":1707360936.4183252,"logger":"http.log","msg":"server running","name":"srv1","protocols":["h1","h2","h3"]}
{"level":"info","ts":1707360936.4185736,"msg":"autosaved config (load with --resume flag)","file":"/config/caddy/autosave.json"}
{"level":"info","ts":1707360936.4185855,"msg":"serving initial configuration"}
{"level":"warn","ts":1707360936.4195309,"logger":"tls","msg":"storage cleaning happened too recently; skipping for now","storage":"FileStorage:/data/caddy","instance":"17dd9229-c91d-4a92-a96f-3d9edf9289e8","try_again":1707447336.4195285,"try_again_in":86399.999999649}
{"level":"info","ts":1707360936.4196296,"logger":"tls","msg":"finished cleaning storage units"}
{"level":"debug","ts":1707360957.3736312,"logger":"events","msg":"event","name":"tls_get_certificate","id":"3a166625-7334-4ca7-9c45-c31f5be9804b","origin":"tls","data":{"client_hello":{"CipherSuites":[49195,49199,49196,49200,52393,52392,49161,49171,49162,49172,156,157,47,53,49170,10,4865,4866,4867],"ServerName":"project.testdom.com","SupportedCurves":[29,23,24,25],"SupportedPoints":"AA==","SignatureSchemes":[2052,1027,2055,2053,2054,1025,1281,1537,1283,1539,513,515],"SupportedProtos":["h2","http/1.1"],"SupportedVersions":[772,771],"RemoteAddr":{"IP":"10.0.2.180","Port":43042,"Zone":""},"LocalAddr":{"IP":"172.26.0.2","Port":443,"Zone":""}}}}
{"level":"debug","ts":1707360957.3739464,"logger":"tls.handshake","msg":"no matching certificate; will choose from all certificates","identifier":"project.testdom.com"}
{"level":"debug","ts":1707360957.3739681,"logger":"tls.handshake","msg":"choosing certificate","identifier":"project.testdom.com","num_choices":1}
{"level":"debug","ts":1707360957.374018,"logger":"tls.handshake","msg":"custom certificate selection results","identifier":"project.testdom.com","subjects":["*.testdom.com","testdom.com"],"managed":false,"issuer_key":"","hash":"584290dbd4acc536535658d78c5e2cb038bc0beddee28578e7ab8f458412c145"}
{"level":"debug","ts":1707360957.374051,"logger":"tls.handshake","msg":"matched certificate in cache","remote_ip":"10.0.2.180","remote_port":"43042","subjects":["*.testdom.com","testdom.com"],"managed":false,"expiration":1721520000,"hash":"584290dbd4acc536535658d78c5e2cb038bc0beddee28578e7ab8f458412c145"}
{"level":"info","ts":1707360957.3778062,"logger":"http.log.access","msg":"handled request","request":{"remote_ip":"10.0.2.180","remote_port":"43042","client_ip":"10.0.2.180","proto":"HTTP/2.0","method":"GET","host":"project-app.testdom.com","uri":"/","headers":{"User-Agent":["curl/7.81.0"],"X-Forwarded-Port":["{server_port}"],"Accept":["*/*"],"X-Forwarded-Proto":["https"],"X-Url-Scheme":["https"],"X-Forwarded-Ssl":["on"],"X-Forwarded-For":["10.0.2.180"],"X-Forwarded-Host":["project-app.testdom.com"],"X-Real-Ip":["10.0.2.180:45730"]},"tls":{"resumed":false,"version":772,"cipher_suite":4865,"proto":"h2","server_name":"project.testdom.com"}},"bytes_read":0,"user_id":"","duration":0.00001175,"size":0,"status":0,"resp_headers":{"Server":["Caddy"],"Alt-Svc":["h3=\":443\"; ma=2592000"]}}

3. Caddy version:

v2.7.6 on both servers

4. How I installed and ran Caddy:

docker-compose.

a. System environment:

Docker on Ubuntu

b. Command:

N/A

c. Service/unit/compose file:

 caddy:
    image: caddy:latest
    container_name: caddy
    hostname: caddy
    restart: unless-stopped
    ports:
      - "80:80"
      - "443:443"
    volumes:
      - /opt/caddy/etc/Caddyfile:/etc/caddy/Caddyfile
      - /opt/caddy/etc/ssl:/etc/caddy/ssl
      - /opt/caddy/data:/data
      - /opt/caddy/config:/config
      - /opt/caddy/log:/log

d. My complete Caddy config:

I’ve tried different configuration options but basically:

{
        debug
}
https://project-app.testdom.com {
        reverse_proxy https://project.testdom.com {
            transport http {
                tls
                tls_insecure_skip_verify
            }
            header_up X-Forwarded-Ssl on
            header_up Host {host}
            header_up X-Real-IP {remote}
            header_up X-Forwarded-Port {server_port}
            header_up X-Url-Scheme {scheme}
        }
        tls /etc/caddy/ssl/fullchain.pem /etc/caddy/ssl/key.pem {
                protocols tls1.3
        }
        log {
                output file /log/access_app-project_443.log {
                        roll_size 1gb
                        roll_keep 5
                        roll_keep_for 720h
                }
        }
}

5. Links to relevant resources:

NA

This is wrong – you’re sending the header project-app.testdom.com upstream, which means if the upstream is matching on project.testdom.com it’ll reject the request.

See the docs:

1 Like

You’re of course, absolutely right! i got so hung up on possible TLS issues that i didn’t even go there. i wish Caddy had a more obvious way to basically state: Received a request with xx Host and i have nothing to do with it. Would make it easier to debug, but now i know to look for it.
Still have some work to deal with some hard coded redirections and possible rewrites, but this moves me along.
Thanks! :slight_smile: