SEE EDIT AT BOTTOM, problem “solved” but not sure why.
1. Caddy version (caddy version): 2.4.5
2. How I run Caddy:
I run caddy using a Dockerfile that is deployed on Fly at https://encore-caddy.fly.dev. I use Caddy as a reverse proxy for custom domains to be proxied to my website https://encoreapp.co which is a NextJS app deployed on Vercel. So for example a domain like https://danielhex.com will be proxied to https://encoreapp.co
c. Service/unit/compose file:
FROM caddy:2.4.5-builder AS builder
RUN xcaddy build \
--with github.com/silinternational/certmagic-storage-dynamodb/v2
FROM caddy:2.4.5
COPY ./app/caddy/src/Caddyfile /etc/caddy/Caddyfile
COPY --from=builder /usr/bin/caddy /usr/bin/caddy
a. System environment: Docker in a Fly VM
d. My complete Caddyfile or JSON config:
{
debug
storage dynamodb caddy_cert {
aws_region "us-east-1"
}
storage_clean_interval 32d
email "support@uncvrd.co"
acme_ca "https://acme-v02.api.letsencrypt.org/directory"
on_demand_tls {
ask "https://encore-graphql.fly.dev/tls/ask"
}
}
:443 {
tls {
on_demand
}
reverse_proxy https://encoreapp.co {
header_up Host {http.reverse_proxy.upstream.hostport}
header_up X-Forwarded-Host {host}
}
}
3. The problem I’m having:
I’m having an issue with my proxy connecting to my https://encoreapp.co on the request. The error the proxy returns is an EOF which I’m having trouble deciphering why it cannot return any information. The proxy was working fine for several months! But if you try and hit Good World by Daniel Hex - Encore you will see that it now returns a 502 Error with a content length of 0. The app correctly responds when accessing the URL without the proxy URL: https://encoreapp.co/goodworld
4. Error messages and/or full log output:
Logs right after i reload with the config mentioned above
2022-01-31T21:50:11.355 app[016c44e6] lax [info] {"level":"info","ts":1643665811.353935,"logger":"admin.api","msg":"received request","method":"POST","host":"localhost:2019","uri":"/load","remote_addr":"127.0.0.1:41058","headers":{"Accept-Encoding":["gzip"],"Content-Length":["715"],"Content-Type":["application/json"],"Origin":["localhost:2019"],"User-Agent":["Go-http-client/1.1"]}}
2022-01-31T21:50:11.355 app[016c44e6] lax [info] {"level":"info","ts":1643665811.3543968,"logger":"admin","msg":"admin endpoint started","address":"tcp/localhost:2019","enforce_origin":false,"origins":["localhost:2019","[::1]:2019","127.0.0.1:2019"]}
2022-01-31T21:50:11.355 app[016c44e6] lax [info] {"level":"info","ts":1643665811.3545587,"logger":"http","msg":"server is listening only on the HTTPS port but has no TLS connection policies; adding one to enable TLS","server_name":"srv0","https_port":443}
2022-01-31T21:50:11.355 app[016c44e6] lax [info] {"level":"info","ts":1643665811.3545732,"logger":"http","msg":"enabling automatic HTTP->HTTPS redirects","server_name":"srv0"}
2022-01-31T21:50:11.355 app[016c44e6] lax [info] {"level":"debug","ts":1643665811.3547354,"logger":"http","msg":"starting server loop","address":"[::]:443","http3":false,"tls":true}
2022-01-31T21:50:11.355 app[016c44e6] lax [info] {"level":"debug","ts":1643665811.354747,"logger":"http","msg":"starting server loop","address":"[::]:80","http3":false,"tls":false}
2022-01-31T21:50:11.355 app[016c44e6] lax [info] {"level":"info","ts":1643665811.3548439,"logger":"tls.cache.maintenance","msg":"started background certificate maintenance","cache":"0xc0000f4620"}
2022-01-31T21:50:11.357 app[016c44e6] lax [info] {"level":"info","ts":1643665811.3570766,"logger":"tls.cache.maintenance","msg":"stopped background certificate maintenance","cache":"0xc000178fc0"}
2022-01-31T21:50:11.357 app[016c44e6] lax [info] {"level":"info","ts":1643665811.3573644,"msg":"autosaved config (load with --resume flag)","file":"/config/caddy/autosave.json"}
2022-01-31T21:50:11.357 app[016c44e6] lax [info] {"level":"info","ts":1643665811.3574514,"logger":"admin.api","msg":"load complete"}
2022-01-31T21:50:11.359 app[016c44e6] lax [info] {"level":"info","ts":1643665811.358709,"logger":"admin","msg":"stopped previous server","address":"tcp/localhost:2019"}
2022-01-31T21:51:10.496 app[016c44e6] lax [info] {"level":"debug","ts":1643665870.4959455,"logger":"tls.handshake","msg":"no matching certificates and no custom selection logic","identifier":"danielhex.com"}
2022-01-31T21:51:10.496 app[016c44e6] lax [info] {"level":"debug","ts":1643665870.4959803,"logger":"tls.handshake","msg":"no matching certificates and no custom selection logic","identifier":"*.com"}
2022-01-31T21:51:10.496 app[016c44e6] lax [info] {"level":"debug","ts":1643665870.4959865,"logger":"tls.handshake","msg":"no matching certificates and no custom selection logic","identifier":"*.*"}
2022-01-31T21:51:11.074 app[016c44e6] lax [info] {"level":"debug","ts":1643665871.074458,"logger":"tls.cache","msg":"added certificate to cache","subjects":["danielhex.com"],"expiration":1651430661,"managed":true,"issuer_key":"acme-v02.api.letsencrypt.org-directory","hash":"30366ae10e9ee098ef43d4cc394e9a4f3e5078e59e52201845ad28ce43491111"}
2022-01-31T21:51:11.075 app[016c44e6] lax [info] {"level":"debug","ts":1643665871.0747228,"logger":"tls.handshake","msg":"loaded certificate from storage","subjects":["danielhex.com"],"managed":true,"expiration":1651430661,"hash":"30366ae10e9ee098ef43d4cc394e9a4f3e5078e59e52201845ad28ce43491111"}
2022-01-31T21:51:11.145 app[016c44e6] lax [info] {"level":"debug","ts":1643665871.1448245,"logger":"http.handlers.reverse_proxy","msg":"upstream roundtrip","upstream":"encoreapp.co:443","request":{"remote_addr":"93.187.216.173:59268","proto":"HTTP/2.0","method":"GET","host":"encoreapp.co:443","uri":"/goodworld","headers":{"Sec-Fetch-User":["?1"],"Sec-Fetch-Dest":["document"],"Accept-Language":["en-US,en;q=0.9"],"Accept":["text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.9"],"X-Forwarded-Proto":["https"],"User-Agent":["Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/96.0.4664.110 Safari/537.36"],"Sec-Ch-Ua":["\" Not A;Brand\";v=\"99\", \"Chromium\";v=\"96\", \"Google Chrome\";v=\"96\""],"Cache-Control":["max-age=0"],"Sec-Ch-Ua-Platform":["\"macOS\""],"Sec-Fetch-Site":["none"],"Sec-Fetch-Mode":["navigate"],"X-Forwarded-For":["93.187.216.173"],"Sec-Ch-Ua-Mobile":["?0"],"Accept-Encoding":["gzip, deflate, br"],"Upgrade-Insecure-Requests":["1"],"X-Forwarded-Host":["danielhex.com"]},"tls":{"resumed":false,"version":772,"cipher_suite":4865,"proto":"h2","proto_mutual":true,"server_name":"danielhex.com"}},"duration":0.021650586,"error":"EOF"}
2022-01-31T21:51:11.145 app[016c44e6] lax [info] {"level":"error","ts":1643665871.1449838,"logger":"http.log.error","msg":"EOF","request":{"remote_addr":"93.187.216.173:59268","proto":"HTTP/2.0","method":"GET","host":"danielhex.com","uri":"/goodworld","headers":{"Cache-Control":["max-age=0"],"Sec-Ch-Ua-Platform":["\"macOS\""],"Upgrade-Insecure-Requests":["1"],"Accept":["text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.9"],"Sec-Fetch-Dest":["document"],"Accept-Encoding":["gzip, deflate, br"],"Accept-Language":["en-US,en;q=0.9"],"Sec-Ch-Ua":["\" Not A;Brand\";v=\"99\", \"Chromium\";v=\"96\", \"Google Chrome\";v=\"96\""],"Sec-Ch-Ua-Mobile":["?0"],"User-Agent":["Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/96.0.4664.110 Safari/537.36"],"Sec-Fetch-Site":["none"],"Sec-Fetch-Mode":["navigate"],"Sec-Fetch-User":["?1"]},"tls":{"resumed":false,"version":772,"cipher_suite":4865,"proto":"h2","proto_mutual":true,"server_name":"danielhex.com"}},"duration":0.022007196,"status":502,"err_id":"xxhh2ye03","err_trace":"reverseproxy.statusError (reverseproxy.go:858)"}
5. What I already tried:
The docs state to use the following in the reverse proxy here: reverse_proxy (Caddyfile directive) — Caddy Documentation
header_up Host {upstream_hostport}
header_up X-Forwarded-Host {host}
but after updating my Caddyfile and running caddy reload, I receive the same error.
6. Links to relevant resources:
I found a similar problem here: Reverse-proxy configuration
And the problem was resolved by disabling https however, I don’t think this is possible since my website is hosted on Vercel and my Reverse Proxy is hosted on Fly so I still need to make sure traffic is encrypted between my proxy and Vercel, correct?
Let me know if I can provide any other information, wish I had more to go with regarding the error.
EDIT RESOLVED, but not sure why…:
Alright…so I’m not sure why this helped but I fully rebuilt the dockerfile and redeployed my Caddy application and now the URL proxies correctly. Is there some sort of ephemeral key that is stored in the container that is used for TLS management that, if the app dies or loses state, could affect the re-signing of certs after a reboot (since I dont have persistent storage on my container)? My reasoning is that updating with caddy reload wasn’t changing anything but a full image rebuild did help so maybe I lost a key during a restart that isn’t stored in a persistent volume 
especially since the logs didn’t really give too much info. Caddy was receiving the request but wasn’t hitting my proxied webpage at Vercel, I figured that something TLS related was happening which caused the upstream server to close preemptively…as we saw was the resolution to the other question I linked in my original post!