HTTP/1.1 502 Bad Gateway while reverse proxy to DOCKER container port from CADDY in host machine

1. Output of caddy version:

v2.4.3 h1:Y1FaV2N4WO3rBqxSYA8UZsZTQdN+PwcoOcAiZTM8C0I=

2. How I run Caddy:

  1. I’m having the caddy executable in my machine at a folder /caddy.
  2. Open the terminal
  3. sudo nohup ./caddy_exe start

a. System environment:

Ubuntu 20.04.5 LTS

b. Command:

sudo nohup ./caddy_exe start

c. Service/unit/compose file:

NOT IN DOCKER

d. My complete Caddy config:

*.csez.zohocorpin.com:9006 {
        log {
                output file live_access.log
        }

        tls internal {
                alpn http1.1
        }

        reverse_proxy /* https://zcem-u20-2.csez.zohocorpin.com:9000 {

                header_up Host {host}
        }
}

3. The problem I’m having:

I have my web application stack running in docker along with CADDY in the same container. The caddy in docker receives 9000 port and redirects it to 8444. I have exposed the docker port 9000. When I access the docker container from my host machine browser using the same 9000 port, its working properly and I am able to get access logs in the caddy (inside docker).

For some reason, I need to run a CADDY in my host machine for reverse proxy. I am trying to redirect PORT : 9006 → 9000 (DOCKER). But I am getting HTTP/1.1 502 Bad Gateway error

4. Error messages and/or full log output:

NO HUP OUT :

{"level":"error","ts":1668606946.9699006,"logger":"http.log.error.log0","msg":"x509: certificate signed by unknown authority (possibly because of \"x509: ECDSA verification failure\" while trying to verify candidate authority certificate \"Caddy Local Authority - 2022 ECC Root\")","request":{"remote_addr":"10.59.2.126:51548","proto":"HTTP/1.1","method":"GET","host":"zcem-u20-2.csez.zohocorpin.com:9006","uri":"/userhome/zcemu2024/admindashboard","headers":{"Cache-Control":["max-age=0"],"Sec-Ch-Ua-Platform":["\"macOS\""],"Accept":["text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.9"],"Sec-Fetch-Site":["none"],"Accept-Language":["en-GB,en-US;q=0.9,en;q=0.8"],"Sec-Ch-Ua":["\"Google Chrome\";v=\"107\", \"Chromium\";v=\"107\", \"Not=A?Brand\";v=\"24\""],"Sec-Fetch-Mode":["navigate"],"Sec-Fetch-Dest":["document"],"Cookie":["_cseziamadt=d348c35eb50c3515d7347508d55debbe0740bfb7f424e5e513538c261ae5dc5c8e83d41dfa52445749f90d88595e571917d4b05b0c3bf0b14f4a39497740dadd; _cseziambdt=0bb60b6af104963cd9ea861be87823c97fad54c67628a9e0f03ab0f40d9f78dd90ba66cd60f02dc97b53d6293249a1545037be4e4b4c115de54a9ab43ab52a26; wms.agent=true; wms-tkp-token=15915807-ac28dc37-3fbf509cd508a05d35ec5925e0bf8448; zccpn=f005668a6988ba0e2887ff5327f9376902df165b7fc3032b9d86d33f872784c2c0cff91ca22c482d3be459a281e9f6ae05fd1fb670df07a6ce3d3a45df751a3e; _zcsr_tmp=f005668a6988ba0e2887ff5327f9376902df165b7fc3032b9d86d33f872784c2c0cff91ca22c482d3be459a281e9f6ae05fd1fb670df07a6ce3d3a45df751a3e; _iampt=15915806.15915807.49473b11c011a0b3519385ce5492fb17824422fc19c90d715dc64871bae9764001058eb5e16c0186c549a491fb86ae2cb756f7586ad2d2348e3b5a3dabb7bcfd; JSESSIONID=E145A60A819F3E851D1F6C65DF0E4DD6"],"Connection":["keep-alive"],"Upgrade-Insecure-Requests":["1"],"Accept-Encoding":["gzip, deflate, br"],"Sec-Ch-Ua-Mobile":["?0"],"User-Agent":["Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/107.0.0.0 Safari/537.36"],"Sec-Fetch-User":["?1"]},"tls":{"resumed":false,"version":772,"cipher_suite":4865,"proto":"","proto_mutual":true,"server_name":"zcem-u20-2.csez.zohocorpin.com"}},"duration":0.003403951,"status":502,"err_id":"n0xn39v9w","err_trace":"reverseproxy.statusError (reverseproxy.go:857)"}

ACCESS LOG :

{"level":"error","ts":1668606946.9700105,"logger":"http.log.access.log0","msg":"handled request","request":{"remote_addr":"10.59.2.126:51548","proto":"HTTP/1.1","method":"GET","host":"zcem-u20-2.csez.zohocorpin.com:9006","uri":"/userhome/zcemu2024/admindashboard","headers":{"Sec-Ch-Ua-Mobile":["?0"],"User-Agent":["Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/107.0.0.0 Safari/537.36"],"Sec-Fetch-User":["?1"],"Cache-Control":["max-age=0"],"Sec-Ch-Ua-Platform":["\"macOS\""],"Accept":["text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.9"],"Sec-Fetch-Site":["none"],"Accept-Language":["en-GB,en-US;q=0.9,en;q=0.8"],"Sec-Ch-Ua":["\"Google Chrome\";v=\"107\", \"Chromium\";v=\"107\", \"Not=A?Brand\";v=\"24\""],"Sec-Fetch-Mode":["navigate"],"Sec-Fetch-Dest":["document"],"Cookie":["_cseziamadt=d348c35eb50c3515d7347508d55debbe0740bfb7f424e5e513538c261ae5dc5c8e83d41dfa52445749f90d88595e571917d4b05b0c3bf0b14f4a39497740dadd; _cseziambdt=0bb60b6af104963cd9ea861be87823c97fad54c67628a9e0f03ab0f40d9f78dd90ba66cd60f02dc97b53d6293249a1545037be4e4b4c115de54a9ab43ab52a26; wms.agent=true; wms-tkp-token=15915807-ac28dc37-3fbf509cd508a05d35ec5925e0bf8448; zccpn=f005668a6988ba0e2887ff5327f9376902df165b7fc3032b9d86d33f872784c2c0cff91ca22c482d3be459a281e9f6ae05fd1fb670df07a6ce3d3a45df751a3e; _zcsr_tmp=f005668a6988ba0e2887ff5327f9376902df165b7fc3032b9d86d33f872784c2c0cff91ca22c482d3be459a281e9f6ae05fd1fb670df07a6ce3d3a45df751a3e; _iampt=15915806.15915807.49473b11c011a0b3519385ce5492fb17824422fc19c90d715dc64871bae9764001058eb5e16c0186c549a491fb86ae2cb756f7586ad2d2348e3b5a3dabb7bcfd; JSESSIONID=E145A60A819F3E851D1F6C65DF0E4DD6"],"Connection":["keep-alive"],"Upgrade-Insecure-Requests":["1"],"Accept-Encoding":["gzip, deflate, br"]},"tls":{"resumed":false,"version":772,"cipher_suite":4865,"proto":"","proto_mutual":true,"server_name":"zcem-u20-2.csez.zohocorpin.com"}},"common_log":"10.59.2.126 - - [16/Nov/2022:19:25:46 +0530] \"GET /userhome/zcemu2024/admindashboard HTTP/1.1\" 502 0","duration":0.003403951,"size":0,"status":502,"resp_headers":{"Server":["Caddy"]}}

Caddy in host :

curl -v --insecure https://zcem-u20-2.csez.zohocorpin.com:9006/userhome/zcemu2024/admindashboard#/
*   Trying 172.24.158.145:9006...
* Connected to zcem-u20-2.csez.zohocorpin.com (172.24.158.145) port 9006 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* successfully set certificate verify locations:
*  CAfile: /etc/ssl/cert.pem
*  CApath: none
* (304) (OUT), TLS handshake, Client hello (1):
* (304) (IN), TLS handshake, Server hello (2):
* (304) (IN), TLS handshake, Unknown (8):
* (304) (IN), TLS handshake, Certificate (11):
* (304) (IN), TLS handshake, CERT verify (15):
* (304) (IN), TLS handshake, Finished (20):
* (304) (OUT), TLS handshake, Finished (20):
* SSL connection using TLSv1.3 / AEAD-CHACHA20-POLY1305-SHA256
* ALPN, server did not agree to a protocol
* Server certificate:
*  subject: [NONE]
*  start date: Nov 16 05:47:19 2022 GMT
*  expire date: Nov 16 17:47:19 2022 GMT
*  issuer: CN=Caddy Local Authority - ECC Intermediate
*  SSL certificate verify result: unable to get local issuer certificate (20), continuing anyway.
> GET /userhome/zcemu2024/admindashboard HTTP/1.1
> Host: zcem-u20-2.csez.zohocorpin.com:9006
> User-Agent: curl/7.79.1
> Accept: */*
> 
* Mark bundle as not supporting multiuse
< HTTP/1.1 502 Bad Gateway
< Server: Caddy
< Date: Wed, 16 Nov 2022 12:45:38 GMT
< Content-Length: 0
< 
* Connection #0 to host zcem-u20-2.csez.zohocorpin.com left intact

Logs while accessing the proxied docker port straightaway :

curl -v --insecure https://zcem-u20-2.csez.zohocorpin.com:9000/userhome/zcemu2024/admindashboard#/
*   Trying 172.24.158.145:9000...
* Connected to zcem-u20-2.csez.zohocorpin.com (172.24.158.145) port 9000 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* successfully set certificate verify locations:
*  CAfile: /etc/ssl/cert.pem
*  CApath: none
* (304) (OUT), TLS handshake, Client hello (1):
* (304) (IN), TLS handshake, Server hello (2):
* (304) (IN), TLS handshake, Unknown (8):
* (304) (IN), TLS handshake, Certificate (11):
* (304) (IN), TLS handshake, CERT verify (15):
* (304) (IN), TLS handshake, Finished (20):
* (304) (OUT), TLS handshake, Finished (20):
* SSL connection using TLSv1.3 / AEAD-CHACHA20-POLY1305-SHA256
* ALPN, server did not agree to a protocol
* Server certificate:
*  subject: [NONE]
*  start date: Nov 16 05:52:22 2022 GMT
*  expire date: Nov 16 17:52:22 2022 GMT
*  issuer: CN=Caddy Local Authority - ECC Intermediate
*  SSL certificate verify result: unable to get local issuer certificate (20), continuing anyway.
> GET /userhome/zcemu2024/admindashboard HTTP/1.1
> Host: zcem-u20-2.csez.zohocorpin.com:9000
> User-Agent: curl/7.79.1
> Accept: */*
> 
* Mark bundle as not supporting multiuse
< HTTP/1.1 302 Found
< Cache-Control: private,no-cache,no-store,max-age=0,must-revalidate
< Content-Length: 0
< Date: Wed, 16 Nov 2022 12:45:32 GMT
< Expires: Thu, 01 Jan 1970 00:00:00 GMT
< Location: /index.jsp?serviceurl=%2Fuserhome%2Fzcemu2024%2Fadmindashboard
< Pragma: no-cache
< Server: Caddy
< Set-Cookie: zccpn=671c8dc1-e3e7-4081-9c40-1bc5af2d97a1;path=/;SameSite=None;Secure;priority=high
< Set-Cookie: _zcsr_tmp=671c8dc1-e3e7-4081-9c40-1bc5af2d97a1;path=/;SameSite=Strict;Secure;priority=high
< X-Content-Type-Options: nosniff
< X-Xss-Protection: 1
< 
* Connection #0 to host zcem-u20-2.csez.zohocorpin.com left intact

5. What I already tried:

I tried changing the reverse proxy in different combinations
https://127.0.0.1:9000
https://localhost:9000
https://hostIP:9000
https://zcem-u20-2.csez.domainname.com:9000

But I got no clue. Please help.

That’s quite an old version. Upgrade to v2.6.2.

If you’re using Ubuntu, you should run Caddy as a systemd service. If you install Caddy using out apt repo, this is set up for your automatically.

Remove this. It’s not useful.

Remove the /* matcher. It does nothing useful.

And remove header_up Host {host}. Caddy already does this by default. See the docs:

That error message is pretty clear. You used tls internal, so you’re using a certificate signed by Caddy itself. You need to add Caddy’s root CA cert to your system’s trust store for it to trust the connection to the upstream.

Are you sure you need to proxy over HTTPS? If you’re just proxying to something on the same machine, you can probably just proxy over HTTP instead. HTTPS only adds overhead, doesn’t add any security because only things on the same system would be able to see the traffic.

1 Like

Thanks @francislavoie .

I tried “caddy trust” and got the below output

{"level":"info","ts":1668637108.4990313,"logger":"ca.local","msg":"root certificate is already trusted by system","path":"storage:pki/authorities/local/root.crt"}

So I hope its already trusted.

The service running in docker is https and it needs https. So I thought the CADDY in host machine also needs https proxy. Please correct me if I am wrong.

You said you have another Caddy instance running inside Docker. If you used tls internal in that one, then that Caddy instance has its own root CA cert used for issuing certs. You’d need to add that one to your system’s trust store.

But that’s complicated, and probably without value. You should probably proxy over HTTP, like I said.

Is there any way to skip this verification like “–insecure” in curl ?
or like we ignore the certificate error in a web browser after ignoring the warning?

It worked after I added the below block inside the reverse proxy block.

transport http {
tls_insecure_skip_verify
}

Thank you @francislavoie for the help and support.

Just be aware you are disabling all security when you do that. (i.e. making TLS pointless)

1 Like

Which is why I suggest using HTTP if this is all on your local network. There’s literally no benefit to using HTTPS if you use tls_insecure_skip_verify, it only adds overhead.

Hi Matt… This setup is just for our local test environment … That’s why…

I tried only HTTP but still i faced the same issue… That’s why i used insecure…

Please be more specific. What issue? Using HTTP should definitely not have the same issues, because it inherently doesn’t involve encryption.

Sorry @francislavoie for the late reply…

Since the caddy I use inside the docker is using HTTPS, (as my web stack needs HTTPS), I’m. cant change it to HTTP…

If I use HTTP in the caddy of my host machine, below are the logs…

curl -v https://docker1.csez.domainame.com:9000/userhome/zcemu2024/admindashboard#/

*   Trying 10.63.18.195:9000...
* Connected to docker1.csez.domainame.com (10.63.18.195) port 9000 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* successfully set certificate verify locations:
*  CAfile: /etc/ssl/cert.pem
*  CApath: none
* (304) (OUT), TLS handshake, Client hello (1):
* (304) (IN), TLS handshake, Server hello (2):
* (304) (IN), TLS handshake, Unknown (8):
* (304) (IN), TLS handshake, Certificate (11):
* SSL certificate problem: unable to get local issuer certificate
* Closing connection 0
curl: (60) SSL certificate problem: unable to get local issuer certificate
More details here: https://curl.se/docs/sslcerts.html

curl failed to verify the legitimacy of the server and therefore could not
establish a secure connection to it. To learn more about this situation and
how to fix it, please visit the web page mentioned above.

Error from browser :

Client sent an HTTP request to an HTTPS server.

Nohup.out :

{"level":"error","ts":1669253417.8591838,"logger":"http.handlers.reverse_proxy","msg":"reading from backend","error":"read tcp 10.63.18.195:40748->10.63.18.195:9001: read: connection reset by peer"}
{"level":"error","ts":1669253417.8592415,"logger":"http.handlers.reverse_proxy","msg":"aborting with incomplete response","error":"read tcp 10.63.18.195:40748->10.63.18.195:9001: read: connection reset by peer"}
{"level":"error","ts":1669253417.933389,"logger":"http.handlers.reverse_proxy","msg":"reading from backend","error":"read tcp 10.63.18.195:40750->10.63.18.195:9001: read: connection reset by peer"}
{"level":"error","ts":1669253417.9334648,"logger":"http.handlers.reverse_proxy","msg":"aborting with incomplete response","error":"read tcp 10.63.18.195:40750->10.63.18.195:9001: read: connection reset by peer"}

The only important thing is that you have HTTPS at the edge. You don’t need to be running your internal Caddy instance with HTTPS. That just adds overhead. That’s what I’m trying to explain.

1 Like

Got it @francislavoie …But my web application stack (which is not designed by me) needs https for some purpose …I cant make it to work without HTTPS…That’s why I had to use this workaround.

Thank you Francis for the support from beginning.

But it does have HTTPS, from your front Caddy.

I’m suggesting you do:

HTTPS -> (HTTP -> App)

You’re doing:

HTTPS -> (HTTPS -> App)

(The parentheses are the parts within your local network)

1 Like