Caddy2 stopped working suddenly. 502s and other odd errors

Help
1

1. Caddy version (caddy version):

Caddy2 docker version caddy:2.3.0

2. How I run Caddy:

Running caddy in a container on a qnap NAS. DuckDNS with caddy as reverse proxy. https://renna.duckdns.org is primary for home assistant but several other subdomains such as: https://blueiris.renna.duckdns.org, https://grafana.renna.duckdns.org

a. System environment:

Docker

b. Command:

c. Service/unit/compose file:

version: "3.7"

services:
  caddy:
    image: caddy:2.3.0
    restart: unless-stopped
    ports:
      - "80:80"
      - "443:443"
    volumes:
      - '/share/Container/caddy2/etc/caddy:/etc/caddy'
      - '/share/Container/caddy2/data:/data'
      - '/share/Container/caddy2/config:/config'

d. My complete Caddyfile or JSON config:

renna.duckdns.org {
    reverse_proxy http://192.168.0.14:8123
	tls mike.renna@live.com
	}

grafana.renna.duckdns.org {
	reverse_proxy http://192.168.0.14:3000 
	tls mike.renna@live.com
	
	}
	
blueiris.renna.duckdns.org {
    reverse_proxy http://192.168.0.3:81
	tls mike.renna@live.com
	}

plex.renna.duckdns.org {
    reverse_proxy http://192.168.0.14:32400
	tls mike.renna@live.com
    }
	
portainer.renna.duckdns.org {
    reverse_proxy http://192.168.0.14:9000
	tls mike.renna@live.com
    }

3. The problem I’m having:

External access recently stopped working and I’m at a loss. Only one reverse proxy still works (blueiris.renna.dickdns.org). I get 502 error in the caddy logs for most for the others. I’ve reviewed networking and it appears to be ok (and unchanged). Strangely while grafana shows a 502 in caddy log, home assistant (base url at https://renna.duckdns.org) doesn’t show anything in the caddy terminal when it fails and the browser will show a home assistant logo in the center of the display and an error, with a retry link, which on click returns a 403: Forbidden, which seems like it’s reaching it but failing. This is the most important url.

if i try the urls using curl from the host machine, for grafana it doesn’t show anything strange to me (see below) but for the base url is references expired certificates, which i’m not sure how to respond.

about six months ago in moving to new NAS, i inadvertently moved to caddy2 and had to update some configurations. any chance that impacted this? could it have used same certs since i did not change the bind mount location? could that cert have expired and not properly renewed because caddy2 config was goofy? Thank you in advance for any insight.

4. Error messages and/or full log output:

[~] # curl -v https://renna.duckdns.org
*   Trying 70.188.225.116:443...
* TCP_NODELAY set
* Connected to renna.duckdns.org (70.188.225.116) port 443 (#0)
* ALPN, offering http/1.1
* successfully set certificate verify locations:
*   CAfile: none
  CApath: /etc/ssl/certs
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* TLSv1.3 (IN), TLS handshake, Server hello (2):
* TLSv1.3 (IN), TLS handshake, Encrypted Extensions (8):
* TLSv1.3 (IN), TLS handshake, Certificate (11):
* TLSv1.3 (OUT), TLS alert, certificate expired (557):
* SSL certificate problem: certificate has expired
* Closing connection 0
curl: (60) SSL certificate problem: certificate has expired

[~] # curl -v grafana.renna.duckdns.org
*   Trying 70.188.225.116:80...
* TCP_NODELAY set
* Connected to grafana.renna.duckdns.org (70.188.225.116) port 80 (#0)
> GET / HTTP/1.1
> Host: grafana.renna.duckdns.org
> User-Agent: curl/7.67.0
> Accept: */*
>
* Mark bundle as not supporting multiuse
< HTTP/1.1 308 Permanent Redirect
< Connection: close
< Location: https://grafana.renna.duckdns.org/
< Server: Caddy
< Date: Thu, 30 Dec 2021 15:30:17 GMT
< Content-Length: 0
<
* Closing connection 0

5. What I already tried:

reading, searching, config changes to move containers around and validate networking configs.

6. Links to relevant resources:

2

additionally, i removed the certificates and could see that a restart of caddy did in fact replace them.

3

Please upgrade to v2.4.6!

Instead of repeating this all over, you can set the email via a global option:

{
	email mike.renna@live.com
}

That’s almost always going to be networking related.

Is 192.168.0.14 another machine entirely?

What’s the error message in Caddy’s logs, exactly?

4

Thank you for the reply. I actually had upgraded to 2.4.6 shortly after my post but no change in behavior. Updated CaddyFile for the email tip. (thanks) Error log for the 502s looks like this:

{"level":"error","ts":1640892003.4026694,"logger":"http.log.error","msg":"dial tcp 192.168.0.14:3000: i/o timeout","request":{"remote_addr":"192.168.0.1:58817","proto":"HTTP/1.1","method":"GET","host":"grafana.renna.duckdns.org","uri":"/","headers":{"Sec-Ch-Ua-Platform":["\"Windows\""],"Dnt":["1"],"Sec-Ch-Ua":["\" Not A;Brand\";v=\"99\", \"Chromium\";v=\"96\", \"Microsoft Edge\";v=\"96\""],"Sec-Fetch-Mode":["navigate"],"Accept-Language":["en-US,en;q=0.9"],"Sec-Ch-Ua-Mobile":["?0"],"Accept":["text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.9"],"Accept-Encoding":["gzip, deflate, br"],"Sec-Fetch-Site":["none"],"Sec-Fetch-User":["?1"],"Sec-Fetch-Dest":["document"],"Connection":["keep-alive"],"Upgrade-Insecure-Requests":["1"],"User-Agent":["Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/96.0.4664.110 Safari/537.36 Edg/96.0.1054.62"]},"tls":{"resumed":false,"version":772,"cipher_suite":4865,"proto":"http/1.1","proto_mutual":true,"server_name":"grafana.renna.duckdns.org"}},"duration":10.009437913,"status":502,"err_id":"k8v2q953b","err_trace":"reverseproxy.statusError (reverseproxy.go:886)"}

192.168.0.14 is the same machine that hosts caddy container and others (grafana, home assistant, etc). The proxy that works is blue iris and that goes from the host (.14) to a totally different machine (.3). And I suspect you are correct about the networking. I’m going to review that all again with container IPs and subnets.

However, i think the biggest problem is the primary proxy which does not throw a 502. In fact no error at all in caddy log. Browser gets me to home assistant logo with forbidden 403. Looking at the home assistant trusted proxies setting again. i’ve checked and checked that based on what i see as the IPs showing in portainer and also in qnaps networking module. is there a caddy command that can help tell me what IP is being presented to home assistant as the proxy so I can check it?

What about the weird expired certificate error I get for primary proxy (home assistant) when I run curl? This is the only url that shows that curl response and its also the one that does not show a 502 but instead the above no error and browser 403: forbidden. Any way the cert is the issue?

Thanks again!

5

Hey did you ever figure out your 502 Gateway issues? Also what version of portainer are you running? It looks like 192.168.0.14 is your Portainer Host?

6

This topic was automatically closed after 30 days. New replies are no longer allowed.