502 Errors with frontend/backend/nextcloud

Help
41

I wanted to follow up in case anyone was wondering.

I restored a backup of my old certs from a couple days ago before I started deleting things. That worked to get around hitting the cert rate limits. The command up above in @francislavoie’s post below looks like it fixed the primary problem. And nice to see v2.5.0 will implement it.

For now, I wonder if adding a line to my systemd service file would help avoid the issue. Something like an ExecStartPre= to run the caddy trust command… I just dont know how to set that up. Would it just be ExecStartPre=-/usr/local/bin/caddy trust ?

Anyway thanks for the help!

42

No, you only need it literally once. You don’t need to do it every time Caddy starts. And running it that way wouldn’t work anyways:

The entire problem is that the caddy user doesn’t have the appropriate permissions to write to the system’s trust location. It does attempt to setup trust when running as a systemd service automatically, but it always fails (without interruptions) because of permissions problems.

Running the command with sudo (or as root) is necessary so that you do have those permissions, and HOME=~caddy is necessary to override the root user’s HOME for that command so that it uses the HOME of the caddy user instead to grab the root cert from storage.

The change landing in v2.5.0 is that you’ll only need to do sudo caddy trust instead of sudo HOME=~caddy caddy trust since the caddy trust command will use the admin API (localhost:2019, similarly to how caddy reload works) to fetch the certificates from the currently running instance of Caddy.

43

Is that once per install on a system with a frontend/backend arrangement like this?

44

This command, only once, on the frontend only. Because it’s just so reverse_proxy can trust that certs served by the backend (which the frontend itself issued). The reverse_proxy module uses the system’s trust store.

45

I spoke too soon and I’m back at error 502 on the nextcloud install.

It was working for a while and then nextcloud was showing 502 errors when I woke up this morning.

curl -v https://cloud.y8s.com
backend:

Feb 21 13:15:30 cloud caddy[527]: {"level":"debug","ts":1645467330.654302,"logger":"tls.handshake","msg":"choosing certificate","identifier":"cloud.y8s.casa","num_choices":1}
Feb 21 13:15:30 cloud caddy[527]: {"level":"debug","ts":1645467330.6548214,"logger":"tls.handshake","msg":"default certificate selection results","identifier":"cloud.y8s.casa","subjects":["cloud.y8s.casa"],"managed":true,"issuer_key":"caddy.y8s.casa-acme-local-directory","hash":"a2a2ddcce28e21b298b806054164aed4469b9785e5ccb1e745904a37f863e3b1"}
Feb 21 13:15:30 cloud caddy[527]: {"level":"debug","ts":1645467330.6549203,"logger":"tls.handshake","msg":"matched certificate in cache","subjects":["cloud.y8s.casa"],"managed":true,"expiration":1645397606,"hash":"a2a2ddcce28e21b298b806054164aed4469b9785e5ccb1e745904a37f863e3b1"}
Feb 21 13:15:30 cloud caddy[527]: {"level":"debug","ts":1645467330.660738,"logger":"http.stdlib","msg":"http: TLS handshake error from 10.10.10.40:37718: remote error: tls: bad certificate"}

frontend:

Feb 21 13:15:30 iot caddy[27429]: {"level":"debug","ts":1645467330.6614866,"logger":"http.handlers.reverse_proxy","msg":"upstream roundtrip","upstream":"cloud.y8s.casa:443","request":{"remote_addr":"173.8.14.69:62113","proto":"HTTP/1.1","method":"GET","host":"cloud.y8s.casa:443","uri":"/","headers":{"User-Agent":["curl/7.79.1"],"Accept":["*/*"],"X-Forwarded-Proto":["https"],"X-Forwarded-Host":["cloud.y8s.com"],"X-Forwarded-For":["173.8.14.69"]},"tls":{"resumed":false,"version":771,"cipher_suite":49196,"proto":"http/1.1","proto_mutual":true,"server_name":"cloud.y8s.com"}},"duration":0.012889179,"error":"x509: certificate has expired or is not yet valid: current time 2022-02-21T13:15:30-05:00 is after 2022-02-20T22:53:26Z"}
Feb 21 13:15:30 iot caddy[27429]: {"level":"error","ts":1645467330.661751,"logger":"http.log.error","msg":"x509: certificate has expired or is not yet valid: current time 2022-02-21T13:15:30-05:00 is after 2022-02-20T22:53:26Z","request":{"remote_addr":"173.8.14.69:62113","proto":"HTTP/1.1","method":"GET","host":"cloud.y8s.com","uri":"/","headers":{"User-Agent":["curl/7.79.1"],"Accept":["*/*"]},"tls":{"resumed":false,"version":771,"cipher_suite":49196,"proto":"http/1.1","proto_mutual":true,"server_name":"cloud.y8s.com"}},"duration":0.013289512,"status":502,"err_id":"4ccxe2418","err_trace":"reverseproxy.statusError (reverseproxy.go:861)"}

Note that I have changed my hostnames back to what they were when this problem originally occurred. test.eastcapitol.us becomes cloud.y8s.com, but the problem is the same.

46

Now I think we have come full circle to:
sudo caddy reload --force --config /etc/caddy/Caddyfile

Also in my two Caddyfiles there was a difference I’m curious about:

Frontend had global option block

{
        debug
        email whatever@whatever.com
}

and backend had

{
	debug
	acme_ca https://caddy.y8s.casa/acme/local/directory
	acme_ca_root /etc/ssl/certs/root.crt

}

in other words, no email address. Does this have any impact? Is the communication between machines using this email address?

I force reloaded and added the same email to the backend and now things work again.
need to go add that cron job.

47

Nope. The email address is just for Let’s Encrypt and/or ZeroSSL to know who to contact in case of a problem with your certificates.

For the backend, it reaches out to the frontend to issue certs, and your logs are the evidence for problems, it doesn’t make sense to Caddy to try to email you (how would it even do that? it would need an SMTP server for example, makes no sense).

As I said before, you need to make sure to force reload the frontend daily for now to work around the intermediate certs issue (where acme_server doesn’t know the intermediate was updated, otherwise it will sign leaf certs with an expired intermediate).

48

I only ask because there are subfolders with the email address name. I was wondering if it used those for something.

My frontend now has a cron entry for the force reload.

49

Yeah, the email address is your “account” for Let’s Encrypt. If you don’t specify an email, Caddy will use a safe default.

50

This topic was automatically closed after 30 days. New replies are no longer allowed.