1. The problem I’m having:
I am trying to use caddy as a reverse proxy as I have done many times, but when visiting the URL I get ERR_SSL_PROTOCOL_ERROR. When looking into the logs it looks like caddy is failing to get the certificate form Let’s Encrypt.
I thing the relevant error is HTTP 400 urn:ietf:params:acme:error:malformed - JWS verification error
I also tried it with the stating server and got an unsafe but working certificate.
2. Error messages and/or full log output:
{"level":"info","ts":1729978385.566533,"logger":"admin","msg":"admin endpoint started","address":"localhost:2019","enforce_origin":false,"origins":["//localhost:2019","//[::1]:2019","//127.0.0.1:2019"]}
{"level":"info","ts":1729978385.5668411,"logger":"http.auto_https","msg":"server is listening only on the HTTPS port but has no TLS connection policies; adding one to enable TLS","server_name":"srv0","https_port":443}
{"level":"info","ts":1729978385.5668623,"logger":"http.auto_https","msg":"enabling automatic HTTP->HTTPS redirects","server_name":"srv0"}
{"level":"info","ts":1729978385.5678353,"logger":"http","msg":"enabling HTTP/3 listener","addr":":443"}
{"level":"info","ts":1729978385.5680788,"logger":"http.log","msg":"server running","name":"srv0","protocols":["h1","h2","h3"]}
{"level":"info","ts":1729978385.568627,"logger":"http.log","msg":"server running","name":"remaining_auto_https_redirects","protocols":["h1","h2","h3"]}
{"level":"info","ts":1729978385.568643,"logger":"http","msg":"enabling automatic TLS certificate management","domains":["test.paulmaier.online","tars.paulmaier.online","sync.paulmaier.online","docker.tars.paulmaier.online","photos.paulmaier.online"]}
{"level":"info","ts":1729978385.5689948,"logger":"tls.cache.maintenance","msg":"started background certificate maintenance","cache":"0xc0002e5b00"}
{"level":"info","ts":1729978385.569079,"msg":"autosaved config (load with --resume flag)","file":"/root/.config/caddy/autosave.json"}
{"level":"info","ts":1729978385.5690916,"msg":"serving initial configuration"}
{"level":"info","ts":1729978385.570076,"logger":"tls.obtain","msg":"acquiring lock","identifier":"test.paulmaier.online"}
{"level":"info","ts":1729978385.5761354,"logger":"tls.obtain","msg":"lock acquired","identifier":"test.paulmaier.online"}
{"level":"info","ts":1729978385.5762367,"logger":"tls.obtain","msg":"obtaining certificate","identifier":"test.paulmaier.online"}
{"level":"info","ts":1729978385.578072,"logger":"http","msg":"waiting on internal rate limiter","identifiers":["test.paulmaier.online"],"ca":"https://acme-v02.api.letsencrypt.org/directory","account":""}
{"level":"info","ts":1729978385.5781412,"logger":"http","msg":"done waiting on internal rate limiter","identifiers":["test.paulmaier.online"],"ca":"https://acme-v02.api.letsencrypt.org/directory","account":""}
{"level":"info","ts":1729978385.5785215,"logger":"http","msg":"using ACME account","account_id":"https://acme-v02.api.letsencrypt.org/acme/acct/2020432947","account_contact":[]}
{"level":"info","ts":1729978385.5797386,"logger":"tls.obtain","msg":"acquiring lock","identifier":"docker.tars.paulmaier.online"}
{"level":"info","ts":1729978385.5799298,"logger":"tls.obtain","msg":"acquiring lock","identifier":"sync.paulmaier.online"}
{"level":"info","ts":1729978385.5812254,"logger":"tls","msg":"storage cleaning happened too recently; skipping for now","storage":"FileStorage:/root/.local/share/caddy","instance":"3aa521ea-37f5-486a-97f0-481c43dbef5a","try_again":1730064785.5812204,"try_again_in":86399.999999248}
{"level":"info","ts":1729978385.581342,"logger":"tls","msg":"finished cleaning storage units"}
{"level":"info","ts":1729978385.581961,"logger":"tls.obtain","msg":"acquiring lock","identifier":"photos.paulmaier.online"}
{"level":"info","ts":1729978385.5894427,"logger":"tls.obtain","msg":"acquiring lock","identifier":"tars.paulmaier.online"}
{"level":"info","ts":1729978385.6013052,"logger":"tls.obtain","msg":"lock acquired","identifier":"docker.tars.paulmaier.online"}
{"level":"info","ts":1729978385.601529,"logger":"tls.obtain","msg":"obtaining certificate","identifier":"docker.tars.paulmaier.online"}
{"level":"info","ts":1729978385.6034245,"logger":"http","msg":"waiting on internal rate limiter","identifiers":["docker.tars.paulmaier.online"],"ca":"https://acme-v02.api.letsencrypt.org/directory","account":""}
{"level":"info","ts":1729978385.6034808,"logger":"http","msg":"done waiting on internal rate limiter","identifiers":["docker.tars.paulmaier.online"],"ca":"https://acme-v02.api.letsencrypt.org/directory","account":""}
{"level":"info","ts":1729978385.6034968,"logger":"http","msg":"using ACME account","account_id":"https://acme-v02.api.letsencrypt.org/acme/acct/2020432947","account_contact":[]}
{"level":"info","ts":1729978385.6036146,"logger":"tls.obtain","msg":"lock acquired","identifier":"sync.paulmaier.online"}
{"level":"info","ts":1729978385.6037016,"logger":"tls.obtain","msg":"obtaining certificate","identifier":"sync.paulmaier.online"}
{"level":"info","ts":1729978385.6120772,"logger":"http","msg":"waiting on internal rate limiter","identifiers":["sync.paulmaier.online"],"ca":"https://acme-v02.api.letsencrypt.org/directory","account":""}
{"level":"info","ts":1729978385.6121445,"logger":"http","msg":"done waiting on internal rate limiter","identifiers":["sync.paulmaier.online"],"ca":"https://acme-v02.api.letsencrypt.org/directory","account":""}
{"level":"info","ts":1729978385.6121666,"logger":"http","msg":"using ACME account","account_id":"https://acme-v02.api.letsencrypt.org/acme/acct/2020432947","account_contact":[]}
{"level":"info","ts":1729978385.6141548,"logger":"tls.obtain","msg":"lock acquired","identifier":"photos.paulmaier.online"}
{"level":"info","ts":1729978385.614274,"logger":"tls.obtain","msg":"obtaining certificate","identifier":"photos.paulmaier.online"}
{"level":"info","ts":1729978385.6148582,"logger":"http","msg":"waiting on internal rate limiter","identifiers":["photos.paulmaier.online"],"ca":"https://acme-v02.api.letsencrypt.org/directory","account":""}
{"level":"info","ts":1729978385.6148717,"logger":"http","msg":"done waiting on internal rate limiter","identifiers":["photos.paulmaier.online"],"ca":"https://acme-v02.api.letsencrypt.org/directory","account":""}
{"level":"info","ts":1729978385.6148822,"logger":"http","msg":"using ACME account","account_id":"https://acme-v02.api.letsencrypt.org/acme/acct/2020432947","account_contact":[]}
{"level":"info","ts":1729978385.6183841,"logger":"tls.obtain","msg":"lock acquired","identifier":"tars.paulmaier.online"}
{"level":"info","ts":1729978385.6185741,"logger":"tls.obtain","msg":"obtaining certificate","identifier":"tars.paulmaier.online"}
{"level":"info","ts":1729978385.6193018,"logger":"http","msg":"waiting on internal rate limiter","identifiers":["tars.paulmaier.online"],"ca":"https://acme-v02.api.letsencrypt.org/directory","account":""}
{"level":"info","ts":1729978385.6193411,"logger":"http","msg":"done waiting on internal rate limiter","identifiers":["tars.paulmaier.online"],"ca":"https://acme-v02.api.letsencrypt.org/directory","account":""}
{"level":"info","ts":1729978385.6193557,"logger":"http","msg":"using ACME account","account_id":"https://acme-v02.api.letsencrypt.org/acme/acct/2020432947","account_contact":[]}
{"level":"error","ts":1729978386.2865307,"logger":"tls.obtain","msg":"could not get certificate from issuer","identifier":"test.paulmaier.online","issuer":"acme-v02.api.letsencrypt.org-directory","error":"HTTP 400 urn:ietf:params:acme:error:malformed - JWS verification error"}
{"level":"error","ts":1729978386.2866676,"logger":"tls.obtain","msg":"will retry","error":"[test.paulmaier.online] Obtain: [test.paulmaier.online] creating new order: attempt 1: https://acme-v02.api.letsencrypt.org/acme/new-order: HTTP 400 urn:ietf:params:acme:error:malformed - JWS verification error (ca=https://acme-v02.api.letsencrypt.org/directory)","attempt":1,"retrying_in":60,"elapsed":0.710500468,"max_duration":2592000}
{"level":"error","ts":1729978386.2888002,"logger":"tls.obtain","msg":"could not get certificate from issuer","identifier":"sync.paulmaier.online","issuer":"acme-v02.api.letsencrypt.org-directory","error":"HTTP 400 urn:ietf:params:acme:error:malformed - JWS verification error"}
{"level":"error","ts":1729978386.2889001,"logger":"tls.obtain","msg":"will retry","error":"[sync.paulmaier.online] Obtain: [sync.paulmaier.online] creating new order: attempt 1: https://acme-v02.api.letsencrypt.org/acme/new-order: HTTP 400 urn:ietf:params:acme:error:malformed - JWS verification error (ca=https://acme-v02.api.letsencrypt.org/directory)","attempt":1,"retrying_in":60,"elapsed":0.685273002,"max_duration":2592000}
{"level":"error","ts":1729978386.2900517,"logger":"tls.obtain","msg":"could not get certificate from issuer","identifier":"photos.paulmaier.online","issuer":"acme-v02.api.letsencrypt.org-directory","error":"HTTP 400 urn:ietf:params:acme:error:malformed - JWS verification error"}
{"level":"error","ts":1729978386.2901862,"logger":"tls.obtain","msg":"will retry","error":"[photos.paulmaier.online] Obtain: [photos.paulmaier.online] creating new order: attempt 1: https://acme-v02.api.letsencrypt.org/acme/new-order: HTTP 400 urn:ietf:params:acme:error:malformed - JWS verification error (ca=https://acme-v02.api.letsencrypt.org/directory)","attempt":1,"retrying_in":60,"elapsed":0.676000225,"max_duration":2592000}
{"level":"error","ts":1729978386.2939076,"logger":"tls.obtain","msg":"could not get certificate from issuer","identifier":"tars.paulmaier.online","issuer":"acme-v02.api.letsencrypt.org-directory","error":"HTTP 400 urn:ietf:params:acme:error:malformed - JWS verification error"}
{"level":"error","ts":1729978386.2940345,"logger":"tls.obtain","msg":"will retry","error":"[tars.paulmaier.online] Obtain: [tars.paulmaier.online] creating new order: attempt 1: https://acme-v02.api.letsencrypt.org/acme/new-order: HTTP 400 urn:ietf:params:acme:error:malformed - JWS verification error (ca=https://acme-v02.api.letsencrypt.org/directory)","attempt":1,"retrying_in":60,"elapsed":0.675598048,"max_duration":2592000}
{"level":"error","ts":1729978386.29948,"logger":"tls.obtain","msg":"could not get certificate from issuer","identifier":"docker.tars.paulmaier.online","issuer":"acme-v02.api.letsencrypt.org-directory","error":"HTTP 400 urn:ietf:params:acme:error:malformed - JWS verification error"}
{"level":"error","ts":1729978386.2996392,"logger":"tls.obtain","msg":"will retry","error":"[docker.tars.paulmaier.online] Obtain: [docker.tars.paulmaier.online] creating new order: attempt 1: https://acme-v02.api.letsencrypt.org/acme/new-order: HTTP 400 urn:ietf:params:acme:error:malformed - JWS verification error (ca=https://acme-v02.api.letsencrypt.org/directory)","attempt":1,"retrying_in":60,"elapsed":0.698269711,"max_duration":2592000}
3. Caddy version:
$ caddy version
v2.8.4 h1:q3pe0wpBj1OcHFZ3n/1nl4V4bxBrYoSoab7rL9BMYNk=
4. How I installed and ran Caddy:
a. System environment:
$ lsb_release -a
No LSB modules are available.
Distributor ID: Ubuntu
Description: Ubuntu 22.04.5 LTS
Release: 22.04
Codename: jammy
Caddy install (like in the docs)
sudo apt install -y debian-keyring debian-archive-keyring apt-transport-https curl
curl -1sLf 'https://dl.cloudsmith.io/public/caddy/stable/gpg.key' | sudo gpg --dearmor -o /usr/share/keyrings/caddy-stable-archive-keyring.gpg
curl -1sLf 'https://dl.cloudsmith.io/public/caddy/stable/debian.deb.txt' | sudo tee /etc/apt/sources.list.d/caddy-stable.list
sudo apt update
sudo apt install caddy
b. Command:
$ sudo caddy start --config Caddyfile
2024/10/26 21:38:58.528 INFO using config from file {"file": "Caddyfile"}
2024/10/26 21:38:58.530 INFO adapted config to JSON {"adapter": "caddyfile"}
2024/10/26 21:38:58.530 INFO redirected default logger {"from": "stderr", "to": "/var/log/caddy/access.log"}
Successfully started Caddy (pid=97138) - Caddy is running in the background
c. My complete Caddy config:
{
log {
output file /var/log/caddy/access.log
format json
}
}
test.paulmaier.online {
respond "Hello world"
}
tars.paulmaier.online {
root * /home/tars/caddy
file_server
try_files tars.txt
}
sync.paulmaier.online {
reverse_proxy http://127.0.0.1:8384
}
photos.paulmaier.online {
reverse_proxy http://localhost:2342
}
docker.tars.paulmaier.online {
reverse_proxy https://127.0.0.1:9443 {
transport http {
tls
tls_insecure_skip_verify
}
}
}
5. What I already tried
a. Checking DNS
$ ip a
[...]
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UP group default qlen 1000
link/ether 00:50:56:55:f5:00 brd ff:ff:ff:ff:ff:ff
altname enp0s18
altname ens18
inet 84.247.131.178/20 brd 84.247.143.255 scope global eth0
valid_lft forever preferred_lft forever
inet6 fe80::250:56ff:fe55:f500/64 scope link
valid_lft forever preferred_lft forever
[...]
tars@vmd152576:~$ dig test.paulmaier.online
; <<>> DiG 9.18.28-0ubuntu0.22.04.1-Ubuntu <<>> test.paulmaier.online
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 10654
;; flags: qr rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 1
;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 65494
;; QUESTION SECTION:
;test.paulmaier.online. IN A
;; ANSWER SECTION:
test.paulmaier.online. 3600 IN A 84.247.131.178
;; Query time: 16 msec
;; SERVER: 127.0.0.53#53(127.0.0.53) (UDP)
;; WHEN: Sat Oct 26 23:44:17 CEST 2024
;; MSG SIZE rcvd: 66
I also don’t have the domain pointing to two ips which I read was the issue in another case.
b. Generating a certificate with certbot
$ sudo certbot certonly -d test.paulmaier.online
Saving debug log to /var/log/letsencrypt/letsencrypt.log
How would you like to authenticate with the ACME CA?
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
1: Spin up a temporary webserver (standalone)
2: Place files in webroot directory (webroot)
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Select the appropriate number [1-2] then [enter] (press 'c' to cancel): 1
Requesting a certificate for test.paulmaier.online
Successfully received certificate.
Certificate is saved at: /etc/letsencrypt/live/test.paulmaier.online/fullchain.pem
Key is saved at: /etc/letsencrypt/live/test.paulmaier.online/privkey.pem
This certificate expires on 2025-01-24.
These files will be updated when the certificate renews.
Certbot has set up a scheduled task to automatically renew this certificate in the background.
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
If you like Certbot, please consider supporting our work by:
* Donating to ISRG / Let's Encrypt: https://letsencrypt.org/donate
* Donating to EFF: https://eff.org/donate-le
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Worked.
c. Checking Firewall
sudo ufw status
Status: active
To Action From
-- ------ ----
80 ALLOW Anywhere
443 ALLOW Anywhere
[...]
80 (v6) ALLOW Anywhere (v6)
443 (v6) ALLOW Anywhere (v6)
[...]
Thanks for you help 