Caddy with Jellyfin via Google Domains noob help

Help
21

Sorry I know I’m spamming but I figure it might be helpful to document the process as I haven’t been able to find the information all written out in one place.

I followed two other solutions that I found googling to solve the permissions issue, neither worked. So I just chmodded to 777. I am aware this is a security risk, but it immediately fixed the issue. Trying to use the commands chown etc did not do anything, so if anyone can tell me how to make it work without have 777 enabled I would be grateful.

Now for the first time ever I actually see a set up page when trying to access my url via the new Cloudflare domain. This leads me to believe that there were some additional settings around Google’s DNS records etc that I was missing.

I follow the instructions on screen for the caddy set up; change ‘:80’ in the Caddyfile to the new settings with the new domain and API token, open up my ports, reload the systemctl process… and… same error.

Error code: SSL_ERROR_INTERNAL_ERROR_ALERT

The new code for my Caddyfile looks as such:

  GNU nano 6.2                                     /etc/caddy/Caddyfile                                              
# The Caddyfile is an easy way to configure your Caddy web server.
#
# Unless the file starts with a global options block, the first
# uncommented line is always the address of your site.
#
# To use your own domain name (with automatic HTTPS), first make
# sure your domain's A/AAAA DNS records are properly pointed to
# this machine's public IP, then replace ":80" below with your
# domain name.

mitchflix.net {
        reverse_proxy 192.168.1.31:8096
        tls {
                dns cloudflare API_KEY
        }
}

What the hell am I doing wrong?

22

After some further investigation, I can see caddy is still failing to get any SSL certificates from Let’s Encrypt. At least I know now that’s the issue. I can confirm this is the case because I do not have any “.caddy” folder in my home directory at all, which as far as I can tell should be created upon the successful signing of a new SSL certificate.

I changed my Caddyfile in /etc/caddy/ to remove my API key, and instead added an environment variable for both that and the email address to the bottom of my .bashrc file in my home folder, in the hopes it would help the caddy service resolve the DNS-01 challenge it is failing. It did not.

I have no idea if this is even correct as I’m totally new to this, but I’m trying everything I can.

23

Just going to keep logging my progress here… not really much else I can do.

I attempted a fix suggested here.

Doing so stopped the process from starting at all, saying that I needed to include the token, which is a problem caddy didn’t seem to be recognising at all… so progress, I guess?

I included the environment variable in the caddyfile as suggested here which fixed the issue, however I am now getting the same original dns-01 error.

I have validated via cloudflare’s suggested curl command that the token is active and working.

I also followed another suggestion here to try including different resolvers for the DNS.

I followed the steps found here to set the zone permissions and make sure I am using a token, not a key.

I again tried the first suggestion, to delete all acme records found in /var/lib/caddy/.local/share/caddy/acme and then restart the service. Same error:

● caddy.service - Caddy
     Loaded: loaded (/etc/systemd/system/caddy.service; enabled; vendor preset: enabled)
     Active: active (running) since Mon 2023-07-24 14:07:50 AEST; 5min ago
       Docs: https://caddyserver.com/docs/
   Main PID: 1236859 (caddy)
      Tasks: 17 (limit: 18817)
     Memory: 11.5M
        CPU: 132ms
     CGroup: /system.slice/caddy.service
             └─1236859 /usr/bin/caddy run --environ --config /etc/caddy/Caddyfile

Jul 24 14:11:09 calcifer caddy[1236859]: {"level":"error","ts":1690171869.5663412,"logger":"tls.obtain","msg":"could not get certificate from issuer","identifier":"mitchflix.net","issuer":"acme.zerossl.com-v2-DV90","error":"[mitchflix.net] solving challenges: presenting for challenge: adding temporary record for zone \"mitchflix.net.\": got error status: HTTP 400: [{Code:6003 Message: Invalid request headers}] (order=https://acme.zerossl.com/v2/DV90/order/M5AtVzTQBEsq9NgaPz3i-g) (ca=https://acme.zerossl.com/v2/DV90)"}
Jul 24 14:11:09 calcifer caddy[1236859]: {"level":"error","ts":1690171869.5663843,"logger":"tls.obtain","msg":"will retry","error":"[mitchflix.net] Obtain: [mitchflix.net] solving challenges: presenting for challenge: adding temporary record for zone \"mitchflix.net.\": got error status: HTTP 400: [{Code:6003 Message:Invalid request headers}] (order=https://acme.zerossl.com/v2/DV90/order/M5AtVzTQBEsq9NgaPz3i-g) (ca=https://acme.zerossl.com/v2/DV90)","attempt":3,"retrying_in":120,"elapsed":199.141759113,"max_duration":2592000}
Jul 24 14:13:09 calcifer caddy[1236859]: {"level":"info","ts":1690171989.566676,"logger":"tls.obtain","msg":"obtaining certificate","identifier":"mitchflix.net"}
Jul 24 14:13:10 calcifer caddy[1236859]: {"level":"info","ts":1690171990.257321,"logger":"http.acme_client","msg":"trying to solve challenge","identifier":"mitchflix.net","challenge_type":"dns-01","ca":"https://acme-staging-v02.api.letsencrypt.org/directory"}
Jul 24 14:13:10 calcifer caddy[1236859]: {"level":"error","ts":1690171990.4952774,"logger":"http.acme_client","msg":"cleaning up solver","identifier":"mitchflix.net","challenge_type":"dns-01","error":"no memory of presenting a DNS record for \"_acme-challenge.mitchflix.net\" (usually OK if presenting also failed)"}
Jul 24 14:13:10 calcifer caddy[1236859]: {"level":"error","ts":1690171990.7239583,"logger":"tls.obtain","msg":"could not get certificate from issuer","identifier":"mitchflix.net","issuer":"acme-v02.api.letsencrypt.org-directory","error":"[mitchflix.net] solving challenges: presenting for challenge: adding temporary record for zone \"mitchflix.net.\": got error status: HTTP 400: [{Code:6003 Message:Invalid request headers}] (order=https://acme-staging-v02.api.letsencrypt.org/acme/order/112115184/9916639504) (ca=https://acme-staging-v02.api.letsencrypt.org/directory)"}
Jul 24 14:13:13 calcifer caddy[1236859]: {"level":"info","ts":1690171993.673248,"logger":"http.acme_client","msg":"trying to solve challenge","identifier":"mitchflix.net","challenge_type":"dns-01","ca":"https://acme.zerossl.com/v2/DV90"}
Jul 24 14:13:13 calcifer caddy[1236859]: {"level":"error","ts":1690171993.8832502,"logger":"http.acme_client","msg":"cleaning up solver","identifier":"mitchflix.net","challenge_type":"dns-01","error":"no memory of presenting a DNS record for \"_acme-challenge.mitchflix.net\" (usually OK if presenting also failed)"}
Jul 24 14:13:14 calcifer caddy[1236859]: {"level":"error","ts":1690171994.8841772,"logger":"tls.obtain","msg":"could not get certificate from issuer","identifier":"mitchflix.net","issuer":"acme.zerossl.com-v2-DV90","error":"[mitchflix.net] solving challenges: presenting for challenge: adding temporary record for zone \"mitchflix.net.\": got error status: HTTP 400: [{Code:6003 Message:Invalid request headers}] (order=https://acme.zerossl.com/v2/DV90/order/xRluGPshWKvv-rHa-lTswg) (ca=https://acme.zerossl.com/v2/DV90)"}
Jul 24 14:13:14 calcifer caddy[1236859]: {"level":"error","ts":1690171994.8842106,"logger":"tls.obtain","msg":"will retry","error":"[mitchflix.net] Obtain: [mitchflix.net] solving challenges: presenting for challenge: adding temporary record for zone \"mitchflix.net.\": got error status: HTTP 400: [{Code:6003 Message:Invalid request headers}] (order=https://acme.zerossl.com/v2/DV90/order/xRluGPshWKvv-rHa-lTswg) (ca=https://acme.zerossl.com/v2/DV90)","attempt":4,"retrying_in":300,"elapsed":324.459585305,"max_duration":2592000}
lines 1-21/21 (END)

My Caddyfile now looks like:

mitchtchflix.net {
        reverse_proxy 192.168.1.31:8096
        tls {
                dns cloudflare {env.CLOUDFLARE_AUTH_TOKEN}
                resolvers 8.8.8.8
        }
}
24

I realised now that I screwed up the env variables, as explained here. I removed the email variable entirely, and changed “CLOUDLARE_API_KEY” to “CLOUDFLARE_AUTH_TOKEN” as suggested in the link to match my Caddyfile.

I then tried again to delete the contents of /var/lib/caddy/.local/share/caddy making sure to stop the systemctl caddy process first, then started the process again.

Same error.

25

So you’re sure the API key is correct now? When you say “same error” what error specifically do you mean now? If it’s that last one about headers being invalid that’s an obscure error from Cloudflare that means your token isn’t correct.

26

The same error being ‘SSL_ERROR_INTERNAL_ERROR_ALERT’ on the webpage when attempting to connect to my new domain, mitchflix.net

Just in case, I rolled a new token on the cloudflare dash and put that token at the bottom of my .bashrc in my root (and only) user’s home folder:

CLOUDFLARE_AUTH_TOKEN=-zvy5_FsqPJWlaDX0iK123456etc0dh6HgjYtwAa

journalctl results:

$ journalctl -f -u caddy
Jul 24 16:43:01 calcifer caddy[1244251]: {"level":"info","ts":1690180981.5333958,"logger":"http","msg":"done waiting on internal rate limiter","identifiers":["mitchflix.net"],"ca":"https://acme-v02.api.letsencrypt.org/directory","account":""}
Jul 24 16:43:03 calcifer caddy[1244251]: {"level":"info","ts":1690180983.1493216,"logger":"http.acme_client","msg":"trying to solve challenge","identifier":"mitchflix.net","challenge_type":"dns-01","ca":"https://acme-v02.api.letsencrypt.org/directory"}
Jul 24 16:43:03 calcifer caddy[1244251]: {"level":"error","ts":1690180983.4815896,"logger":"http.acme_client","msg":"cleaning up solver","identifier":"mitchflix.net","challenge_type":"dns-01","error":"no memory of presenting a DNS record for \"_acme-challenge.mitchflix.net\" (usually OK if presenting also failed)"}
Jul 24 16:43:03 calcifer caddy[1244251]: {"level":"error","ts":1690180983.7752678,"logger":"tls.obtain","msg":"could not get certificate from issuer","identifier":"mitchflix.net","issuer":"acme-v02.api.letsencrypt.org-directory","error":"[mitchflix.net] solving challenges: presenting for challenge: adding temporary record for zone \"mitchflix.net.\": got error status: HTTP 400: [{Code:6003 Message:Invalid request headers}] (order=https://acme-v02.api.letsencrypt.org/acme/order/1221956417/196951955567) (ca=https://acme-v02.api.letsencrypt.org/directory)"}
Jul 24 16:43:03 calcifer caddy[1244251]: {"level":"info","ts":1690180983.7755237,"logger":"http","msg":"waiting on internal rate limiter","identifiers":["mitchflix.net"],"ca":"https://acme.zerossl.com/v2/DV90","account":"caddy@zerossl.com"}
Jul 24 16:43:03 calcifer caddy[1244251]: {"level":"info","ts":1690180983.775531,"logger":"http","msg":"done waiting on internal rate limiter","identifiers":["mitchflix.net"],"ca":"https://acme.zerossl.com/v2/DV90","account":"caddy@zerossl.com"}
Jul 24 16:43:07 calcifer caddy[1244251]: {"level":"info","ts":1690180987.896306,"logger":"http.acme_client","msg":"trying to solve challenge","identifier":"mitchflix.net","challenge_type":"dns-01","ca":"https://acme.zerossl.com/v2/DV90"}
Jul 24 16:43:08 calcifer caddy[1244251]: {"level":"error","ts":1690180988.1270409,"logger":"http.acme_client","msg":"cleaning up solver","identifier":"mitchflix.net","challenge_type":"dns-01","error":"no memory of presenting a DNS record for \"_acme-challenge.mitchflix.net\" (usually OK if presenting also failed)"}
Jul 24 16:43:09 calcifer caddy[1244251]: {"level":"error","ts":1690180989.5768328,"logger":"tls.obtain","msg":"could not get certificate from issuer","identifier":"mitchflix.net","issuer":"acme.zerossl.com-v2-DV90","error":"[mitchflix.net] solving challenges: presenting for challenge: adding temporary record for zone \"mitchflix.net.\": got error status: HTTP 400: [{Code:6003 Message:Invalid request headers}] (order=https://acme.zerossl.com/v2/DV90/order/sjN4V5fOg-J55rRaCWsK3g) (ca=https://acme.zerossl.com/v2/DV90)"}
Jul 24 16:43:09 calcifer caddy[1244251]: {"level":"error","ts":1690180989.5768673,"logger":"tls.obtain","msg":"will retry","error":"[mitchflix.net] Obtain: [mitchflix.net] solving challenges: presenting for challenge: adding temporary record for zone \"mitchflix.net.\": got error status: HTTP 400: [{Code:6003 Message:Invalid request headers}] (order=https://acme.zerossl.com/v2/DV90/order/sjN4V5fOg-J55rRaCWsK3g) (ca=https://acme.zerossl.com/v2/DV90)","attempt":1,"retrying_in":60,"elapsed":8.04396134,"max_duration":2592000}
Jul 24 16:44:09 calcifer caddy[1244251]: {"level":"info","ts":1690181049.577499,"logger":"tls.obtain","msg":"obtaining certificate","identifier":"mitchflix.net"}
Jul 24 16:44:10 calcifer caddy[1244251]: {"level":"info","ts":1690181050.9394023,"logger":"http.acme_client","msg":"trying to solve challenge","identifier":"mitchflix.net","challenge_type":"dns-01","ca":"https://acme-staging-v02.api.letsencrypt.org/directory"}
Jul 24 16:44:11 calcifer caddy[1244251]: {"level":"error","ts":1690181051.173163,"logger":"http.acme_client","msg":"cleaning up solver","identifier":"mitchflix.net","challenge_type":"dns-01","error":"no memory of presenting a DNS record for \"_acme-challenge.mitchflix.net\" (usually OK if presenting also failed)"}
Jul 24 16:44:11 calcifer caddy[1244251]: {"level":"error","ts":1690181051.425305,"logger":"tls.obtain","msg":"could not get certificate from issuer","identifier":"mitchflix.net","issuer":"acme-v02.api.letsencrypt.org-directory","error":"[mitchflix.net] solving challenges: presenting for challenge: adding temporary record for zone \"mitchflix.net.\": got error status: HTTP 400: [{Code:6003 Message:Invalid request headers}] (order=https://acme-staging-v02.api.letsencrypt.org/acme/order/112117674/9918401374) (ca=https://acme-staging-v02.api.letsencrypt.org/directory)"}
Jul 24 16:44:14 calcifer caddy[1244251]: {"level":"info","ts":1690181054.0701044,"logger":"http.acme_client","msg":"trying to solve challenge","identifier":"mitchflix.net","challenge_type":"dns-01","ca":"https://acme.zerossl.com/v2/DV90"}
Jul 24 16:44:14 calcifer caddy[1244251]: {"level":"error","ts":1690181054.2926908,"logger":"http.acme_client","msg":"cleaning up solver","identifier":"mitchflix.net","challenge_type":"dns-01","error":"no memory of presenting a DNS record for \"_acme-challenge.mitchflix.net\" (usually OK if presenting also failed)"}
Jul 24 16:44:15 calcifer caddy[1244251]: {"level":"error","ts":1690181055.007349,"logger":"tls.obtain","msg":"could not get certificate from issuer","identifier":"mitchflix.net","issuer":"acme.zerossl.com-v2-DV90","error":"[mitchflix.net] solving challenges: presenting for challenge: adding temporary record for zone \"mitchflix.net.\": got error status: HTTP 400: [{Code:6003 Message:Invalid request headers}] (order=https://acme.zerossl.com/v2/DV90/order/VGJ-1hHKDR9ZivViq_TL_A) (ca=https://acme.zerossl.com/v2/DV90)"}
Jul 24 16:44:15 calcifer caddy[1244251]: {"level":"error","ts":1690181055.0073886,"logger":"tls.obtain","msg":"will retry","error":"[mitchflix.net] Obtain: [mitchflix.net] solving challenges: presenting for challenge: adding temporary record for zone \"mitchflix.net.\": got error status: HTTP 400: [{Code:6003 Message:Invalid request headers}] (order=https://acme.zerossl.com/v2/DV90/order/VGJ-1hHKDR9ZivViq_TL_A) (ca=https://acme.zerossl.com/v2/DV90)","attempt":2,"retrying_in":120,"elapsed":73.474482544,"max_duration":2592000}

Verifying the token:

curl -X GET "https://api.cloudflare.com/client/v4/user/tokens/verify" \
     -H "Authorization: Bearer -zvy5_FsqPJWlaDX0iK123456etc0dh6HgjYtwAa" \
     -H "Content-Type:application/json"
{"result":{"id":"0b5903ebe18a05d81af7c6dc5ed3a220","status":"active","not_before":"2023-07-24T00:00:00Z","expires_on":"2024-07-24T23:59:59Z"},"success":true,"errors":[],"messages":[{"code":10000,"message":"This API Token is valid and active","type":null}]}

I have the zones set as zone: read and DNS: edit as advised in the documentation linked above, and have made sure to use an API token, not the API key.

I found another link where you suggest a command to check the environment caddy sees. Is there a different command for the systemd process? Using:

caddy environ

Does not show the EV ‘CLOUDFLARE_AUTH_TOKEN’ at all, but I am wondering if that is checking the previous package install and not the systemd process?

27

Went back and changed the EV via the instructions here.

Still receiving a SERVFAIL error

Jul 24 17:10:21 calcifer caddy[1245467]: {"level":"error","ts":1690182621.5905697,"logger":"tls.obtain","msg":"will retry","error":"[mitchflix.net] Obtain: [mitchflix.net] solving challenges: presenting for challenge: could not determine zone for domain \"_acme-challenge.mitchflix.net\": unexpected response code 'SERVFAIL' for _acme-challenge.mitchflix.net. (order=https://acme.zerossl.com/v2/DV90/order/c3zbuZdOkIH908vfKA9k5A) (ca=https://acme.zerossl.com/v2/DV90)","attempt":2,"retrying_in":120,"elapsed":76.622900654,"max_duration":2592000}

Absolutely pulling my hair out here, because it’s the same error across both google domains and cloudflare

28

Also tried this solution to include lines of code to change the timeout times. No difference. My caddyfile now looks like:

mitchflix.net {
        reverse_proxy 192.168.1.31:8096
        tls {
                issuer acme {
                        dns cloudflare {env.CF_API_TOKEN}
                        propagation_delay 60s
                        propagation_timeout -1
                }
        }
}
29

OMG I FIXED IT!!!
Holy crap 2 weeks of troubleshooting paid off.

I had to add the resolver address specific to my domain provider, as well as the propagation delay instructions. Because I’m new to this I didn’t understand that in other forum posts, when it was suggested to add the “resolver” line, it was meant to be pointing to the IP address for the DNS server relevant to my domain provider.

I also just in case change the connection settings for my server hosting caddy to point to the same cloudlfare DNS server, 1.1.1.1. Not sure if that is necessary or not.

The final code in my caddyfile looks like:

mitchflix.net {
        reverse_proxy 192.168.1.31:8096
        tls {
                issuer acme {
                        dns cloudflare {env.CF_API_TOKEN}
                        resolvers 1.1.1.1
                        propagation_delay 60s
                        propagation_timeout -1
                }
        }
}

Thankyou to matt and francis for taking the time to reply to me!

2 Likes
30

Glad you figured it out, nice job persisting. Sounds like one of your local DNS responders on your network or machine were misconfigured, that’s usually why specifying a resolver works.

1 Like
31

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.