Caddy with Jellyfin via Google Domains noob help

1. The problem I’m having:

Hi there. I’m attempting to set up a Caddy web server to reverse proxy via HTTPS from my google domains website url to my locally hosted jellyfin server.

I have been through like 4 different tutorials and am having 0 luck and absolutely pulling my hair out. I am finding the tutorials quite difficult as a beginner because it will often tell me to do something without any context or explanation for what I need to do or where.

At first I didn’t have access so i changed permissions. Then the ports wouldn’t bind, so I had to uninstall, download the bin for google domains and put it in my /usr/bin/

I then ran through the set up for installing at as a systemctl service. Then it seemed to all be working on my end, but when I went to the website it would say connection refused, despite being able to access the jellyfin server via the direct IP address. All ports are forwarded.

I then after much googling (because the tutorial doesn’t mention it at all) discovered I needed to include an API for the google domain. So I added that to the Caddyfile, but now I am getting errors when trying to run the service.

#2:

I enter:

sudo systemctl start caddy

and I get back

○ caddy.service - Caddy
     Loaded: loaded (/etc/systemd/system/caddy.service; enabled; vendor preset: enabled)
     Active: inactive (dead) since Wed 2023-07-12 19:48:01 AEST; 17min ago
       Docs: https://caddyserver.com/docs/
    Process: 8195 ExecStart=/usr/bin/caddy run --environ --config /etc/caddy/Caddyfile (code=exited, status=0/SUCCESS)
   Main PID: 8195 (code=exited, status=0/SUCCESS)
        CPU: 69ms

Jul 12 19:48:01 calcifer caddy[8195]: {"level":"info","ts":1689155281.1891105,"logger":"tls.obtain","msg":"releasing lock","identifier":"mitchflix.mov"}
Jul 12 19:48:01 calcifer caddy[8195]: {"level":"info","ts":1689155281.189118,"logger":"tls.obtain","msg":"releasing lock","identifier":"www.mitchflix.mov"}
Jul 12 19:48:01 calcifer caddy[8195]: {"level":"error","ts":1689155281.1891944,"logger":"tls.obtain","msg":"unable to unlock","identifier":"www.mitchflix.mov","lock_key":"issue_cert_www.mitchflix.mov",">
Jul 12 19:48:01 calcifer caddy[8195]: {"level":"error","ts":1689155281.1891963,"logger":"tls.obtain","msg":"unable to unlock","identifier":"mitchflix.mov","lock_key":"issue_cert_mitchflix.mov","error":">
Jul 12 19:48:01 calcifer caddy[8195]: {"level":"error","ts":1689155281.1892118,"logger":"tls","msg":"job failed","error":"www.mitchflix.mov: obtaining certificate: context canceled"}
Jul 12 19:48:01 calcifer caddy[8195]: {"level":"error","ts":1689155281.1892147,"logger":"tls","msg":"job failed","error":"mitchflix.mov: obtaining certificate: context canceled"}
Jul 12 19:48:01 calcifer caddy[8195]: {"level":"info","ts":1689155281.1892295,"logger":"admin","msg":"stopped previous server","address":"localhost:2019"}
Jul 12 19:48:01 calcifer caddy[8195]: {"level":"info","ts":1689155281.1892374,"msg":"shutdown complete","signal":"SIGTERM","exit_code":0}
Jul 12 19:48:01 calcifer systemd[1]: caddy.service: Deactivated successfully.
Jul 12 19:48:01 calcifer systemd[1]: Stopped Caddy.

3. Caddy version:

v2.6.4 h1:2hwYqiRwk1tf3VruhMpLcYTg+11fCdr8S3jhNAdnPy8=

4. How I installed and ran Caddy:

First via the package manager, via the tutorial. Then I found a google link saying that won’t work at all, so I uninstalled.

I then attempted to install manually via get-apt. However it kept saying it couldn’t see the caddy package, and the online conversations on how to fix this were unhelpful. So instead I went to the main website, downloaded the linux/ubuntu binary, and just renamed it to “caddy” and put it in /usr/bin.

It is worth noting the tutorial never mentioned many of the steps required to do this without prior knowledge. I had to just assume they were referring to the downloaded binary on the site, imply that i was meant to rename it (because it downloads as something completely different than what the tutorial refers to it as) then move that to my home folder and continue following the steps in the tutorial. It also off-handedly mentions permissions, but never specifies what commands are required to do this.

When it still wouldn’t work, I then realised I needed a different binary, specifically for the google domain name servers. The tutorial didn’t mention this either. Looking at the documentation for the relevant package showed 2 commands, but no info on how to deploy them. To use them I had to find another google post sharing their Caddyfile for google domains and copy the syntax for the DNS and API key commands and how to format them.

When I input that, it broke again, saying the syntax was wrong. The error was something like:

Unkown global command "mitchflix.mov"

Which I assumed to mean my parsing/syntax was wrong in the config. I edited it to the below, and now it is at least loading the service but it doesn’t seem to be able to get the info from google.

I have tried multiple attempts at rewriting the config file but none of the attempts have worked.

a. System environment:

Ubuntu 22.04
Intel 10th Gen CPU
Erying motherboard (Intel HM570)

b. Command:

sudo systemctl start caddy

c. Service/unit/compose file:

# caddy.service
#
# For using Caddy with a config file.
#
# Make sure the ExecStart and ExecReload commands are correct
# for your installation.
#
# See https://caddyserver.com/docs/install for instructions.
#
# WARNING: This service does not use the --resume flag, so if you
# use the API to make changes, they will be overwritten by the
# Caddyfile next time the service is restarted. If you intend to
# use Caddy's API to configure it, add the --resume flag to the
# `caddy run` command or use the caddy-api.service file instead.

[Unit]
Description=Caddy
Documentation=https://caddyserver.com/docs/
After=network.target network-online.target
Requires=network-online.target

[Service]
Type=notify
User=caddy
Group=caddy
ExecStart=/usr/bin/caddy run --environ --config /etc/caddy/Caddyfile
ExecReload=/usr/bin/caddy reload --config /etc/caddy/Caddyfile --force
TimeoutStopSec=5s
LimitNOFILE=1048576
LimitNPROC=512
PrivateTmp=true
ProtectSystem=full
AmbientCapabilities=CAP_NET_ADMIN CAP_NET_BIND_SERVICE

[Install]
WantedBy=multi-user.target

d. My complete Caddy config:

{
	acme_dns google_domains ("API_KEY_REMOVED")
}

mitchflix.mov, www.mitchflix.mov {
	reverse_proxy localhost:8096
}

I have kept looking and found some documentation on one page for API usage saying to not have any parentheses around the key. I removed those and no change unfortunately. Not really sure what else to try here

Your logs are truncated (notice the > character at the end) so the full error message is not visible.

Please use the command mentioned in the docs to see your full logs: Keep Caddy Running — Caddy Documentation

Did you use parentheses in your config? Don’t do that, parentheses are not valid syntax in the Caddyfile.

Either way, Google Domains has just been transferred to Squarespace, so I recommend switching away from them. We don’t know whether the API will still exist after the transition, so you’re at risk.

Hi, thanks for taking the time to reply to me. Here is the print out from the journalctl command:

Jul 13 13:46:14 calcifer systemd[1]: Starting Caddy...
░░ Subject: A start job for unit caddy.service has begun execution
░░ Defined-By: systemd
░░ Support: http://www.ubuntu.com/support
░░ 
░░ A start job for unit caddy.service has begun execution.
░░ 
░░ The job identifier is 6295.
Jul 13 13:46:14 calcifer caddy[498810]: caddy.HomeDir=/var/lib/caddy
Jul 13 13:46:14 calcifer caddy[498810]: caddy.AppDataDir=/var/lib/caddy/.local/share/caddy
Jul 13 13:46:14 calcifer caddy[498810]: caddy.AppConfigDir=/var/lib/caddy/.config/caddy
Jul 13 13:46:14 calcifer caddy[498810]: caddy.ConfigAutosavePath=/var/lib/caddy/.config/caddy/autosave.json
Jul 13 13:46:14 calcifer caddy[498810]: caddy.Version=v2.6.4 h1:2hwYqiRwk1tf3VruhMpLcYTg+11fCdr8S3jhNAdnPy8=
Jul 13 13:46:14 calcifer caddy[498810]: runtime.GOOS=linux
Jul 13 13:46:14 calcifer caddy[498810]: runtime.GOARCH=amd64
Jul 13 13:46:14 calcifer caddy[498810]: runtime.Compiler=gc
Jul 13 13:46:14 calcifer caddy[498810]: runtime.NumCPU=12
Jul 13 13:46:14 calcifer caddy[498810]: runtime.GOMAXPROCS=12
Jul 13 13:46:14 calcifer caddy[498810]: runtime.Version=go1.20.5
Jul 13 13:46:14 calcifer caddy[498810]: os.Getwd=/
Jul 13 13:46:14 calcifer caddy[498810]: LANG=en_AU.UTF-8
Jul 13 13:46:14 calcifer caddy[498810]: LANGUAGE=en_AU:en
Jul 13 13:46:14 calcifer caddy[498810]: PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/snap/bin
Jul 13 13:46:14 calcifer caddy[498810]: NOTIFY_SOCKET=/run/systemd/notify
Jul 13 13:46:14 calcifer caddy[498810]: HOME=/var/lib/caddy
Jul 13 13:46:14 calcifer caddy[498810]: LOGNAME=caddy
Jul 13 13:46:14 calcifer caddy[498810]: USER=caddy
Jul 13 13:46:14 calcifer caddy[498810]: INVOCATION_ID=13da95469b274f2d8879b665a3243e68
Jul 13 13:46:14 calcifer caddy[498810]: JOURNAL_STREAM=8:1651242
Jul 13 13:46:14 calcifer caddy[498810]: SYSTEMD_EXEC_PID=498810
Jul 13 13:46:14 calcifer caddy[498810]: {"level":"info","ts":1689219974.2605836,"msg":"using provided configuration","config_file":"/etc/caddy/Caddyfile","config_adapter":""}
Jul 13 13:46:14 calcifer caddy[498810]: Error: adapting config using caddyfile: server block without any key is global configuration, and if used, it must be first
Jul 13 13:46:14 calcifer systemd[1]: caddy.service: Main process exited, code=exited, status=1/FAILURE
░░ Subject: Unit process exited
░░ Defined-By: systemd
░░ Support: http://www.ubuntu.com/support
░░ 
░░ An ExecStart= process belonging to unit caddy.service has exited.
░░ 
░░ The process' exit code is 'exited' and its exit status is 1.
Jul 13 13:46:14 calcifer systemd[1]: caddy.service: Failed with result 'exit-code'.
░░ Subject: Unit failed
░░ Defined-By: systemd
░░ Support: http://www.ubuntu.com/support
░░ 
░░ The unit caddy.service has entered the 'failed' state with result 'exit-code'.
Jul 13 13:46:14 calcifer systemd[1]: Failed to start Caddy.
░░ Subject: A start job for unit caddy.service has failed
░░ Defined-By: systemd
░░ Support: http://www.ubuntu.com/support
░░ 
░░ A start job for unit caddy.service has finished with a failure.
░░ 
░░ The job identifier is 6295 and the job result is failed.
lines 1404-1457/1457 (END)

My Caddyfile has been updated, it now looks like this:

{
	acme_dns google_domains WFAKEAPIKEYETCETCETC==
}

mitchflix.mov {
	reverse_proxy 192.168.1.28:8096
}

www.mitchflix.mov {
        reverse_proxy 192.168.1.28:8096
}

I am still unable to launch caddy. I have tried rebooting multiple times also to no avail.

I think your config might not have the contents you think it does. Are you sure that’s all that’s in /etc/caddy/Caddyfile? Is there other text in the file?

image

Screenshot from 2023-07-13 22-24-43

Here’s a screenshot of the folder, and then the file contents (cropped to hide API key). The text is the same as my last pasted, and you can see no lines of code after line 11.

That’s very strange. Are there any weird hidden characters in the text file somehow? Wrong text encoding? (Try making sure it’s UTF-8.) Check for mixed newline characters.

That error happens when the Caddyfile parser thinks there’s no “keys” for a server block, i.e. no domains preceding the {. Visually the file looks fine, so I can only assume it’s an invisible text encoding problem.

Ok, so I checked the file, no information at the start of it before the first {. However, the file type is returning as ascii:

nimda@calcifer:~$ file -bi /etc/caddy/Caddyfile
text/plain; charset=us-ascii

I then checked if it passes utf-8 following a command found on google:

nimda@calcifer:~$ iconv -f utf8 /etc/caddy/Caddyfile -t utf8 -o /dev/null
nimda@calcifer:~$ 

If I’m understanding the advice right, the command should return a 0 value if the file passes as utf8, which it didn’t.

However when I try to convert the file type, it doesn’t do anything:

nimda@calcifer:~$ file -bi /etc/caddy/Caddyfile
text/plain; charset=us-ascii
nimda@calcifer:~$ iconv -f us-ascii -t UTF-8 /etc/caddy/Caddyfile -o Caddyfile
nimda@calcifer:~$ file -bi /etc/caddy/Caddyfile
text/plain; charset=us-ascii
nimda@calcifer:~$ iconv -f ascii -t UTF-8//TRANSLIT Caddyfile -o Caddyfile
nimda@calcifer:~$ sudo iconv -f ascii -t UTF-8//TRANSLIT Caddyfile -o Caddyfile
[sudo] password for nimda: 
nimda@calcifer:~$ file -i /etc/caddy/Caddyfile
/etc/caddy/Caddyfile: text/plain; charset=us-ascii

You didn’t have it overwrite itself, you outputted the file to your current directory. Do -o /etc/caddy/Caddyfile instead.

That said, my Caddyfiles show text/plain; charset=us-ascii as well, so that should be fine.

Hmm. I’m really stumped. This is really weird.

Can you try running caddy adapt --config /etc/caddy/Caddyfile ? This should spit out your config as JSON.

nimda@calcifer:~$ caddy adapt --config /etc/caddy/Caddyfile
{"apps":{"http":{"servers":{"srv0":{"listen":[":443"],"routes":[{"match":[{"host":["www.mitchflix.mov"]}],"handle":[{"handler":"subroute","routes":[{"handle":[{"handler":"reverse_proxy","upstreams":[{"dial":"192.168.1.28:8096"}]}]}]}],"terminal":true},{"match":[{"host":["mitchflix.mov"]}],"handle":[{"handler":"subroute","routes":[{"handle":[{"handler":"reverse_proxy","upstreams":[{"dial":"192.168.1.28:8096"}]}]}]}],"terminal":true}]}}},"tls":{"automation":{"policies":[{"subjects":["www.mitchflix.mov","mitchflix.mov"],"issuers":[{"challenges":{"dns":{"provider":{"access_token":"WUF0VEVLSEswRzJtOHVfbFg0ajNZUQ==","name":"google_domains"}}},"module":"acme"},{"challenges":{"dns":{"provider":{"access_token":"removed","name":"google_domains"}}},"module":"zerossl"}]}]}}}}
2023/07/14 06:44:00.109	WARN	caddyfile	Caddyfile input is not formatted; run the 'caddy fmt' command to fix inconsistencies	{"file": "/etc/caddy/Caddyfile", "line": 8}

I did run the ‘caddy fmt’ command by the way, it didn’t change anything.

Ubuntu also now randomly has lost access to the mounted drives, that all my other computers have no problem accessing. I’m starting to think linux doesn’t want me using it :sweat_smile:

Wild. So running caddy adapt directly works.

It makes no sense to me that Caddy running as a service wouldn’t adapt the same way and would return an error.

Try restarting the systemd service one more time with sudo systemctl restart caddy.

:man_shrugging:

Sorry, I should mention I received an error when running the 'fmt command:

Error: Caddyfile:8: Caddyfile input is not formatted

Restarting the service did work, but when attempting to access the site I still receive an error

ERR_SSL_PROTOCOL_ERROR

I checked journalctl again and now there seems to be different content listing a server side error.

Jul 14 18:10:45 calcifer caddy[8337]: {"level":"error","ts":1689322245.437629,"logger":"tls.obtain","msg":"could not get certificate from issuer","identifier":"www.mitchflix.mov","issuer":"acme.zerossl.com-v2-DV90","error":"[www.mitchflix.mov] solving challenges: presenting for challenge: could not determine zone for domain \"_acme-challenge.www.mitchflix.mov\": unexpected response code 'SERVFAIL' for _acme-challenge.www.mitchflix.mov. (order=https://acme.zerossl.com/v2/DV90/order/7oLZJbLxerI7R7fhzrl0_g) (ca=https://acme.zerossl.com/v2/DV90)"}
Jul 14 18:10:45 calcifer caddy[8337]: {"level":"error","ts":1689322245.437665,"logger":"tls.obtain","msg":"will retry","error":"[www.mitchflix.mov] Obtain: [www.mitchflix.mov] solving challenges: presenting for challenge: could not determine zone for domain \"_acme-challenge.www.mitchflix.mov\": unexpected response code 'SERVFAIL' for _acme-challenge.www.mitchflix.mov. (order=https://acme.zerossl.com/v2/DV90/order/7oLZJbLxerI7R7fhzrl0_g) (ca=https://acme.zerossl.com/v2/DV90)","attempt":4,"retrying_in":300,"elapsed":397.19306532,"max_duration":2592000}
Jul 14 18:10:45 calcifer caddy[8337]: {"level":"error","ts":1689322245.4780235,"logger":"tls.obtain","msg":"could not get certificate from issuer","identifier":"mitchflix.mov","issuer":"acme.zerossl.com-v2-DV90","error":"[mitchflix.mov] solving challenges: presenting for challenge: could not determine zone for domain \"_acme-challenge.mitchflix.mov\": unexpected response code 'SERVFAIL' for _acme-challenge.mitchflix.mov. (order=https://acme.zerossl.com/v2/DV90/order/l1-PsfzRlCejW9hhFjJ_vw) (ca=https://acme.zerossl.com/v2/DV90)"}
Jul 14 18:10:45 calcifer caddy[8337]: {"level":"error","ts":1689322245.478045,"logger":"tls.obtain","msg":"will retry","error":"[mitchflix.mov] Obtain: [mitchflix.mov] solving challenges: presenting for challenge: could not determine zone for domain \"_acme-challenge.mitchflix.mov\": unexpected response code 'SERVFAIL' for _acme-challenge.mitchflix.mov. (order=https://acme.zerossl.com/v2/DV90/order/l1-PsfzRlCejW9hhFjJ_vw) (ca=https://acme.zerossl.com/v2/DV90)","attempt":4,"retrying_in":300,"elapsed":397.231989828,"max_duration":2592000}

There was a LOT of text so I only quoted the last couple paragraphs. This has led me to check my public ports, which are still closed despite being routed and port forwarded.

SERVFAIL is usually an error in the DNS server (it’s like a 500 HTTP error, but for DNS) – or the resolver. I would verify your SOA / DNS records / resolver config.

Is there a specific tutorial or set of instructions I can follow for google domains? I have tried googling multiple times but can only find stuff for sites like cloudflare, and as such aren’t sure if I’m doing it right

EDIT: I have attempted to just redo it myself, it seems to be successfully rerouting to my IP which shows up in the browser, but I am receiving another error:

SSL_ERROR_INTERNAL_ERROR_ALERT

The page you are trying to view cannot be shown because the authenticity of the received data could not be verified.

I’m guessing it’s a certificate issue now? Any ideas on how I can fix this?

The thing that is killing me is most of the advice is for people using docker, but I’m on Ubuntu. For example I found this solution which I’d like to try, but I don’t have the same files, nor does googling show me where they would be stored. It suggests putting the following code into an .env file:

TLS_OPTS="tls {
  on_demand
}"

:exploding_head: :exploding_head:

Honestly I’d suggest you just move away from Google Domains ASAP because its future is clearly going to be rocky at best since it was just acquired by Squarespace. There’s no guarantee the API will continue to work after the transition. There are much better domain registrars with much better DNS APIs.

1 Like

While I would be happy to, I just bought the google domain so it would be nice to get that working at least for the duration I paid for, rather than paying again

1 Like

I wonder if the Caddyfile error was because of some hidden characters? Can you upload the file directly (not copy+paste) and we can examine it? (Remove any credentials first.)

Can you update me on where you’re at specifically. Are you still getting SERVFAIL?

I’m going to try a complete reinstall from scratch because I am completely stumped. If I continue getting this issue I will upload the caddyfile directly

So immediately I have an issue in the first step of the tutorial.

I have purchased a new domain name, downloaded the relevant binary, renamed it to caddy, moved it to the home folder. Then after the first step of the manual installation:

sudo mv caddy /usr/bin/

Although the file is successfully moved, verifying it does not work:

$ sudo caddy version
sudo: caddy: command not found

EDIT: It turns out it is a permissions issue. I managed to successfully go around the manual install of the cloudflare binary, and follow the curl instructions to install via the apt command;

sudo apt install -y debian-keyring debian-archive-keyring apt-transport-https
curl -1sLf 'https://dl.cloudsmith.io/public/caddy/stable/gpg.key' | sudo gpg --dearmor -o /usr/share/keyrings/caddy-stable-archive-keyring.gpg
curl -1sLf 'https://dl.cloudsmith.io/public/caddy/stable/debian.deb.txt' | sudo tee /etc/apt/sources.list.d/caddy-stable.list
sudo apt update
sudo apt install caddy

And then manually putting the cloudflare binary, again named ‘caddy’ in to /usr/bin/, however when attempting to run the service through the systemctl I get a permission denied error.

$ systemctl status caddy
× caddy.service - Caddy
     Loaded: loaded (/etc/systemd/system/caddy.service; enabled; vendor preset: enabled)
     Active: failed (Result: exit-code) since Sun 2023-07-23 21:22:31 AEST; 12s ago
       Docs: https://caddyserver.com/docs/
    Process: 1138438 ExecStart=/usr/bin/caddy run --environ --config /etc/caddy/Caddyfile (code=exited, status=203/EXEC)
   Main PID: 1138438 (code=exited, status=203/EXEC)
        CPU: 8ms

Jul 23 21:22:31 calcifer systemd[1]: Starting Caddy...
Jul 23 21:22:31 calcifer systemd[1138438]: caddy.service: Failed to locate executable /usr/bin/caddy: Permission denied
Jul 23 21:22:31 calcifer systemd[1138438]: caddy.service: Failed at step EXEC spawning /usr/bin/caddy: Permission denied
Jul 23 21:22:31 calcifer systemd[1]: caddy.service: Main process exited, code=exited, status=203/EXEC
Jul 23 21:22:31 calcifer systemd[1]: caddy.service: Failed with result 'exit-code'.
Jul 23 21:22:31 calcifer systemd[1]: Failed to start Caddy.