Novice trying to reverse proxy with caddy for Jellyfin - problems

1. Caddy version (caddy version):

v2.1.1

2. How I run Caddy:

installed from the command line using sudo apt install caddy

a. System environment:

Ubuntu 20, VPS (CLI only), no docker

b. Command:

caddy reverse-proxy --from xxxx.xxxx.com --to 127.0.0.1:8096

c. Service/unit/compose file:

N/A

d. My complete Caddyfile or JSON config:

No config file, trying to do the ONE LINER reverse proxy method

3. The problem I’m having:

caddy cannot startup on port 80, tells me something else is already on port 80 (but I can’t figure out what)

4. Error messages and/or full log output:

root@XXX:~# caddy reverse-proxy --from xxx.xxx.com --to 127.0.0.1:8096
2020/08/11 21:49:20.777 WARN    admin   admin endpoint disabled
2020/08/11 21:49:20.777 INFO    http    server is listening only on the HTTPS port but has no TLS connection policies; adding one to enable TLS {"server_name":"proxy", "https_port": 443}
2020/08/11 21:49:20.777 INFO    http    enabling automatic HTTP->HTTPS redirects                                              {"server_name": "proxy"}
2020/08/11 21:49:20 [INFO][cache:0xc0000a5680] Started certificate maintenance routine
2020/08/11 21:49:20.777 INFO    tls     cleaned up storage units
reverse-proxy: loading new config: http app module: start: tcp: listening on :80                                              : listen tcp :80: bind: **address already in use**

5. What I already tried:

I checked the ports with
sudo netstat -tunlp | grep :80

Nothing seems to be interfering:

root@xxxxxx:~# sudo netstat -tunlp | grep :80
tcp6       0      0 :::8096                 :::*                    LISTEN      1061/jellyfin
tcp6       0      0 :::80                   :::*                    LISTEN      13132/caddy

6. Links to relevant resources:

Using install guide found here

And Jellyfin specific instructions found here

If you installed Caddy with apt, then you’ll already have a Caddy instance running via a systemd service, using port 80.

You can find the config file for that service at /etc/caddy/Caddyfile and the service file at /lib/systemd/system/caddy.service.

I would recommend running Caddy using the service instead of the reverse-proxy command. It’s very simple. You’ll just need to replace the contents of /etc/caddy/Caddyfile with something like this:

domain.com {
	reverse_proxy 127.0.0.1:8096
}

Then you’ll need to reload the Caddy instance with the new config by running the following:

sudo systemctl reload caddy

And you should be good to go. You can check the logs with:

journalctl --no-pager -u caddy | less
2 Likes

Huge help thank you. I followed these steps. This step:

sudo systemctl caddy reload

Threw an error but I just rebooted the server to restart the service. The service doesn’t appear to be working, any thoughts? subdomain.mydomain.com is a REDACTED version. The actual line in the log is my real domain, it’s a subdomain of a domain I own that is forwarded to my server domain IP (which is static) I’ve tried a bunch of different ways but I keep getting the UNRECOGNIZED DIRECTIVE error. Here’s the output of the logs:

-- Reboot --
Aug 12 23:26:38 XXXXXXX systemd[1]: Started Caddy.
Aug 12 23:26:39 XXXXXXX caddy[1028]: caddy.HomeDir=/var/lib/caddy
Aug 12 23:26:39 XXXXXXX caddy[1028]: caddy.AppDataDir=/var/lib/caddy/.local/share/caddy
Aug 12 23:26:39 XXXXXXX caddy[1028]: caddy.AppConfigDir=/var/lib/caddy/.config/caddy
Aug 12 23:26:39 XXXXXXX caddy[1028]: caddy.ConfigAutosavePath=/var/lib/caddy/.config/caddy/autosave.json
Aug 12 23:26:39 XXXXXXX caddy[1028]: runtime.GOOS=linux
Aug 12 23:26:39 XXXXXXX caddy[1028]: runtime.GOARCH=amd64
Aug 12 23:26:39 XXXXXXX caddy[1028]: runtime.Compiler=gc
Aug 12 23:26:39 XXXXXXX caddy[1028]: runtime.NumCPU=8
Aug 12 23:26:39 XXXXXXX caddy[1028]: runtime.GOMAXPROCS=8
Aug 12 23:26:39 XXXXXXX caddy[1028]: runtime.Version=go1.14.4
Aug 12 23:26:39 XXXXXXX caddy[1028]: os.Getwd=/
Aug 12 23:26:39 XXXXXXX caddy[1028]: LANG=C.UTF-8
Aug 12 23:26:39 XXXXXXX caddy[1028]: PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/snap/bin
Aug 12 23:26:39 XXXXXXX caddy[1028]: HOME=/var/lib/caddy
Aug 12 23:26:39 XXXXXXX caddy[1028]: LOGNAME=caddy
Aug 12 23:26:39 XXXXXXX caddy[1028]: USER=caddy
Aug 12 23:26:39 XXXXXXX caddy[1028]: INVOCATION_ID=7fc8b9ee93d94a2a8a6ee88422c6d2aa
Aug 12 23:26:39 XXXXXXX caddy[1028]: JOURNAL_STREAM=9:19083
Aug 12 23:26:39 XXXXXXX caddy[1028]: {"level":"info","ts":1597274799.2519379,"msg":"using provided configuration","config_file":"/etc/caddy/Caddyfile","config_adapter":""}
Aug 12 23:26:39 XXXXXXX caddy[1028]: run: adapting config using caddyfile: /etc/caddy/Caddyfile:20: unrecognized directive: subdomain.mydomain.com
Aug 12 23:26:39 XXXXXXX systemd[1]: caddy.service: Main process exited, code=exited, status=1/FAILURE
Aug 12 23:26:39 XXXXXXX systemd[1]: caddy.service: Failed with result 'exit-code'.

That’s clearly due to invalid configuration. Please post the contents of your Caddyfile so we can help, it’s hard to point at a specific problem without seeing the file. But as a primer, reading this page should help you understand how a Caddyfile should look:

Thanks for taking a look!

  #
  # Unless the file starts with a global options block, the first
  # uncommented line is always the address of your site.
  #
  # To use your own domain name (with automatic HTTPS), first make
  # sure your domain's A/AAAA DNS records are properly pointed to
  # this machine's public IP, then replace the line below with your
  # domain name.
  :80

  # Set this path to your site's directory.
  root * /usr/share/caddy

  # Enable the static file server.
  file_server

  # Another common task is to set up a reverse proxy:
  # reverse_proxy localhost:8080
  http://subdomain.mydomain.com     {

  reverse_proxy 127.0.0.1:8096

  }```

You’ll need to get rid of this stuff or wrap the directives following :80 in { } braces. As written in the docs I linked above:

In this case, you have two site blocks, i.e. :80 and http://subdomain.mydomain.com, so those clash, and you need to have all site blocks using braces.

The contents that were in that file are just meant as an introductory example, not meant to be left in.

Thank you, again, for all your help. Progress has been made but a new error has been thrown off.

# The Caddyfile is an easy way to configure your Caddy web server.
#
# Unless the file starts with a global options block, the first
# uncommented line is always the address of your site.

http://subdomain.xxxxxx.com    {

reverse_proxy 127.0.0.1:8096

}

When I reload I get:

reload: sending configuration to instance: performing request: Post "http://localhost:2019/load": dial tcp 127.0.0.1:2019: connect: connection refused

Hmm, that might be because Caddy failed to load in the first place. You probably need a sudo systemctl caddy start now to give it a kick. Reload won’t work if it hadn’t been started first (or failed to start due to whatever issue, i.e. bad config).

To clarify, Caddy listens on 127.0.0.1:2019 to provide its configuration API. It’s how it implements graceful reloads and online configuration changes. The reload command essentially just tries to push the new configuration to the running Caddy instance by making an HTTP request to its config API.

Unknown operation caddy.

I can reboot the server but I just wanted to let you know. When I use sudo systemctl caddy I get this error.

^ that’s what you ran?

You can also run sudo systemctl caddy status to see what’s going on with the service.

If I just ‘caddy reload’ it works. EDIT sorry, it doesn’t work, it just reloads and throws that error.

  root@xxxxx:/etc/caddy# sudo systemctl caddy status
    Unknown operation caddy.
    root@xxxxx:/etc/caddy#

Oops, sorry I confused two different CLI param orderings…

There’s both systemctl and service commands, which are pretty similar, but they have different parameter orders.

For example:

sudo service caddy status
sudo systemctl status caddy

Both of these should do the same thing. I just gave you the wrong one.

root@XXXXXXX:~# sudo service caddy status
● caddy.service - Caddy
   Loaded: loaded (/lib/systemd/system/caddy.service; enabled; vendor preset: en
   Active: active (running) since Thu 2020-08-13 01:00:15 UTC; 1min 26s ago
     Docs: https://caddyserver.com/docs/
 Main PID: 977 (caddy)
    Tasks: 12 (limit: 4915)
   CGroup: /system.slice/caddy.service
           └─977 /usr/bin/caddy run --environ --config /etc/caddy/Caddyfile

Aug 13 01:00:16 XXXXXXX caddy[977]: USER=caddy
Aug 13 01:00:16 XXXXXXX caddy[977]: INVOCATION_ID=f050199ec89c452984f11b70dbd
Aug 13 01:00:16 XXXXXXX caddy[977]: JOURNAL_STREAM=9:16230
Aug 13 01:00:16 XXXXXXX caddy[977]: {"level":"info","ts":1597280416.2935572,"
Aug 13 01:00:16 XXXXXXX caddy[977]: {"level":"info","ts":1597280416.308594,"l
Aug 13 01:00:16 XXXXXXX caddy[977]: {"level":"info","ts":1597280416.3092139,"
Aug 13 01:00:16 XXXXXXX caddy[977]: {"level":"info","ts":1597280416.313084,"l
Aug 13 01:00:16 XXXXXXX caddy[977]: 2020/08/13 01:00:16 [INFO][cache:0xc0000b
Aug 13 01:00:16 XXXXXXX caddy[977]: {"level":"info","ts":1597280416.3170059,"
Aug 13 01:00:16 XXXXXXX caddy[977]: {"level":"info","ts":1597280416.3170598,"
lines 1-19/19 (END)
1 Like

Great! So you should be good to go.

Remember you can use the following to see your full logs:

Would this indicate it’s up and running and working? I’m not getting the server when I visit the domain but that could be a forwarding issue, I guess.

Aug 13 01:00:16 XXXXXXX caddy[977]: caddy.HomeDir=/var/lib/caddy
Aug 13 01:00:16 XXXXXXX caddy[977]: caddy.AppDataDir=/var/lib/caddy/.local/share/caddy
Aug 13 01:00:16 XXXXXXX caddy[977]: caddy.AppConfigDir=/var/lib/caddy/.config/caddy
Aug 13 01:00:16 XXXXXXX caddy[977]: caddy.ConfigAutosavePath=/var/lib/caddy/.config/caddy/autosave.json
Aug 13 01:00:16 XXXXXXX caddy[977]: runtime.GOOS=linux
Aug 13 01:00:16 XXXXXXX caddy[977]: runtime.GOARCH=amd64
Aug 13 01:00:16 XXXXXXX caddy[977]: runtime.Compiler=gc
Aug 13 01:00:16 XXXXXXX caddy[977]: runtime.NumCPU=8
Aug 13 01:00:16 XXXXXXX caddy[977]: runtime.GOMAXPROCS=8
Aug 13 01:00:16 XXXXXXX caddy[977]: runtime.Version=go1.14.4
Aug 13 01:00:16 XXXXXXX caddy[977]: os.Getwd=/
Aug 13 01:00:16 XXXXXXX caddy[977]: LANG=C.UTF-8
Aug 13 01:00:16 XXXXXXX caddy[977]: PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/snap/bin
Aug 13 01:00:16 XXXXXXX caddy[977]: HOME=/var/lib/caddy
Aug 13 01:00:16 XXXXXXX caddy[977]: LOGNAME=caddy
Aug 13 01:00:16 XXXXXXX caddy[977]: USER=caddy
Aug 13 01:00:16 XXXXXXX caddy[977]: INVOCATION_ID=f050199ec89c452984f11b70dbd9862c
Aug 13 01:00:16 XXXXXXX caddy[977]: JOURNAL_STREAM=9:16230
Aug 13 01:00:16 XXXXXXX caddy[977]: {"level":"info","ts":1597280416.2935572,"msg":"using provided configuration","config_file":"/etc/caddy/Caddyfile","config_adapter":""}
Aug 13 01:00:16 XXXXXXX caddy[977]: {"level":"info","ts":1597280416.308594,"logger":"admin","msg":"admin endpoint started","address":"tcp/localhost:2019","enforce_origin":false,"origins":["[::1]:2019","127.0.0.1:2019","localhost:2019"]}
Aug 13 01:00:16 XXXXXXX caddy[977]: {"level":"info","ts":1597280416.3092139,"logger":"http","msg":"server is listening only on the HTTP port, so no automatic HTTPS will be applied to this server","server_name":"srv0","http_port":80}
Aug 13 01:00:16 XXXXXXX caddy[977]: {"level":"info","ts":1597280416.313084,"logger":"tls","msg":"cleaned up storage units"}
Aug 13 01:00:16 XXXXXXX caddy[977]: 2020/08/13 01:00:16 [INFO][cache:0xc0000b4780] Started certificate maintenance routine
Aug 13 01:00:16 XXXXXXX caddy[977]: {"level":"info","ts":1597280416.3170059,"msg":"autosaved config","file":"/var/lib/caddy/.config/caddy/autosave.json"}
Aug 13 01:00:16 XXXXXXX caddy[977]: {"level":"info","ts":1597280416.3170598,"msg":"serving initial configuration"}
(END)```

What does that mean exactly? What behaviour are you seeing in your browser?

Caddy is running correctly - but I’m not sure I understand the problem with the current information.

I own a domain that is registered on Google domains. I’ve setup a subdomain through the DNS panel to forward to my.server.ip.

When I put my.google.domain into the URL bar of a browser, it gives me a SITE CANNOT BE REACHED error but it shows me the IP address of my server - so the forward is working. And caddy is working. But for some reason the URL is not taking me to the Jellyfin server.

Can you get us the output of:

curl -IL googledomain.example.com

Thanks so much for your continued help with this. Here’s the output:

root@xxxxxx:~# curl -IL xxxxx.xxxx.com
HTTP/1.1 301 Moved Permanently
Location: http://xxx.xxxx.xx.xxx  (this correctly points at the IP of my VPS)
Date: Fri, 14 Aug 2020 14:58:21 GMT
Content-Type: text/html; charset=UTF-8
Server: ghs
Content-Length: 218
X-XSS-Protection: 0
X-Frame-Options: SAMEORIGIN

HTTP/1.1 200 OK
Server: Caddy
Date: Fri, 14 Aug 2020 14:58:21 GMT

To test the server, I OPENED port 8096 and tried to visit my.vps.ip:8096 and it did successfully connect to the Jellyfin server. So the server is there and working. I CLOSED 8096 again.

Is there a chance some port I need to have open is not open?

#
# Unless the file starts with a global options block, the first
# uncommented line is always the address of your site.

http:/google.subdomain.com    {

reverse_proxy 127.0.0.1:8096

}```