Badly configured Caddy causing Kodi crashes with Jellyfin

  1. Caddy version (caddy version):

Latest stable on Ubuntu Desktop 20.04. It should be v.2

  1. How I run Caddy:

Caddyfile configuration

a. System environment:

Ubuntu Desktop 20.04
Intel i5-8400T
32GB RAM
500GB NVME SSD
Jellyfin Media Server
Syncthing Media Server

b. Command:

sudo smartctl status caddy
sudo smartctl reload caddy

d. My complete Caddyfile or JSON config:

# The Caddyfile is an easy way to configure your Caddy web server.
#
# Unless the file starts with a global options block, the first
# uncommented line is always the address of your site.
#
# To use your own domain name (with automatic HTTPS), first make
# sure your domain's A/AAAA DNS records are properly pointed to
# this machine's public IP, then replace the line below with your
# domain name.

jellymmeirolas.ddnss.de {
        reverse_proxy localhost:8096
}

syncmmeirolas.ddnss.de {
        reverse_proxy localhost:8384 {
                header_up Host localhost
        }
}

# Another common task is to set up a reverse proxy:
# reverse_proxy localhost:8080

# Or serve a PHP site through php-fpm:
# php_fastcgi localhost:9000

# Refer to the Caddy docs for more information:
# https://caddyserver.com/docs/caddyfile
  1. The problem I’m having:

Using Kodi v19.0 in conjunction with the Jellyfin Kodi Addon, Kodi randomly crashes.

Sometimes it’s when watching a Video, but most of the times it’s right after the Jellyfin greetings message.

  1. Error messages and/or full log output:
â—Ź caddy.service - Caddy
     Loaded: loaded (/lib/systemd/system/caddy.service; enabled; vendor preset: enabled)
     Active: active (running) since Fri 2021-05-07 17:17:39 CEST; 5h 19min ago
       Docs: https://caddyserver.com/docs/
   Main PID: 1620 (caddy)
      Tasks: 12 (limit: 38074)
     Memory: 39.2M
     CGroup: /system.slice/caddy.service
             └─1620 /usr/bin/caddy run --environ --config /etc/caddy/Caddyfile

Mai 07 17:17:39 mmeiraserv caddy[1620]: {"level":"info","ts":1620400659.7911048,"logger":"tls","msg":"cleaned up storage units"}
Mai 07 17:17:39 mmeiraserv caddy[1620]: {"level":"info","ts":1620400659.7999945,"msg":"autosaved config","file":"/var/lib/caddy/.config/caddy/autosave.json"}
Mai 07 17:17:39 mmeiraserv caddy[1620]: {"level":"info","ts":1620400659.8000119,"msg":"serving initial configuration"}
Mai 07 21:46:58 mmeiraserv caddy[1620]: {"level":"error","ts":1620416818.3571558,"logger":"http.handlers.reverse_proxy","msg":"aborting with incomplete response","error":"write tcp 192.168.178.23:443->5.10.50.141:58156: >
Mai 07 21:47:12 mmeiraserv caddy[1620]: {"level":"error","ts":1620416832.0390239,"logger":"http.handlers.reverse_proxy","msg":"aborting with incomplete response","error":"write tcp 192.168.178.23:443->5.10.50.141:58160: >
Mai 07 21:48:05 mmeiraserv caddy[1620]: {"level":"error","ts":1620416885.494674,"logger":"http.handlers.reverse_proxy","msg":"aborting with incomplete response","error":"write tcp 192.168.178.23:443->5.10.50.141:58196: w>
Mai 07 21:48:05 mmeiraserv caddy[1620]: {"level":"error","ts":1620416885.5029018,"logger":"http.handlers.reverse_proxy","msg":"aborting with incomplete response","error":"write tcp 192.168.178.23:443->5.10.50.141:58200: >
Mai 07 21:56:51 mmeiraserv caddy[1620]: {"level":"error","ts":1620417411.2047021,"logger":"http.handlers.reverse_proxy","msg":"aborting with incomplete response","error":"write tcp 192.168.178.23:443->5.10.50.141:58474: >
Mai 07 21:56:51 mmeiraserv caddy[1620]: {"level":"error","ts":1620417411.2132554,"logger":"http.handlers.reverse_proxy","msg":"aborting with incomplete response","error":"write tcp 192.168.178.23:443->5.10.50.141:58472: >
Mai 07 21:57:08 mmeiraserv caddy[1620]: {"level":"error","ts":1620417428.4119852,"logger":"http.handlers.reverse_proxy","msg":"aborting with incomplete response","error":"write tcp 192.168.178.23:443->5.10.50.141:58476: >
â—Ź jellyfin.service - Jellyfin Media Server
     Loaded: loaded (/lib/systemd/system/jellyfin.service; enabled; vendor preset: enabled)
    Drop-In: /etc/systemd/system/jellyfin.service.d
             └─jellyfin.service.conf
     Active: active (running) since Fri 2021-05-07 17:17:34 CEST; 5h 16min ago
   Main PID: 1006 (jellyfin)
      Tasks: 29 (limit: 38074)
     Memory: 7.9G
     CGroup: /system.slice/jellyfin.service
             └─1006 /usr/bin/jellyfin --webdir=/usr/share/jellyfin/web --restartpath=/usr/lib/jellyfin/restart.sh --ffmpeg=/usr/lib/jellyfin-ffmpeg/ffmpeg

Mai 07 22:31:05 mmeiraserv jellyfin[1006]: [22:31:05] [WRN] WS 5.10.50.141 error receiving data: The remote party closed the WebSocket connection without completing the close handshake.
Mai 07 22:31:05 mmeiraserv jellyfin[1006]: [22:31:05] [INF] WS 5.10.50.141 closed
Mai 07 22:31:17 mmeiraserv jellyfin[1006]: [22:31:17] [INF] WS 5.10.50.141 request
Mai 07 22:31:18 mmeiraserv jellyfin[1006]: [22:31:18] [INF] Server Time Requested...
Mai 07 22:31:18 mmeiraserv jellyfin[1006]: [22:31:18] [INF] Sync Requested for UserID: 'e61f70dcdd75437793d45be9a9525c80' with LastUpdateDT: '2021-05-07T20:25:04z'
Mai 07 22:31:18 mmeiraserv jellyfin[1006]: [22:31:18] [INF] Added: 0, Removed: 0, Updated: 0, Changed User Data: 2
Mai 07 22:31:18 mmeiraserv jellyfin[1006]: [22:31:18] [INF] Request Finished Taking 00:00:00.0071901
Mai 07 22:32:05 mmeiraserv jellyfin[1006]: [22:32:05] [INF] Sending ForceKeepAlive message to 1 inactive WebSockets.
Mai 07 22:32:12 mmeiraserv jellyfin[1006]: [22:32:12] [WRN] WS 5.10.50.141 error receiving data: The remote party closed the WebSocket connection without completing the close handshake.
Mai 07 22:32:12 mmeiraserv jellyfin[1006]: [22:32:12] [INF] WS 5.10.50.141 closed
  1. What I already tried:

Reinstalling Kodi and the Jellyfin addon several times. Trying JellyCon instead of the Jellyfin addon.
Several settings within those addons.
Several settings inside Jellyfin Server itself.
Config Caddyfile in different ways.

Sorry, not ver versed in Linux to try much more.

I don’t see any way this could be Caddy’s fault. I think you should ask the Kodi people why their it crashes (and give them as much detail as possible).

Hello francislavoie

Thx for your reply and looking into it

I am not sure it is Caddys fault because i have been using Kodi for over a year with Jellyfin without Caddy and never had crashes.

And as well because the logs say something about “WebSockets” and “my internal ip->my external ip:port”… Which leads me to think it is server related.

But you’re right, i am a noob so i asked already in the Jellyfin Reddit group and will ask Kodi as well.

Just pne question tho: which is the recommended folder to place the Caddyfile? And am i supposed to issue a “run” command since it never works.

Thx

If you installed it via the Debian package, then /etc/caddy/Caddyfile, which is where the systemd service config looks for it.

Ok. I think that is where i have it

Thx

I am a noob i know but this output from Caddy:

mmeiraserv caddy[1620]: {"level":"error","ts":1620416818.3571558,"logger":"http.handlers.reverse_proxy","msg":"aborting with incomplete response","error":"write tcp 192.168.178.23:443->5.10.50.141:58156

I mean it says explicitly “http.handlers.REVERSE_PROXY” which is what Caddy does…

It points to an IP adress with a port number :58156… Don’t know where that comes from since Jellyfin runs on 8096, which is what i put in the Caddyfile.

Weird…

That log line is truncated. Please run journalctl -u caddy --no-pager | less to see the full log lines.

But like I said, I don’t think this is a problem with Caddy. It has to be a problem with Kodi or Jellyfin, either due to misconfiguration or a bug.

Caddy is known to work well with Jellyfin, there are many users who use Caddy to proxy to Jellyfin (myself included).

There’s nothing in Caddy that would trigger other programs to spontaneously explode, it had to be a problem with those programs.

Yes, you are right.

You were the one that helped me with my previous server, and it worked perfectly… So i suspect Jellyfin or the new Kodi version… Or even the Jellyfin Addon Maybe some dependency not ported to Kodi 19, since i heard about broken Addons after people upgraded to v. 19.

Will post the full log when i am at home.

Cheers and thx for your time

1 Like

When i enter that “journalctl command”, how many lines should i paste here?

I ask because it’s huuuuge :slight_smile:

Just hit Shift+G and it’ll jump to the bottom.

here it is:

Mai 07 22:42:20 mmeiraserv systemd[1]: Reloaded Caddy.
Mai 07 22:53:59 mmeiraserv systemd[1]: Reloading Caddy.
Mai 07 22:53:59 mmeiraserv caddy[9168]: {"level":"info","ts":1620420839.9110525,"msg":"using provided configuration","config_file":"/etc/caddy/Caddyfile","config_adapter":""}
Mai 07 22:53:59 mmeiraserv caddy[1620]: {"level":"info","ts":1620420839.9121513,"logger":"admin.api","msg":"received request","method":"POST","host":"localhost:2019","uri":"/load","remote_addr":"127.0.0.1:56752","headers":{"Accept-Encoding":["gzip"],"Content-Length":["503"],"Content-Type":["application/json"],"Origin":["localhost:2019"],"User-Agent":["Go-http-client/1.1"]}}
Mai 07 22:53:59 mmeiraserv caddy[1620]: {"level":"info","ts":1620420839.9122555,"logger":"admin.api","msg":"config is unchanged"}
Mai 07 22:53:59 mmeiraserv caddy[1620]: {"level":"info","ts":1620420839.9122615,"logger":"admin.api","msg":"load complete"}
Mai 07 22:53:59 mmeiraserv systemd[1]: Reloaded Caddy.
Mai 07 23:17:39 mmeiraserv caddy[1620]: {"level":"info","ts":1620422259.9495852,"logger":"tls.cache.maintenance","msg":"advancing OCSP staple","identifiers":["jellymmeirolas.ddnss.de"],"from":1620723045,"to":1620906371}
Mai 08 17:17:39 mmeiraserv caddy[1620]: {"level":"info","ts":1620487059.7921438,"logger":"tls","msg":"cleaned up storage units"}
Mai 08 19:17:39 mmeiraserv caddy[1620]: {"level":"info","ts":1620494259.9439743,"logger":"tls.cache.maintenance","msg":"advancing OCSP staple","identifiers":["syncmmeirolas.ddnss.de"],"from":1620796193,"to":1620961794}

Why it is so short now is beyond me

That’s just Caddy’s startup stuff, no errors there. You’ll need to do things until you can replicate the problem again, or scroll up a bit (use arrow keys to scroll) until you see the earlier errors. But you should make sure you can actually replicate the problem anyways.

ok. will try later on and get back to you.

thank you for your time

Hey there

When i run the the journalctl command and hit shift + G it jumps to the supposed end.

I then try to select everything from bottom to the top but the terminal window just won’t scroll up for me to select everything…

How can i do this?

Btw, i managed to replicate the Kodi crash at 2:30 a.m but journalctl only shows until 1:59…

less is a program that only renders a screen’s worth of text at a time, but is very fast. Since you have so many logs, this is much easier to navigate.

You navigate it with arrow keys/pg up/pg dn keys, and sometimes it supports mouse scrolling. If there’s more than a screen’s worth that is relevant to copy, then copy chunks at a time. Then hit q to edit.

Alternatively, you can pipe into tail -50 instead to output the last 50 lines for example, which just prints them out in your terminal instead of rendering in a managed screen.

So, i think i managed to do it, but no sign of the 2 crashes around 2:30 a.m:

Mai 09 03:06:34 mmeiraserv caddy[3665]: {"level":"info","ts":1620522394.2368453,"msg":"shutting down apps then terminating","signal":"SIGTERM"}
Mai 09 03:06:34 mmeiraserv systemd[1]: Stopping Caddy...
Mai 09 03:06:35 mmeiraserv caddy[3665]: {"level":"info","ts":1620522395.2373018,"logger":"tls.cache.maintenance","msg":"stopped background certificate maintenance","cache":"0xc0003cf960"}
Mai 09 03:06:35 mmeiraserv caddy[3665]: {"level":"info","ts":1620522395.7377515,"logger":"admin","msg":"stopped previous server"}
Mai 09 03:06:35 mmeiraserv caddy[3665]: {"level":"info","ts":1620522395.7378047,"msg":"shutdown done","signal":"SIGTERM"}
Mai 09 03:06:35 mmeiraserv systemd[1]: caddy.service: Succeeded.
Mai 09 03:06:35 mmeiraserv systemd[1]: Stopped Caddy.
Mai 09 03:06:35 mmeiraserv systemd[1]: Started Caddy.
Mai 09 03:06:35 mmeiraserv caddy[5738]: caddy.HomeDir=/var/lib/caddy
Mai 09 03:06:35 mmeiraserv caddy[5738]: caddy.AppDataDir=/var/lib/caddy/.local/share/caddy
Mai 09 03:06:35 mmeiraserv caddy[5738]: caddy.AppConfigDir=/var/lib/caddy/.config/caddy
Mai 09 03:06:35 mmeiraserv caddy[5738]: caddy.ConfigAutosavePath=/var/lib/caddy/.config/caddy/autosave.json
Mai 09 03:06:35 mmeiraserv caddy[5738]: caddy.Version=v2.3.0
Mai 09 03:06:35 mmeiraserv caddy[5738]: runtime.GOOS=linux
Mai 09 03:06:35 mmeiraserv caddy[5738]: runtime.GOARCH=amd64
Mai 09 03:06:35 mmeiraserv caddy[5738]: runtime.Compiler=gc
Mai 09 03:06:35 mmeiraserv caddy[5738]: runtime.NumCPU=6
Mai 09 03:06:35 mmeiraserv caddy[5738]: runtime.GOMAXPROCS=6
Mai 09 03:06:35 mmeiraserv caddy[5738]: runtime.Version=go1.15.6
Mai 09 03:06:35 mmeiraserv caddy[5738]: os.Getwd=/
Mai 09 03:06:35 mmeiraserv caddy[5738]: LANG=pt_PT.UTF-8
Mai 09 03:06:35 mmeiraserv caddy[5738]: LANGUAGE=pt:pt_BR:en
Mai 09 03:06:35 mmeiraserv caddy[5738]: LC_ADDRESS=de_DE.UTF-8
Mai 09 03:06:35 mmeiraserv caddy[5738]: LC_IDENTIFICATION=de_DE.UTF-8
Mai 09 03:06:35 mmeiraserv caddy[5738]: LC_MEASUREMENT=de_DE.UTF-8
Mai 09 03:06:35 mmeiraserv caddy[5738]: LC_MONETARY=de_DE.UTF-8
Mai 09 03:06:35 mmeiraserv caddy[5738]: LC_NAME=de_DE.UTF-8
Mai 09 03:06:35 mmeiraserv caddy[5738]: LC_NUMERIC=de_DE.UTF-8
Mai 09 03:06:35 mmeiraserv caddy[5738]: LC_PAPER=de_DE.UTF-8
Mai 09 03:06:35 mmeiraserv caddy[5738]: LC_TELEPHONE=de_DE.UTF-8
Mai 09 03:06:35 mmeiraserv caddy[5738]: LC_TIME=de_DE.UTF-8
Mai 09 03:06:35 mmeiraserv caddy[5738]: PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/snap/bin
Mai 09 03:06:35 mmeiraserv caddy[5738]: HOME=/var/lib/caddy
Mai 09 03:06:35 mmeiraserv caddy[5738]: LOGNAME=caddy
Mai 09 03:06:35 mmeiraserv caddy[5738]: USER=caddy
Mai 09 03:06:35 mmeiraserv caddy[5738]: INVOCATION_ID=574818cfde51471c877037f5a7e891c2
Mai 09 03:06:35 mmeiraserv caddy[5738]: JOURNAL_STREAM=8:70051
Mai 09 03:06:35 mmeiraserv caddy[5738]: {"level":"info","ts":1620522395.7760048,"msg":"using provided configuration","config_file":"/etc/caddy/Caddyfile","config_adapter":""}
Mai 09 03:06:35 mmeiraserv caddy[5738]: {"level":"info","ts":1620522395.7773366,"logger":"admin","msg":"admin endpoint started","address":"tcp/localhost:2019","enforce_origin":false,"origins":["[::1]:2019","127.0.0.1:2019","localhost:2019"]}
Mai 09 03:06:35 mmeiraserv caddy[5738]: {"level":"info","ts":1620522395.7775056,"logger":"tls.cache.maintenance","msg":"started background certificate maintenance","cache":"0xc000369260"}
Mai 09 03:06:35 mmeiraserv caddy[5738]: {"level":"info","ts":1620522395.777512,"logger":"http","msg":"server is listening only on the HTTPS port but has no TLS connection policies; adding one to enable TLS","server_name":"srv0","https_port":443}
Mai 09 03:06:35 mmeiraserv caddy[5738]: {"level":"info","ts":1620522395.7775342,"logger":"http","msg":"enabling automatic HTTP->HTTPS redirects","server_name":"srv0"}
Mai 09 03:06:35 mmeiraserv caddy[5738]: {"level":"info","ts":1620522395.7778988,"logger":"http","msg":"enabling automatic TLS certificate management","domains":["syncmmeirolas.ddnss.de","jellymmeirolas.ddnss.de"]}
Mai 09 03:06:35 mmeiraserv caddy[5738]: {"level":"info","ts":1620522395.778621,"logger":"tls","msg":"cleaned up storage units"}
Mai 09 03:06:35 mmeiraserv caddy[5738]: {"level":"info","ts":1620522395.7881315,"msg":"autosaved config","file":"/var/lib/caddy/.config/caddy/autosave.json"}
Mai 09 03:06:35 mmeiraserv caddy[5738]: {"level":"info","ts":1620522395.788148,"msg":"serving initial configuration"}

I suppose this means this issue has nothing to do with Caddy?

Like I said, I don’t think it ever had anything to do with Caddy. I’m just trying to help you find the logs that might be relevant. Your earlier logs with the error were from May 7th, but what you just posted are timestamped May 9th. Go further up (hit the up arrow on your keyboard, or pgup) and you should eventually find them when using less.

This topic was automatically closed after 30 days. New replies are no longer allowed.