Jellyfin behind Caddy on an other server using internal tls

Help
1

1. The problem I’m having:

I’m trying to configure caddy for having internal ssl but it looks like I’m having a SSL problem.

2. Error messages and/or full log output:

Curl the external name

curl https://jelly.billkidd.fr/ -v
*   Trying 192.168.11.41...
* TCP_NODELAY set
* Connected to jelly.billkidd.fr (192.168.11.41) port 443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* successfully set certificate verify locations:
*   CAfile: /etc/ssl/cert.pem
  CApath: none
* TLSv1.2 (OUT), TLS handshake, Client hello (1):
* TLSv1.2 (IN), TLS alert, internal error (592):
* error:14004438:SSL routines:CONNECT_CR_SRVR_HELLO:tlsv1 alert internal error
* Closing connection 0
curl: (35) error:14004438:SSL routines:CONNECT_CR_SRVR_HELLO:tlsv1 alert internal error

Curl the local name:

curl https://jelly.roadrunner -v
*   Trying 192.168.11.41...
* TCP_NODELAY set
* Connected to jelly.roadrunner (192.168.11.41) port 443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* successfully set certificate verify locations:
*   CAfile: /etc/ssl/cert.pem
  CApath: none
* TLSv1.2 (OUT), TLS handshake, Client hello (1):
* TLSv1.2 (IN), TLS handshake, Server hello (2):
* TLSv1.2 (IN), TLS handshake, Certificate (11):
* TLSv1.2 (OUT), TLS alert, unknown CA (560):
* SSL certificate problem: unable to get local issuer certificate
* Closing connection 0
curl: (60) SSL certificate problem: unable to get local issuer certificate
More details here: https://curl.haxx.se/docs/sslcerts.html

curl failed to verify the legitimacy of the server and therefore could not
establish a secure connection to it. To learn more about this situation and
how to fix it, please visit the web page mentioned above.

Caddy logs backend

Jul 16 15:56:23 corneille systemd[1]: Reloading Caddy...
Jul 16 15:56:23 corneille caddy[57382]: {"level":"info","ts":1689537383.836844,"msg":"using provided configuration","config_file":"/etc/caddy/Caddyfile","config_adapter":""}
Jul 16 15:56:23 corneille caddy[57382]: {"level":"warn","ts":1689537383.8376567,"msg":"Caddyfile input is not formatted; run the 'caddy fmt' command to fix inconsistencies","adapter
":"caddyfile","file":"/etc/caddy/Caddyfile","line":10}
Jul 16 15:56:23 corneille caddy[609]: {"level":"info","ts":1689537383.8388731,"logger":"admin.api","msg":"received request","method":"POST","host":"localhost:2019","uri":"/load","re
mote_ip":"127.0.0.1","remote_port":"58230","headers":{"Accept-Encoding":["gzip"],"Cache-Control":["must-revalidate"],"Content-Length":["628"],"Content-Type":["application/json"],"Or
igin":["http://localhost:2019"],"User-Agent":["Go-http-client/1.1"]}}
Jul 16 15:56:23 corneille caddy[609]: {"level":"info","ts":1689537383.8397205,"logger":"admin","msg":"admin endpoint started","address":"localhost:2019","enforce_origin":false,"orig
ins":["//localhost:2019","//[::1]:2019","//127.0.0.1:2019"]}
Jul 16 15:56:23 corneille caddy[609]: {"level":"info","ts":1689537383.839917,"logger":"tls.cache.maintenance","msg":"started background certificate maintenance","cache":"0xc0003d2d2
0"}
Jul 16 15:56:23 corneille caddy[609]: {"level":"info","ts":1689537383.8401237,"logger":"http","msg":"server is listening only on the HTTPS port but has no TLS connection policies; a
dding one to enable TLS","server_name":"srv0","https_port":443}
Jul 16 15:56:23 corneille caddy[609]: {"level":"info","ts":1689537383.840154,"logger":"http","msg":"enabling automatic HTTP->HTTPS redirects","server_name":"srv0"}
Jul 16 15:56:23 corneille caddy[609]: {"level":"info","ts":1689537383.8402743,"logger":"pki.ca.local","msg":"root certificate is already trusted by system","path":"storage:pki/autho
rities/local/root.crt"}
Jul 16 15:56:23 corneille caddy[609]: {"level":"info","ts":1689537383.8403614,"logger":"http","msg":"enabling HTTP/3 listener","addr":":443"}
Jul 16 15:56:23 corneille caddy[609]: {"level":"debug","ts":1689537383.8403864,"logger":"http","msg":"starting server loop","address":"[::]:443","tls":true,"http3":true}
Jul 16 15:56:23 corneille caddy[609]: {"level":"info","ts":1689537383.840401,"logger":"http.log","msg":"server running","name":"srv0","protocols":["h1","h2","h3"]}
Jul 16 15:56:23 corneille caddy[609]: {"level":"debug","ts":1689537383.840432,"logger":"http","msg":"starting server loop","address":"[::]:80","tls":false,"http3":false}
Jul 16 15:56:23 corneille caddy[609]: {"level":"info","ts":1689537383.8404412,"logger":"http.log","msg":"server running","name":"remaining_auto_https_redirects","protocols":["h1","h2","h3"]}
Jul 16 15:56:23 corneille caddy[609]: {"level":"info","ts":1689537383.840447,"logger":"http","msg":"enabling automatic TLS certificate management","domains":["jelly.roadrunner"]}
Jul 16 15:56:23 corneille caddy[609]: {"level":"warn","ts":1689537383.8406663,"logger":"tls","msg":"stapling OCSP","error":"no OCSP stapling for [jelly.roadrunner]: no OCSP server specified in certificate","identifiers":["jelly.roadrunner"]}
Jul 16 15:56:23 corneille caddy[609]: {"level":"debug","ts":1689537383.8406832,"logger":"tls.cache","msg":"added certificate to cache","subjects":["jelly.roadrunner"],"expiration":1689552929,"managed":true,"issuer_key":"local","hash":"f78de98e28bc6bca8ee3abd5e2d5aea414b9fef951a1f88d88d802d78ad526bf","cache_size":1,"cache_capacity":10000}
Jul 16 15:56:23 corneille caddy[609]: {"level":"debug","ts":1689537383.840701,"logger":"events","msg":"event","name":"cached_managed_cert","id":"f321d77b-44e6-4574-a2c5-158d03db9c57","origin":"tls","data":{"sans":["jelly.roadrunner"]}}
Jul 16 15:56:23 corneille caddy[609]: {"level":"debug","ts":1689537383.8407419,"logger":"http","msg":"servers shutting down with eternal grace period"}
Jul 16 15:56:23 corneille caddy[609]: {"level":"info","ts":1689537383.840891,"logger":"tls.cache.maintenance","msg":"stopped background certificate maintenance","cache":"0xc0001d8f50"}
Jul 16 15:56:23 corneille caddy[609]: {"level":"info","ts":1689537383.8410387,"msg":"autosaved config (load with --resume flag)","file":"/var/lib/caddy/.config/caddy/autosave.json"}
Jul 16 15:56:23 corneille caddy[609]: {"level":"info","ts":1689537383.8410914,"logger":"admin.api","msg":"load complete"}
Jul 16 15:56:23 corneille systemd[1]: Reloaded Caddy.
Jul 16 15:56:23 corneille caddy[609]: {"level":"info","ts":1689537383.8442667,"logger":"admin","msg":"stopped previous server","address":"localhost:2019"}
Jul 16 15:56:41 corneille caddy[609]: {"level":"debug","ts":1689537401.998924,"logger":"events","msg":"event","name":"tls_get_certificate","id":"84110155-aeb4-4939-be68-01706ba13994","origin":"tls","data":{"client_hello":{"CipherSuites":[49200,49196,49192,49188,49172,49162,159,107,57,52393,52392,52394,65413,196,136,129,157,61,53,192,132,49199,49195,49191,49187,49171,49161,158,103,51,190,69,156,60,47,186,65,49169,49159,5,4,49170,49160,22,10,255],"ServerName":"jelly.billkidd.fr","SupportedCurves":[29,23,24],"SupportedPoints":"AA==","SignatureSchemes":[1537,1539,61423,1281,1283,1025,1027,61166,60909,769,771,513,515],"SupportedProtos":["h2","http/1.1"],"SupportedVersions":[771,770,769],"Conn":{}}}}
Jul 16 15:56:41 corneille caddy[609]: {"level":"debug","ts":1689537401.9989855,"logger":"tls.handshake","msg":"no matching certificates and no custom selection logic","identifier":"
jelly.billkidd.fr"}
Jul 16 15:56:41 corneille caddy[609]: {"level":"debug","ts":1689537401.998994,"logger":"tls.handshake","msg":"no matching certificates and no custom selection logic","identifier":"*
.billkidd.fr"}
Jul 16 15:56:41 corneille caddy[609]: {"level":"debug","ts":1689537401.9989994,"logger":"tls.handshake","msg":"no matching certificates and no custom selection logic","identifier":"
*.*.fr"}
Jul 16 15:56:41 corneille caddy[609]: {"level":"debug","ts":1689537401.9990046,"logger":"tls.handshake","msg":"no matching certificates and no custom selection logic","identifier":"
*.*.*"}
Jul 16 15:56:41 corneille caddy[609]: {"level":"debug","ts":1689537401.9990108,"logger":"tls.handshake","msg":"all external certificate managers yielded no certificates and no error
s","remote_ip":"192.168.11.13","remote_port":"50226","sni":"jelly.billkidd.fr"}
Jul 16 15:56:41 corneille caddy[609]: {"level":"debug","ts":1689537401.9990172,"logger":"tls.handshake","msg":"no certificate matching TLS ClientHello","remote_ip":"192.168.11.13","
remote_port":"50226","server_name":"jelly.billkidd.fr","remote":"192.168.11.13:50226","identifier":"jelly.billkidd.fr","cipher_suites":[49200,49196,49192,49188,49172,49162,159,107
,57,52393,52392,52394,65413,196,136,129,157,61,53,192,132,49199,49195,49191,49187,49171,49161,158,103,51,190,69,156,60,47,186,65,49169,49159,5,4,49170,49160,22,10,255],"cert_cache_f
ill":0.0001,"load_if_necessary":true,"obtain_if_necessary":true,"on_demand":false}
Jul 16 15:56:41 corneille caddy[609]: {"level":"debug","ts":1689537401.999072,"logger":"http.stdlib","msg":"http: TLS handshake error from 192.168.11.13:50226: no certificate availa
ble for 'jelly.billkidd.fr'"}
(END)

3. Caddy version:

v2.6.4

4. How I installed and ran Caddy:

a. System environment:

Distributor ID: Ubuntu
Description: Ubuntu 22.04.2 LTS
Release: 22.04
Codename: jammy

b. Command:

None

c. Service/unit/compose file:

d. My complete Caddy config:

Caddy frontend file

# Global Options Block
{
        # General Options
        debug
        acme_dns cloudflare mycloudflaretoken
}

#ACME Server
frontcaddy.home {
        acme_server
        tls internal
}

jelly.billkidd.fr {
        #reverse_proxy 192.168.11.41:8096
        reverse_proxy https://jelly.roadrunner {
                header_up Host {upstream_hostport}
        }
        log {
                output file /var/log/caddy/jellyfin-access.log
                format transform "{common_log}"
        }
}

Caddy backend

{
debug

acme_ca https://frontcaddy.home/acme/local/directory
acme_ca_root /etc/ssl/certs/root.crt
}

jelly.roadrunner {
        tls internal
        reverse_proxy localhost:8096
        log {
                output file /var/log/caddy/jellyfin-access.log
        }
}

5. Links to relevant resources:

2

Your backend shouldn’t be receiving requests for jelly.billkidd.fr if I’m understanding what you’re trying to set up. Are you sure your DNS is correct?

3

As far as I understood the concept, when I go to https://jelly.billkidd.fr, the frontend would send the request to the caddy backend which is supposed to translate with https://jelly.roadrunner:8096.
This is what I’m trying to do by using caddy with split-dns.
So I have jelly.billkidd.fr and jelly.roadrunner with the IP 192.168.11.41.
Isn’t it this way?

4

Yes, but only if DNS resolves jelly.billkidd.fr to your frontend’s IP address. It seems like it’s resolving to your backend’s IP address.

5

You’re right!
I was answering on this way.

I changed that in my DNS entry.
Now the webpage is blank.

curl https://jelly.billkidd.fr/ -v
*   Trying 192.168.11.46...
* TCP_NODELAY set
* Connected to jelly.billkidd.fr (192.168.11.46) port 443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* successfully set certificate verify locations:
*   CAfile: /etc/ssl/cert.pem
  CApath: none
* TLSv1.2 (OUT), TLS handshake, Client hello (1):
* TLSv1.2 (IN), TLS handshake, Server hello (2):
* TLSv1.2 (IN), TLS handshake, Certificate (11):
* TLSv1.2 (IN), TLS handshake, Server key exchange (12):
* TLSv1.2 (IN), TLS handshake, Server finished (14):
* TLSv1.2 (OUT), TLS handshake, Client key exchange (16):
* TLSv1.2 (OUT), TLS change cipher, Change cipher spec (1):
* TLSv1.2 (OUT), TLS handshake, Finished (20):
* TLSv1.2 (IN), TLS change cipher, Change cipher spec (1):
* TLSv1.2 (IN), TLS handshake, Finished (20):
* SSL connection using TLSv1.2 / ECDHE-ECDSA-CHACHA20-POLY1305
* ALPN, server accepted to use h2
* Server certificate:
*  subject: CN=jelly.billkidd.fr
*  start date: Jul 16 18:21:23 2023 GMT
*  expire date: Oct 14 18:21:22 2023 GMT
*  subjectAltName: host "jelly.billkidd.fr" matched cert's "jelly.billkidd.fr"
*  issuer: C=US; O=Let's Encrypt; CN=R3
*  SSL certificate verify ok.
* Using HTTP2, server supports multi-use
* Connection state changed (HTTP/2 confirmed)
* Copying HTTP/2 data in stream buffer to connection buffer after upgrade: len=0
* Using Stream ID: 1 (easy handle 0x7f885800a800)
> GET / HTTP/2
> Host: jelly.billkidd.fr
> User-Agent: curl/7.64.1
> Accept: */*
>
* Connection state changed (MAX_CONCURRENT_STREAMS == 250)!
< HTTP/2 502
< alt-svc: h3=":443"; ma=2592000
< server: Caddy
< content-length: 0
< date: Sun, 16 Jul 2023 21:56:02 GMT
<
* Connection #0 to host jelly.billkidd.fr left intact
* Closing connection 0

logs from the frontend:

Jul 16 17:56:01 frontcaddy caddy[5192]: {"level":"debug","ts":1689544561.9318452,"logger":"events","msg":"event","name":"tls_get_certificate","id":"5acc5160-004e-497b-8c9e-e5844f71a704","origin":"tls","data":{"client_hello":{"CipherSuites":[49200,49196,49192,49188,49172,49162,159,107,57,52393,52392,52394,65413,196,136,129,157,61,53,192,132,49199,49195,49191,49187,49171,49161,158,103,51,190,69,156,60,47,186,65,49169,49159,5,4,49170,49160,22,10,255],"ServerName":"jelly.billkidd.fr","SupportedCurves":[29,23,24],"SupportedPoints":"AA==","SignatureSchemes":[1537,1539,61423,1281,1283,1025,1027,61166,60909,769,771,513,515],"SupportedProtos":["h2","http/1.1"],"SupportedVersions":[771,770,769],"Conn":{}}}}
Jul 16 17:56:01 frontcaddy caddy[5192]: {"level":"debug","ts":1689544561.931911,"logger":"tls.handshake","msg":"choosing certificate","identifier":"jelly.billkidd.fr","num_choices":1}
Jul 16 17:56:01 frontcaddy caddy[5192]: {"level":"debug","ts":1689544561.9319258,"logger":"tls.handshake","msg":"default certificate selection results","identifier":"jelly.billkidd.fr","subjects":["jelly.billkidd.fr"],"managed":true,"issuer_key":"acme-v02.api.letsencrypt.org-directory","hash":"c4b68e961cdf9299ce697ba2f9702cee4d4870bea066d4236be2330b398d5f65"}
Jul 16 17:56:01 frontcaddy caddy[5192]: {"level":"debug","ts":1689544561.9319348,"logger":"tls.handshake","msg":"matched certificate in cache","remote_ip":"192.168.11.13","remote_port":"51996","subjects":["jelly.billkidd.fr"],"managed":true,"expiration":1697307683,"hash":"c4b68e961cdf9299ce697ba2f9702cee4d4870bea066d4236be2330b398d5f65"}
Jul 16 17:56:02 frontcaddy caddy[5192]: {"level":"debug","ts":1689544562.1006372,"logger":"http.handlers.reverse_proxy","msg":"selected upstream","dial":"jelly.roadrunner:443","total_upstreams":1}
Jul 16 17:56:02 frontcaddy caddy[5192]: {"level":"debug","ts":1689544562.1037245,"logger":"http.handlers.reverse_proxy","msg":"upstream roundtrip","upstream":"jelly.roadrunner:443","duration":0.003025586,"request":{"remote_ip":"192.168.11.13","remote_port":"51996","proto":"HTTP/2.0","method":"GET","host":"jelly.roadrunner:443","uri":"/","headers":{"User-Agent":["curl/7.64.1"],"Accept":["*/*"],"X-Forwarded-For":["192.168.11.13"],"X-Forwarded-Proto":["https"],"X-Forwarded-Host":["jelly.billkidd.fr"]},"tls":{"resumed":false,"version":771,"cipher_suite":52393,"proto":"h2","server_name":"jelly.billkidd.fr"}},"error":"tls: failed to verify certificate: x509: certificate signed by unknown authority (possibly because of \"x509: ECDSA verification failure\" while trying to verify candidate authority certificate \"Caddy Local Authority - 2023 ECC Root\")"}
Jul 16 17:56:02 frontcaddy caddy[5192]: {"level":"error","ts":1689544562.104077,"logger":"http.log.error.log1","msg":"tls: failed to verify certificate: x509: certificate signed by unknown authority (possibly because of \"x509: ECDSA verification failure\" while trying to verify candidate authority certificate \"Caddy Local Authority - 2023 ECC Root\")","request":{"remote_ip":"192.168.11.13","remote_port":"51996","proto":"HTTP/2.0","method":"GET","host":"jelly.billkidd.fr","uri":"/","headers":{"User-Agent":["curl/7.64.1"],"Accept":["*/*"]},"tls":{"resumed":false,"version":771,"cipher_suite":52393,"proto":"h2","server_name":"jelly.billkidd.fr"}},"duration":0.003465213,"status":502,"err_id":"6zf7e97rz","err_trace":"reverseproxy.statusError (reverseproxy.go:1299)"}

logs from the backend

Jul 16 17:56:02 corneille caddy[609]: {"level":"debug","ts":1689544562.1072347,"logger":"events","msg":"event","name":"tls_get_certificate","id":"792ce1f7-24df-4a79-a57b-da1295c284ab","origin":"tls","data":{"client_hello":{"CipherSuites":[52393,52392,49195,49199,49196,49200,49161,49171,49162,49172,156,157,47,53,49170,10,4867,4865,4866],"ServerName":"jelly.roadrunner","SupportedCurves":[29,23,24,25],"SupportedPoints":"AA==","SignatureSchemes":[2052,1027,2055,2053,2054,1025,1281,1537,1283,1539,513,515],"SupportedProtos":["h2","http/1.1"],"SupportedVersions":[772,771],"Conn":{}}}}
Jul 16 17:56:02 corneille caddy[609]: {"level":"debug","ts":1689544562.107284,"logger":"tls.handshake","msg":"choosing certificate","identifier":"jelly.roadrunner","num_choices":1}
Jul 16 17:56:02 corneille caddy[609]: {"level":"debug","ts":1689544562.1072981,"logger":"tls.handshake","msg":"default certificate selection results","identifier":"jelly.roadrunner","subjects":["jelly.roadrunner"],"managed":true,"issuer_key":"local","hash":"1159418c768bee4d551c0f55f843c59aa3fa8cd0959cc304faef4c3ef24004d6"}
Jul 16 17:56:02 corneille caddy[609]: {"level":"debug","ts":1689544562.1073067,"logger":"tls.handshake","msg":"matched certificate in cache","remote_ip":"192.168.11.46","remote_port":"49448","subjects":["jelly.roadrunner"],"managed":true,"expiration":1689581784,"hash":"1159418c768bee4d551c0f55f843c59aa3fa8cd0959cc304faef4c3ef24004d6"}
Jul 16 17:56:02 corneille caddy[609]: {"level":"debug","ts":1689544562.1088142,"logger":"http.stdlib","msg":"http: TLS handshake error from 192.168.11.46:49448: remote error: tls: bad certificate"}

I’ve the root certificate (/etc/ssl/certs/root.crt): it’s the same that the one of the frontend (/var/lib/caddy/.local/share/caddy/pki/authorities/local/root.crt).

This is what I get from the backend:

curl -v https://frontcaddy.home/acme/local/directory
*   Trying 192.168.11.46:443...
* Connected to frontcaddy.home (192.168.11.46) port 443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
*  CAfile: /etc/ssl/certs/ca-certificates.crt
*  CApath: /etc/ssl/certs
* TLSv1.0 (OUT), TLS header, Certificate Status (22):
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* TLSv1.2 (IN), TLS header, Certificate Status (22):
* TLSv1.3 (IN), TLS handshake, Server hello (2):
* TLSv1.2 (IN), TLS header, Finished (20):
* TLSv1.2 (IN), TLS header, Supplemental data (23):
* TLSv1.3 (IN), TLS handshake, Encrypted Extensions (8):
* TLSv1.2 (IN), TLS header, Supplemental data (23):
* TLSv1.3 (IN), TLS handshake, Certificate (11):
* TLSv1.2 (OUT), TLS header, Unknown (21):
* TLSv1.3 (OUT), TLS alert, certificate unknown (558):
* SSL certificate problem: authority and subject key identifier mismatch
* Closing connection 0
curl: (60) SSL certificate problem: authority and subject key identifier mismatch
More details here: https://curl.se/docs/sslcerts.html

curl failed to verify the legitimacy of the server and therefore could not
establish a secure connection to it. To learn more about this situation and
how to fix it, please visit the web page mentioned above.
6

You might need to run sudo caddy trust on the frontend so that it properly installs the root cert in the frontend’s trust store. The proxy uses the system’s trust store, so if it wasn’t properly installed then it won’t be able to trust certs signed by the internal CA, even if it’s from the same Caddy instance.

7

It looks like it was already installed.

sudo caddy trust
2023/07/16 22:33:37.143	INFO	root certificate is already trusted by system	{"path": "localhost:2019/pki/ca/local"}
8

Maybe try deleting the certs from your backend (i.e. in /var/lib/caddy) then restart the backend to force it to re-issue a cert using your frontend ACME. It might’ve been issued incorrectly for some reason.

9

I’ve deleted everything in /var/lib/caddy/.local/share/caddythen rebooted the backend:
It still doesn’t work but the error message seemed to have changed.
I even have a blank page with https://jelly.roadrunner.
So the problem could come from the backend.
Jellyfin works with IP address

This is what happens when I reload caddy service backend

Jul 16 19:16:59 corneille systemd[1]: Reloading Caddy...
Jul 16 19:16:59 corneille caddy[2505]: {"level":"info","ts":1689549419.8773804,"msg":"using provided configuration","config_file":"/etc/caddy/Caddyfile","config_adapter":""}
Jul 16 19:16:59 corneille caddy[610]: {"level":"info","ts":1689549419.8793018,"logger":"admin.api","msg":"received request","method":"POST","host":"localhost:2019","uri":"/load","re
mote_ip":"127.0.0.1","remote_port":"58670","headers":{"Accept-Encoding":["gzip"],"Cache-Control":["must-revalidate"],"Content-Length":["700"],"Content-Type":["application/json"],"Or
igin":["http://localhost:2019"],"User-Agent":["Go-http-client/1.1"]}}
Jul 16 19:16:59 corneille caddy[610]: {"level":"info","ts":1689549419.8799837,"logger":"admin","msg":"admin endpoint started","address":"localhost:2019","enforce_origin":false,"orig
ins":["//localhost:2019","//[::1]:2019","//127.0.0.1:2019"]}
Jul 16 19:16:59 corneille caddy[610]: {"level":"info","ts":1689549419.8801954,"logger":"tls.cache.maintenance","msg":"started background certificate maintenance","cache":"0xc000128e
00"}
Jul 16 19:16:59 corneille caddy[610]: {"level":"info","ts":1689549419.8804164,"logger":"http","msg":"server is listening only on the HTTPS port but has no TLS connection policies; a
dding one to enable TLS","server_name":"srv0","https_port":443}
Jul 16 19:16:59 corneille caddy[610]: {"level":"info","ts":1689549419.880439,"logger":"http","msg":"enabling automatic HTTP->HTTPS redirects","server_name":"srv0"}
Jul 16 19:16:59 corneille caddy[610]: {"level":"warn","ts":1689549419.8808355,"logger":"pki.ca.local","msg":"installing root certificate (you might be prompted for password)","path"
:"storage:pki/authorities/local/root.crt"}
Jul 16 19:16:59 corneille caddy[610]: {"level":"info","ts":1689549419.8809829,"msg":"not NSS security databases found"}
Jul 16 19:16:59 corneille caddy[610]: {"level":"info","ts":1689549419.8809958,"msg":"define JAVA_HOME environment variable to use the Java trust"}
Jul 16 19:16:59 corneille caddy[610]: {"level":"error","ts":1689549419.8826358,"logger":"pki.ca.local","msg":"failed to install root certificate","error":"failed to execute sudo: ex
it status 1","certificate_file":"storage:pki/authorities/local/root.crt"}
Jul 16 19:16:59 corneille caddy[610]: {"level":"info","ts":1689549419.882724,"logger":"http","msg":"enabling HTTP/3 listener","addr":":443"}
Jul 16 19:16:59 corneille caddy[610]: {"level":"debug","ts":1689549419.8827744,"logger":"http","msg":"starting server loop","address":"[::]:443","tls":true,"http3":true}
Jul 16 19:16:59 corneille caddy[610]: {"level":"info","ts":1689549419.8827887,"logger":"http.log","msg":"server running","name":"srv0","protocols":["h1","h2","h3"]}
Jul 16 19:16:59 corneille caddy[610]: {"level":"debug","ts":1689549419.8828313,"logger":"http","msg":"starting server loop","address":"[::]:80","tls":false,"http3":false}
Jul 16 19:16:59 corneille caddy[610]: {"level":"info","ts":1689549419.8828435,"logger":"http.log","msg":"server running","name":"remaining_auto_https_redirects","protocols":["h1","h
2","h3"]}
Jul 16 19:16:59 corneille caddy[610]: {"level":"info","ts":1689549419.882852,"logger":"http","msg":"enabling automatic TLS certificate management","domains":["jelly.roadrunner"]}
Jul 16 19:16:59 corneille caddy[610]: {"level":"warn","ts":1689549419.883208,"logger":"tls","msg":"stapling OCSP","error":"no OCSP stapling for [jelly.roadrunner]: no OCSP server sp
ecified in certificate","identifiers":["jelly.roadrunner"]}
Jul 16 19:16:59 corneille caddy[610]: {"level":"debug","ts":1689549419.8832262,"logger":"tls.cache","msg":"added certificate to cache","subjects":["jelly.roadrunner"],"expiration":1
689591000,"managed":true,"issuer_key":"local","hash":"4184640d66a8dc2046dc7dbb809942649afe1f23a2f8a51bbf7b7df63d18a8a9","cache_size":1,"cache_capacity":10000}
Jul 16 19:16:59 corneille caddy[610]: {"level":"debug","ts":1689549419.8832502,"logger":"events","msg":"event","name":"cached_managed_cert","id":"493b8778-9988-493e-8ea5-e6dfb6ca797
8","origin":"tls","data":{"sans":["jelly.roadrunner"]}}
Jul 16 19:16:59 corneille caddy[610]: {"level":"debug","ts":1689549419.8832996,"logger":"http","msg":"servers shutting down with eternal grace period"}
Jul 16 19:16:59 corneille caddy[610]: {"level":"info","ts":1689549419.8833957,"logger":"tls.cache.maintenance","msg":"stopped background certificate maintenance","cache":"0xc0005197
a0"}
Jul 16 19:16:59 corneille caddy[610]: {"level":"info","ts":1689549419.8835185,"msg":"autosaved config (load with --resume flag)","file":"/var/lib/caddy/.config/caddy/autosave.json"}
Jul 16 19:16:59 corneille caddy[610]: {"level":"info","ts":1689549419.8835795,"logger":"admin.api","msg":"load complete"}
Jul 16 19:16:59 corneille systemd[1]: Reloaded Caddy.
Jul 16 19:16:59 corneille caddy[610]: {"level":"info","ts":1689549419.8887556,"logger":"admin","msg":"stopped previous server","address":"localhost:2019"}

These are the logs connecting with https://jelly.roadrunner:

Jul 16 19:28:48 corneille caddy[610]: {"level":"debug","ts":1689550128.0704672,"logger":"http.handlers.reverse_proxy","msg":"selected upstream","dial":"192.168.11.41:8096","total_upstreams":1}
Jul 16 19:28:48 corneille caddy[610]: {"level":"debug","ts":1689550128.071455,"logger":"http.handlers.reverse_proxy","msg":"upstream roundtrip","upstream":"192.168.11.41:8096","duration":0.000928397,"request":{"remote_ip":"192.168.11.13","remote_port":"53560","proto":"HTTP/2.0","method":"GET","host":"jelly.roadrunner","uri":"/","headers":{"User-Agent":["Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:109.0) Gecko/20100101 Firefox/115.0"],"Accept":["text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,*/*;q=0.8"],"Dnt":["1"],"Accept-Encoding":["gzip, deflate, br"],"Upgrade-Insecure-Requests":["1"],"Sec-Fetch-User":["?1"],"X-Forwarded-Host":["jelly.roadrunner"],"Accept-Language":["fr,fr-FR;q=0.8,en-US;q=0.5,en;q=0.3"],"Te":["trailers"],"X-Forwarded-Proto":["https"],"Sec-Fetch-Dest":["document"],"Sec-Fetch-Site":["none"],"Sec-Fetch-Mode":["navigate"],"X-Forwarded-For":["192.168.11.13"]},"tls":{"resumed":true,"version":772,"cipher_suite":4867,"proto":"h2","server_name":"jelly.roadrunner"}},"error":"tls: first record does not look like a TLS handshake"}
Jul 16 19:28:48 corneille caddy[610]: {"level":"error","ts":1689550128.0715115,"logger":"http.log.error.log0","msg":"tls: first record does not look like a TLS handshake","request":{"remote_ip":"192.168.11.13","remote_port":"53560","proto":"HTTP/2.0","method":"GET","host":"jelly.roadrunner","uri":"/","headers":{"Te":["trailers"],"Accept-Encoding":["gzip, deflate, br"],"Upgrade-Insecure-Requests":["1"],"Sec-Fetch-Mode":["navigate"],"Sec-Fetch-Site":["none"],"Sec-Fetch-User":["?1"],"User-Agent":["Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:109.0) Gecko/20100101 Firefox/115.0"],"Accept":["text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,*/*;q=0.8"],"Accept-Language":["fr,fr-FR;q=0.8,en-US;q=0.5,en;q=0.3"],"Dnt":["1"],"Sec-Fetch-Dest":["document"]},"tls":{"resumed":true,"version":772,"cipher_suite":4867,"proto":"h2","server_name":"jelly.roadrunner"}},"duration":0.001062593,"status":502,"err_id":"992e7rg4c","err_trace":"reverseproxy.statusError (reverseproxy.go:1299)"}

These are the logs connecting with https://jelly.billkidd.fr in the backend:

Jul 16 19:28:36 corneille caddy[610]: {"level":"debug","ts":1689550116.5622404,"logger":"events","msg":"event","name":"tls_get_certificate","id":"7a892419-734d-44e8-a56e-8b1d64118269","origin":"tls","data":{"client_hello":{"CipherSuites":[52393,52392,49195,49199,49196,49200,49161,49171,49162,49172,156,157,47,53,49170,10,4867,4865,4866],"ServerName":"jelly.roadrunner","SupportedCurves":[29,23,24,25],"SupportedPoints":"AA==","SignatureSchemes":[2052,1027,2055,2053,2054,1025,1281,1537,1283,1539,513,515],"SupportedProtos":["h2","http/1.1"],"SupportedVersions":[772,771],"Conn":{}}}}
Jul 16 19:28:36 corneille caddy[610]: {"level":"debug","ts":1689550116.5622988,"logger":"tls.handshake","msg":"choosing certificate","identifier":"jelly.roadrunner","num_choices":1}
Jul 16 19:28:36 corneille caddy[610]: {"level":"debug","ts":1689550116.562314,"logger":"tls.handshake","msg":"default certificate selection results","identifier":"jelly.roadrunner","subjects":["jelly.roadrunner"],"managed":true,"issuer_key":"local","hash":"4184640d66a8dc2046dc7dbb809942649afe1f23a2f8a51bbf7b7df63d18a8a9"}
Jul 16 19:28:36 corneille caddy[610]: {"level":"debug","ts":1689550116.5623226,"logger":"tls.handshake","msg":"matched certificate in cache","remote_ip":"192.168.11.46","remote_port":"59494","subjects":["jelly.roadrunner"],"managed":true,"expiration":1689591000,"hash":"4184640d66a8dc2046dc7dbb809942649afe1f23a2f8a51bbf7b7df63d18a8a9"}
Jul 16 19:28:36 corneille caddy[610]: {"level":"debug","ts":1689550116.563738,"logger":"http.stdlib","msg":"http: TLS handshake error from 192.168.11.46:59494: remote error: tls: bad certificate"}

These are the logs connecting with https://jelly.billkidd.fr in the frontend:

Jul 16 19:28:36 frontcaddy caddy[5192]: {"level":"debug","ts":1689550116.5544174,"logger":"http.handlers.reverse_proxy","msg":"upstream roundtrip","upstream":"jelly.roadrunner:443","duration":0.00297241,"request":{"remote_ip":"192.168.11.13","remote_port":"53558","proto":"HTTP/2.0","method":"GET","host":"jelly.roadrunner:443","uri":"/","headers":{"Accept-Language":["fr,fr-FR;q=0.8,en-US;q=0.5,en;q=0.3"],"X-Forwarded-Host":["jelly.billkidd.fr"],"X-Forwarded-Proto":["https"],"X-Forwarded-For":["192.168.11.13"],"Accept":["text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,*/*;q=0.8"],"Dnt":["1"],"User-Agent":["Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:109.0) Gecko/20100101 Firefox/115.0"],"Upgrade-Insecure-Requests":["1"],"Accept-Encoding":["gzip, deflate, br"],"Sec-Fetch-Dest":["document"],"Sec-Fetch-Site":["none"],"Te":["trailers"],"Sec-Fetch-Mode":["navigate"],"Sec-Fetch-User":["?1"]},"tls":{"resumed":true,"version":772,"cipher_suite":4867,"proto":"h2","server_name":"jelly.billkidd.fr"}},"error":"tls: failed to verify certificate: x509: certificate signed by unknown authority (possibly because of \"x509: ECDSA verification failure\" while trying to verify candidate authority certificate \"Caddy Local Authority - 2023 ECC Root\")"}
Jul 16 19:28:36 frontcaddy caddy[5192]: {"level":"error","ts":1689550116.554801,"logger":"http.log.error.log1","msg":"tls: failed to verify certificate: x509: certificate signed by unknown authority (possibly because of \"x509: ECDSA verification failure\" while trying to verify candidate authority certificate \"Caddy Local Authority - 2023 ECC Root\")","request":{"remote_ip":"192.168.11.13","remote_port":"53558","proto":"HTTP/2.0","method":"GET","host":"jelly.billkidd.fr","uri":"/","headers":{"Sec-Fetch-Mode":["navigate"],"Sec-Fetch-Site":["none"],"Sec-Fetch-User":["?1"],"Te":["trailers"],"Accept-Encoding":["gzip, deflate, br"],"Dnt":["1"],"Sec-Fetch-Dest":["document"],"Upgrade-Insecure-Requests":["1"],"User-Agent":["Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:109.0) Gecko/20100101 Firefox/115.0"],"Accept":["text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,*/*;q=0.8"],"Accept-Language":["fr,fr-FR;q=0.8,en-US;q=0.5,en;q=0.3"]},"tls":{"resumed":true,"version":772,"cipher_suite":4867,"proto":"h2","server_name":"jelly.billkidd.fr"}},"duration":0.00396733,"status":502,"err_id":"2972ugtw1","err_trace":"reverseproxy.statusError (reverseproxy.go:1299)"}
10

Hmm. That’s really strange. You could try configuring tls_trusted_ca_certs in reverse_proxy (Caddyfile directive) — Caddy Documentation to the path to the root cert on your frontend. That should make it explicitly trust the root cert. I’m not sure why it’s not picking it up normally :grimacing:

11

So, I don’t know why/how but Jellyfin works with https://jelly.roadrunner.
I still have the same problem with https://jelly.billkidd.fr.

I put the option in the frontend caddyfile.

jelly.billkidd.fr {
        #reverse_proxy 192.168.11.41:8096
        reverse_proxy https://jelly.roadrunner {
                header_up Host {upstream_hostport}
                transport http {
                        tls_trusted_ca_certs /var/lib/caddy/.local/share/caddy/pki/authorities/local/root.crt
                }
        }
        log {
                output file /var/log/caddy/jellyfin-access.log
                format transform "{common_log}"
        }
}
12

I found the solution.
With all my research, I did a lot of things.
There were two problems in my configuration:

  • DNS: I didn’t redirect to the right IP address
  • Backend Caddy file: the line tls internal made the website blank.

In the end, my frontend caddy file looks like this:

jelly.billkidd.fr {
        reverse_proxy https://jelly.roadrunner {
                header_up Host {upstream_hostport}
        }
        log {
                output file /var/log/caddy/jellyfin-access.log
                format transform "{common_log}"
        }

And my backend caddy file looks like this:

jelly.roadrunner {
        reverse_proxy localhost:8096 {
        }
        log {
                output file /var/log/caddy/jellyfin-access.log
        }
}

Thank you very much for your help @francislavoie

1 Like
13

Ooooh, right that makes sense. It was using the backend’s own self-generated CA to issue the certs, and not using the configured acme_ca to issue the cert.

I missed that detail :man_facepalming: sorry about that.

1 Like
14

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.