Tailscale, HTTPS, TLS no matching certificate

1. The problem I’m having:

I just updated to Fedora 40 - after the update, my tailscale domain “server.halosaur-tautara.ts.net” would report “no matching certificates” (caddy logs) or SSL_ERROR_INTERNAL_ERROR_ALERT (firefox).

Prior to the Fedora update, the url was working fine with https + caddy + tailscale.

Caddy and Tailscale are running as system services.
I have added the line “TS_PERMIT_CERT_UID=caddy” to my tailscaled in /etc/default/tailscaled

I have removed my email from the caddy file as suggested in this thread

I have also tried it with the email in the Caddyfile and the option

tls {
    get_certificate tailscale
}

I’m far from an expert on these things, so any help would be greatly appreciated!

Curl output:

* Host server.halosaur-tautara.ts.net:443 was resolved.
* IPv6: (none)
* IPv4: 100.68.218.106
*   Trying 100.68.218.106:443...
* Connected to server.halosaur-tautara.ts.net (100.68.218.106) port 443
* ALPN: curl offers h2,http/1.1
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
*  CAfile: /etc/pki/tls/certs/ca-bundle.crt
*  CApath: none
* TLSv1.3 (IN), TLS alert, internal error (592):
* OpenSSL/3.2.1: error:0A000438:SSL routines::tlsv1 alert internal error
* Closing connection
curl: (35) OpenSSL/3.2.1: error:0A000438:SSL routines::tlsv1 alert internal error

2. Error messages and/or full log output:

{"level":"debug","ts":1714705597.3186367,"logger":"events","msg":"event","name":"tls_get_certificate","id":"94a6ed39-37eb-4e40-9f94-8e147b7c2386","origin":"tls","data":{"client_hello":{"CipherSuites":[4865,4867,4866,49195,49199,52393,52392,49196,49200,49162,49161,49171,49172,156,157,47,53],"ServerName":"server.halosaur-tautara.ts.net","SupportedCurves":[29,23,24,25,256,257],"SupportedPoints":"AA==","SignatureSchemes":[1027,1283,1539,2052,2053,2054,1025,1281,1537,515,513],"SupportedProtos":["h2","http/1.1"],"SupportedVersions":[772,771],"RemoteAddr":{"IP":"100.104.129.98","Port":45756,"Zone":""},"LocalAddr":{"IP":"100.68.218.106","Port":443,"Zone":""}}}}
{"level":"debug","ts":1714705597.3189664,"logger":"tls.handshake","msg":"no matching certificates and no custom selection logic","identifier":"server.halosaur-tautara.ts.net"}
{"level":"debug","ts":1714705597.3189964,"logger":"tls.handshake","msg":"no matching certificates and no custom selection logic","identifier":"*.halosaur-tautara.ts.net"}
{"level":"debug","ts":1714705597.3190138,"logger":"tls.handshake","msg":"no matching certificates and no custom selection logic","identifier":"*.*.ts.net"}
{"level":"debug","ts":1714705597.319075,"logger":"tls.handshake","msg":"no matching certificates and no custom selection logic","identifier":"*.*.*.net"}
{"level":"debug","ts":1714705597.3190897,"logger":"tls.handshake","msg":"no matching certificates and no custom selection logic","identifier":"*.*.*.*"}
{"level":"debug","ts":1714705597.3191369,"logger":"tls.handshake","msg":"no certificate matching TLS ClientHello","remote_ip":"100.104.129.98","remote_port":"45756","server_name":"server.halosaur-tautara.ts.net","remote":"100.104.129.98:45756","identifier":"server.halosaur-tautara.ts.net","cipher_suites":[4865,4867,4866,49195,49199,52393,52392,49196,49200,49162,49161,49171,49172,156,157,47,53],"cert_cache_fill":0.0015,"load_or_obtain_if_necessary":true,"on_demand":false}
{"level":"debug","ts":1714705597.3197656,"logger":"http.stdlib","msg":"http: TLS handshake error from 100.104.129.98:45756: no certificate available for 'server.halosaur-tautara.ts.net'"}

3. Caddy version:

$ caddy version
v2.7.6

4. How I installed and ran Caddy:

Caddy is run as a system sevice, using “sudo dnf install caddy”

“dnf install caddy” output:
Package caddy-2.7.6-2.fc40.x86_64 is already installed.

a. System environment:

Fedora release 40 (Forty), systemd running Caddy and Tailscale

b. Command:

$ sudo systemctl enable caddy
$ sudo systemctl start caddy
$ sudo systemctl adapt --config ./etc/caddy/Caddyfile
$ sudo systemctl reload caddy

c. Service/unit/compose file:

N/A

d. My complete Caddy config:

{
	log {
		level debug
		output file /var/log/caddy/caddy.log
	}
}

joplin.lab {
	tls internal
	reverse_proxy localhost:22300
}

server.halosaur-tautara.ts.net {
	reverse_proxy localhost:22300
}

server.lab {
	tls internal
	reverse_proxy http://localhost:4080
}

https://pihole.lab {
	tls internal
	redir http://pihole.lab/admin 301
}

http://pihole.lab {
	tls internal
	route / {
		redir http://pihole.lab/admin 301
	}
	reverse_proxy localhost:15428
}

portainer.server.lab {
	tls internal
	reverse_proxy http://localhost:9000
}

admin.lab 192.168.1.4 {
	tls internal
	reverse_proxy localhost:9090
}

bookstack.lab {
	tls internal
	reverse_proxy localhost:6875
}

filestash.lab {
	log {
		output file /home/fedora/caddy/filestash.log
	}
	tls internal
	reverse_proxy localhost:8334
}

filestash-private.server.lab {
	tls internal
	reverse_proxy localhost:8339
}

http://storedowndb.server.lab {
	tls internal
	reverse_proxy http://localhost:5984
}

http://grocy.server.lab {
	tls internal
	reverse_proxy http://localhost:5080
}

http://barcodebuddy.server.lab {
	tls internal
	reverse_proxy http://localhost:1180
}

paperless.server.lab {
	tls internal
	reverse_proxy http://localhost:7040
}

stirling.server.lab {
	tls internal
	reverse_proxy localhost:11800
}

hass.lab {
	tls internal
	reverse_proxy localhost:8123
}

http://seth.server.lab {
	tls internal
	reverse_proxy http://localhost:1234
}

seth.server.lab {
	tls internal
	reverse_proxy https://localhost:1235
}

grafana.server.lab {
	tls internal
	reverse_proxy http://localhost:3000
}

http://door.server.lab {
	tls internal
	reverse_proxy http://127.0.0.1:11501
}

http://jelly.lab {
	tls internal
	reverse_proxy http://localhost:8096
}

wazuh.lab {
	tls internal
	reverse_proxy https://127.0.0.1:14000 {
		transport http {
			tls_insecure_skip_verify
		}
	}
}

5. Links to relevant resources:

Forum post
Github Issue 4779

Are you sure tailscaled is running? Did you grant it permission to the caddy user?

1 Like

@francislavoie That’s what this is supposed to do, but it requires resarting the daemon.

1 Like

Yes, tailscaled is running - and it has been restarted multiple times after adding the TS_PERMIT_CERT_UID=caddy to /etc/default/tailscaled.

I had this running with no issues for a minimum of 6 months, then upgraded from Fedora 38 to Fedora 40, and it stopped working.

Tailscale works normally, Caddy works normally (besides the Tailscale cert). I can use all of the other reverse proxies normally.

Okay, so after letting it rest for a few days I’m no longer getting the certificate errors. I’m now getting “Client sent an HTTP request to an HTTPS server” in the browser, and if I’m not mistaken that is indicating a configuration issue on the specific application side (Joplin server, in case you were curious.)

I don’t know what solved the certificate errors, I’m sorry.

Please enable the debug global option and show your logs.

Are you sure you’re proxying to the correct port of your app? Proxy to the HTTP port, not the HTTPS port.

1 Like

Hey! Thank you for the response francislavoie.

With the new issue (http request to https server) it was indeed a configuration issue. Throughout my troubleshooting I kept messing with bits and tweaking bobs.

In the end, I had to set the APP_BASE_URL for Joplin Server to http://server.halosaur-tautara.ts.net , and port to 22300, noting the http://, and the caddy configuration to localhost:22300.

docker-compose.yml for Joplin Server

app:
        image: joplin/server:latest
        depends_on:
            - db
        ports:
          - "22300:22300"
        restart: unless-stopped
        environment:
            - APP_PORT=22300
            - APP_BASE_URL=http://server.halosaur-tautara.ts.net

Caddyfile:

server.halosaur-tautara.ts.net {
	reverse_proxy localhost:22300
	tls {
		get_certificate tailscale
	}
}

This issue is resolved now, thank you everyone for your help!

For the future internet visitors:

enable debug logging for caddy, and tail -f the log while attempting to access the site to get the verbose errors.

At the same time, have another terminal windows running the docker container interactively (launch it with “docker-compose up” instead of “docker compose up -d”

This way you see both how Caddy is handling the reverse proxy and the app is handling the attempts to connect to it (if they’re even getting there).

1 Like