Unable to get TLS certificate issued for a Tailscale node

1. Caddy version (caddy version):

v2.4.5 h1:P1mRs6V2cMcagSPn+NWpD+OEYUYLIf6ecOa48cFGeUg=

2. How I run Caddy:

a. System environment:

pi@raspberrypi:~ $ cat /etc/os-release
PRETTY_NAME="Raspbian GNU/Linux 10 (buster)"
NAME="Raspbian GNU/Linux"
VERSION_ID="10"
VERSION="10 (buster)"
VERSION_CODENAME=buster
ID=raspbian
ID_LIKE=debian
HOME_URL="http://www.raspbian.org/"
SUPPORT_URL="http://www.raspbian.org/RaspbianForums"
BUG_REPORT_URL="http://www.raspbian.org/RaspbianBugs"

b. Command:

sudo systemctl start caddy

c. Service/unit/compose file:

Paste full file contents here.
Make sure backticks stay on their own lines,
and the post looks nice in the preview pane.

d. My complete Caddyfile or JSON config:

:80 {
	reverse_proxy localhost:1080
}

mew.tailnet-b593.ts.net {
	reverse_proxy localhost:1080
}

3. The problem I’m having:

I am unable to get a TLS certificate issued for my Tailscale node. It’s a new feature that came out in 2.50 release:

4. Error messages and/or full log output:

By running journalctl -u caddy --no-pager | less +G, I see these logs:

Mar 16 08:56:44 raspberrypi caddy[6740]: {"level":"info","ts":1647421004.5081987,"logger":"tls.issuance.acme.acme_client","msg":"trying to solve challenge","identifier":"mew.tailnet-b59
3.ts.net","challenge_type":"http-01","ca":"https://acme-staging-v02.api.letsencrypt.org/directory"}

Mar 16 08:56:45 raspberrypi caddy[6740]: {"level":"error","ts":1647421005.2800276,"logger":"tls.issuance.acme.acme_client","msg":"challenge failed","identifier":"mew.tailnet-b593.ts.net
","challenge_type":"http-01","status_code":400,"problem_type":"urn:ietf:params:acme:error:dns","error":"DNS problem: NXDOMAIN looking up A for mew.tailnet-b593.ts.net - check that a DNS
 record exists for this domain; DNS problem: NXDOMAIN looking up AAAA for mew.tailnet-b593.ts.net - check that a DNS record exists for this domain"}

Mar 16 08:56:45 raspberrypi caddy[6740]: {"level":"error","ts":1647421005.2802334,"logger":"tls.issuance.acme.acme_client","msg":"validating authorization","identifier":"mew.tailnet-b59
3.ts.net","error":"authorization failed: HTTP 400 urn:ietf:params:acme:error:dns - DNS problem: NXDOMAIN looking up A for mew.tailnet-b593.ts.net - check that a DNS record exists for th
is domain; DNS problem: NXDOMAIN looking up AAAA for mew.tailnet-b593.ts.net - check that a DNS record exists for this domain","order":"https://acme-staging-v02.api.letsencrypt.org/acme
/order/47329398/2042287218","attempt":1,"max_attempts":3}

Mar 16 08:56:46 raspberrypi caddy[6740]: {"level":"info","ts":1647421006.816509,"logger":"tls.issuance.acme.acme_client","msg":"trying to solve challenge","identifier":"mew.tailnet-b593
.ts.net","challenge_type":"tls-alpn-01","ca":"https://acme-staging-v02.api.letsencrypt.org/directory"}

Mar 16 08:56:47 raspberrypi caddy[6740]: {"level":"error","ts":1647421007.6190846,"logger":"tls.issuance.acme.acme_client","msg":"challenge failed","identifier":"mew.tailnet-b593.ts.net
","challenge_type":"tls-alpn-01","status_code":400,"problem_type":"urn:ietf:params:acme:error:dns","error":"DNS problem: NXDOMAIN looking up A for mew.tailnet-b593.ts.net - check that a
 DNS record exists for this domain; DNS problem: NXDOMAIN looking up AAAA for mew.tailnet-b593.ts.net - check that a DNS record exists for this domain"}

Mar 16 08:56:47 raspberrypi caddy[6740]: {"level":"error","ts":1647421007.6193137,"logger":"tls.issuance.acme.acme_client","msg":"validating authorizati
on","identifier":"mew.tailnet-b593.ts.net","error":"authorization failed: HTTP 400 urn:ietf:params:acme:error:dns - DNS problem: NXDOMAIN looking up A for mew.tailnet-b593.ts.net - check that a DNS record exists for this domain; DNS problem: NXDOMAIN looking up AAAA for mew.tailnet-b593.ts.net - check that a DNS record exists for this domain","order":"https://acme-staging-v02.api.letsencrypt.org/acme/order/47329398/2042287408","attempt":2,"max_attempts":3}

Mar 16 08:56:49 raspberrypi caddy[6740]: {"level":"error","ts":1647421009.4231,"logger":"tls.obtain","msg":"could not get certificate from issuer","identifier":"mew.tailnet-b593.ts.net","issuer":"acme-v02.api.letsencrypt.org-directory","error":"[mew.tailnet-b593.ts.net] solving challenges: mew.tailnet-b593.ts.net: no solvers available for remaining challenges (configured=[http-01 tls-alpn-01] offered=[http-01 dns-01 tls-alpn-01] remaining=[dns-01]) (order=https://acme-staging-v02.api.letsencrypt.org/acme/order/47329398/2042287518) (ca=https://acme-staging-v02.api.letsencrypt.org/directory)"}

Mar 16 08:56:49 raspberrypi caddy[6740]: {"level":"warn","ts":1647421009.4292324,"logger":"tls.issuance.zerossl","msg":"missing email address for ZeroSSL; it is strongly recommended to set one for next time"}

Mar 16 08:56:50 raspberrypi caddy[6740]: {"level":"info","ts":1647421010.7975643,"logger":"tls.issuance.zerossl","msg":"generated EAB credentials","key_id":"XclwP9ZCJfZ7t1KlSw-fYw"}

Mar 16 08:57:16 raspberrypi caddy[6740]: {"level":"info","ts":1647421036.003566,"logger":"tls.issuance.acme.acme_client","msg":"trying to solve challenge","identifier":"mew.tailnet-b593.ts.net","challenge_type":"http-01","ca":"https://acme.zerossl.com/v2/DV90"}

Mar 16 09:02:29 raspberrypi caddy[6740]: {"level":"error","ts":1647421349.5678914,"logger":"tls.obtain","msg":"could not get certificate from issuer","identifier":"mew.tailnet-b593.ts.net","issuer":"acme.zerossl.com-v2-DV90","error":"[mew.tailnet-b593.ts.net] solving challenges: [mew.tailnet-b593.ts.net] authorization took too long (order=https://acme.zerossl.com/v2/DV90/order/6eV0ds9XPiy5hYoaNh17ig) (ca=https://acme.zerossl.com/v2/DV90)"}

Mar 16 09:02:29 raspberrypi caddy[6740]: {"level":"error","ts":1647421349.568119,"logger":"tls.obtain","msg":"will retry","error":"[mew.tailnet-b593.ts.net] Obtain: [mew.tailnet-b593.ts.net] solving challenges: [mew.tailnet-b593.ts.net] authorization took too long (order=https://acme.zerossl.com/v2/DV90/order/6eV0ds9XPiy5hYoaNh17ig) (ca=https://acme.zerossl.com/v2/DV90)","attempt":13,"retrying_in":21600,"elapsed":47453.046438203,"max_duration":2592000}

5. What I already tried:

I am not too sure where to start. My knowledge of both Tailscale and Caddy is limited. I am happy to check whatever you want me to check next, thanks!

6. Links to relevant resources:

Doh, I just realized I am on Caddy 2.4.5, not 2.5. :man_facepalming: I have been testing with two different Raspberry Pis, so there was a mixup. I’ll re-test with Caddy 2.5 now.

1 Like

I am stuck. The systemd service is stuck at 2.4.5 despite installing 2.5.0~beta.1.

pi@raspberrypi:~ $ sudo apt install caddy
Reading package lists... Done
Building dependency tree... 47%
Building dependency tree
Reading state information... Done
caddy is already the newest version (2.5.0~beta.1).
0 upgraded, 0 newly installed, 0 to remove and 103 not upgraded.
pi@raspberrypi:~ $
pi@raspberrypi:~ $ caddy version
v2.4.5 h1:P1mRs6V2cMcagSPn+NWpD+OEYUYLIf6ecOa48cFGeUg=
pi@raspberrypi:~ $ sudo systemctl start caddy
pi@raspberrypi:~ $ caddy version
v2.4.5 h1:P1mRs6V2cMcagSPn+NWpD+OEYUYLIf6ecOa48cFGeUg=

I have tried restarting but I don’t see a change.

pi@raspberrypi:~ $ sudo systemctl reload caddy
pi@raspberrypi:~ $ sudo systemctl stop caddy
pi@raspberrypi:~ $ systemctl status caddy
● caddy.service - Caddy
   Loaded: loaded (/lib/systemd/system/caddy.service; enabled; vendor preset: enabled)
   Active: failed (Result: timeout) since Wed 2022-03-16 12:58:35 GMT; 8s ago
     Docs: https://caddyserver.com/docs/
  Process: 16831 ExecStart=/usr/bin/caddy run --environ --config /etc/caddy/Caddyfile (code=killed, signal=KILL)
  Process: 16854 ExecReload=/usr/bin/caddy reload --config /etc/caddy/Caddyfile (code=exited, status=0/SUCCESS)
 Main PID: 16831 (code=killed, signal=KILL)

Mar 16 12:58:29 raspberrypi caddy[16831]: {"level":"info","ts":1647435509.1970358,"msg":"shutting down apps, then terminating","signal":"SI
Mar 16 12:58:29 raspberrypi caddy[16831]: {"level":"warn","ts":1647435509.1972976,"msg":"exiting; byeee!! 👋","signal":"SIGTERM"}
Mar 16 12:58:29 raspberrypi caddy[16831]: {"level":"info","ts":1647435509.200875,"logger":"tls.cache.maintenance","msg":"stopped background
Mar 16 12:58:29 raspberrypi caddy[16831]: {"level":"info","ts":1647435509.2013898,"logger":"tls.obtain","msg":"releasing lock","identifier"
Mar 16 12:58:29 raspberrypi systemd[1]: Stopping Caddy...
Mar 16 12:58:34 raspberrypi systemd[1]: caddy.service: State 'stop-sigterm' timed out. Killing.
Mar 16 12:58:34 raspberrypi systemd[1]: caddy.service: Killing process 16831 (caddy) with signal SIGKILL.
Mar 16 12:58:35 raspberrypi systemd[1]: caddy.service: Main process exited, code=killed, status=9/KILL
Mar 16 12:58:35 raspberrypi systemd[1]: caddy.service: Failed with result 'timeout'.
Mar 16 12:58:35 raspberrypi systemd[1]: Stopped Caddy.
pi@raspberrypi:~ $ sudo systemctl start caddy
pi@raspberrypi:~ $ systemctl status caddy
● caddy.service - Caddy
   Loaded: loaded (/lib/systemd/system/caddy.service; enabled; vendor preset: enabled)
   Active: active (running) since Wed 2022-03-16 12:58:49 GMT; 1s ago
     Docs: https://caddyserver.com/docs/
 Main PID: 16872 (caddy)
    Tasks: 10 (limit: 2062)
   CGroup: /system.slice/caddy.service
           └─16872 /usr/bin/caddy run --environ --config /etc/caddy/Caddyfile

Mar 16 12:58:49 raspberrypi caddy[16872]: {"level":"info","ts":1647435529.136453,"logger":"tls.cache.maintenance","msg":"started background
Mar 16 12:58:49 raspberrypi caddy[16872]: {"level":"info","ts":1647435529.13663,"logger":"http","msg":"server is listening only on the HTTP
Mar 16 12:58:49 raspberrypi caddy[16872]: {"level":"info","ts":1647435529.1384761,"logger":"tls","msg":"cleaning storage unit","description
Mar 16 12:58:49 raspberrypi caddy[16872]: {"level":"info","ts":1647435529.1387384,"logger":"http","msg":"enabling automatic TLS certificate
Mar 16 12:58:49 raspberrypi caddy[16872]: {"level":"info","ts":1647435529.1391566,"logger":"tls","msg":"finished cleaning storage units"}
Mar 16 12:58:49 raspberrypi caddy[16872]: {"level":"info","ts":1647435529.139853,"msg":"autosaved config (load with --resume flag)","file":
Mar 16 12:58:49 raspberrypi caddy[16872]: {"level":"info","ts":1647435529.140213,"msg":"serving initial configuration"}
Mar 16 12:58:49 raspberrypi caddy[16872]: {"level":"info","ts":1647435529.1411986,"logger":"tls.obtain","msg":"acquiring lock","identifier"
Mar 16 12:58:49 raspberrypi caddy[16872]: {"level":"error","ts":1647435529.1418004,"logger":"tls","msg":"job failed","error":"mew.tailnet-b
Mar 16 12:58:49 raspberrypi systemd[1]: Started Caddy.

pi@raspberrypi:~ $ caddy version
v2.4.5 h1:P1mRs6V2cMcagSPn+NWpD+OEYUYLIf6ecOa48cFGeUg=

Tips on how to navigate this appreciated!

I uninstalled Caddy completely:

Installed 2.5 following this guide:

Yet I see 2.4.5 when I run caddy version:

Preparing to unpack .../caddy_2.5.0~beta.1_armhf.deb ...
Unpacking caddy (2.5.0~beta.1) ...
Setting up caddy (2.5.0~beta.1) ...
pi@raspberrypi:~ $ caddy version
v2.4.5 h1:P1mRs6V2cMcagSPn+NWpD+OEYUYLIf6ecOa48cFGeUg=
pi@raspberrypi:~ $ systemctl status caddy
● caddy.service - Caddy
   Loaded: loaded (/lib/systemd/system/caddy.service; enabled; vendor preset: enabled)
   Active: active (running) since Wed 2022-03-16 13:40:17 GMT; 8s ago
     Docs: https://caddyserver.com/docs/
 Main PID: 18066 (caddy)
    Tasks: 9 (limit: 2062)
   CGroup: /system.slice/caddy.service
           └─18066 /usr/bin/caddy run --environ --config /etc/caddy/Caddyfile

Mar 16 13:40:17 raspberrypi caddy[18066]: {"level":"info","ts":1647438017.6912637,"logger":"tls.cache.maintenance","msg":"started backgr
Mar 16 13:40:17 raspberrypi caddy[18066]: {"level":"info","ts":1647438017.6914306,"logger":"http","msg":"server is listening only on the
Mar 16 13:40:17 raspberrypi caddy[18066]: {"level":"info","ts":1647438017.6933014,"logger":"http","msg":"enabling automatic TLS certific
Mar 16 13:40:17 raspberrypi caddy[18066]: {"level":"info","ts":1647438017.6963093,"logger":"tls.obtain","msg":"acquiring lock","identifi
Mar 16 13:40:17 raspberrypi caddy[18066]: {"level":"error","ts":1647438017.6966279,"logger":"tls","msg":"job failed","error":"mew.tailne
Mar 16 13:40:17 raspberrypi caddy[18066]: {"level":"info","ts":1647438017.6969547,"logger":"tls","msg":"cleaning storage unit","descript
Mar 16 13:40:17 raspberrypi caddy[18066]: {"level":"info","ts":1647438017.697212,"logger":"tls","msg":"finished cleaning storage units"}
Mar 16 13:40:17 raspberrypi caddy[18066]: {"level":"info","ts":1647438017.7627392,"msg":"autosaved config (load with --resume flag)","fi
Mar 16 13:40:17 raspberrypi caddy[18066]: {"level":"info","ts":1647438017.7630439,"msg":"serving initial configuration"}
Mar 16 13:40:17 raspberrypi systemd[1]: Started Caddy.
pi@raspberrypi:~ $ journalctl -u caddy --no-pager | less +G
pi@raspberrypi:~ $ caddy version
v2.4.5 h1:P1mRs6V2cMcagSPn+NWpD+OEYUYLIf6ecOa48cFGeUg=

It’s as if I have two versions of Caddy running on my Raspberry Pi. Not sure how to untangle this just yet.

Please run which caddy, you might have put your v2.4.5 binary in a location that “hides” the v2.5.0-beta.1 binary :thinking:

When you run journalctl -u caddy --no-pager | less +G to see your logs, scroll up to just before Caddy starts; Caddy prints out its environment, it should show you the version there that the service actually ran with.

Did you install Caddy with these instructions possibly?

If so, then you’ll need to redo some of these steps to make sure the divert uses the right one

1 Like

Thanks for the pointer, @francislavoie! Now I remember creating a custom Caddy binary a few months ago to use a plugin. I have managed to change to the default installation now and I see the correct version output:

pi@raspberrypi:~ $ update-alternatives --config caddy
There are 2 choices for the alternative caddy (providing /usr/bin/caddy).

  Selection    Path                    Priority   Status
------------------------------------------------------------
  0            /usr/bin/caddy.custom    50        auto mode
  1            /usr/bin/caddy.custom    50        manual mode
* 2            /usr/bin/caddy.default   10        manual mode

Press <enter> to keep the current choice[*], or type selection number: 2
pi@raspberrypi:~ $ caddy version
v2.5.0-beta.1 h1:lF5wWqqDJ6HjETbnBILvTAeKcThsz1+OeWB+d1tWxp4=

But the problem with Tailscale node not having a certificate remains. I see this when I visit the site:

pi@raspberrypi:~ $ journalctl -u caddy --no-pager | less +G

Mar 16 15:38:02 raspberrypi systemd[1]: Starting Caddy...
Mar 16 15:38:02 raspberrypi caddy[19140]: caddy.HomeDir=/var/lib/caddy
Mar 16 15:38:02 raspberrypi caddy[19140]: caddy.AppDataDir=/var/lib/caddy/.local/share/caddy
Mar 16 15:38:02 raspberrypi caddy[19140]: caddy.AppConfigDir=/var/lib/caddy/.config/caddy
Mar 16 15:38:02 raspberrypi caddy[19140]: caddy.ConfigAutosavePath=/var/lib/caddy/.config/caddy/autosave.json
Mar 16 15:38:02 raspberrypi caddy[19140]: caddy.Version=v2.5.0-beta.1 h1:lF5wWqqDJ6HjETbnBILvTAeKcThsz1+OeWB+d1tWxp4=
Mar 16 15:38:02 raspberrypi caddy[19140]: runtime.GOOS=linux
Mar 16 15:38:02 raspberrypi caddy[19140]: runtime.GOARCH=arm
Mar 16 15:38:02 raspberrypi caddy[19140]: runtime.Compiler=gc
Mar 16 15:38:02 raspberrypi caddy[19140]: runtime.NumCPU=4
Mar 16 15:38:02 raspberrypi caddy[19140]: runtime.GOMAXPROCS=4
Mar 16 15:38:02 raspberrypi caddy[19140]: runtime.Version=go1.17.7
Mar 16 15:38:02 raspberrypi caddy[19140]: os.Getwd=/
Mar 16 15:38:02 raspberrypi caddy[19140]: LANG=en_GB.UTF-8
Mar 16 15:38:02 raspberrypi caddy[19140]: PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
Mar 16 15:38:02 raspberrypi caddy[19140]: NOTIFY_SOCKET=/run/systemd/notify
Mar 16 15:38:02 raspberrypi caddy[19140]: HOME=/var/lib/caddy
Mar 16 15:38:02 raspberrypi caddy[19140]: LOGNAME=caddy
Mar 16 15:38:02 raspberrypi caddy[19140]: USER=caddy
Mar 16 15:38:02 raspberrypi caddy[19140]: INVOCATION_ID=beb0d7d5a43d43d9a203c657bbc08018
Mar 16 15:38:02 raspberrypi caddy[19140]: JOURNAL_STREAM=8:9758146
Mar 16 15:38:02 raspberrypi caddy[19140]: {"level":"info","ts":1647445082.91355,"msg":"using provided configuration","config_file":"/etc/caddy/Caddyfile","config_adapter":""}
Mar 16 15:38:02 raspberrypi caddy[19140]: {"level":"info","ts":1647445082.9234872,"logger":"admin","msg":"admin endpoint started","address":"tcp/localhost:2019","enforce_origin":false,"origins":["//localhost:2019","//[::1]:2019","//127.0.0.1:2019"]}
Mar 16 15:38:02 raspberrypi caddy[19140]: {"level":"info","ts":1647445082.9246805,"logger":"http","msg":"server is listening only on the HTTPS port but has no TLS connection policies; adding one to enable TLS","server_name":"srv0","https_port":443}
Mar 16 15:38:02 raspberrypi caddy[19140]: {"level":"info","ts":1647445082.925247,"logger":"http","msg":"enabling automatic HTTP->HTTPS redirects","server_name":"srv0"}
Mar 16 15:38:02 raspberrypi caddy[19140]: {"level":"warn","ts":1647445082.9257178,"logger":"http","msg":"server is listening only on the HTTP port, so no automatic HTTPS will be applied to this server","server_name":"srv1","http_port":80}
Mar 16 15:38:02 raspberrypi caddy[19140]: {"level":"info","ts":1647445082.9246705,"logger":"tls.cache.maintenance","msg":"started background certificate maintenance","cache":"0x2822140"}
Mar 16 15:38:02 raspberrypi caddy[19140]: {"level":"info","ts":1647445082.928237,"logger":"tls","msg":"cleaning storage unit","description":"FileStorage:/var/lib/caddy/.local/share/caddy"}
Mar 16 15:38:02 raspberrypi caddy[19140]: {"level":"info","ts":1647445082.92883,"logger":"tls","msg":"finished cleaning storage units"}
Mar 16 15:38:02 raspberrypi caddy[19140]: {"level":"info","ts":1647445082.9299302,"msg":"autosaved config (load with --resume flag)","file":"/var/lib/caddy/.config/caddy/autosave.json"}
Mar 16 15:38:02 raspberrypi caddy[19140]: {"level":"info","ts":1647445082.9305859,"msg":"serving initial configuration"}
Mar 16 15:38:02 raspberrypi systemd[1]: Started Caddy.
Mar 16 15:38:08 raspberrypi caddy[19140]: {"level":"error","ts":1647445088.7104971,"logger":"tls.handshake","msg":"getting certificate from external certificate manager","sni":"mew.tailnet-b593.ts.net","cert_manager":0,"error":"Access denied: cert access denied"}
Mar 16 15:38:09 raspberrypi caddy[19140]: {"level":"error","ts":1647445089.5363169,"logger":"tls.handshake","msg":"getting certificate from external certificate manager","sni":"mew.tailnet-b593.ts.net","cert_manager":0,"error":"Access denied: cert access denied"}
Mar 16 15:38:10 raspberrypi caddy[19140]: {"level":"error","ts":1647445090.095257,"logger":"tls.handshake","msg":"getting certificate from external certificate manager","sni":"mew.tailnet-b593.ts.net","cert_manager":0,"error":"Access denied: cert access denied"}
Mar 16 15:38:10 raspberrypi caddy[19140]: {"level":"error","ts":1647445090.7159898,"logger":"tls.handshake","msg":"getting certificate from external certificate manager","sni":"mew.tailnet-b593.ts.net","cert_manager":0,"error":"Access denied: cert access denied"}
Mar 16 15:40:35 raspberrypi caddy[19140]: {"level":"error","ts":1647445235.1340349,"logger":"tls.handshake","msg":"getting certificate from external certificate manager","sni":"mew.tailnet-b593.ts.net","cert_manager":0,"error":"Access denied: cert access denied"}

Do you have any pointers on what might be happening here? I have already configured Caddy user to have access to Tailscale’s socket:

pi@raspberrypi:~ $ sudo nano /etc/default/tailscaled
# Set the port to listen on for incoming VPN packets.
# Remote nodes will automatically be informed about the new port number,
# but you might want to configure this in order to set external firewall
# settings.
PORT="41641"

# Extra flags you might want to pass to tailscaled.
FLAGS=""
TS_PERMIT_CERT_UID=caddy

Thanks for taking a look.

Make sure the caddy process is actually being run by the caddy user. Caddy is successfully using the TS socket, but TS is rejecting access, likely due to user mismatch.

Hmm, did you reload the tailscale daemon to make that TS_PERMIT change take effect?

Also make sure that you’re running the very latest Tailscale version. I think it was just released this week. I had to download the dev tip to build and test this feature.

(Prior versions only allow authorizing by UID, not usernames.)

Also make sure that you’re running the very latest Tailscale version. I think it was just released this week. I had to download the dev tip to build and test this feature.

I am on Tailscale version 1.22.1, which is the latest per Tailscale dashboard:

I’d see a message to update otherwise.

I am not seeing a word about the required version on the Tailscale blog post or support document either, which suggests 1.22.1 is sufficient. :thinking:

Hmm, did you reload the tailscale daemon to make that TS_PERMIT change take effect?

As far as my understanding goes, tailscale down and tailscale up is sufficient for tailscaled service to restart. I have done that now but the issue persists.

Make sure the caddy process is actually being run by the caddy user. Caddy is successfully using the TS socket, but TS is rejecting access, likely due to user mismatch.

Could you confirm if I am checking this correctly? I ran ps aux and I see this:

caddy    19140  0.0  3.0 826152 28852 ?        Ssl  15:38   0:02 /usr/bin/caddy run --environ --config /etc/caddy/Caddyfile

It matches the process ID I see when I run journalctl -u caddy --no-pager | less +G:

Mar 16 17:06:14 raspberrypi caddy[19140]: {"level":"error","ts":1647450374.450569,"logger":"tls.handshake","msg":"getting certificate from external certificate manager","sni":"mew.tailnet-b593.ts.net","cert_manager":0,"error":"Access denied: cert access denied"}
Mar 16 17:06:15 raspberrypi caddy[19140]: {"level":"error","ts":1647450375.3657193,"logger":"tls.handshake","msg":"getting certificate from external certificate manager","sni":"mew.tailnet-b593.ts.net","cert_manager":0,"error":"Access denied: cert access denied"}
Mar 16 17:06:15 raspberrypi caddy[19140]: {"level":"error","ts":1647450375.9666233,"logger":"tls.handshake","msg":"getting certificate from external certificate manager","sni":"mew.tailnet-b593.ts.net","cert_manager":0,"error":"Access denied: cert access denied"}

The changes landed in 1.22.0 so yeah you should be fine:

That also looks fine. We can also see in your logs at the start, it shows USER=caddy as the env.

I’m not sure what the problem is. Maybe we need to get help from the Tailscale team :thinking:

I can send them a support ticket and see if one of them can participate in this conversation. Thanks for taking a look so far!

As far as my understanding goes, tailscale down and tailscale up is sufficient for tailscaled service to restart. I have done that now but the issue persists.

Unfortunately that isn’t enough. tailscaled checks environment variables like TS_PERMIT when it is starting up. Taking the interface down and up tells tailscaled to disconnect and reconnect from the tailnet but doesn’t actually restart the daemon.

This is some kind of Linux system? systemctl restart tailscaled would do it.

6 Likes

Bingo! That helped. :tada: Thanks so much, @DentonGentry.

4 Likes

This topic was automatically closed after 30 days. New replies are no longer allowed.