Error: requesting CA info: 2019: connection refused

edwardoo · February 11, 2023, 11:55pm

1. Caddy version:

v2.6.2 h1:wKoFIxpmOJLGl3QXoo6PNbYvGW4xLEgo32GPBEjWL8o=

2. How I installed, and run Caddy:

a. System environment:

> cat /etc/*release
NAME="Oracle Linux Server"
VERSION="9.1"
ID_LIKE="fedora"
PLATFORM_ID="platform:el9"
...
> uname -a
Linux oracle-linux 5.15.0-6.80.3.1.el9uek.aarch64 #2 SMP Tue Jan 10 11:35:24 PST 2023 aarch64 aarch64 aarch64 GNU/Linux

b. Command:

sudo dnf copr enable @caddy/caddy
sudo dnf install caddy
sudo mkdir -p /etc/caddy ; sudo cp Caddyfile /etc/caddy/
sudo systemctl enable caddy
sudo systemctl start caddy

c. Service/unit/compose file:

no modifications

d. My complete Caddy config:

cat /etc/caddy/Caddyfile
localhost {
  respond "Hello world!"
}

localhost:8080 {
  file_server browse
}

3. The problem I’m having / Error messages and/or full log output / What I already tried / Links to relevant resources:

I’m trying to setup caddy, first as a simple file server, on Oracle cloud free tier.

First it seems I hit

github.com/caddyserver/caddy

Caddy fails to install root certificate on linux when using caddy.service

opened 08:21PM - 19 Jul 21 UTC

closed 09:32PM - 19 Jul 21 UTC

tim-hm

discussion

documentation

Hi firstly longtime nginx user who has just discovered caddy ... loving it so th…ank you 🙏 # Problem Caddy is unable to install the root cert on Ubuntu 20.10 installation when following the [install process](https://caddyserver.com/docs/install). As you can see from the logs below, the issue is that the provided `caddy` user does not have permissions to write to the certificate stores: ``` Jul 19 20:59:32 ubuntu-vm sudo[10043]: pam_unix(sudo:auth): conversation failed Jul 19 20:59:32 ubuntu-vm sudo[10043]: pam_unix(sudo:auth): auth could not identify password for [caddy] Jul 19 20:59:32 ubuntu-vm sudo[10043]: caddy : user NOT in sudoers ; TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/usr/bin/tee /usr/local/share/ca-certificates/Caddy_Local_Authority_-_2021_ECC_Root_158313951851744774645843564988098209363.crt ``` # Reproduce - Install fresh ubuntu 20.10 in a vm - Install caddy and other prereqs - Check that the caddy service has started with `systemctl status caddy` and if not start it - Run `curl https://localhost` and you should get a certificate error ``` # Caddyfile localhost { respond "hi" } ``` # What I've tried If I run `sudo caddy run` then certificates work correctly for curl but not Firefox or chrome. However, this doesn't resolve the issue for the caddy user because the cert is in the root user's nssdb and I'd like to get it working from the browser for local dev. If I change the user inside `/lib/systemd/system/caddy.service` to root then the cert is installed correctly for curl but not for firefox or chrome (haven't tested JAVA yet). Running `caddy trust` as root and then starting the service does work either. # Workaround So for the moment I've changed the caddy.server user to root, which successfully installs the cert when it starts up and then i use certutil to manually install the certs for firefox and chrome and keytool to import it for java.

Full log of my `systemctl start caddy`

Feb 11 23:51:51 oracle-linux systemd[1]: Starting Caddy...
Feb 11 23:51:51 oracle-linux caddy[6141]: caddy.HomeDir=/var/lib/caddy
Feb 11 23:51:51 oracle-linux caddy[6141]: caddy.AppDataDir=/var/lib/caddy/.local/share/caddy
Feb 11 23:51:51 oracle-linux caddy[6141]: caddy.AppConfigDir=/var/lib/caddy/.config/caddy
Feb 11 23:51:51 oracle-linux caddy[6141]: caddy.ConfigAutosavePath=/var/lib/caddy/.config/caddy/autosave.json
Feb 11 23:51:51 oracle-linux caddy[6141]: caddy.Version=v2.6.2 h1:wKoFIxpmOJLGl3QXoo6PNbYvGW4xLEgo32GPBEjWL8o=
Feb 11 23:51:51 oracle-linux caddy[6141]: runtime.GOOS=linux
Feb 11 23:51:51 oracle-linux caddy[6141]: runtime.GOARCH=arm64
Feb 11 23:51:51 oracle-linux caddy[6141]: runtime.Compiler=gc
Feb 11 23:51:51 oracle-linux caddy[6141]: runtime.NumCPU=4
Feb 11 23:51:51 oracle-linux caddy[6141]: runtime.GOMAXPROCS=4
Feb 11 23:51:51 oracle-linux caddy[6141]: runtime.Version=go1.18.6
Feb 11 23:51:51 oracle-linux caddy[6141]: os.Getwd=/
Feb 11 23:51:51 oracle-linux caddy[6141]: LANG=en_US.UTF-8
Feb 11 23:51:51 oracle-linux caddy[6141]: PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin
Feb 11 23:51:51 oracle-linux caddy[6141]: NOTIFY_SOCKET=/run/systemd/notify
Feb 11 23:51:51 oracle-linux caddy[6141]: HOME=/var/lib/caddy
Feb 11 23:51:51 oracle-linux caddy[6141]: LOGNAME=caddy
Feb 11 23:51:51 oracle-linux caddy[6141]: USER=caddy
Feb 11 23:51:51 oracle-linux caddy[6141]: INVOCATION_ID=37143b1cb36647119efe1321feaab3e2
Feb 11 23:51:51 oracle-linux caddy[6141]: JOURNAL_STREAM=8:35768
Feb 11 23:51:51 oracle-linux caddy[6141]: SYSTEMD_EXEC_PID=6141
Feb 11 23:51:51 oracle-linux caddy[6141]: {"level":"info","ts":1676159511.7426624,"msg":"using provided configuration","config_file":"/etc/caddy/Caddyfile","config_adapter":""}
Feb 11 23:51:51 oracle-linux caddy[6141]: {"level":"warn","ts":1676159511.7433949,"msg":"Caddyfile input is not formatted; run the 'caddy fmt' command to fix inconsistencies","adapter":"caddyfile","file":"/etc/caddy/Caddyfile","line":2}
Feb 11 23:51:51 oracle-linux caddy[6141]: {"level":"info","ts":1676159511.7447956,"logger":"admin","msg":"admin endpoint started","address":"localhost:2019","enforce_origin":false,"origins":["//[::1]:2019","//127.0.0.1:2019","//localhost:2019"]}
Feb 11 23:51:51 oracle-linux caddy[6141]: {"level":"info","ts":1676159511.745006,"logger":"http","msg":"server is listening only on the HTTPS port but has no TLS connection policies; adding one to enable TLS","server_name":"srv0","https_port":443}
Feb 11 23:51:51 oracle-linux caddy[6141]: {"level":"info","ts":1676159511.745073,"logger":"http","msg":"enabling automatic HTTP->HTTPS redirects","server_name":"srv0"}
Feb 11 23:51:51 oracle-linux caddy[6141]: {"level":"info","ts":1676159511.7451272,"logger":"http","msg":"enabling automatic HTTP->HTTPS redirects","server_name":"srv1"}
Feb 11 23:51:51 oracle-linux caddy[6141]: {"level":"info","ts":1676159511.7451527,"logger":"tls.cache.maintenance","msg":"started background certificate maintenance","cache":"0x40001b10a0"}
Feb 11 23:51:51 oracle-linux caddy[6141]: {"level":"info","ts":1676159511.7457712,"logger":"tls","msg":"cleaning storage unit","description":"FileStorage:/var/lib/caddy/.local/share/caddy"}
Feb 11 23:51:51 oracle-linux caddy[6141]: {"level":"info","ts":1676159511.7457967,"logger":"tls","msg":"finished cleaning storage units"}
Feb 11 23:51:51 oracle-linux caddy[6141]: {"level":"warn","ts":1676159511.7587595,"logger":"pki.ca.local","msg":"installing root certificate (you might be prompted for password)","path":"storage:pki/authorities/local/root.crt"}
Feb 11 23:51:51 oracle-linux caddy[6141]: {"level":"info","ts":1676159511.7590356,"msg":"Warning: \"certutil\" is not available, install \"certutil\" with \"apt install libnss3-tools\" or \"yum install nss-tools\" and try again"}
Feb 11 23:51:51 oracle-linux caddy[6141]: {"level":"info","ts":1676159511.7590518,"msg":"define JAVA_HOME environment variable to use the Java trust"}
Feb 11 23:51:51 oracle-linux sudo[6151]:    caddy : unable to open /run/sudo/ts/caddy : Permission denied ; PWD=/ ; USER=root ; COMMAND=/bin/tee /etc/pki/ca-trust/source/anchors/Caddy_Local_Authority_-_2023_ECC_Root_291589996991120136616000285617640621002.pem
Feb 11 23:51:51 oracle-linux sudo[6151]: pam_unix(sudo:auth): conversation failed
Feb 11 23:51:51 oracle-linux sudo[6151]: pam_unix(sudo:auth): auth could not identify password for [caddy]
Feb 11 23:51:51 oracle-linux sudo[6151]: PAM audit_open() failed: Permission denied
Feb 11 23:51:53 oracle-linux sudo[6151]: PAM audit_open() failed: Permission denied
Feb 11 23:51:53 oracle-linux caddy[6141]: {"level":"error","ts":1676159513.8407798,"logger":"pki.ca.local","msg":"failed to install root certificate","error":"failed to execute sudo: exit status 1","certificate_file":"storage:pki/authorities/local/root.crt"}
Feb 11 23:51:53 oracle-linux caddy[6141]: {"level":"info","ts":1676159513.841017,"logger":"http.log","msg":"server running","name":"remaining_auto_https_redirects","protocols":["h1","h2","h3"]}
Feb 11 23:51:53 oracle-linux caddy[6141]: {"level":"info","ts":1676159513.8410473,"logger":"http","msg":"enabling HTTP/3 listener","addr":":443"}
Feb 11 23:51:53 oracle-linux caddy[6141]: {"level":"info","ts":1676159513.8411539,"msg":"failed to sufficiently increase receive buffer size (was: 208 kiB, wanted: 2048 kiB, got: 416 kiB). See https://github.com/lucas-clemente/quic-go/wiki/UDP-Receive-Buffer-Size for details."}
Feb 11 23:51:53 oracle-linux caddy[6141]: {"level":"info","ts":1676159513.8412209,"logger":"http.log","msg":"server running","name":"srv0","protocols":["h1","h2","h3"]}
Feb 11 23:51:53 oracle-linux caddy[6141]: {"level":"info","ts":1676159513.841267,"logger":"http","msg":"enabling HTTP/3 listener","addr":":8080"}
Feb 11 23:51:53 oracle-linux caddy[6141]: {"level":"info","ts":1676159513.8413582,"logger":"tls.cache.maintenance","msg":"stopped background certificate maintenance","cache":"0x40001b10a0"}
Feb 11 23:51:53 oracle-linux caddy[6141]: Error: loading initial config: loading new config: http app module: start: listen udp :8080: bind: permission denied
Feb 11 23:51:53 oracle-linux systemd[1]: caddy.service: Main process exited, code=exited, status=1/FAILURE
Feb 11 23:51:53 oracle-linux systemd[1]: caddy.service: Failed with result 'exit-code'.
Feb 11 23:51:53 oracle-linux systemd[1]: Failed to start Caddy.

So I tried the workaround suggested there, first “adding to sudoers” (which should be wheel on this system), but sudo usermod -aG wheel caddy did not change anything, then the trust thing:

sudo HOME=~caddy caddy trust

but I get Error: requesting CA info: performing request: Get “http://localhost:2019/pki/ca/local”: dial tcp [::1]:2019: connect: connection refused

Any tips on solving this one is welcome, I understand port 2019 should be exposed by caddy, but it’s not started because of the certificate issue right?

edwardoo · February 12, 2023, 12:12am

Ok, so I mixed up 2 things:

The linked #4248 certificate issue does not prevent caddy from starting what prevents it is later in the full log above, and is “Error: loading initial config: loading new config: http app module: start: listen udp :8080: bind: permission denied”

And so, because the second issue prevents caddy from starting, port 2019 is not available to the trust command… which fails with this rather obtuse message (it could at least say “ensure caddy is running”).

Anyway, I removed this part of the Caddyfile

localhost:8080 {
  file_server browse
}

And this allowed caddy to start and trust to work. I then restarted caddy and got rid of #4248. Progress, at last \o/

francislavoie · February 12, 2023, 7:19am

The reason for this error is that the caddy trust command was rewritten recently to read the root CA cert via the admin API, and not from file anymore. So wherever you read that suggestion from is now outdated unfortunately.

I don’t understand how that would be happening. Binding on UDP should only happen for HTTP/3, but you didn’t override the https_port global option to make that use a port other than 443. So I’m confused.

edwardoo:

Anyway, I removed this part of the Caddyfile
localhost:8080 {
  file_server browse
}
And this allowed caddy to start and trust to work. I then restarted caddy and got rid of #4248. Progress, at last \o/

Now you’re not using HTTPS at all, Caddy will be listening only for HTTP on port 8080. So running caddy trust is unnecessary because there’s no HTTPS going on.

system · March 14, 2023, 7:20am

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.