Caddy not forwarding anymore after installing YUNOHOST on a different machine (but forwarding to it with Caddy)

1. The problem I’m having:

I had a caddy configuration correctly forwarding requests. A browser would request some data from https://cache.mapcomplete.org/summary/status.json and caddy would do the https-stuff and forward it to one of my services on 127.0.0.1:2345. This worked fine.

This is in a semi-residential/small business setup. Our ISP (Proximus, Belgium) forwards all requests to the machine this is running on (we named her lain).

Now, we also want to setup a YUNOHOST at this location in this network. As lain is the exposed host, I setup forwording for nerdlab.nohost.me (now offline again).

This (probably) caused major issues, as I suddenly got ‘SSL_INTERNAL_ERROR’

The exposed host/DMZ forwarding is still working, as http://109.128.57.178:2345/status.json is reachable.
The DynDNS is fine too, as I can ssh into the exposed machine using `cache.mapcomplete.org

2. Error messages and/or full log output:

Jun 12 12:21:12 lain systemd[1]: Starting Caddy...
Jun 12 12:21:12 lain caddy[713748]: caddy.HomeDir=/var/lib/caddy
Jun 12 12:21:12 lain caddy[713748]: caddy.AppDataDir=/var/lib/caddy/.local/share/caddy
Jun 12 12:21:12 lain caddy[713748]: caddy.AppConfigDir=/var/lib/caddy/.config/caddy
Jun 12 12:21:12 lain caddy[713748]: caddy.ConfigAutosavePath=/var/lib/caddy/.config/caddy/autosave.json
Jun 12 12:21:12 lain caddy[713748]: caddy.Version=v2.8.4 h1:q3pe0wpBj1OcHFZ3n/1nl4V4bxBrYoSoab7rL9BMYNk=
Jun 12 12:21:12 lain caddy[713748]: runtime.GOOS=linux
Jun 12 12:21:12 lain caddy[713748]: runtime.GOARCH=amd64
Jun 12 12:21:12 lain caddy[713748]: runtime.Compiler=gc
Jun 12 12:21:12 lain caddy[713748]: runtime.NumCPU=4
Jun 12 12:21:12 lain caddy[713748]: runtime.GOMAXPROCS=4
Jun 12 12:21:12 lain caddy[713748]: runtime.Version=go1.22.3
Jun 12 12:21:12 lain caddy[713748]: os.Getwd=/
Jun 12 12:21:12 lain caddy[713748]: LANG=en_US.UTF-8
Jun 12 12:21:12 lain caddy[713748]: PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/snap/bin
Jun 12 12:21:12 lain caddy[713748]: NOTIFY_SOCKET=/run/systemd/notify
Jun 12 12:21:12 lain caddy[713748]: HOME=/var/lib/caddy
Jun 12 12:21:12 lain caddy[713748]: LOGNAME=caddy
Jun 12 12:21:12 lain caddy[713748]: USER=caddy
Jun 12 12:21:12 lain caddy[713748]: INVOCATION_ID=95f2187bc34a49a5b4faf7a39c33e4d3
Jun 12 12:21:12 lain caddy[713748]: JOURNAL_STREAM=8:6299848
Jun 12 12:21:12 lain caddy[713748]: SYSTEMD_EXEC_PID=713748
Jun 12 12:21:12 lain caddy[713748]: {"level":"info","ts":1718194872.5312371,"msg":"using config from file","file":"/etc/caddy/Caddyfile"}
Jun 12 12:21:12 lain caddy[713748]: {"level":"info","ts":1718194872.5322998,"msg":"adapted config to JSON","adapter":"caddyfile"}
Jun 12 12:21:12 lain caddy[713748]: Error: loading initial config: loading new config: starting caddy administration endpoint: listen tcp 127.0.0.1:2019: bind: address already in use
Jun 12 12:21:12 lain systemd[1]: caddy.service: Main process exited, code=exited, status=1/FAILURE
Jun 12 12:21:12 lain systemd[1]: caddy.service: Failed with result 'exit-code'.
Jun 12 12:21:12 lain systemd[1]: Failed to start Caddy.
-- Boot a07ec418c21145dabc3b8b032dd1079b --
Jun 12 12:38:57 lain systemd[1]: Starting Caddy...
Jun 12 12:39:00 lain caddy[758]: caddy.HomeDir=/var/lib/caddy
Jun 12 12:39:00 lain caddy[758]: caddy.AppDataDir=/var/lib/caddy/.local/share/caddy
Jun 12 12:39:00 lain caddy[758]: caddy.AppConfigDir=/var/lib/caddy/.config/caddy
Jun 12 12:39:00 lain caddy[758]: caddy.ConfigAutosavePath=/var/lib/caddy/.config/caddy/autosave.json
Jun 12 12:39:00 lain caddy[758]: caddy.Version=v2.8.4 h1:q3pe0wpBj1OcHFZ3n/1nl4V4bxBrYoSoab7rL9BMYNk=
Jun 12 12:39:00 lain caddy[758]: runtime.GOOS=linux
Jun 12 12:39:00 lain caddy[758]: runtime.GOARCH=amd64
Jun 12 12:39:00 lain caddy[758]: runtime.Compiler=gc
Jun 12 12:39:00 lain caddy[758]: runtime.NumCPU=4
Jun 12 12:39:00 lain caddy[758]: runtime.GOMAXPROCS=4
Jun 12 12:39:00 lain caddy[758]: runtime.Version=go1.22.3
Jun 12 12:39:00 lain caddy[758]: os.Getwd=/
Jun 12 12:39:00 lain caddy[758]: LANG=en_US.UTF-8
Jun 12 12:39:00 lain caddy[758]: PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/snap/bin
Jun 12 12:39:00 lain caddy[758]: NOTIFY_SOCKET=/run/systemd/notify
Jun 12 12:39:00 lain caddy[758]: HOME=/var/lib/caddy
Jun 12 12:39:00 lain caddy[758]: LOGNAME=caddy
Jun 12 12:39:00 lain caddy[758]: USER=caddy
Jun 12 12:39:00 lain caddy[758]: INVOCATION_ID=7dcb486aab0c4cdaa50527996c0f5d84
Jun 12 12:39:00 lain caddy[758]: JOURNAL_STREAM=8:20374
Jun 12 12:39:00 lain caddy[758]: SYSTEMD_EXEC_PID=758
Jun 12 12:39:00 lain caddy[758]: {"level":"info","ts":1718195940.7244382,"msg":"using config from file","file":"/etc/caddy/Caddyfile"}
Jun 12 12:39:00 lain caddy[758]: {"level":"info","ts":1718195940.7259715,"msg":"adapted config to JSON","adapter":"caddyfile"}
Jun 12 12:39:00 lain caddy[758]: {"level":"info","ts":1718195940.7296157,"logger":"admin","msg":"admin endpoint started","address":"localhost:2019","enforce_origin":false,"origins":["//localhost:2019","//[::1]:2019","//127.0.0.1:2019"]}
Jun 12 12:39:00 lain caddy[758]: {"level":"warn","ts":1718195940.7299738,"logger":"http.auto_https","msg":"server is listening only on the HTTP port, so no automatic HTTPS will be applied to this server","server_name":"srv0","http_port":80}
Jun 12 12:39:00 lain caddy[758]: {"level":"info","ts":1718195940.7310135,"logger":"http.log","msg":"server running","name":"srv0","protocols":["h1","h2","h3"]}
Jun 12 12:39:00 lain caddy[758]: {"level":"info","ts":1718195940.748611,"msg":"autosaved config (load with --resume flag)","file":"/var/lib/caddy/.config/caddy/autosave.json"}
Jun 12 12:39:00 lain caddy[758]: {"level":"info","ts":1718195940.7486765,"msg":"serving initial configuration"}
Jun 12 12:39:00 lain systemd[1]: Started Caddy.
Jun 12 12:39:00 lain caddy[758]: {"level":"info","ts":1718195940.752533,"logger":"tls.cache.maintenance","msg":"started background certificate maintenance","cache":"0xc000395d80"}
Jun 12 12:39:00 lain caddy[758]: {"level":"info","ts":1718195940.800717,"logger":"tls","msg":"cleaning storage unit","storage":"FileStorage:/var/lib/caddy/.local/share/caddy"}
Jun 12 12:39:00 lain caddy[758]: {"level":"info","ts":1718195940.800943,"logger":"tls","msg":"finished cleaning storage units"}

3. Caddy version:

v2.8.4 h1:q3pe0wpBj1OcHFZ3n/1nl4V4bxBrYoSoab7rL9BMYNk=
(latest as of writing)

4. How I installed and ran Caddy

Installation via stable release for ubuntu:

sudo apt install -y debian-keyring debian-archive-keyring apt-transport-https curl
curl -1sLf 'https://dl.cloudsmith.io/public/caddy/stable/gpg.key' | sudo gpg --dearmor -o /usr/share/keyrings/caddy-stable-archive-keyring.gpg
curl -1sLf 'https://dl.cloudsmith.io/public/caddy/stable/debian.deb.txt' | sudo tee /etc/apt/sources.list.d/caddy-stable.list
sudo apt update
sudo apt install caddy

a. System environment:

Ubuntu server 22.04 on bare metal

b. Command:

sudo caddy start Caddyfile

which prints the following to the console:

2024/06/12 12:26:52.782	INFO	using adjacent Caddyfile
2024/06/12 12:26:52.784	INFO	adapted config to JSON	{"adapter": "caddyfile"}
2024/06/12 12:26:52.784	WARN	Caddyfile input is not formatted; run 'caddy fmt --overwrite' to fix inconsistencies	{"adapter": "caddyfile", "file": "Caddyfile", "line": 2}
2024/06/12 12:26:52.785	INFO	admin	admin endpoint started	{"address": "localhost:2019", "enforce_origin": false, "origins": ["//localhost:2019", "//[::1]:2019", "//127.0.0.1:2019"]}
2024/06/12 12:26:52.785	INFO	http.auto_https	server is listening only on the HTTPS port but has no TLS connection policies; adding one to enable TLS	{"server_name": "srv0", "https_port": 443}
2024/06/12 12:26:52.785	INFO	http.auto_https	enabling automatic HTTP->HTTPS redirects	{"server_name": "srv0"}
2024/06/12 12:26:52.785	INFO	tls.cache.maintenance	started background certificate maintenance	{"cache": "0xc000257280"}
2024/06/12 12:26:52.785	DEBUG	http.auto_https	adjusted config	{"tls": {"automation":{"policies":[{}]}}, "http": {"servers":{"remaining_auto_https_redirects":{"listen":[":80"],"routes":[{},{}]},"srv0":{"listen":[":443"],"routes":[{"handle":[{"handler":"subroute","routes":[{"handle":[{"handler":"reverse_proxy","upstreams":[{"dial":"127.0.0.1:2345"}]}],"match":[{"path":["/summary/*"]}]},{"handle":[{"handler":"reverse_proxy","upstreams":[{"dial":"127.0.0.1:7800"}]}],"match":[{"path":["/*"]}]}]}],"terminal":true}],"tls_connection_policies":[{}],"automatic_https":{}}}}}
2024/06/12 12:26:52.786	INFO	http	enabling HTTP/3 listener	{"addr": ":443"}
2024/06/12 12:26:52.786	DEBUG	http	starting server loop	{"address": "[::]:443", "tls": true, "http3": true}
2024/06/12 12:26:52.786	INFO	http.log	server running	{"name": "srv0", "protocols": ["h1", "h2", "h3"]}
2024/06/12 12:26:52.786	DEBUG	http	starting server loop	{"address": "[::]:80", "tls": false, "http3": false}
2024/06/12 12:26:52.786	INFO	http.log	server running	{"name": "remaining_auto_https_redirects", "protocols": ["h1", "h2", "h3"]}
2024/06/12 12:26:52.786	INFO	http	enabling automatic TLS certificate management	{"domains": ["cache.mapcomplete.org"]}
2024/06/12 12:26:52.786	INFO	autosaved config (load with --resume flag)	{"file": "/root/.config/caddy/autosave.json"}
2024/06/12 12:26:52.786	INFO	serving initial configuration
2024/06/12 12:26:52.786	INFO	tls.obtain	acquiring lock	{"identifier": "cache.mapcomplete.org"}
2024/06/12 12:26:52.787	INFO	tls	storage cleaning happened too recently; skipping for now	{"storage": "FileStorage:/root/.local/share/caddy", "instance": "c40bb21c-0294-4704-af32-5a0f2e91fa03", "try_again": "2024/06/13 12:26:52.787", "try_again_in": 86399.99999935}
2024/06/12 12:26:52.787	INFO	tls	finished cleaning storage units
2024/06/12 12:30:26.843	DEBUG	http.stdlib	http: TLS handshake error from 127.0.0.1:45710: EOF

d. My complete Caddy config:

{
  debug
}

cache.mapcomplete.org {
    reverse_proxy /summary/* {
        to http://127.0.0.1:2345
    }
    
    reverse_proxy /* {
        to http://127.0.0.1:7800
    }

}

5. Links to relevant resources:

When testing on local host, I get an internal error for SSL::

curl -vL http://127.0.0.1:80
*   Trying 127.0.0.1:80...
* Connected to 127.0.0.1 (127.0.0.1) port 80 (#0)
> GET / HTTP/1.1
> Host: 127.0.0.1
> User-Agent: curl/7.81.0
> Accept: */*
> 
* Mark bundle as not supporting multiuse
< HTTP/1.1 308 Permanent Redirect
< Connection: close
< Location: https://127.0.0.1/
< Server: Caddy
< Date: Wed, 12 Jun 2024 13:17:15 GMT
< Content-Length: 0
< 
* Closing connection 0
* Clear auth, redirects to port from 80 to 443
* Issue another request to this URL: 'https://127.0.0.1/'
*   Trying 127.0.0.1:443...
* Connected to 127.0.0.1 (127.0.0.1) port 443 (#1)
* ALPN, offering h2
* ALPN, offering http/1.1
*  CAfile: /etc/ssl/certs/ca-certificates.crt
*  CApath: /etc/ssl/certs
* TLSv1.0 (OUT), TLS header, Certificate Status (22):
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* TLSv1.2 (IN), TLS header, Unknown (21):
* TLSv1.3 (IN), TLS alert, internal error (592):
* error:0A000438:SSL routines::tlsv1 alert internal error
* Closing connection 1
curl: (35) error:0A000438:SSL routines::tlsv1 alert internal error

Looks like at some point, you either had a 2nd Caddy instance running, or another piece of software binding to port 2019, which Caddy uses by default for its admin endpoint.

You should only have one Caddy instance running. Don’t run caddy start at any point, always use the systemd commands to start/stop/restart Caddy to ensure you only have one instance.

If you have another app using that port, then either configure that app to use a different port than 2019 (to let Caddy use it), or configure Caddy with the admin global option to use a different port (e.g. admin localhost:2020).

I don’t know if this is still an issue because after that, I see Caddy was able to start successfully, but just wanted to clarify that anyway.

Yeah this is a problem, don’t use sudo caddy start. Make sure to run sudo caddy stop if you still have an instance running this way, then follow the instructions on Keep Caddy Running — Caddy Documentation to run Caddy as a systemd service.

You can simplify this a bit:

	reverse_proxy /summary/* 127.0.0.1:2345
	reverse_proxy 127.0.0.1:7800

You don’t need a /* matcher, omitting the matcher is very slightly more efficient because it skips needing to perform a path comparison. Also you don’t need to use to because you can inline the address, and you don’t need http:// because that’s implied.

The problem here is you’re making a request for the hostname 127.0.0.1, but Caddy doesn’t have a certificate for that name. You configured Caddy with the domain cache.mapcomplete.org so it can only serve requests with that hostname. Caddy selects a TLS certificate using TLS-SNI, i.e. using the domain name. Caddy wasn’t configured to manage a certificate for 127.0.0.1 so it can’t handle that request.

1 Like

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.