Caddy on a multihomed host with strict IP binding

Igor · December 21, 2022, 4:32pm

1. Output of `caddy version`:

2.4.6-4.fc36

2. How I run Caddy:

a. System environment:

Fedora 36
systemd
baremetal
Caddy installed from official fedora repo

b. Command:

systemctl start caddy

c. Service/unit/compose file:

# caddy.service
#
# For using Caddy with a config file.
#
# WARNING: This service does not use the --resume flag, so if you
# use the API to make changes, they will be overwritten by the
# Caddyfile next time the service is restarted. If you intend to
# use Caddy's API to configure it, add the --resume flag to the
# `caddy run` command or use the caddy-api.service file instead.

[Unit]
Description=Caddy web server
Documentation=https://caddyserver.com/docs/
After=network.target

[Service]
Type=notify
User=caddy
Group=caddy
ExecStartPre=/usr/bin/caddy validate --config /etc/caddy/Caddyfile
ExecStart=/usr/bin/caddy run --environ --config /etc/caddy/Caddyfile
ExecReload=/usr/bin/caddy reload --config /etc/caddy/Caddyfile
TimeoutStopSec=5s
LimitNOFILE=1048576
LimitNPROC=512
PrivateTmp=true
ProtectHome=true
ProtectSystem=full
AmbientCapabilities=CAP_NET_BIND_SERVICE

[Install]
WantedBy=multi-user.target

d. My complete Caddy config:

/etc/caddy/Caddyfile:

# The Caddyfile is an easy way to configure your Caddy web server.
#
# https://caddyserver.com/docs/caddyfile

# As an alternative to editing the above site block, you can add your own site
# block files in the Caddyfile.d directory, and they will be included as long
# as they use the .caddyfile extension.
import Caddyfile.d/*.caddyfile

/etc/caddy/Caddyfile.d/00global.caddyfile:

{
        # TLS Options
        email "ssladmin@example.org"
}

/etc/caddy/Caddyfile.d/a.example.org.caddyfile:

a.example.org {
        bind 203.0.113.2 2001:db8::2

        respond "Welcome to a.example.org"
}

/etc/caddy/Caddyfile.d/b.example.org.caddyfile:

http://b.example.org {
        bind 203.0.113.3

        respond "Welcome to b.example.org"
}

3. The problem I’m having:

The IP addresses used in this example are based on RFC 5737 (IPv4) and RFC 3849 (IPv6).

I have a system with 10 IP addresses:

203.0.113.1
203.0.113.2
203.0.113.3
203.0.113.4
203.0.113.5
2001:db8::1
2001:db8::2
2001:db8::3
2001:db8::4
2001:db8::5

IP address 203.0.113.1 is used by Apache web server.

I now have 3 websites:

a.example.org, should be available via HTTP and HTTPS and served via 203.0.113.2 and 2001:db8::2.
b.example.org, should only be available via HTTP and served via 203.0.113.3.
c.example.org, should be available via HTTPS only and served via 203.0.113.4 and 2001:db8::4.

I came up with shown config files. Without b.example.org.caddyfile in place, a.example.org works as expected. I.e. caddy is only listening on 203.0.113.2 and 2001:db8::2 for both, HTTP and HTTPS and does not clash with Apache.

But once I add config for b.example.org like shown, caddy stops listening on 203.0.113.2:80 and [2001:db8::2]:80:

# ss -tlnp | grep -F 'caddy'
LISTEN 0      4096     203.0.113.3:80        0.0.0.0:*    users:(("caddy",pid=31890,fd=10))
LISTEN 0      4096     203.0.113.2:443       0.0.0.0:*    users:(("caddy",pid=31890,fd=7))
LISTEN 0      4096       127.0.0.1:2019      0.0.0.0:*    users:(("caddy",pid=31890,fd=3))
LISTEN 0      4096   [2001:db8::2]:443          [::]:*    users:(("caddy",pid=31890,fd=8))

Connection test:

$ curl -kL --resolve a.example.org:80:203.0.113.2 http://a.example.org/
curl: (7) Failed to connect to a.example.org port 80 after 2115 ms: Connection refused

But a.example.org is accessible via 203.0.113.3:

$ curl -kL --resolve a.example.org:80:203.0.113.3 http://a.example.org/
Welcome to a.example.org

This raises an additional question: I don’t want that caddy serves a.example.org via any other listener.

I also don’t want that caddy arbitrary redirects sites I did not add to caddy:

$ dig +short caddyserver.com
165.227.20.207

$ curl -vkL --resolve badsite.example.org:80:165.227.20.207 http://badsite.example.org/
* Added badsite.example.org:80:165.227.20.207 to DNS cache
* Hostname badsite.example.org was found in DNS cache
*   Trying 165.227.20.207:80...
* Connected to badsite.example.org (165.227.20.207) port 80 (#0)
> GET / HTTP/1.1
> Host: badsite.example.org
> User-Agent: curl/7.83.1
> Accept: */*
>
* Mark bundle as not supporting multiuse
< HTTP/1.1 308 Permanent Redirect
< Connection: close
< Location: https://badsite.example.org/
< Server: Caddy
< Date: Wed, 21 Dec 2022 15:58:55 GMT
< Content-Length: 0
<
* Closing connection 0

francislavoie · December 22, 2022, 12:59am

That’s quite an old version at this point. Please upgrade to v2.6.2, the latest.

We’ve made significant changes to how we manage listeners since, so you should upgrade and try again before we drill down any further.

Igor · December 22, 2022, 3:10pm

I upgraded to v2.6.2 h1:wKoFIxpmOJLGl3QXoo6PNbYvGW4xLEgo32GPBEjWL8o= using xcaddy but nothing has changed. Same problems. Still only listening on 203.0.113.3 for HTTP.

francislavoie · December 22, 2022, 8:12pm

Strange.

Please adapt your config by running caddy adapt --config /etc/caddy/Caddyfile --pretty and show us what it looks like. I’d like to see what the entire config looks like, to know if it’s a problem with the Caddyfile adapter, or if the JSON config looks okay and it being a possible runtime bug.

Igor · December 22, 2022, 10:42pm

{
	"apps": {
		"http": {
			"servers": {
				"srv0": {
					"listen": [
						"203.0.113.2:443",
						"[2001:db8::2]:443"
					],
					"routes": [
						{
							"match": [
								{
									"host": [
										"a.example.org"
									]
								}
							],
							"handle": [
								{
									"handler": "subroute",
									"routes": [
										{
											"handle": [
												{
													"body": "Welcome to a.example.org!",
													"handler": "static_response"
												}
											]
										}
									]
								}
							],
							"terminal": true
						}
					]
				},
				"srv1": {
					"listen": [
						"203.0.113.3:80"
					],
					"routes": [
						{
							"match": [
								{
									"host": [
										"b.example.org"
									]
								}
							],
							"handle": [
								{
									"handler": "subroute",
									"routes": [
										{
											"handle": [
												{
													"body": "Welcome to b.example.org!",
													"handler": "static_response"
												}
											]
										}
									]
								}
							],
							"terminal": true
						}
					]
				}
			}
		},
		"tls": {
			"automation": {
				"policies": [
					{
						"subjects": [
							"a.example.org"
						],
						"issuers": [
							{
								"email": "ssladmin@example.org",
								"module": "acme"
							},
							{
								"email": "ssladmin@example.org",
								"module": "zerossl"
							}
						]
					}
				]
			}
		}
	}
}

So looks like caddy doesn’t generate a listener for 203.0.113.2:80 and [2001:db8::2]:80. However, even without /etc/caddy/Caddyfile.d/b.example.org.caddyfile file in place, the JSON output only lists listener for :443 but a.example.org is accessible via HTTP.

system · January 20, 2023, 4:33pm

This topic was automatically closed after 30 days. New replies are no longer allowed.