How does Caddy determine wether to use IPv6 to forward traffic?

Help
1

1. The problem I’m having:

I’m trying to understand how the reverse proxy determines wether to use IPv6 or IPv4 to forward requests. There’s nothing necessarily wrong with it, I’m just trying to understand the following behavior:

Caddy is setup as a reverse proxy for a server that handles IPv4 traffic only. That server runs in a container managed by podman. Podman listens to trafic on the host and redirects it to the container. If podman listens to trafic on all addresses, caddy will try to forward trafic to IPv6. If podman listens to trafic on 127.0.0.1, caddy will use IPv4.

Between those two examples, Caddy’s config did not change, but its behaviour seems to.

Is it documented somewhere that Caddy checks for listening ports to determine IPv4/6 ? Or does it fall back to IPv4 if IPv6 fails?

2. Error messages and/or full log output:

When podman is bound on all addresses:
ss output:

sudo ss -lp | grep 8096
tcp   LISTEN 0      128                                                  *:8096                            *:*    users:(("pasta",pid=13630,fd=8))

Error log:

user@hostname:~$ journalctl -f caddy.service
Failed to add match 'caddy.service': Argument invalide
user@hostname:~$ journalctl -f -u caddy
mai 02 23:22:51 hostname caddy[1253]: 2025/05/02 21:22:51.289        ERROR        http.log.error.subdomain        read tcp [::1]:35834->[::1]:8096: read: connection reset by peer        {"request": {"remote_ip": "192.168.1.1", "remote_port": "39066", "client_ip": "192.168.1.1", "proto": "HTTP/2.0", "method": "GET", "host": "subdomain.domain.com", "uri": "/", "headers": {"User-Agent": ["Mozilla/5.0 (X11; Linux x86_64; rv:137.0) Gecko/20100101 Firefox/137.0"], "Accept-Encoding": ["gzip, deflate, br, zstd"], "Dnt": ["1"], "Sec-Fetch-Dest": ["document"], "Sec-Fetch-Site": ["none"], "Sec-Fetch-User": ["?1"], "Priority": ["u=0, i"], "Accept": ["text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8"], "Accept-Language": ["fr"], "Sec-Gpc": ["1"], "Upgrade-Insecure-Requests": ["1"], "Sec-Fetch-Mode": ["navigate"], "Te": ["trailers"]}, "tls": {"resumed": false, "version": 772, "cipher_suite": 4867, "proto": "h2", "server_name": "subdomain.domain.com"}}, "duration": 0.001415803, "status": 502, "err_id": "snhrqybrc", "err_trace": "reverseproxy.statusError (reverseproxy.go:1269)"}
mai 02 23:22:51 hostname caddy[1253]: 2025/05/02 21:22:51.289        ERROR        http.log.access.subdomain        handled request        {"request": {"remote_ip": "192.168.1.1", "remote_port": "39066", "client_ip": "192.168.1.1", "proto": "HTTP/2.0", "method": "GET", "host": "subdomain.domain.com", "uri": "/", "headers": {"Accept-Encoding": ["gzip, deflate, br, zstd"], "Dnt": ["1"], "Sec-Fetch-Dest": ["document"], "Sec-Fetch-Site": ["none"], "Sec-Fetch-User": ["?1"], "Priority": ["u=0, i"], "User-Agent": ["Mozilla/5.0 (X11; Linux x86_64; rv:137.0) Gecko/20100101 Firefox/137.0"], "Accept-Language": ["fr"], "Sec-Gpc": ["1"], "Upgrade-Insecure-Requests": ["1"], "Sec-Fetch-Mode": ["navigate"], "Te": ["trailers"], "Accept": ["text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8"]}, "tls": {"resumed": false, "version": 772, "cipher_suite": 4867, "proto": "h2", "server_name": "subdomain.domain.com"}}, "bytes_read": 0, "user_id": "", "duration": 0.001415803, "size": 0, "status": 502, "resp_headers": {"Server": ["Caddy"], "Alt-Svc": ["h3=\":443\"; ma=2592000"]}}
mai 02 23:23:03 hostname caddy[1253]: 2025/05/02 21:23:03.566        DEBUG        http.handlers.reverse_proxy        selected upstream        {"dial": "localhost:8096", "total_upstreams": 1}
mai 02 23:23:03 hostname caddy[1253]: 2025/05/02 21:23:03.567        DEBUG        http.handlers.reverse_proxy        upstream roundtrip        {"upstream": "localhost:8096", "duration": 0.001279118, "request": {"remote_ip": "192.168.1.1", "remote_port": "39066", "client_ip": "192.168.1.1", "proto": "HTTP/2.0", "method": "GET", "host": "subdomain.domain.com", "uri": "/", "headers": {"Upgrade-Insecure-Requests": ["1"], "Sec-Gpc": ["1"], "X-Forwarded-For": ["192.168.1.1"], "Accept-Language": ["fr"], "Sec-Fetch-Dest": ["document"], "Sec-Fetch-User": ["?1"], "Priority": ["u=0, i"], "X-Forwarded-Host": ["subdomain.domain.com"], "Te": ["trailers"], "User-Agent": ["Mozilla/5.0 (X11; Linux x86_64; rv:137.0) Gecko/20100101 Firefox/137.0"], "Accept": ["text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8"], "Dnt": ["1"], "X-Forwarded-Proto": ["https"], "Accept-Encoding": ["gzip, deflate, br, zstd"], "Sec-Fetch-Mode": ["navigate"], "Sec-Fetch-Site": ["none"]}, "tls": {"resumed": false, "version": 772, "cipher_suite": 4867, "proto": "h2", "server_name": "subdomain.domain.com"}}, "error": "read tcp [::1]:58808->[::1]:8096: read: connection reset by peer"}
mai 02 23:23:03 hostname caddy[1253]: 2025/05/02 21:23:03.567        ERROR        http.log.error.subdomain        read tcp [::1]:58808->[::1]:8096: read: connection reset by peer        {"request": {"remote_ip": "192.168.1.1", "remote_port": "39066", "client_ip": "192.168.1.1", "proto": "HTTP/2.0", "method": "GET", "host": "subdomain.domain.com", "uri": "/", "headers": {"Sec-Fetch-Dest": ["document"], "Sec-Fetch-Mode": ["navigate"], "Sec-Fetch-Site": ["none"], "Sec-Fetch-User": ["?1"], "Priority": ["u=0, i"], "User-Agent": ["Mozilla/5.0 (X11; Linux x86_64; rv:137.0) Gecko/20100101 Firefox/137.0"], "Accept": ["text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8"], "Accept-Language": ["fr"], "Accept-Encoding": ["gzip, deflate, br, zstd"], "Dnt": ["1"], "Sec-Gpc": ["1"], "Upgrade-Insecure-Requests": ["1"], "Te": ["trailers"]}, "tls": {"resumed": false, "version": 772, "cipher_suite": 4867, "proto": "h2", "server_name": "subdomain.domain.com"}}, "duration": 0.001508753, "status": 502, "err_id": "52vbxyjj9", "err_trace": "reverseproxy.statusError (reverseproxy.go:1269)"}
mai 02 23:23:03 hostname caddy[1253]: 2025/05/02 21:23:03.567        ERROR        http.log.access.subdomain        handled request        {"request": {"remote_ip": "192.168.1.1", "remote_port": "39066", "client_ip": "192.168.1.1", "proto": "HTTP/2.0", "method": "GET", "host": "subdomain.domain.com", "uri": "/", "headers": {"Sec-Fetch-Dest": ["document"], "Sec-Fetch-Mode": ["navigate"], "Sec-Fetch-Site": ["none"], "Sec-Fetch-User": ["?1"], "Priority": ["u=0, i"], "Sec-Gpc": ["1"], "Upgrade-Insecure-Requests": ["1"], "Te": ["trailers"], "User-Agent": ["Mozilla/5.0 (X11; Linux x86_64; rv:137.0) Gecko/20100101 Firefox/137.0"], "Accept": ["text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8"], "Accept-Language": ["fr"], "Accept-Encoding": ["gzip, deflate, br, zstd"], "Dnt": ["1"]}, "tls": {"resumed": false, "version": 772, "cipher_suite": 4867, "proto": "h2", "server_name": "subdomain.domain.com"}}, "bytes_read": 0, "user_id": "", "duration": 0.001508753, "size": 0, "status": 502, "resp_headers": {"Server": ["Caddy"], "Alt-Svc": ["h3=\":443\"; ma=2592000"]}}
mai 02 23:23:04 hostname caddy[1253]: 2025/05/02 21:23:04.261        DEBUG        http.handlers.reverse_proxy        selected upstream        {"dial": "localhost:8096", "total_upstreams": 1}
mai 02 23:23:04 hostname caddy[1253]: 2025/05/02 21:23:04.262        DEBUG        http.handlers.reverse_proxy        upstream roundtrip        {"upstream": "localhost:8096", "duration": 0.000596084, "request": {"remote_ip": "192.168.1.1", "remote_port": "33260", "client_ip": "192.168.1.1", "proto": "HTTP/2.0", "method": "GET", "host": "subdomain.domain.com", "uri": "/", "headers": {"Sec-Fetch-Site": ["none"], "Sec-Gpc": ["1"], "Sec-Fetch-Dest": ["document"], "Sec-Fetch-Mode": ["navigate"], "Te": ["trailers"], "Priority": ["u=0, i"], "X-Forwarded-Proto": ["https"], "Cache-Control": ["no-cache"], "User-Agent": ["Mozilla/5.0 (X11; Linux x86_64; rv:137.0) Gecko/20100101 Firefox/137.0"], "Dnt": ["1"], "Accept-Encoding": ["gzip, deflate, br, zstd"], "Pragma": ["no-cache"], "Accept-Language": ["fr"], "X-Forwarded-Host": ["subdomain.domain.com"], "Upgrade-Insecure-Requests": ["1"], "Accept": ["text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8"], "Sec-Fetch-User": ["?1"], "X-Forwarded-For": ["192.168.1.1"]}, "tls": {"resumed": true, "version": 772, "cipher_suite": 4867, "proto": "h2", "server_name": "subdomain.domain.com"}}, "error": "read tcp [::1]:58816->[::1]:8096: read: connection reset by peer"}

When podman is bound on 127.0.0.1:
ss output:

sudo ss -lp | grep 8096
tcp   LISTEN 0      128                                          127.0.0.1:8096                      0.0.0.0:*    users:(("pasta",pid=20407,fd=6))

No error in that case (traffic is forwarded to 127.0.0.1 as expected).

3. Caddy version:

v2.8.4

4. How I installed and ran Caddy:

Caddy is installed from the package manager. It runs as a systemd service.

a. System environment:

Fedora 41

b. Command:

caddy run --environ --config /etc/caddy/caddy.json

c. Service/unit/compose file:

# caddy.service
#
# For using Caddy with a config file.
#
# WARNING: This service does not use the --resume flag, so if you
# use the API to make changes, they will be overwritten by the
# Caddyfile next time the service is restarted. If you intend to
# use Caddy's API to configure it, add the --resume flag to the
# `caddy run` command or use the caddy-api.service file instead.

[Unit]
Description=Caddy web server
Documentation=https://caddyserver.com/docs/
After=network.target

[Service]
Type=notify
User=caddy
Group=caddy
ExecStartPre=/usr/bin/caddy validate --config /etc/caddy/caddy.json
ExecStart=/usr/bin/caddy run --environ --config /etc/caddy/caddy.json
ExecReload=/usr/bin/caddy reload --config /etc/caddy/caddy.json
TimeoutStopSec=5s
LimitNOFILE=1048576
PrivateTmp=true
ProtectHome=true
ProtectSystem=full
AmbientCapabilities=CAP_NET_ADMIN CAP_NET_BIND_SERVICE

[Install]
WantedBy=multi-user.target

d. My complete Caddy config:

{
	"logging": {
		"logs": {
			"default": {
				"level": "DEBUG"
			},
			"blabla": {
				"writer": {
					"filename": "/var/log/caddy/blabla1.log",
					"output": "file",
					"roll_local_time": true
				},
				"include": [
					"http.log.access.blabla1",
					"http.log.error.blabla1"
				]
			},
			"blabla": {
				"writer": {
					"filename": "/var/log/caddy/blabla2.log",
					"output": "file",
					"roll_local_time": true
				},
				"include": [
					"http.log.access.blabla2",
					"http.log.error.blabla2"
				]
			},
			"blabla": {
				"writer": {
					"filename": "/var/log/caddy/blabla3.log",
					"output": "file",
					"roll_local_time": true
				},
				"include": [
					"http.log.access.blabla3",
					"http.log.error.blabla3"
				]
			}
		}
	},
	"apps": {
		"http": {
			"servers": {
				"srv0": {
					"listen": [
						":443"
					],
					"routes": [
						{
							"handle": [
								{
									"handler": "subroute",
									"routes": [
										{
											"handle": [
												{
													"handler": "reverse_proxy",
													"upstreams": [
														{
															"dial": "localhost:8096"
														}
													]
												}
											]
										}
									]
								}
							],
							"match": [
								{
									"host": [
										"blabla.domain1.duckdns.org"
									]
								},
								{
									"host": [
										"blabla.domain1.net"
									]
								}
							],
							"terminal": true
						},
						{
							"handle": [
								{
									"handler": "subroute",
									"routes": [
										{
											"handle": [
												{
													"handler": "reverse_proxy",
													"upstreams": [
														{
															"dial": "localhost:8080"
														}
													]
												}
											]
										}
									]
								}
							],
							"match": [
								{
									"host": [
										"blabla.domain1.duckdns.org"
									]
								},
								{
									"host": [
										"blabla.domain1.net"
									]
								}
							],
							"terminal": true
						},
						{
							"match": [
								{
									"host": [
										"blabla.domain1.duckdns.org"
									]
								},
								{
									"host": [
										"blabla.domain1.net"
									]
								},
								{
									"host": [
										"fichiers.domain2.fr"
									]
								}
							],
							"handle": [
								{
									"handler": "file_server",
									"browse": {},
									"root": "/home/user/Public/caddy"
								}
							],
							"terminal": true
						}
					],
					"logs": {
						"logger_names": {
							"blabla.domain1.net": "blabla1",
							"blabla.domain1.duckdns.org": "blabla1",
							"blabla.domain1.net": "blabla2",
							"blabla.domain1.duckdns.org": "blabla2",
							"blabla.domain1.net": "blabla3",
							"blabla.domain1.duckdns.org": "blabla3",
							"fichiers.domain2.fr": "blabla3"
						},
						"should_log_credentials": true
					}
				}
			}
		},
		"tls": {
			"certificates": {
				"automate": [
					"blabla1.domain1.duckdns.org",
					"blabla2.domain1.duckdns.org",
					"blabla3.domain1.duckdns.org",
					"blabla1.domain1.net",
					"blabla2.domain1.net",
					"blabla3.domain1.net",
					"fichiers.domain2.fr"
				]
			}
		}
	}
}

5. Links to relevant resources:

2

It’s not Caddy who decides. It’s whatever the DNS resolver or the operating system gives.

1 Like
3

There’s no dns resolving involved here. My question is about the call to either [::1] or 127.0.0.1.

4

There’s one from localhost to either [::1] or 127.0.0.1. In any way, that is not something Caddy decides. It gets it from the OS.

2 Likes
5

What does it request from the OS exactly ? The IP listening on port 8096 ?

It seems that the OS returns [::1] even though it is not listening; ss shows *:8096 as the listening IP range, which does not include [::1]:8096 if I understand correctly.

6

Caddy asks the OS (technically the DNS resolver of the network, which likely starts with the OS) to translate the given host name to an IP address, regardless of whether there’s someone listening on the subject port or not. I don’t know why Podman fails to bind the IPv6 correctly to the upstream address, but this is not something Caddy controls. You’ll have to check why Podman does what it does.

2 Likes