How does Caddy determine wether to use IPv6 to forward traffic?

1. The problem I’m having:

I’m trying to understand how the reverse proxy determines wether to use IPv6 or IPv4 to forward requests. There’s nothing necessarily wrong with it, I’m just trying to understand the following behavior:

Caddy is setup as a reverse proxy for a server that handles IPv4 traffic only. That server runs in a container managed by podman. Podman listens to trafic on the host and redirects it to the container. If podman listens to trafic on all addresses, caddy will try to forward trafic to IPv6. If podman listens to trafic on 127.0.0.1, caddy will use IPv4.

Between those two examples, Caddy’s config did not change, but its behaviour seems to.

Is it documented somewhere that Caddy checks for listening ports to determine IPv4/6 ? Or does it fall back to IPv4 if IPv6 fails?

2. Error messages and/or full log output:

When podman is bound on all addresses:
ss output:

sudo ss -lp | grep 8096
tcp   LISTEN 0      128                                                  *:8096                            *:*    users:(("pasta",pid=13630,fd=8))   

Error log:

user@hostname:~$ journalctl -f caddy.service
Failed to add match 'caddy.service': Argument invalide
user@hostname:~$ journalctl -f -u caddy
mai 02 23:22:51 hostname caddy[1253]: 2025/05/02 21:22:51.289        ERROR        http.log.error.subdomain        read tcp [::1]:35834->[::1]:8096: read: connection reset by peer        {"request": {"remote_ip": "192.168.1.1", "remote_port": "39066", "client_ip": "192.168.1.1", "proto": "HTTP/2.0", "method": "GET", "host": "subdomain.domain.com", "uri": "/", "headers": {"User-Agent": ["Mozilla/5.0 (X11; Linux x86_64; rv:137.0) Gecko/20100101 Firefox/137.0"], "Accept-Encoding": ["gzip, deflate, br, zstd"], "Dnt": ["1"], "Sec-Fetch-Dest": ["document"], "Sec-Fetch-Site": ["none"], "Sec-Fetch-User": ["?1"], "Priority": ["u=0, i"], "Accept": ["text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8"], "Accept-Language": ["fr"], "Sec-Gpc": ["1"], "Upgrade-Insecure-Requests": ["1"], "Sec-Fetch-Mode": ["navigate"], "Te": ["trailers"]}, "tls": {"resumed": false, "version": 772, "cipher_suite": 4867, "proto": "h2", "server_name": "subdomain.domain.com"}}, "duration": 0.001415803, "status": 502, "err_id": "snhrqybrc", "err_trace": "reverseproxy.statusError (reverseproxy.go:1269)"}
mai 02 23:22:51 hostname caddy[1253]: 2025/05/02 21:22:51.289        ERROR        http.log.access.subdomain        handled request        {"request": {"remote_ip": "192.168.1.1", "remote_port": "39066", "client_ip": "192.168.1.1", "proto": "HTTP/2.0", "method": "GET", "host": "subdomain.domain.com", "uri": "/", "headers": {"Accept-Encoding": ["gzip, deflate, br, zstd"], "Dnt": ["1"], "Sec-Fetch-Dest": ["document"], "Sec-Fetch-Site": ["none"], "Sec-Fetch-User": ["?1"], "Priority": ["u=0, i"], "User-Agent": ["Mozilla/5.0 (X11; Linux x86_64; rv:137.0) Gecko/20100101 Firefox/137.0"], "Accept-Language": ["fr"], "Sec-Gpc": ["1"], "Upgrade-Insecure-Requests": ["1"], "Sec-Fetch-Mode": ["navigate"], "Te": ["trailers"], "Accept": ["text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8"]}, "tls": {"resumed": false, "version": 772, "cipher_suite": 4867, "proto": "h2", "server_name": "subdomain.domain.com"}}, "bytes_read": 0, "user_id": "", "duration": 0.001415803, "size": 0, "status": 502, "resp_headers": {"Server": ["Caddy"], "Alt-Svc": ["h3=\":443\"; ma=2592000"]}}
mai 02 23:23:03 hostname caddy[1253]: 2025/05/02 21:23:03.566        DEBUG        http.handlers.reverse_proxy        selected upstream        {"dial": "localhost:8096", "total_upstreams": 1}
mai 02 23:23:03 hostname caddy[1253]: 2025/05/02 21:23:03.567        DEBUG        http.handlers.reverse_proxy        upstream roundtrip        {"upstream": "localhost:8096", "duration": 0.001279118, "request": {"remote_ip": "192.168.1.1", "remote_port": "39066", "client_ip": "192.168.1.1", "proto": "HTTP/2.0", "method": "GET", "host": "subdomain.domain.com", "uri": "/", "headers": {"Upgrade-Insecure-Requests": ["1"], "Sec-Gpc": ["1"], "X-Forwarded-For": ["192.168.1.1"], "Accept-Language": ["fr"], "Sec-Fetch-Dest": ["document"], "Sec-Fetch-User": ["?1"], "Priority": ["u=0, i"], "X-Forwarded-Host": ["subdomain.domain.com"], "Te": ["trailers"], "User-Agent": ["Mozilla/5.0 (X11; Linux x86_64; rv:137.0) Gecko/20100101 Firefox/137.0"], "Accept": ["text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8"], "Dnt": ["1"], "X-Forwarded-Proto": ["https"], "Accept-Encoding": ["gzip, deflate, br, zstd"], "Sec-Fetch-Mode": ["navigate"], "Sec-Fetch-Site": ["none"]}, "tls": {"resumed": false, "version": 772, "cipher_suite": 4867, "proto": "h2", "server_name": "subdomain.domain.com"}}, "error": "read tcp [::1]:58808->[::1]:8096: read: connection reset by peer"}
mai 02 23:23:03 hostname caddy[1253]: 2025/05/02 21:23:03.567        ERROR        http.log.error.subdomain        read tcp [::1]:58808->[::1]:8096: read: connection reset by peer        {"request": {"remote_ip": "192.168.1.1", "remote_port": "39066", "client_ip": "192.168.1.1", "proto": "HTTP/2.0", "method": "GET", "host": "subdomain.domain.com", "uri": "/", "headers": {"Sec-Fetch-Dest": ["document"], "Sec-Fetch-Mode": ["navigate"], "Sec-Fetch-Site": ["none"], "Sec-Fetch-User": ["?1"], "Priority": ["u=0, i"], "User-Agent": ["Mozilla/5.0 (X11; Linux x86_64; rv:137.0) Gecko/20100101 Firefox/137.0"], "Accept": ["text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8"], "Accept-Language": ["fr"], "Accept-Encoding": ["gzip, deflate, br, zstd"], "Dnt": ["1"], "Sec-Gpc": ["1"], "Upgrade-Insecure-Requests": ["1"], "Te": ["trailers"]}, "tls": {"resumed": false, "version": 772, "cipher_suite": 4867, "proto": "h2", "server_name": "subdomain.domain.com"}}, "duration": 0.001508753, "status": 502, "err_id": "52vbxyjj9", "err_trace": "reverseproxy.statusError (reverseproxy.go:1269)"}
mai 02 23:23:03 hostname caddy[1253]: 2025/05/02 21:23:03.567        ERROR        http.log.access.subdomain        handled request        {"request": {"remote_ip": "192.168.1.1", "remote_port": "39066", "client_ip": "192.168.1.1", "proto": "HTTP/2.0", "method": "GET", "host": "subdomain.domain.com", "uri": "/", "headers": {"Sec-Fetch-Dest": ["document"], "Sec-Fetch-Mode": ["navigate"], "Sec-Fetch-Site": ["none"], "Sec-Fetch-User": ["?1"], "Priority": ["u=0, i"], "Sec-Gpc": ["1"], "Upgrade-Insecure-Requests": ["1"], "Te": ["trailers"], "User-Agent": ["Mozilla/5.0 (X11; Linux x86_64; rv:137.0) Gecko/20100101 Firefox/137.0"], "Accept": ["text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8"], "Accept-Language": ["fr"], "Accept-Encoding": ["gzip, deflate, br, zstd"], "Dnt": ["1"]}, "tls": {"resumed": false, "version": 772, "cipher_suite": 4867, "proto": "h2", "server_name": "subdomain.domain.com"}}, "bytes_read": 0, "user_id": "", "duration": 0.001508753, "size": 0, "status": 502, "resp_headers": {"Server": ["Caddy"], "Alt-Svc": ["h3=\":443\"; ma=2592000"]}}
mai 02 23:23:04 hostname caddy[1253]: 2025/05/02 21:23:04.261        DEBUG        http.handlers.reverse_proxy        selected upstream        {"dial": "localhost:8096", "total_upstreams": 1}
mai 02 23:23:04 hostname caddy[1253]: 2025/05/02 21:23:04.262        DEBUG        http.handlers.reverse_proxy        upstream roundtrip        {"upstream": "localhost:8096", "duration": 0.000596084, "request": {"remote_ip": "192.168.1.1", "remote_port": "33260", "client_ip": "192.168.1.1", "proto": "HTTP/2.0", "method": "GET", "host": "subdomain.domain.com", "uri": "/", "headers": {"Sec-Fetch-Site": ["none"], "Sec-Gpc": ["1"], "Sec-Fetch-Dest": ["document"], "Sec-Fetch-Mode": ["navigate"], "Te": ["trailers"], "Priority": ["u=0, i"], "X-Forwarded-Proto": ["https"], "Cache-Control": ["no-cache"], "User-Agent": ["Mozilla/5.0 (X11; Linux x86_64; rv:137.0) Gecko/20100101 Firefox/137.0"], "Dnt": ["1"], "Accept-Encoding": ["gzip, deflate, br, zstd"], "Pragma": ["no-cache"], "Accept-Language": ["fr"], "X-Forwarded-Host": ["subdomain.domain.com"], "Upgrade-Insecure-Requests": ["1"], "Accept": ["text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8"], "Sec-Fetch-User": ["?1"], "X-Forwarded-For": ["192.168.1.1"]}, "tls": {"resumed": true, "version": 772, "cipher_suite": 4867, "proto": "h2", "server_name": "subdomain.domain.com"}}, "error": "read tcp [::1]:58816->[::1]:8096: read: connection reset by peer"}

When podman is bound on 127.0.0.1:
ss output:

sudo ss -lp | grep 8096
tcp   LISTEN 0      128                                          127.0.0.1:8096                      0.0.0.0:*    users:(("pasta",pid=20407,fd=6))

No error in that case (traffic is forwarded to 127.0.0.1 as expected).

3. Caddy version:

v2.8.4

4. How I installed and ran Caddy:

Caddy is installed from the package manager. It runs as a systemd service.

a. System environment:

Fedora 41

b. Command:

caddy run --environ --config /etc/caddy/caddy.json

c. Service/unit/compose file:

# caddy.service
#
# For using Caddy with a config file.
#
# WARNING: This service does not use the --resume flag, so if you
# use the API to make changes, they will be overwritten by the
# Caddyfile next time the service is restarted. If you intend to
# use Caddy's API to configure it, add the --resume flag to the
# `caddy run` command or use the caddy-api.service file instead.

[Unit]
Description=Caddy web server
Documentation=https://caddyserver.com/docs/
After=network.target

[Service]
Type=notify
User=caddy
Group=caddy
ExecStartPre=/usr/bin/caddy validate --config /etc/caddy/caddy.json
ExecStart=/usr/bin/caddy run --environ --config /etc/caddy/caddy.json
ExecReload=/usr/bin/caddy reload --config /etc/caddy/caddy.json
TimeoutStopSec=5s
LimitNOFILE=1048576
PrivateTmp=true
ProtectHome=true
ProtectSystem=full
AmbientCapabilities=CAP_NET_ADMIN CAP_NET_BIND_SERVICE

[Install]
WantedBy=multi-user.target

d. My complete Caddy config:

{
	"logging": {
		"logs": {
			"default": {
				"level": "DEBUG"
			},
			"blabla": {
				"writer": {
					"filename": "/var/log/caddy/blabla1.log",
					"output": "file",
					"roll_local_time": true
				},
				"include": [
					"http.log.access.blabla1",
					"http.log.error.blabla1"
				]
			},
			"blabla": {
				"writer": {
					"filename": "/var/log/caddy/blabla2.log",
					"output": "file",
					"roll_local_time": true
				},
				"include": [
					"http.log.access.blabla2",
					"http.log.error.blabla2"
				]
			},
			"blabla": {
				"writer": {
					"filename": "/var/log/caddy/blabla3.log",
					"output": "file",
					"roll_local_time": true
				},
				"include": [
					"http.log.access.blabla3",
					"http.log.error.blabla3"
				]
			}
		}
	},
	"apps": {
		"http": {
			"servers": {
				"srv0": {
					"listen": [
						":443"
					],
					"routes": [
						{
							"handle": [
								{
									"handler": "subroute",
									"routes": [
										{
											"handle": [
												{
													"handler": "reverse_proxy",
													"upstreams": [
														{
															"dial": "localhost:8096"
														}
													]
												}
											]
										}
									]
								}
							],
							"match": [
								{
									"host": [
										"blabla.domain1.duckdns.org"
									]
								},
								{
									"host": [
										"blabla.domain1.net"
									]
								}
							],
							"terminal": true
						},
						{
							"handle": [
								{
									"handler": "subroute",
									"routes": [
										{
											"handle": [
												{
													"handler": "reverse_proxy",
													"upstreams": [
														{
															"dial": "localhost:8080"
														}
													]
												}
											]
										}
									]
								}
							],
							"match": [
								{
									"host": [
										"blabla.domain1.duckdns.org"
									]
								},
								{
									"host": [
										"blabla.domain1.net"
									]
								}
							],
							"terminal": true
						},
						{
							"match": [
								{
									"host": [
										"blabla.domain1.duckdns.org"
									]
								},
								{
									"host": [
										"blabla.domain1.net"
									]
								},
								{
									"host": [
										"fichiers.domain2.fr"
									]
								}
							],
							"handle": [
								{
									"handler": "file_server",
									"browse": {},
									"root": "/home/user/Public/caddy"
								}
							],
							"terminal": true
						}
					],
					"logs": {
						"logger_names": {
							"blabla.domain1.net": "blabla1",
							"blabla.domain1.duckdns.org": "blabla1",
							"blabla.domain1.net": "blabla2",
							"blabla.domain1.duckdns.org": "blabla2",
							"blabla.domain1.net": "blabla3",
							"blabla.domain1.duckdns.org": "blabla3",
							"fichiers.domain2.fr": "blabla3"
						},
						"should_log_credentials": true
					}
				}
			}
		},
		"tls": {
			"certificates": {
				"automate": [
					"blabla1.domain1.duckdns.org",
					"blabla2.domain1.duckdns.org",
					"blabla3.domain1.duckdns.org",
					"blabla1.domain1.net",
					"blabla2.domain1.net",
					"blabla3.domain1.net",
					"fichiers.domain2.fr"
				]
			}
		}
	}
}

5. Links to relevant resources:

It’s not Caddy who decides. It’s whatever the DNS resolver or the operating system gives.

2 Likes

There’s no dns resolving involved here. My question is about the call to either [::1] or 127.0.0.1.

There’s one from localhost to either [::1] or 127.0.0.1. In any way, that is not something Caddy decides. It gets it from the OS.

2 Likes

What does it request from the OS exactly ? The IP listening on port 8096 ?

It seems that the OS returns [::1] even though it is not listening; ss shows *:8096 as the listening IP range, which does not include [::1]:8096 if I understand correctly.

Caddy asks the OS (technically the DNS resolver of the network, which likely starts with the OS) to translate the given host name to an IP address, regardless of whether there’s someone listening on the subject port or not. I don’t know why Podman fails to bind the IPv6 correctly to the upstream address, but this is not something Caddy controls. You’ll have to check why Podman does what it does.

2 Likes

I think it actually succeeds in binding IPv6 but Jellyfin doesn’t answer to IPv6 requests.

Is it possible to restrict "dial": "localhost:8096" to IPv4 ?

You will have to do this through Podman, not Caddy. The other option is to use 127.0.0.1:8096 instead of localhost:8096.

Caddy requests the ip addresses belonging to an hostname via getaddrinfo (via /run/systemd/resolve/io.systemd.Resolve), this will return IP6 ::1 and IP4 127.0.0.1.

Caddy tries to connect to the first IP address. If it returns a failure during the connect call, it tries the second one. (verified this using strace -f caddy ...)

The problem in your case is that the connect call succeeds, so Caddy start sending the body, but then your remote server (podman) closes the connection with a reset (likely because podman tried to connect using IPv6 to your container, but failed, so it closed the connection directly after it accepted it)

Fix podman to not listen on IPv6 if it cannot proxy calls on IPv6

EDIT: If podman acts like docker, if the container has access to IPv6, make sure that the program in the container is listing on :: instead of 0.0.0.0

2 Likes