1. The problem I’m having:
I am working on setting up a matrix homeserver using NixOS modules, and it appears that once everything is set up the caddy reverse proxy attempts to make requests to invalid ip addresses like 0.0.30.200:80 and I cannot determine where such ip addresses are coming from, since they are not present in the config.
2. Error messages and/or full log output:
Mar 10 08:47:09 nixos-matrix-homeserver-testing caddy[19307]: {"level":"error","ts":1773146829.1132128,"logger":"http.log.error.log0","msg":"dial tcp 0.0.24.23:80: i/o timeout","request":{"remote_ip":"173.49.123.17","remote_port":"54876","client_ip":"173.49.123.17","proto":"HTTP/2.0","method":"GET","host":"matrix2.srasu.org","uri":"/","headers":{"Sec-Fetch-Site":["none"],"Priority":["u=0, i"],"Te":["trailers"],"Accept-Language":["en-US,en;q=0.9"],"Sec-Fetch-Mode":["navigate"],"Accept-Encoding":["gzip, deflate, br, zstd"],"Upgrade-Insecure-Requests":["1"],"Sec-Fetch-Dest":["document"],"User-Agent":["Mozilla/5.0 (X11; Linux x86_64; rv:148.0) Gecko/20100101 Firefox/148.0"],"Accept":["text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8"]},"tls":{"resumed":false,"version":772,"cipher_suite":4865,"proto":"h2","server_name":"matrix2.srasu.org","ech":false}},"duration":3.000683803,"status":502,"err_id":"mhdgdpduf","err_trace":"reverseproxy.statusError (reverseproxy.go:1473)"}
Mar 10 09:21:37 nixos-matrix-homeserver-testing caddy[19307]: {"level":"error","ts":1773148897.452203,"logger":"http.log.error.log0","msg":"dial tcp 0.0.24.23:80: i/o timeout","request":{"remote_ip":"16.144.17.106","remote_port":"37658","client_ip":"16.144.17.106","proto":"HTTP/1.1","method":"GET","host":"matrix2.srasu.org","uri":"/","headers":{"Connection":["close"],"User-Agent":["Mozilla/5.0 (Android; Linux armv7l; rv:10.0.1) Gecko/20100101 Firefox/10.0.1 Fennec/10.0.1"],"Accept-Charset":["utf-8"],"Accept-Encoding":["gzip"]},"tls":{"resumed":false,"version":772,"cipher_suite":4865,"proto":"","server_name":"matrix2.srasu.org","ech":false}},"duration":3.001503004,"status":502,"err_id":"svnsqvnge","err_trace":"reverseproxy.statusError (reverseproxy.go:1473)"}
Mar 10 09:37:49 nixos-matrix-homeserver-testing caddy[19307]: {"level":"error","ts":1773149869.5751345,"logger":"http.log.error.log1","msg":"dial tcp 0.0.30.200:80: i/o timeout","request":{"remote_ip":"34.67.39.62","remote_port":"57208","client_ip":"34.67.39.62","proto":"HTTP/2.0","method":"GET","host":"sfu.matrix2.srasu.org","uri":"/.env","headers":{"Te":["trailers"],"Accept-Language":["en-US,en;q=0.5"],"Sec-Fetch-Site":["none"],"Accept":["text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8"],"Priority":["u=0, i"],"User-Agent":["Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:135.0) Gecko/20100101 Firefox/135.0"],"Upgrade-Insecure-Requests":["1"],"Sec-Fetch-Dest":["document"],"Accept-Encoding":["gzip, deflate, br, zstd"],"Sec-Fetch-Mode":["navigate"],"Sec-Fetch-User":["?1"]},"tls":{"resumed":false,"version":772,"cipher_suite":4865,"proto":"h2","server_name":"sfu.matrix2.srasu.org","ech":false}},"duration":3.000765468,"status":502,"err_id":"p18bwchua","err_trace":"reverseproxy.statusError (reverseproxy.go:1473)"}
3. Caddy version:
2.11.1
4. How I installed and ran Caddy:
I installed caddy by way of the NixOS module.
a. System environment:
NixOS 26.05 (unstable), x86_64-linux. The NixOS module installs caddy via a systemd service.
b. Command:
/nix/store/0y8w75a33h8qxxmg5jglxk0kvibcgx4p-caddy-2.11.1/bin/caddy run --environ --config /etc/caddy/Caddyfile
c. Service/unit/compose file:
# section of the nixos configuration relevant to caddy
services.caddy = {
enable = true;
openFirewall = true;
virtualHosts = {
"matrix2.srasu.org" = {
extraConfig = ''
reverse_proxy 6167
'';
};
"sfu.matrix2.srasu.org" = {
extraConfig = ''
@jwt_service {
path /sfu/get* /heathz*
}
handle @jwt_service {
reverse_proxy 8081
}
handle {
reverse_proxy {
to 7880
header_up Connection "upgrade"
header_up Upgrade "{http.request.header.Upgrade}"
}
}
'';
};
};
};
# caddy.service
#
# For using Caddy with a config file.
#
# Make sure the ExecStart and ExecReload commands are correct
# for your installation.
#
# See https://caddyserver.com/docs/install for instructions.
#
# WARNING: This service does not use the --resume flag, so if you
# use the API to make changes, they will be overwritten by the
# Caddyfile next time the service is restarted. If you intend to
# use Caddy's API to configure it, add the --resume flag to the
# `caddy run` command or use the caddy-api.service file instead.
[Unit]
Description=Caddy
Documentation=https://caddyserver.com/docs/
After=network.target network-online.target
Requires=network-online.target
[Service]
Type=notify
User=caddy
Group=caddy
ExecStart=/nix/store/0y8w75a33h8qxxmg5jglxk0kvibcgx4p-caddy-2.11.1/bin/caddy run --environ --config /etc/caddy/Caddyfile
ExecReload=/nix/store/0y8w75a33h8qxxmg5jglxk0kvibcgx4p-caddy-2.11.1/bin/caddy reload --config /etc/caddy/Caddyfile --force
TimeoutStopSec=5s
LimitNOFILE=1048576
PrivateTmp=true
ProtectSystem=full
AmbientCapabilities=CAP_NET_ADMIN CAP_NET_BIND_SERVICE
[Install]
WantedBy=multi-user.target
d. My complete Caddy config:
This config is auto-generated from the nixos module listed above.
{
log {
level ERROR
}
}
matrix2.srasu.org {
log {
output file /var/log/caddy/access-matrix2.srasu.org.log
}
reverse_proxy 6167
}
sfu.matrix2.srasu.org {
log {
output file /var/log/caddy/access-sfu.matrix2.srasu.org.log
}
@jwt_service {
path /sfu/get\* /heathz\*
}
handle @jwt_service {
reverse_proxy 8081
}
handle {
reverse_proxy {
to 7880
header_up Connection "upgrade"
header_up Upgrade "{http.request.header.Upgrade}"
}
}
}