Random timeouts

Explain what you are trying to do

Run caddy

Issues

Caddy randomly times out. I haven’t managed to get patterns such as specific times or how long it times out for

Show what you have already tried

I’ve setup logging, errors, etc to all output to journald (since I’m managing caddy with systemd, all stdout output goes to journald by default)

    # logging
    log / stdout "{host},{remote},{user},[{when_iso}],\"{method} {uri} {proto}\",{status},{size},{latency},\"{>Referer}\",\"{>User-Agent}\""
    errors stdout {
        404 www/404.html
        418 www/418.html
    }

I’ve also ran tcpdumps when things timeout. There is no activity at all from my external IP (work) to caddy (home)

Other relevant information

DNS is cloudflare, NOT utilizing the cloudlfare proxy. Router is pfsense port-forwarding 80 and 443 TCP to a Virtual IP (VIP) manage by keepalived. Keepalived is managing a floating IP among multiple nodes. As far as I can tell, this doesn’t seem to be an issue with a floating IP, as caddy is bound to *:443, and when it times out, hosts I’ve defined in my hosts file to point to the static IP of a single node time out as well. There are no relevant logs in caddy (it just stops logging because no activity is happening, apparently). The service doesn’t fail, so there is no failover. ss -plnt shows that it is still bound and listening to interfaces. It will randomly come back and serve requests usually between 5 and 10 minutes, with no manual intervention. The issue is not a network issues - connections like RDP, SSH, ping/ICMP through the same network drop no packets and have no issues. The exact setup with traefik instead of caddy proxied/load-balanced the exact same sites with the same variables (IP, router, network, virtual hosts, etc) for months with no issues, timeouts, etc.

build info

caddy -version
Caddy 0.11.4 (+33b00dc Tue Feb 19 09:57:34 UTC 2019) (unofficial)
(manually compiled - official build server was broken)

caddy -plugins

Server types:
  http

Caddyfile loaders:
  short
  flag
  default

Other plugins:
  http.basicauth
  http.bind
  http.browse
  http.cors
  http.errors
  http.expvar
  http.ext
  http.fastcgi
  http.gzip
  http.header
  http.index
  http.internal
  http.limits
  http.log
  http.markdown
  http.mime
  http.minify
  http.pprof
  http.proxy
  http.proxyprotocol
  http.push
  http.realip
  http.redir
  http.request_id
  http.rewrite
  http.root
  http.s3browser
  http.status
  http.templates
  http.timeouts
  http.websocket
  on
  tls
  tls.cluster.file
  tls.dns.cloudflare

Hi @Gorian, welcome to the Caddy community. Thanks for all the info on this one.

I’m assuming your unit file is still this one here, as well: Want Caddy to run automatically on boot - #2 by Gorian

Can you pcap the HTTP ports during a period of downtime and see whether those packets are hitting the interface?

@Whitestrake thank you for replying! I think I might have found the issue and fixed it (simple dumb configuration error with clustering setup). Testing to see if that fixed it. …

1 Like

This topic was automatically closed 90 days after the last reply. New replies are no longer allowed.