Sometimes systemctl restart caddy yelds timeout

Help
1

1. The problem I’m having:

I’m writing a custom log within caddy and, in order to rotate it, I need to restart the daemon daily (which I have configured in a logrotate file using systemctl restart caddy.service).

However, every once in a while the start part times out (even when the daemon is starting) and systemd kills the recently started caddy.

If you check the log below, at 00:01:26 systemd starts stopping caddy, and by 00:01:29 it succeeds and starts starting caddy.

Caddy actually starts and by 00:01:31 it is already handling requests.

However, systemd doesn’t seem to acknowledge this and, by 00:02:59 (90 seconds after it started) it says that the start operation timed out and sends a SIGTERM. Even when caddy does acknowledge this and shuts down, systemd doesn’t detect this and sends a SIGKILL.

Looks like systemd is loosing track of the controlled caddy process.

2. Error messages and/or full log output:

Apr 03 00:01:25 ep-it-010 caddy[1006405]: {"level":"info","ts":1712113285.912979,"logger":"http.log.access","msg":"handled request","request":{"remote_ip":"51.222.253.1","remote_port":"31990","proto":"HTTP/1.1","method":"GET","host":"bibliotecadigital.cin.edu.ar","uri":"/simple-search?query=&sort_by=score&order=desc&rpp=10&filter_field_1=subject&filter_type_1=equals&filter_value_1=REVISTA&filter_field_2=subject&filter_type_2=equals&filter_value_2=CIENCIA&filter_field_3=subject&filter_type_3=equals&filter_value_3=ALIMENTOS&filter_field_4=subject&filter_type_4=equals&filter_value_4=CIN&filter_field_5=subject&filter_type_5=equals&filter_value_5=SALUD&etal=0&filtername=subject&filterquery=Conferencia+de+Rectores+de+las+Universidades+Espa%C3%B1olas+%28CRUE%29&filtertype=equals","headers":{"User-Agent":["Mozilla/5.0(compatible; AhrefsBot/7.0; +http://ahrefs.com/robot/)"],"Accept":["*/*"],"Accept-Encoding":["deflate, gzip, br"]}},"user_id":"" ,"duration":0.000102638,"size":0,"status":308,"resp_headers":{"Server":["Caddy"],"Connection":["close"],"Location":["https://bibliotecadigital.cin.edu.ar/simple-search?query=&sort_by=score&order=desc&rpp=10&filter_field_1=subject&filter_type_1=equals&filter_value_1=REVISTA&filter_field_2=subject&filter_type_2=equals&filter_value_2=CIENCIA&filter_field_3=subject&filter_type_3=equals&filter_value_3=ALIMENTOS&filter_field_4=subject&filter_type_4=equals&filter_value_4=CIN&filter_field_5=subject&filter_type_5=equals&filter_value_5=SALUD&etal=0&filtername=subject&filterquery=Conferencia+de+Rectores+de+las+Universidades+Espa%C3%B1olas+%28CRUE%29&filtertype=equals"],"Content-Type":[]}}
Apr 03 00:01:26 ep-it-010 caddy[1006405]: {"level":"info","ts":1712113286.5010617,"msg":"shutting down apps, then terminating","signal":"SIGTERM"}
Apr 03 00:01:26 ep-it-010 caddy[1006405]: {"level":"warn","ts":1712113286.5013669,"msg":"exiting; byeee!! <F0><9F><91><8B>","signal":"SIGTERM"}
Apr 03 00:01:26 ep-it-010 systemd[1]: Stopping Caddy...
Apr 03 00:01:26 ep-it-010 caddy[1006405]: {"level":"error","ts":1712113286.5397124,"logger":"http.log","msg":"setting HTTP/3 Alt-Svc header","error":"no port can be announced, specify it explicitly using Server.Port or Server.Addr"}
Apr 03 00:01:26 ep-it-010 caddy[1006405]: {"level":"error","ts":1712113286.6919575,"logger":"http.log","msg":"setting HTTP/3 Alt-Svc header","error":"no port can be announced, specify it explicitly using Server.Port or Server.Addr"}
Apr 03 00:01:26 ep-it-010 caddy[1006405]: {"level":"error","ts":1712113286.7689574,"logger":"http.log","msg":"setting HTTP/3 Alt-Svc header","error":"no port can be announced, specify it explicitly using Server.Port or Server.Addr"}
Apr 03 00:01:29 ep-it-010 caddy[1006405]: {"level":"error","ts":1712113289.2034917,"logger":"http.log.error.log5","msg":"dial tcp: lookup ep-wb-002.siu.edu.ar: i/o timeout","request":{"remote_ip":"85.208.96.205","remote_port":"37468","proto":"HTTP/1.1","method":"GET","host":"encuestas.siu.edu.ar","uri":"/aplicacion.php?ah=st6607021933b756.73731163&ai=kolla%7C%7C12000094","headers":{"Accept":["text/html, application/rss+xml, application/atom+xml, text/xml, text/rss+xml, application/xhtml+xml"],"Accept-Encoding":["gzip,deflate"],"Connection":["close"],"User-Agent":["Mozilla/5.0 (compatible; SemrushBot/7~bl; +http://www.semrush.com/bot.html)"]},"tls":{"resumed":false,"version":772,"cipher_suite":4867,"proto":"","server_name":"encuestas.siu.edu.ar"}},"duration":3.000732449,"status":502,"err_id":"u6jsd2hih","err_trace":"reverseproxy.statusError (reverseproxy.go:1299)"}
Apr 03 00:01:29 ep-it-010 caddy[1006405]: {"level":"info","ts":1712113289.6887517,"logger":"tls.cache.maintenance","msg":"stopped background certificate maintenance","cache":"0xc0001ecfc0"}
Apr 03 00:01:29 ep-it-010 caddy[1006405]: {"level":"info","ts":1712113289.6890624,"logger":"admin","msg":"stopped previous server","address":"localhost:2019"}
Apr 03 00:01:29 ep-it-010 caddy[1006405]: {"level":"info","ts":1712113289.6891007,"msg":"shutdown complete","signal":"SIGTERM","exit_code":0}
Apr 03 00:01:29 ep-it-010 systemd[1]: caddy.service: Succeeded.
Apr 03 00:01:29 ep-it-010 systemd[1]: Stopped Caddy.
Apr 03 00:01:29 ep-it-010 systemd[1]: caddy.service: Consumed 1h 28min 35.446s CPU time.
Apr 03 00:01:29 ep-it-010 systemd[1]: Starting Caddy...
Apr 03 00:01:29 ep-it-010 caddy[1015084]: caddy.HomeDir=/var/lib/caddy
Apr 03 00:01:29 ep-it-010 caddy[1015084]: caddy.AppDataDir=/var/lib/caddy/.local/share/caddy
Apr 03 00:01:29 ep-it-010 caddy[1015084]: caddy.AppConfigDir=/var/lib/caddy/.config/caddy
Apr 03 00:01:29 ep-it-010 caddy[1015084]: caddy.ConfigAutosavePath=/var/lib/caddy/.config/caddy/autosave.json
Apr 03 00:01:29 ep-it-010 caddy[1015084]: caddy.Version=v2.6.4 h1:2hwYqiRwk1tf3VruhMpLcYTg+11fCdr8S3jhNAdnPy8=
Apr 03 00:01:29 ep-it-010 caddy[1015084]: runtime.GOOS=linux
Apr 03 00:01:29 ep-it-010 caddy[1015084]: runtime.GOARCH=amd64
Apr 03 00:01:29 ep-it-010 caddy[1015084]: runtime.Compiler=gc
Apr 03 00:01:29 ep-it-010 caddy[1015084]: runtime.NumCPU=8
Apr 03 00:01:29 ep-it-010 caddy[1015084]: runtime.GOMAXPROCS=8
Apr 03 00:01:29 ep-it-010 caddy[1015084]: runtime.Version=go1.20.5
Apr 03 00:01:29 ep-it-010 caddy[1015084]: os.Getwd=/
Apr 03 00:01:29 ep-it-010 caddy[1015084]: LANG=en_US.UTF-8
Apr 03 00:01:29 ep-it-010 caddy[1015084]: PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
Apr 03 00:01:29 ep-it-010 caddy[1015084]: NOTIFY_SOCKET=/run/systemd/notify
Apr 03 00:01:29 ep-it-010 caddy[1015084]: HOME=/var/lib/caddy
Apr 03 00:01:29 ep-it-010 caddy[1015084]: LOGNAME=caddy
Apr 03 00:01:29 ep-it-010 caddy[1015084]: USER=caddy
Apr 03 00:01:29 ep-it-010 caddy[1015084]: INVOCATION_ID=117630af1d70495c9f6c5987385707db
Apr 03 00:01:29 ep-it-010 caddy[1015084]: JOURNAL_STREAM=8:155783553
Apr 03 00:01:29 ep-it-010 caddy[1015084]: {"level":"info","ts":1712113289.8104048,"msg":"using provided configuration","config_file":"/etc/caddy/Caddyfile","config_adapter":""}
Apr 03 00:01:29 ep-it-010 caddy[1015084]: {"level":"warn","ts":1712113289.8803763,"msg":"Caddyfile input is not formatted; run the 'caddy fmt' command to fix inconsistencies","adapter":"caddyfile","file":"/etc/caddy/Caddyfile","line":10}
Apr 03 00:01:29 ep-it-010 caddy[1015084]: {"level":"info","ts":1712113289.9338868,"logger":"admin","msg":"admin endpoint started","address":"localhost:2019","enforce_origin":false,"origins":["//127.0.0.1:2019","//localhost:2019","//[::1]:2019"]}
Apr 03 00:01:29 ep-it-010 caddy[1015084]: {"level":"info","ts":1712113289.9374,"logger":"tls.cache.maintenance","msg":"started background certificate maintenance","cache":"0xc00039ee00"}
Apr 03 00:01:29 ep-it-010 caddy[1015084]: {"level":"info","ts":1712113289.9402037,"logger":"http","msg":"enabling automatic HTTP->HTTPS redirects","server_name":"srv0"}
Apr 03 00:01:29 ep-it-010 caddy[1015084]: {"level":"warn","ts":1712113289.9404554,"logger":"http","msg":"server is listening only on the HTTP port, so no automatic HTTPS will be applied to this server","server_name":"srv1","http_port":80}
Apr 03 00:01:30 ep-it-010 caddy[1015084]: {"level":"warn","ts":1712113290.000977,"logger":"pki.ca.local","msg":"installing root certificate (you might be prompted for password)","path":"storage:pki/authorities/local/root.crt"}
Apr 03 00:01:30 ep-it-010 caddy[1015084]: {"level":"info","ts":1712113290.0014095,"msg":"warning: \"certutil\" is not available, install \"certutil\" with \"apt install libnss3-tools\" or \"yum install nss-tools\" and try again"}
Apr 03 00:01:30 ep-it-010 caddy[1015084]: {"level":"info","ts":1712113290.001463,"msg":"define JAVA_HOME environment variable to use the Java trust"}
Apr 03 00:01:30 ep-it-010 caddy[1015084]: {"level":"error","ts":1712113290.0069468,"logger":"pki.ca.local","msg":"failed to install root certificate","error":"failed to execute sudo: exit status 1","certificate_file":"storage:pki/authorities/local/root.crt"}
Apr 03 00:01:30 ep-it-010 caddy[1015084]: {"level":"info","ts":1712113290.0074062,"logger":"tls","msg":"cleaning storage unit","description":"FileStorage:/var/lib/caddy/.local/share/caddy"}
Apr 03 00:01:30 ep-it-010 caddy[1015084]: {"level":"info","ts":1712113290.0075202,"logger":"http","msg":"enabling HTTP/3 listener","addr":":443"}
Apr 03 00:01:30 ep-it-010 caddy[1015084]: {"level":"info","ts":1712113290.0078375,"msg":"failed to sufficiently increase receive buffer size (was: 208 kiB, wanted: 2048 kiB, got: 416 kiB). See https://github.com/quic-go/quic-go/wiki/UDP-Receive-Buffer-Size for details."}
Apr 03 00:01:30 ep-it-010 caddy[1015084]: {"level":"info","ts":1712113290.0081155,"logger":"http.log","msg":"server running","name":"srv0","protocols":["h1","h2","h3"]}
Apr 03 00:01:30 ep-it-010 caddy[1015084]: {"level":"info","ts":1712113290.0083115,"logger":"http.log","msg":"server running","name":"srv1","protocols":["h1","h2","h3"]}
Apr 03 00:01:30 ep-it-010 caddy[1015084]: {"level":"info","ts":1712113290.0083733,"logger":"http","msg":"enabling automatic TLS certificate management","domains":["idear.siu.edu.ar","marcademo.siu.edu.ar","planves.siu.edu.ar","simulacionclinica.siu.edu.ar","desarrollouniv.siu.edu.ar","gestorinformes.siu.edu.ar","informacion.siu.edu.ar","sinergia.siu.edu.ar","posgradosasociados.siu.edu.ar","piesci.siu.edu.ar","abrealas.siu.edu.ar","mapadeproyectoscsu.siu.edu.ar","informacion.cin.edu.ar","contextoencierro.siu.edu.ar","portal.comunidad.siu.edu.ar","fortalecimientoext.siu.edu.ar","proyectocin.siu.edu.ar","guaranivirtual.siu.edu.ar","sied.siu.edu.ar","arfagri.siu.edu.ar","perhid-becas.siu.edu.ar","promce.siu.edu.ar","doctorados.siu.edu.ar","proyectos.marca.siu.edu.ar","proav.siu.edu.ar","esps.siu.edu.ar","kiwi.siu.edu.ar","conveniosfrance.siu.edu.ar","joomlahuarpe.siu.edu.ar","odsuniversidades.siu.edu.ar","simulacionodonto.siu.edu.ar","evc.siu.edu.ar","voluntariado.siu.edu.ar","generoydiversidad.siu.edu.ar","ies.siu.edu.ar","calchaquitest.siu.edu.ar","cytuniversidades.siu.edu.ar","calidaduniversitaria.siu.edu.ar","cuaa-dahz.siu.edu.ar","precalchaqui.siu.edu.ar","idear2017.siu.edu.ar","px00.siu.edu.ar","csuencarceles.siu.edu.ar","spusceu.siu.edu.ar","ppuaadmin.siu.edu.ar","preinscripcion.guarani.siu.edu.ar","info.siu.edu.ar","chat.siu.edu.ar","formarte.siu.edu.ar","semilla2017.siu.edu.ar","mapavoluntariado.siu.edu.ar","mundou.siu.edu.ar","unagranlibertadmas.siu.edu.ar","ofertasuperior.siu.edu.ar","demogestor.siu.edu.ar","disenioydesarrollo.siu.edu.ar","xn--diseoydesarrollo-9tb.siu.edu.ar","proun.siu.edu.ar","ed-wb-002-forodocker.siu.edu.ar","practicaseducativas.siu.edu.ar","cargaconveniosfrance.siu.edu.ar","estudiarenargentina.siu.edu.ar","rtfsimulador.siu.edu.ar","eei-desarrollo.siu.edu.ar","sipes.siu.edu.ar","pueep.siu.edu.ar","csucontextosdeprivacion.siu.edu.ar","200.123.128.68","compromisosocialuniv.siu.edu.ar","culturaysociedad.siu.edu.ar","laboratoriosremotos.siu.edu.ar","nube.cin.edu.ar","universidadydesarrollo.siu.edu.ar","forocultural.siu.edu.ar","autogestion.guarani.siu.edu.ar","prombiogen.siu.edu.ar","idear2014.siu.edu.ar","agregandovalor.siu.edu.ar","snraregistro.siu.edu.ar","perhid-becas.cin.edu.ar","protocolos2019.siu.edu.ar","registrograduados.siu.edu.ar","usc.siu.edu.ar","cienciaytecnologia.siu.edu.ar","bancodeevaluadores.cin.edu.ar","siu.edu.ar","siac.siu.edu.ar","disenoydesarrollo.siu.edu.ar","graduadosuniversitarios.siu.edu.ar","conveniospu.siu.edu.ar","nochedelareforma.siu.edu.ar","fortalecimientoosc.siu.edu.ar","111mil.siu.edu.ar","sicerconsulta.siu.edu.ar","marca.siu.edu.ar","csu.siu.edu.ar","cpres.siu.edu.ar","sacau.siu.edu.ar","gestion.guarani.siu.edu.ar","universocial.siu.edu.ar","doblecarrera.siu.edu.ar","calchaqui.siu.edu.ar","www.siu.edu.ar","encuestas.siu.edu.ar","guarani.siu.edu.ar","nomina.siu.edu.ar","cafp.siu.edu.ar","postgradosasociados.siu.edu.ar","extension.siu.edu.ar","innovacionsocial.siu.edu.ar","sirvat.siu.edu.ar","infraestructura.siu.edu.ar","172.27.100.240","redesemprendedoras.siu.edu.ar","sigamosestudiando.siu.edu.ar","inscripciones.siu.edu.ar","perhid-redes.siu.edu.ar","kollaspu.siu.edu.ar","querandies.siu.edu.ar","moodlevinculacion.siu.edu.ar","csucontextosvulnerables.siu.edu.ar","genero.siu.edu.ar","infosiu.siu.edu.ar","titulosoficiales.siu.edu.ar","pedidosciie.siu.edu.ar","api-app.guarani.siu.edu.ar","kollacin.siu.edu.ar","demo.guarani.siu.edu.ar","bibliotecadigital.cin.edu.ar","toba-usuarios.guarani.siu.edu.ar","fortalecimientovt.siu.edu.ar","normativa.cin.edu.ar","acompaniarsinide.siu.edu.ar","ppua.siu.edu.ar","studyinargentina.siu.edu.ar","spupedidos.siu.edu.ar","emprendedorismo.siu.edu.ar"]}
Apr 03 00:01:30 ep-it-010 caddy[1015084]: {"level":"info","ts":1712113290.0885773,"logger":"http.log.access","msg":"handled request","request":{"remote_ip":"51.222.253.4","remote_port":"21964","proto":"HTTP/1.1","method":"GET","host":"bibliotecadigital.cin.edu.ar","uri":"/simple-search?query=&sort_by=score&order=desc&rpp=10&filter_field_1=subject&filter_type_1=equals&filter_value_1=EDUCACI%C3%93N&filter_field_2=author&filter_type_2=equals&filter_value_2=Garc%C3%ADa+de+Fanelli%2C+Ana+Mar%C3%ADa&filter_field_3=subject&filter_type_3=equals&filter_value_3=CALIDAD+DE+LA+EDUCACI%C3%93N&filter_field_4=subject&filter_type_4=equals&filter_value_4=EMISION+DE+TITULOS&filter_field_5=subject&filter_type_5=equals&filter_value_5=ENSENANZA+EN+EL+EXTRANJERO&etal=0&filtername=subject&filterquery=ESTUDIOS+DE+POSTGRADO&filtertype=equals","headers":{"User-Agent":["Mozilla/5.0 (compatible; AhrefsBot/7.0; +http://ahrefs.com/robot/)"],"Accept":["*/*"],"Accept-Encoding":["deflate, gzip, br"]}},"user_id":"","duration":0.000122208,"size":0,"status":308,"resp_headers":{"Content-Type":[],"Server":["Caddy"],"Connection":["close"],"Location":["https://bibliotecadigital.cin.edu.ar/simple-search?query=&sort_by=score&order=desc&rpp=10&filter_field_1=subject&filter_type_1=equals&filter_value_1=EDUCACI%C3%93N&filter_field_2=author&filter_type_2=equals&filter_value_2=Garc%C3%ADa+de+Fanelli%2C+Ana+Mar%C3%ADa&filter_field_3=subject&filter_type_3=equals&filter_value_3=CALIDAD+DE+LA+EDUCACI%C3%93N&filter_field_4=subject&filter_type_4=equals&filter_value_4=EMISION+DE+TITULOS&filter_field_5=subject&filter_type_5=equals&filter_value_5=ENSENANZA+EN+EL+EXTRANJERO&etal=0&filtername=subject&filterquery=ESTUDIOS+DE+POSTGRADO&filtertype=equals"]}}
Apr 03 00:01:30 ep-it-010 caddy[1015084]: {"level":"info","ts":1712113290.1191974,"logger":"tls","msg":"finished cleaning storage units"}
Apr 03 00:01:31 ep-it-010 caddy[1015084]: {"level":"info","ts":1712113291.9665122,"logger":"http.log.access","msg":"handled request","request":{"remote_ip":"51.222.253.16","remote_port":"31214","proto":"HTTP/1.1","method":"GET","host":"bibliotecadigital.cin.edu.ar","uri":"/simple-search?query=&sort_by=score&order=desc&rpp=10&filter_field_1=subject&filter_type_1=equals&filter_value_1=EDUCACI%C3%93N&filter_field_2=author&filter_type_2=equals&filter_value_2=Labont%C3%A9%2C+Huguette&filter_field_3=dateIssued&filter_type_3=equals&filter_value_3=1993&filter_field_4=subject&filter_type_4=equals&filter_value_4=COSTA+RICA&filter_field_5=subject&filter_type_5=equals&filter_value_5=ARGENTINA&filter_field_6=subject&filter_type_6=equals&filter_value_6=AM%C3%89RICA+DEL+NORTE&etal=0&filtername=subject&filterquery=ECUADOR&filtertype=equals","headers":{"User-Agent":["Mozilla/5.0 (compatible; AhrefsBot/7.0; +http://ahrefs.com/robot/)"],"Accept":["*/*"],"Accept-Encoding":["deflate, gzip, br"]}},"user_id":"","duration":0.000106944,"size":0,"status":308,"resp_headers":{"Server":["Caddy"],"Connection":["close"],"Location":["https://bibliotecadigital.cin.edu.ar/simple-search?query=&sort_by=score&order=desc&rpp=10&filter_field_1=subject&filter_type_1=equals&filter_value_1=EDUCACI%C3%93N&filter_field_2=author&filter_type_2=equals&filter_value_2=Labont%C3%A9%2C+Huguette&filter_field_3=dateIssued&filter_type_3=equals&filter_value_3=1993&filter_field_4=subject&filter_type_4=equals&filter_value_4=COSTA+RICA&filter_field_5=subject&filter_type_5=equals&filter_value_5=ARGENTINA&filter_field_6=subject&filter_type_6=equals&filter_value_6=AM%C3%89RICA+DEL+NORTE&etal=0&filtername=subject&filterquery=ECUADOR&filtertype=equals"],"Content-Type":[]}}
<SNIP>
Apr 03 00:02:56 ep-it-010 caddy[1015084]: {"level":"info","ts":1712113376.6805432,"logger":"http.log.access","msg":"handled request","request":{"remote_ip":"51.222.253.13","remote_port":"51438","proto":"HTTP/1.1","method":"GET","host":"bibliotecadigital.cin.edu.ar","uri":"/handle/123456789/2198/simple-search?query=&sort_by=score&order=desc&rpp=10&filter_field_1=author&filter_type_1=equals&filter_value_1=Correa%2C+Mar%C3%ADa+Laura&filter_field_2=author&filter_type_2=equals&filter_value_2=Higa%2C+Diego&filter_field_3=author&filter_type_3=equals&filter_value_3=Morales%2C+Mariana&filter_field_4=subject&filter_type_4=equals&filter_value_4=SOFTWARE+PARA+UNIVERSIDADES&filter_field_5=subject&filter_type_5=equals&filter_value_5=INTEROPERABILIDAD&etal=0&filtername=dateIssued&filterquery=2016&filtertype=equals","headers":{"Accept-Encoding":["deflate, gzip, br"],"User-Agent":["Mozilla/5.0 (compatible; AhrefsBot/7.0; +http://ahrefs.com/robot/)"],"Accept":["*/*"]}},"user_id":"","duration":0.000101577,"size":0,"status":308,"resp_headers":{"Server":["Caddy"],"Connection":["close"],"Location":["https://bibliotecadigital.cin.edu.ar/handle/123456789/2198/simple-search?query=&sort_by=score&order=desc&rpp=10&filter_field_1=author&filter_type_1=equals&filter_value_1=Correa%2C+Mar%C3%ADa+Laura&filter_field_2=author&filter_type_2=equals&filter_value_2=Higa%2C+Diego&filter_field_3=author&filter_type_3=equals&filter_value_3=Morales%2C+Mariana&filter_field_4=subject&filter_type_4=equals&filter_value_4=SOFTWARE+PARA+UNIVERSIDADES&filter_field_5=subject&filter_type_5=equals&filter_value_5=INTEROPERABILIDAD&etal=0&filtername=dateIssued&filterquery=2016&filtertype=equals"],"Content-Type":[]}}Apr 03 00:02:58 ep-it-010 caddy[1015084]: {"level":"info","ts":1712113378.2061694,"logger":"http.log.access","msg":"handled request","request":{"remote_ip":"51.222.253.10","remote_port":"48444","proto":"HTTP/1.1","method":"GET","host":"bibliotecadigital.cin.edu.ar","uri":"/simple-search?query=&sort_by=score&order=desc&rpp=10&filter_field_1=author&filter_type_1=equals&filter_value_1=Lafforgue+%28dir.%29&filter_field_2=subject&filter_type_2=equals&filter_value_2=ACREDITACI%C3%93N+INSTITUCIONES&filter_field_3=subject&filter_type_3=equals&filter_value_3=CALIDAD+DE+LA+EDUCACI%C3%93N&etal=0&filtername=subject&filterquery=DOCENCIA&filtertype=equals","headers":{"Accept-Encoding":["deflate, gzip, br"],"User-Agent":["Mozilla/5.0 (compatible; AhrefsBot/7.0; +http://ahrefs.com/robot/)"],"Accept":["*/*"]}},"user_id":"","duration":0.000096715,"size":0,"status":308,"resp_headers":{"Server":["Caddy"],"Connection":["close"],"Location":["https://bibliotecadigital.cin.edu.ar/simple-search?query=&sort_by=score&order=desc&rpp=10&filter_field_1=author&filter_type_1=equals&filter_value_1=Lafforgue+%28dir.%29&filter_field_2=subject&filter_type_2=equals&filter_value_2=ACREDITACI%C3%93N+INSTITUCIONES&filter_field_3=subject&filter_type_3=equals&filter_value_3=CALIDAD+DE+LA+EDUCACI%C3%93N&etal=0&filtername=subject&filterquery=DOCENCIA&filtertype=equals"],"Content-Type":[]}}
Apr 03 00:02:59 ep-it-010 systemd[1]: caddy.service: start operation timed out. Terminating.
Apr 03 00:02:59 ep-it-010 caddy[1015084]: {"level":"info","ts":1712113379.8983665,"msg":"shutting down apps, then terminating","signal":"SIGTERM"}
Apr 03 00:02:59 ep-it-010 caddy[1015084]: {"level":"warn","ts":1712113379.898576,"msg":"exiting; byeee!! <F0><9F><91><8B>","signal":"SIGTERM"}
Apr 03 00:03:00 ep-it-010 caddy[1015084]: {"level":"info","ts":1712113380.0814826,"logger":"http.log.access","msg":"handled request","request":{"remote_ip":"51.222.253.20","remote_port":"45574","proto":"HTTP/1.1","method":"GET","host":"bibliotecadigital.cin.edu.ar","uri":"/simple-search?query=&sort_by=score&order=desc&rpp=10&filter_field_1=author&filter_type_1=equals&filter_value_1=Labont%C3%A9%2C+Huguette&filter_field_2=dateIssued&filter_type_2=equals&filter_value_2=1997&filter_field_3=subject&filter_type_3=equals&filter_value_3=AM%C3%89RICA+CENTRAL&filter_field_4=subject&filter_type_4=equals&filter_value_4=COSTA+RICA&etal=0&filtername=subject&filterquery=AM%C3%89RICA+LATINA&filtertype=equals","headers":{"Accept":["*/*"],"Accept-Encoding":["deflate, gzip, br"],"User-Agent":["Mozilla/5.0 (compatible; AhrefsBot/7.0; +http://ahrefs.com/robot/)"]}},"user_id":"","duration":0.000098032,"size":0,"status":308,"resp_headers":{"Server":["Caddy"],"Connection":["close"],"Location":["https://bibliotecadigital.cin.edu.ar/simple-search?query=&sort_by=score&order=desc&rpp=10&filter_field_1=author&filter_type_1=equals&filter_value_1=Labont%C3%A9%2C+Huguette&filter_field_2=dateIssued&filter_type_2=equals&filter_value_2=1997&filter_field_3=subject&filter_type_3=equals&filter_value_3=AM%C3%89RICA+CENTRAL&filter_field_4=subject&filter_type_4=equals&filter_value_4=COSTA+RICA&etal=0&filtername=subject&filterquery=AM%C3%89RICA+LATINA&filtertype=equals"],"Content-Type":[]}}
<SNIP>
Apr 03 00:03:03 ep-it-010 caddy[1015084]: {"level":"info","ts":1712113383.4083805,"logger":"http.log.access","msg":"handled request","request":{"remote_ip":"51.222.253.1","remote_port":"34018","proto":"HTTP/1.1","method":"GET","host":"bibliotecadigital.cin.edu.ar","uri":"/simple-search?query=&sort_by=score&order=desc&rpp=10&filter_field_1=subject&filter_type_1=equals&filter_value_1=ENSE%C3%91ANZA+SUPERIOR&filter_field_2=dateIssued&filter_type_2=equals&filter_value_2=%5B1970+TO+1979%5D&filter_field_3=author&filter_type_3=equals&filter_value_3=Cea%2C+Rosa+Marta+de&filter_field_4=subject&filter_type_4=equals&filter_value_4=EDUCACI%C3%93N&etal=0&filtername=subject&filterquery=CREACI%C3%93N+DE+UNIVERSIDADES&filtertype=equals","headers":{"Accept-Encoding":["deflate, gzip, br"],"User-Agent":["Mozilla/5.0 (compatible; AhrefsBot/7.0; +http://ahrefs.com/robot/)"],"Accept":["*/*"]}},"user_id":"","duration":0.000105358,"size":0,"status":308,"resp_headers":{"Connection":["close"],"Location":["https://bibliotecadigital.cin.edu.ar/simple-search?query=&sort_by=score&order=desc&rpp=10&filter_field_1=subject&filter_type_1=equals&filter_value_1=ENSE%C3%91ANZA+SUPERIOR&filter_field_2=dateIssued&filter_type_2=equals&filter_value_2=%5B1970+TO+1979%5D&filter_field_3=author&filter_type_3=equals&filter_value_3=Cea%2C+Rosa+Marta+de&filter_field_4=subject&filter_type_4=equals&filter_value_4=EDUCACI%C3%93N&etal=0&filtername=subject&filterquery=CREACI%C3%93N+DE+UNIVERSIDADES&filtertype=equals"],"Content-Type":[],"Server":["Caddy"]}}
Apr 03 00:03:05 ep-it-010 systemd[1]: caddy.service: State 'stop-sigterm' timed out. Killing.
Apr 03 00:03:05 ep-it-010 systemd[1]: caddy.service: Killing process 1015084 (caddy) with signal SIGKILL.
Apr 03 00:03:05 ep-it-010 systemd[1]: caddy.service: Killing process 1015086 (caddy) with signal SIGKILL.
Apr 03 00:03:05 ep-it-010 systemd[1]: caddy.service: Main process exited, code=killed, status=9/KILL
Apr 03 00:03:05 ep-it-010 systemd[1]: caddy.service: Failed with result 'timeout'.
Apr 03 00:03:05 ep-it-010 systemd[1]: Failed to start Caddy.
Apr 03 00:03:05 ep-it-010 systemd[1]: caddy.service: Consumed 2.092s CPU time.

3. Caddy version:

v2.6.4 h1:2hwYqiRwk1tf3VruhMpLcYTg+11fCdr8S3jhNAdnPy8=

4. How I installed and ran Caddy:

sudo apt-get install caddy

Plus, I added the transform-encoder module using caddy add-package

a. System environment:

Debian 11 running in a proxmox VM.

systemd version 247.3-7+deb11u2

b. Command:

sudo systemctl start caddy.service

Actually, this daily process is triggered by the following logrotate configuration:

# Este archivo debe ir en /etc/logrotate.d
#
/var/log/caddy/analisis.log {
    daily
    rotate 60
    compress
    delaycompress
    missingok
    notifempty
    create 0640 caddy caddy
    sharedscripts
    postrotate
        sudo systemctl restart caddy.service
        caddylog_404_error_report.sh /var/log/caddy/analisis.log.1 /var/log/caddy/reportes
    endscript
}

c. Service/unit/compose file:

[Unit]
Description=Caddy
Documentation=https://caddyserver.com/docs/
After=network.target network-online.target
Requires=network-online.target

[Service]
Type=notify
User=caddy
Group=caddy
ExecStart=/usr/bin/caddy run --environ --config /etc/caddy/Caddyfile
ExecReload=/usr/bin/caddy reload --config /etc/caddy/Caddyfile --force
TimeoutStopSec=5s
LimitNOFILE=1048576
LimitNPROC=512
PrivateDevices=yes
PrivateTmp=true
ProtectSystem=full
AmbientCapabilities=CAP_NET_BIND_SERVICE

[Install]
WantedBy=multi-user.target

d. My complete Caddy config:

The full Caddyfile is so long that the forum kept rejecting my post for being over 32K chars long.

I uploaded the complete file here

{
	email it+tls-caddy@siu.edu.ar

	default_sni px00.siu.edu.ar
}

# snippets
(REDES_INTERNAS_EXTERNAS) {
	@redes_internas {
		remote_ip 172.16.0.0/12 192.168.0.0/16 100.64.0.0/10 10.0.1.0/24 10.0.2.0/23 10.0.4.0/22 10.0.8.0/21 10.0.16.0/20 10.0.32.0/19 10.0.64.0/18 10.0.128.0/17 10.1.0.0/16 10.2.0.0/15 10.4.0.0/14 10.8.0.0/13 10.16.0.0/12 10.32.0.0/11 10.64.0.0/10 10.128.0.0/9
	}
	@redes_externas {
		not remote_ip 172.16.0.0/12 192.168.0.0/16 100.64.0.0/10 10.0.1.0/24 10.0.2.0/23 10.0.4.0/22 10.0.8.0/21 10.0.16.0/20 10.0.32.0/19 10.0.64.0/18 10.0.128.0/17 10.1.0.0/16 10.2.0.0/15 10.4.0.0/14 10.8.0.0/13 10.16.0.0/12 10.32.0.0/11 10.64.0.0/10 10.128.0.0/9
	}
}

(LOG_ANALISIS) {
	log {
		output file /var/log/caddy/analisis.log {
			roll_disabled
		}
		format transform `{ts},{request>remote_ip},{request>host},{request>uri},method:{request>method},proto:{request>proto},status:{status},size:{size},referer:{request>headers>Referer},user-agent:{request>headers>User-Agent}` {
			time_format "rfc3339"
		}
	}
}

(LOG_FLAT) {
	log {
		output file /var/log/caddy/caddy-access.log
		format transform `{request>remote_ip} - {request>host} - [{ts}] - {request>headers>User-Agent} "{request>method} {request>uri} {request>proto}" status:{status} size:{size} referer:{request>headers>Referer}` {
			time_format "common_log"
		}
	}
}

(LOG_FLAT_NOBLOCK) {
	log {
		output file /var/log/caddy/noblock-caddy-access.log
		format transform `{request>remote_ip} - {request>host} - [{ts}] - {request>headers>User-Agent} "{request>method} {request>uri} {request>proto}" status:{status} size:{size} referer:{request>headers>Referer}` {
			time_format "common_log"
		}
	}
}

px00.siu.edu.ar,
200.123.128.68:80,
200.123.128.68,
172.27.100.240:80,
172.27.100.240 {
	redir https://www.siu.edu.ar/
	import LOG_ANALISIS
}

siu.edu.ar {
	redir https://www.siu.edu.ar{uri}
	import LOG_ANALISIS
}

infosiu.siu.edu.ar,
info.siu.edu.ar {
	redir https://www.siu.edu.ar/infosiu permanent
	import LOG_ANALISIS
}

inscripciones.siu.edu.ar {
	redir https://www.siu.edu.ar/eventos permanent
	import LOG_ANALISIS
}

estudiarenargentina.siu.edu.ar,
studyinargentina.siu.edu.ar {
	redir https://estudia-en-argentina.com.ar permanent
	import LOG_ANALISIS
}

encuestas.siu.edu.ar,
kollacin.siu.edu.ar,
kollaspu.siu.edu.ar {
	reverse_proxy ep-wb-002.siu.edu.ar
	import LOG_ANALISIS
}

www.siu.edu.ar,
joomlahuarpe.siu.edu.ar,
portal.comunidad.siu.edu.ar {
	reverse_proxy ep-wb-003.siu.edu.ar
	import LOG_ANALISIS
}
firmadigital.siu.edu.ar {
	import REDES_INTERNAS_EXTERNAS
	reverse_proxy @redes_internas ep-wb-003.siu.edu.ar
	redir @redes_externas https://www.siu.edu.ar/
	import LOG_ANALISIS
}

guarani.siu.edu.ar {
	reverse_proxy ep-wb-003.siu.edu.ar
	import LOG_ANALISIS
}
gestion.guarani.siu.edu.ar {
	reverse_proxy daldoelectico.siu.edu.ar:7271
	import LOG_ANALISIS
}
autogestion.guarani.siu.edu.ar {
	reverse_proxy daldoelectico.siu.edu.ar:7272
	import LOG_ANALISIS
}
preinscripcion.guarani.siu.edu.ar {
	reverse_proxy daldoelectico.siu.edu.ar:7273
	import LOG_ANALISIS
}
api-app.guarani.siu.edu.ar {
	reverse_proxy daldoelectico.siu.edu.ar:7274
	import LOG_ANALISIS
}
toba-usuarios.guarani.siu.edu.ar {
	redir https://gestion.guarani.siu.edu.ar/toba_usuarios/03201{uri}
	import LOG_ANALISIS
}
demo.guarani.siu.edu.ar {
	reverse_proxy daldoelectico.siu.edu.ar
	import LOG_ANALISIS
}

infraestructura.siu.edu.ar,
querandies.siu.edu.ar {
	reverse_proxy ep-wb-006.siu.edu.ar
	import LOG_ANALISIS
}

pedidosciie.siu.edu.ar {
	reverse_proxy ep-wb-001.siu.edu.ar
	import LOG_ANALISIS
}

cpres.siu.edu.ar {
	reverse_proxy ep-wb-008.siu.edu.ar
	import LOG_ANALISIS
}

ppua.siu.edu.ar,
ppuaadmin.siu.edu.ar {
	reverse_proxy ep-wb-009.siu.edu.ar
	import LOG_ANALISIS
}

cafp.siu.edu.ar,
proyectos.marca.siu.edu.ar,
sicerconsulta.siu.edu.ar {
	reverse_proxy ep-wb-013.siu.edu.ar
	import LOG_ANALISIS
}

bibliotecadigital.cin.edu.ar {
	reverse_proxy agnito.siu.edu.ar
	import LOG_ANALISIS
}


informacion.cin.edu.ar,
informacion.siu.edu.ar,
acompaniarsinide.siu.edu.ar,
spupedidos.siu.edu.ar {
	reverse_proxy lechuga.siu.edu.ar
	import LOG_ANALISIS
}

conveniosfrance.siu.edu.ar,
cargaconveniosfrance.siu.edu.ar {
	reverse_proxy lechuga.siu.edu.ar
	import LOG_ANALISIS
}

guaranivirtual.siu.edu.ar {
	reverse_proxy lechuga.siu.edu.ar
	import LOG_ANALISIS
}

moodlevinculacion.siu.edu.ar {
	reverse_proxy lechuga.siu.edu.ar
	import LOG_ANALISIS
}

normativa.cin.edu.ar,
http://normativa.siu.edu.ar {
	reverse_proxy ep-wb-012.siu.edu.ar
	import LOG_ANALISIS
}

posgradosasociados.siu.edu.ar,
postgradosasociados.siu.edu.ar {
	reverse_proxy lechuga.siu.edu.ar
	import LOG_ANALISIS
}

graduadosuniversitarios.siu.edu.ar {
	redir https://registrograduados.siu.edu.ar{uri}
	import LOG_ANALISIS
}

titulosoficiales.siu.edu.ar {
	redir https://sipes.siu.edu.ar/buscar_titulos.php
	import LOG_ANALISIS
}

sied.siu.edu.ar {
	redir https://sipes.siu.edu.ar/consulta_sied.php
	import LOG_ANALISIS
}

sirvat.siu.edu.ar {
	redir https://sipes.siu.edu.ar{uri}
	import LOG_ANALISIS
}

bancodeevaluadores.cin.edu.ar,
calchaqui.siu.edu.ar,
conveniospu.siu.edu.ar,
cytuniversidades.siu.edu.ar,
demogestor.siu.edu.ar,
generoydiversidad.siu.edu.ar,
marcademo.siu.edu.ar,
nomina.siu.edu.ar,
ofertasuperior.siu.edu.ar,
precalchaqui.siu.edu.ar,
registrograduados.siu.edu.ar,
rtfsimulador.siu.edu.ar,
sipes.siu.edu.ar,
snraregistro.siu.edu.ar,
spusceu.siu.edu.ar
{
	reverse_proxy emperador.siu.edu.ar
	import LOG_ANALISIS
}

<SNIP>

Note: redacted for LENGTH. The forum wouldn’t let me write a topic longer than 32K chars

5. Links to relevant resources:

2

That’s quite an old version. Please upgrade to the latest version, v2.7.6. It might resolve your issue. If not, we can dig deeper afterwards.

1 Like
3

Thanx for your answer.

I just upgraded my two caddy boxes.

Incidentally, I also made a couple of changes to the caddy systemd service unit via sudo systemctl edit caddy.service, I added the following lines:

[Service]
TimeoutStartSec=180s
Restart=on-failure
RestartSec=10s

The first one extends the timeout for the start operation to 3 minutes (instead of the default 90 seconds), and the other two hopefully will try to restart the daemon again if it fails (or if it keeps suiciding every now and then).

4

Keep in mind we advise restarting only in the case of certain exit codes:

2 Likes
5

See Keep Caddy Running — Caddy Documentation, we document the proper way to set up restarts here.

2 Likes
6

Excellent. I just added the RestartPreventExitStatus=1 in the systemd unit definition override.

Thanx a lot for your help

3 Likes
7

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.