Sudden "no certificate available" / ERR_SSL_PROTOCOL_ERROR

[einarpersson]

part 1

{"level":"info","ts":1587240433.091952,"msg":"using provided configuration","config_file":"/etc/caddy/Caddyfile","config_adapter":""}
{"level":"info","ts":1587240433.0952182,"logger":"admin","msg":"admin endpoint started","address":"tcp/localhost:2019","enforce_origin":false,"origins":["127.0.0.1:2019","localhost:2019","[::1]:2019"]}
{"level":"info","ts":1587240433.095644,"logger":"http","msg":"server is listening only on the HTTPS port but has no TLS connection policies; adding one to enable TLS","server_name":"srv0","https_port":443}
{"level":"info","ts":1587240433.09576,"logger":"http","msg":"enabling automatic HTTP->HTTPS redirects","server_name":"srv0"}
{"level":"info","ts":1587240433.0974782,"logger":"tls","msg":"cleaned up storage units"}
{"level":"info","ts":1587240433.097753,"logger":"http","msg":"enabling automatic TLS certificate management","domains":["booking.tullingelabs.se","tullingelabs.se"]}
2020/04/18 20:07:13 [INFO][cache:0xc00056e0a0] Started certificate maintenance routine
{"level":"info","ts":1587240433.1091232,"msg":"autosaved config","file":"/config/caddy/autosave.json"}
{"level":"info","ts":1587240433.1091497,"msg":"serving initial configuration"}
{"level":"info","ts":1587240433.1092033,"logger":"watcher","msg":"watching config file for changes","config_file":"/etc/caddy/Caddyfile"}
{"level":"info","ts":1587240657.1096542,"logger":"watcher","msg":"config file changed; reloading","config_file":"/etc/caddy/Caddyfile"}
{"level":"info","ts":1587240657.1098604,"msg":"using provided configuration","config_file":"/etc/caddy/Caddyfile","config_adapter":""}
{"level":"info","ts":1587240657.1113791,"logger":"admin","msg":"admin endpoint started","address":"tcp/localhost:2019","enforce_origin":false,"origins":["localhost:2019","[::1]:2019","127.0.0.1:2019"]}
{"level":"info","ts":1587240657.111816,"logger":"http","msg":"server is listening only on the HTTPS port but has no TLS connection policies; adding one to enable TLS","server_name":"srv0","https_port":443}
{"level":"info","ts":1587240657.1118422,"logger":"http","msg":"enabling automatic HTTP->HTTPS redirects","server_name":"srv0"}
{"level":"info","ts":1587240657.1122282,"logger":"http","msg":"enabling automatic TLS certificate management","domains":["booking.tullingelabs.se","tullingelabs.se"]}
{"level":"info","ts":1587240657.1144376,"logger":"admin","msg":"stopped previous server"}
2020/04/18 20:10:57 [INFO][cache:0xc00056e0a0] Stopped certificate maintenance routine
{"level":"info","ts":1587240657.1148515,"msg":"autosaved config","file":"/config/caddy/autosave.json"}
2020/04/18 20:10:57 [INFO][cache:0xc0003c6730] Started certificate maintenance routine
{"level":"info","ts":1587240723.109411,"logger":"watcher","msg":"config file changed; reloading","config_file":"/etc/caddy/Caddyfile"}
{"level":"info","ts":1587240723.1096232,"msg":"using provided configuration","config_file":"/etc/caddy/Caddyfile","config_adapter":""}
{"level":"info","ts":1587240723.1106136,"logger":"admin.api","msg":"config is unchanged"}
{"level":"info","ts":1587241954.1095564,"logger":"watcher","msg":"config file changed; reloading","config_file":"/etc/caddy/Caddyfile"}
{"level":"info","ts":1587241954.109876,"msg":"using provided configuration","config_file":"/etc/caddy/Caddyfile","config_adapter":""}
{"level":"info","ts":1587241954.1118648,"logger":"admin","msg":"admin endpoint started","address":"tcp/localhost:2019","enforce_origin":false,"origins":["localhost:2019","[::1]:2019","127.0.0.1:2019"]}
{"level":"info","ts":1587241954.1121392,"logger":"admin","msg":"stopped previous server"}
2020/04/18 20:32:34 [INFO][cache:0xc00056e820] Started certificate maintenance routine
{"level":"info","ts":1587241954.112419,"logger":"http","msg":"server is listening only on the HTTPS port but has no TLS connection policies; adding one to enable TLS","server_name":"srv0","https_port":443}
{"level":"info","ts":1587241954.1124508,"logger":"http","msg":"enabling automatic HTTP->HTTPS redirects","server_name":"srv0"}
{"level":"info","ts":1587241954.1126702,"logger":"http","msg":"enabling automatic TLS certificate management","domains":["booking.tullingelabs.se","tullingelabs.se"]}
2020/04/18 20:32:34 [INFO][cache:0xc0003c6730] Stopped certificate maintenance routine
{"level":"info","ts":1587241954.114788,"msg":"autosaved config","file":"/config/caddy/autosave.json"}
2020/04/18 22:18:48 http: TLS handshake error from 159.89.81.186:63309: EOF
2020/04/18 22:18:48 http: TLS handshake error from 159.89.81.186:60042: no certificate available for '185.157.221.99'
2020/04/18 22:18:49 http: TLS handshake error from 159.89.81.186:53690: no certificate available for '185.157.221.99'
2020/04/18 22:18:49 http: TLS handshake error from 159.89.81.186:53283: no certificate available for '185.157.221.99'
2020/04/18 22:18:49 http: TLS handshake error from 159.89.81.186:54621: no certificate available for '185.157.221.99'
2020/04/18 22:18:49 http: TLS handshake error from 159.89.81.186:54559: no certificate available for '185.157.221.99'
2020/04/18 22:18:49 http: TLS handshake error from 159.89.81.186:52849: no certificate available for '185.157.221.99'
2020/04/18 22:18:50 http: TLS handshake error from 159.89.81.186:52545: no certificate available for '185.157.221.99'
2020/04/18 22:18:50 http: TLS handshake error from 159.89.81.186:49724: no certificate available for '185.157.221.99'
2020/04/18 22:18:50 http: TLS handshake error from 159.89.81.186:51027: no certificate available for '185.157.221.99'
2020/04/18 22:40:17 http: TLS handshake error from 94.140.114.17:443: tls: client offered only unsupported versions: [302 301]
2020/04/19 00:49:18 http: TLS handshake error from 74.82.47.5:20038: no certificate available for '185.157.221.99'
2020/04/19 01:11:37 http: TLS handshake error from 203.154.59.166:48536: no certificate available for '185.157.221.99'
2020/04/19 01:11:38 http: TLS handshake error from 203.154.59.166:53934: no certificate available for '185.157.221.99'
2020/04/19 01:26:22 http: TLS handshake error from 51.91.247.125:50808: no certificate available for '185.157.221.99'
2020/04/19 01:51:02 http: TLS handshake error from 45.143.220.143:54214: tls: client offered only unsupported versions: [301]
2020/04/19 03:30:03 http: TLS handshake error from 198.108.66.208:63644: no certificate available for '185.157.221.99'
2020/04/19 03:57:28 http: TLS handshake error from 171.67.71.243:41306: no certificate available for '185.157.221.99'
2020/04/19 03:59:37 http: TLS handshake error from 45.13.93.82:44786: tls: first record does not look like a TLS handshake
2020/04/19 04:15:46 http: TLS handshake error from 208.93.152.17:60317: EOF
2020/04/19 04:15:46 http: TLS handshake error from 208.93.152.17:53721: tls: unsupported SSLv2 handshake received
2020/04/19 04:15:47 http: TLS handshake error from 208.93.152.17:19587: tls: client offered only unsupported versions: []
2020/04/19 04:15:47 http: TLS handshake error from 208.93.152.17:62029: tls: client offered only unsupported versions: [301]
2020/04/19 04:15:49 http: TLS handshake error from 208.93.152.17:23354: tls: client offered only unsupported versions: [302 301]
2020/04/19 04:15:49 http: TLS handshake error from 208.93.152.17:56811: no certificate available for '185.157.221.99'
2020/04/19 04:15:50 http: TLS handshake error from 208.93.152.17:50591: no certificate available for '185.157.221.99'
2020/04/19 05:23:43 http: TLS handshake error from 35.187.98.101:36063: no certificate available for '185.157.221.99'
2020/04/19 05:24:17 http: TLS handshake error from 5.101.0.209:58314: no certificate available for '185.157.221.99'
2020/04/19 05:31:15 http: TLS handshake error from 198.108.66.240:52604: no certificate available for '185.157.221.99'
2020/04/19 06:46:13 http: TLS handshake error from 5.101.0.209:37356: no certificate available for '185.157.221.99'
2020/04/19 07:33:48 http: TLS handshake error from 128.14.133.58:44144: no certificate available for '185.157.221.99'
{"level":"info","ts":1587285154.1173973,"logger":"tls","msg":"cleaned up storage units"}
{"level":"info","ts":1587290703.1094599,"logger":"watcher","msg":"config file changed; reloading","config_file":"/etc/caddy/Caddyfile"}
{"level":"info","ts":1587290703.1098342,"msg":"using provided configuration","config_file":"/etc/caddy/Caddyfile","config_adapter":""}
{"level":"info","ts":1587290703.1120558,"logger":"admin","msg":"admin endpoint started","address":"tcp/localhost:2019","enforce_origin":false,"origins":["localhost:2019","[::1]:2019","127.0.0.1:2019"]}
{"level":"info","ts":1587290703.1123614,"logger":"admin","msg":"stopped previous server"}
2020/04/19 10:05:03 [INFO][cache:0xc00056e640] Started certificate maintenance routine
{"level":"info","ts":1587290703.1126301,"logger":"http","msg":"server is listening only on the HTTPS port but has no TLS connection policies; adding one to enable TLS","server_name":"srv0","https_port":443}
{"level":"info","ts":1587290703.1126661,"logger":"http","msg":"enabling automatic HTTP->HTTPS redirects","server_name":"srv0"}
{"level":"info","ts":1587290703.1129873,"logger":"http","msg":"enabling automatic TLS certificate management","domains":["booking.tullingelabs.se","tullingelabs.se"]}
2020/04/19 10:05:03 [INFO][cache:0xc00056e820] Stopped certificate maintenance routine
{"level":"info","ts":1587290703.115577,"msg":"autosaved config","file":"/config/caddy/autosave.json"}
{"level":"info","ts":1587292014.1097796,"logger":"watcher","msg":"config file changed; reloading","config_file":"/etc/caddy/Caddyfile"}
{"level":"info","ts":1587292014.1101437,"msg":"using provided configuration","config_file":"/etc/caddy/Caddyfile","config_adapter":""}
{"level":"info","ts":1587292014.1122632,"logger":"admin","msg":"admin endpoint started","address":"tcp/localhost:2019","enforce_origin":false,"origins":["localhost:2019","[::1]:2019","127.0.0.1:2019"]}
{"level":"info","ts":1587292014.112575,"logger":"admin","msg":"stopped previous server"}
2020/04/19 10:26:54 [INFO][cache:0xc0003c6460] Started certificate maintenance routine
{"level":"info","ts":1587292014.113162,"logger":"http","msg":"server is listening only on the HTTPS port but has no TLS connection policies; adding one to enable TLS","server_name":"srv0","https_port":443}
{"level":"info","ts":1587292014.1132185,"logger":"http","msg":"enabling automatic HTTP->HTTPS redirects","server_name":"srv0"}
{"level":"info","ts":1587292014.1134763,"logger":"http","msg":"enabling automatic TLS certificate management","domains":["booking.tullingelabs.se","tullingelabs.se"]}
2020/04/19 10:26:54 [INFO][cache:0xc00056e640] Stopped certificate maintenance routine
{"level":"info","ts":1587292014.1162918,"msg":"autosaved config","file":"/config/caddy/autosave.json"}
2020/04/19 10:58:27 http: TLS handshake error from 103.132.161.226:3575: EOF
2020/04/19 11:02:24 http: TLS handshake error from 5.101.64.77:64205: tls: first record does not look like a TLS handshake
2020/04/19 11:11:21 http: TLS handshake error from 5.101.64.77:2629: tls: first record does not look like a TLS handshake
{"level":"info","ts":1587300847.1096327,"logger":"watcher","msg":"config file changed; reloading","config_file":"/etc/caddy/Caddyfile"}
{"level":"info","ts":1587300847.1101148,"msg":"using provided configuration","config_file":"/etc/caddy/Caddyfile","config_adapter":""}
{"level":"info","ts":1587300847.1125798,"logger":"admin","msg":"admin endpoint started","address":"tcp/localhost:2019","enforce_origin":false,"origins":["127.0.0.1:2019","localhost:2019","[::1]:2019"]}
{"level":"info","ts":1587300847.1129549,"logger":"admin","msg":"stopped previous server"}
{"level":"info","ts":1587300847.1131675,"logger":"http","msg":"server is listening only on the HTTPS port but has no TLS connection policies; adding one to enable TLS","server_name":"srv0","https_port":443}
{"level":"info","ts":1587300847.113214,"logger":"http","msg":"enabling automatic HTTP->HTTPS redirects","server_name":"srv0"}
2020/04/19 12:54:07 [INFO][cache:0xc00019b9f0] Started certificate maintenance routine
{"level":"info","ts":1587300847.113555,"logger":"http","msg":"enabling automatic TLS certificate management","domains":["attendance.tullingelabs.se","booking.tullingelabs.se","tullingelabs.se"]}
2020/04/19 12:54:07 [INFO][cache:0xc0003c6460] Stopped certificate maintenance routine
{"level":"info","ts":1587300847.1173499,"msg":"autosaved config","file":"/config/caddy/autosave.json"}
2020/04/19 12:54:07 [INFO][attendance.tullingelabs.se] Obtain certificate; acquiring lock...
2020/04/19 12:54:07 [INFO][attendance.tullingelabs.se] Obtain: Lock acquired; proceeding...
2020/04/19 12:54:07 [INFO][attendance.tullingelabs.se] Waiting on rate limiter...
2020/04/19 12:54:07 [INFO][attendance.tullingelabs.se] Done waiting
2020/04/19 12:54:07 [INFO] [attendance.tullingelabs.se] acme: Obtaining bundled SAN certificate given a CSR
2020/04/19 12:54:08 [INFO] [attendance.tullingelabs.se] AuthURL: https://acme-v02.api.letsencrypt.org/acme/authz-v3/4053683933
2020/04/19 12:54:08 [INFO] [attendance.tullingelabs.se] acme: use tls-alpn-01 solver
2020/04/19 12:54:08 [INFO] [attendance.tullingelabs.se] acme: Trying to solve TLS-ALPN-01
2020/04/19 12:54:08 http: TLS handshake error from 127.0.0.1:54856: EOF
2020/04/19 12:54:08 [INFO][attendance.tullingelabs.se] Served key authentication certificate (TLS-ALPN challenge)
2020/04/19 12:54:09 [INFO][attendance.tullingelabs.se] Served key authentication certificate (TLS-ALPN challenge)
2020/04/19 12:54:09 [INFO][attendance.tullingelabs.se] Served key authentication certificate (TLS-ALPN challenge)
2020/04/19 12:54:09 [INFO][attendance.tullingelabs.se] Served key authentication certificate (TLS-ALPN challenge)
2020/04/19 12:54:14 [INFO] [attendance.tullingelabs.se] The server validated our request
2020/04/19 12:54:14 [INFO] [attendance.tullingelabs.se] acme: Validations succeeded; requesting certificates
2020/04/19 12:54:15 [INFO] [attendance.tullingelabs.se] Server responded with a certificate.
2020/04/19 12:54:15 [INFO][attendance.tullingelabs.se] Certificate obtained successfully
2020/04/19 12:54:15 [INFO][attendance.tullingelabs.se] Obtain: Releasing lock
2020/04/19 13:28:56 http: TLS handshake error from 128.14.134.170:57890: no certificate available for '185.157.221.99'
2020/04/19 14:46:34 http: TLS handshake error from 149.56.129.207:56086: no certificate available for '185.157.221.99'
2020/04/19 17:10:06 http: TLS handshake error from 139.162.116.133:52218: no certificate available for '185.157.221.99'
2020/04/19 18:02:18 http: TLS handshake error from 172.104.100.219:43146: no certificate available for '185.157.221.99'
2020/04/19 18:02:19 http: TLS handshake error from 172.104.100.219:43256: tls: first record does not look like a TLS handshake
2020/04/19 19:05:51 http: TLS handshake error from 172.104.242.173:56136: tls: client offered only unsupported versions: [302 301]
2020/04/19 20:43:05 http: TLS handshake error from 146.88.240.14:35062: no certificate available for '185.157.221.99'
2020/04/19 22:00:46 http: TLS handshake error from 66.240.205.34:35224: tls: first record does not look like a TLS handshake
2020/04/19 23:18:05 http: TLS handshake error from 137.226.113.26:43844: no certificate available for 'www.tullingelabs.se'
2020/04/20 00:36:52 http: TLS handshake error from 216.218.206.69:16660: no certificate available for '185.157.221.99'
{"level":"info","ts":1587344047.1415713,"logger":"tls","msg":"cleaned up storage units"}
2020/04/20 01:44:47 http: TLS handshake error from 163.172.68.20:61194: tls: first record does not look like a TLS handshake
2020/04/20 02:33:05 http: TLS handshake error from 172.104.242.173:45516: tls: client offered only unsupported versions: [302 301]
2020/04/20 03:59:21 http: TLS handshake error from 171.67.71.243:48782: no certificate available for '185.157.221.99'
2020/04/20 05:05:50 http: TLS handshake error from 128.14.134.170:49870: no certificate available for '185.157.221.99'
2020/04/20 05:14:31 http: TLS handshake error from 172.105.89.161:35532: no certificate available for '185.157.221.99'
2020/04/20 06:25:04 http: TLS handshake error from 198.108.66.64:33250: no certificate available for '185.157.221.99'
2020/04/20 06:27:13 http: TLS handshake error from 164.52.24.162:60673: no certificate available for '185.157.221.99'
2020/04/20 06:27:13 http: TLS handshake error from 164.52.24.162:52474: no certificate available for '185.157.221.99'
2020/04/20 06:27:14 http: TLS handshake error from 164.52.24.162:44591: tls: client offered only unsupported versions: [302 301]
2020/04/20 06:27:14 http: TLS handshake error from 164.52.24.162:59956: tls: client offered only unsupported versions: [301]
2020/04/20 06:27:18 http: TLS handshake error from 164.52.24.162:35972: read tcp 185.157.221.99:443->164.52.24.162:35972: read: connection reset by peer
2020/04/20 06:41:54 http: TLS handshake error from 162.243.133.71:41936: no certificate available for '185.157.221.99'
{"level":"error","ts":1587367414.5514584,"logger":"http.log.error","msg":"dial tcp [::1]:5000: connect: connection refused","request":{"method":"GET","uri":"/","proto":"HTTP/2.0","remote_addr":"158.174.82.32:49776","host":"attendance.tullingelabs.se","headers":{"User-Agent":["Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/81.0.4044.113 Safari/537.36"],"Sec-Fetch-Site":["none"],"Sec-Fetch-Mode":["navigate"],"Sec-Fetch-User":["?1"],"Upgrade-Insecure-Requests":["1"],"Accept":["text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.9"],"Sec-Fetch-Dest":["document"],"Accept-Encoding":["gzip, deflate, br"],"Accept-Language":["sv-SE,sv;q=0.9,en-US;q=0.8,en;q=0.7"]},"tls":{"resumed":false,"version":772,"ciphersuite":4865,"proto":"h2","proto_mutual":true,"server_name":"attendance.tullingelabs.se"}},"status":502,"err_id":"dpxmixq9b","err_trace":"reverseproxy.(*Handler).ServeHTTP (reverseproxy.go:362)"}
{"level":"error","ts":1587367420.8997176,"logger":"http.log.error","msg":"dial tcp [::1]:5000: connect: connection refused","request":{"method":"GET","uri":"/","proto":"HTTP/2.0","remote_addr":"35.227.62.178:49172","host":"attendance.tullingelabs.se","headers":{"Accept":["*/*"],"User-Agent":["Mozilla/5.0 (compatible; Discordbot/2.0; +https://discordapp.com)"],"Content-Length":["0"]},"tls":{"resumed":false,"version":772,"ciphersuite":4865,"proto":"h2","proto_mutual":true,"server_name":"attendance.tullingelabs.se"}},"status":502,"err_id":"v8p16w8da","err_trace":"reverseproxy.(*Handler).ServeHTTP (reverseproxy.go:362)"}
{"level":"error","ts":1587367430.1295934,"logger":"http.log.error","msg":"dial tcp [::1]:5000: connect: connection refused","request":{"method":"GET","uri":"/","proto":"HTTP/2.0","remote_addr":"213.100.223.139:55737","host":"attendance.tullingelabs.se","headers":{"Sec-Fetch-Mode":["navigate"],"Sec-Fetch-User":["?1"],"Accept-Encoding":["gzip, deflate, br"],"Accept-Language":["en-US,en;q=0.9"],"Upgrade-Insecure-Requests":["1"],"User-Agent":["Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/80.0.3987.163 Safari/537.36"],"Sec-Fetch-Dest":["document"],"Accept":["text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.9"],"Sec-Fetch-Site":["none"]},"tls":{"resumed":false,"version":772,"ciphersuite":4865,"proto":"h2","proto_mutual":true,"server_name":"attendance.tullingelabs.se"}},"status":502,"err_id":"isxddzr4u","err_trace":"reverseproxy.(*Handler).ServeHTTP (reverseproxy.go:362)"}
{"level":"error","ts":1587367432.0286484,"logger":"http.log.error","msg":"dial tcp [::1]:5000: connect: connection refused","request":{"method":"GET","uri":"/","proto":"HTTP/2.0","remote_addr":"213.100.223.139:55737","host":"attendance.tullingelabs.se","headers":{"Cache-Control":["max-age=0"],"Upgrade-Insecure-Requests":["1"],"Accept-Language":["en-US,en;q=0.9"],"Accept-Encoding":["gzip, deflate, br"],"User-Agent":["Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/80.0.3987.163 Safari/537.36"],"Sec-Fetch-Dest":["document"],"Accept":["text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.9"],"Sec-Fetch-Site":["cross-site"],"Sec-Fetch-Mode":["navigate"],"Sec-Fetch-User":["?1"]},"tls":{"resumed":false,"version":772,"ciphersuite":4865,"proto":"h2","proto_mutual":true,"server_name":"attendance.tullingelabs.se"}},"status":502,"err_id":"6bgmz4w6y","err_trace":"reverseproxy.(*Handler).ServeHTTP (reverseproxy.go:362)"}
{"level":"error","ts":1587367436.4105563,"logger":"http.log.error","msg":"dial tcp [::1]:5000: connect: connection refused","request":{"method":"GET","uri":"/","proto":"HTTP/2.0","remote_addr":"213.100.223.139:55737","host":"attendance.tullingelabs.se","headers":{"User-Agent":["Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/80.0.3987.163 Safari/537.36"],"Sec-Fetch-Site":["cross-site"],"Sec-Fetch-Mode":["navigate"],"Cache-Control":["max-age=0"],"Sec-Fetch-Dest":["document"],"Accept":["text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.9"],"Sec-Fetch-User":["?1"],"Accept-Encoding":["gzip, deflate, br"],"Accept-Language":["en-US,en;q=0.9"],"Upgrade-Insecure-Requests":["1"]},"tls":{"resumed":false,"version":772,"ciphersuite":4865,"proto":"h2","proto_mutual":true,"server_name":"attendance.tullingelabs.se"}},"status":502,"err_id":"ni3fexctv","err_trace":"reverseproxy.(*Handler).ServeHTTP (reverseproxy.go:362)"}
{"level":"error","ts":1587367437.3184068,"logger":"http.log.error","msg":"dial tcp [::1]:5000: connect: connection refused","request":{"method":"GET","uri":"/","proto":"HTTP/2.0","remote_addr":"213.100.223.139:55737","host":"attendance.tullingelabs.se","headers":{"User-Agent":["Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/80.0.3987.163 Safari/537.36"],"Sec-Fetch-Dest":["document"],"Accept":["text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.9"],"Sec-Fetch-Site":["cross-site"],"Sec-Fetch-Mode":["navigate"],"Sec-Fetch-User":["?1"],"Cache-Control":["max-age=0"],"Upgrade-Insecure-Requests":["1"],"Accept-Encoding":["gzip, deflate, br"],"Accept-Language":["en-US,en;q=0.9"]},"tls":{"resumed":false,"version":772,"ciphersuite":4865,"proto":"h2","proto_mutual":true,"server_name":"attendance.tullingelabs.se"}},"status":502,"err_id":"k0jq496pf","err_trace":"reverseproxy.(*Handler).ServeHTTP (reverseproxy.go:362)"}
{"level":"error","ts":1587367437.7173903,"logger":"http.log.error","msg":"dial tcp [::1]:5000: connect: connection refused","request":{"method":"GET","uri":"/","proto":"HTTP/2.0","remote_addr":"213.100.223.139:55737","host":"attendance.tullingelabs.se","headers":{"Cache-Control":["max-age=0"],"Sec-Fetch-Dest":["document"],"Accept":["text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.9"],"Sec-Fetch-Mode":["navigate"],"Accept-Language":["en-US,en;q=0.9"],"Upgrade-Insecure-Requests":["1"],"User-Agent":["Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/80.0.3987.163 Safari/537.36"],"Sec-Fetch-Site":["cross-site"],"Sec-Fetch-User":["?1"],"Accept-Encoding":["gzip, deflate, br"]},"tls":{"resumed":false,"version":772,"ciphersuite":4865,"proto":"h2","proto_mutual":true,"server_name":"attendance.tullingelabs.se"}},"status":502,"err_id":"6f2sjwkqj","err_trace":"reverseproxy.(*Handler).ServeHTTP (reverseproxy.go:362)"}
{"level":"error","ts":1587367438.1668909,"logger":"http.log.error","msg":"dial tcp [::1]:5000: connect: connection refused","request":{"method":"GET","uri":"/","proto":"HTTP/2.0","remote_addr":"213.100.223.139:55737","host":"attendance.tullingelabs.se","headers":{"Cache-Control":["max-age=0"],"Upgrade-Insecure-Requests":["1"],"Sec-Fetch-Site":["cross-site"],"Sec-Fetch-Mode":["navigate"],"Accept-Language":["en-US,en;q=0.9"],"User-Agent":["Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/80.0.3987.163 Safari/537.36"],"Sec-Fetch-Dest":["document"],"Accept":["text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.9"],"Sec-Fetch-User":["?1"],"Accept-Encoding":["gzip, deflate, br"]},"tls":{"resumed":false,"version":772,"ciphersuite":4865,"proto":"h2","proto_mutual":true,"server_name":"attendance.tullingelabs.se"}},"status":502,"err_id":"0pxmk0r3i","err_trace":"reverseproxy.(*Handler).ServeHTTP (reverseproxy.go:362)"}
{"level":"error","ts":1587367438.7446542,"logger":"http.log.error","msg":"dial tcp [::1]:5000: connect: connection refused","request":{"method":"GET","uri":"/","proto":"HTTP/2.0","remote_addr":"213.100.223.139:55737","host":"attendance.tullingelabs.se","headers":{"Sec-Fetch-Site":["cross-site"],"Sec-Fetch-Mode":["navigate"],"Accept-Encoding":["gzip, deflate, br"],"Accept-Language":["en-US,en;q=0.9"],"Cache-Control":["max-age=0"],"User-Agent":["Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/80.0.3987.163 Safari/537.36"],"Sec-Fetch-Dest":["document"],"Upgrade-Insecure-Requests":["1"],"Accept":["text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.9"],"Sec-Fetch-User":["?1"]},"tls":{"resumed":false,"version":772,"ciphersuite":4865,"proto":"h2","proto_mutual":true,"server_name":"attendance.tullingelabs.se"}},"status":502,"err_id":"trit9vz27","err_trace":"reverseproxy.(*Handler).ServeHTTP (reverseproxy.go:362)"}
{"level":"error","ts":1587367444.8467648,"logger":"http.log.error","msg":"dial tcp [::1]:5000: connect: connection refused","request":{"method":"GET","uri":"/","proto":"HTTP/2.0","remote_addr":"158.174.82.32:49776","host":"attendance.tullingelabs.se","headers":{"Sec-Fetch-Dest":["document"],"Accept":["text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.9"],"Sec-Fetch-Site":["cross-site"],"Sec-Fetch-Mode":["navigate"],"Sec-Fetch-User":["?1"],"Accept-Language":["sv-SE,sv;q=0.9,en-US;q=0.8,en;q=0.7"],"Cache-Control":["max-age=0"],"Upgrade-Insecure-Requests":["1"],"User-Agent":["Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/81.0.4044.113 Safari/537.36"],"Accept-Encoding":["gzip, deflate, br"]},"tls":{"resumed":false,"version":772,"ciphersuite":4865,"proto":"h2","proto_mutual":true,"server_name":"attendance.tullingelabs.se"}},"status":502,"err_id":"zmi08syaq","err_trace":"reverseproxy.(*Handler).ServeHTTP (reverseproxy.go:362)"}
{"level":"error","ts":1587367446.3833764,"logger":"http.log.error","msg":"dial tcp [::1]:5000: connect: connection refused","request":{"method":"GET","uri":"/","proto":"HTTP/2.0","remote_addr":"158.174.82.32:49776","host":"attendance.tullingelabs.se","headers":{"User-Agent":["Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/81.0.4044.113 Safari/537.36"],"Sec-Fetch-Mode":["navigate"],"Sec-Fetch-User":["?1"],"Sec-Fetch-Dest":["document"],"Accept-Language":["sv-SE,sv;q=0.9,en-US;q=0.8,en;q=0.7"],"Cache-Control":["max-age=0"],"Accept":["text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.9"],"Sec-Fetch-Site":["cross-site"],"Accept-Encoding":["gzip, deflate, br"],"Upgrade-Insecure-Requests":["1"]},"tls":{"resumed":false,"version":772,"ciphersuite":4865,"proto":"h2","proto_mutual":true,"server_name":"attendance.tullingelabs.se"}},"status":502,"err_id":"sbcarj5yh","err_trace":"reverseproxy.(*Handler).ServeHTTP (reverseproxy.go:362)"}
{"level":"error","ts":1587367681.7180984,"logger":"http.log.error","msg":"dial tcp [::1]:5000: connect: connection refused","request":{"method":"GET","uri":"/","proto":"HTTP/2.0","remote_addr":"213.100.223.139:55737","host":"attendance.tullingelabs.se","headers":{"Sec-Fetch-Dest":["document"],"Sec-Fetch-Site":["none"],"Sec-Fetch-Mode":["navigate"],"Accept-Encoding":["gzip, deflate, br"],"Upgrade-Insecure-Requests":["1"],"Accept":["text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.9"],"Sec-Fetch-User":["?1"],"Accept-Language":["en-US,en;q=0.9"],"User-Agent":["Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/80.0.3987.163 Safari/537.36"]},"tls":{"resumed":false,"version":772,"ciphersuite":4865,"proto":"h2","proto_mutual":true,"server_name":"attendance.tullingelabs.se"}},"status":502,"err_id":"qt6ria8nx","err_trace":"reverseproxy.(*Handler).ServeHTTP (reverseproxy.go:362)"}
2020/04/20 11:45:22 http: TLS handshake error from 195.37.190.77:49422: no certificate available for '185.157.221.99'
2020/04/20 11:52:36 http: TLS handshake error from 185.153.196.243:1088: tls: first record does not look like a TLS handshake
2020/04/20 11:57:19 http: TLS handshake error from 185.153.196.243:1836: tls: first record does not look like a TLS handshake
2020/04/20 14:35:58 http: TLS handshake error from 164.68.112.178:51018: tls: client offered only unsupported versions: [302 301]
2020/04/20 15:58:27 http: TLS handshake error from 172.105.117.26:46186: EOF
2020/04/20 15:58:27 http: TLS handshake error from 172.105.117.26:46426: no certificate available for '185.157.221.99'
2020/04/20 15:58:27 http: TLS handshake error from 172.105.117.26:47008: no certificate available for '185.157.221.99'
2020/04/20 15:58:28 http: TLS handshake error from 172.105.117.26:47664: no certificate available for '185.157.221.99'
2020/04/20 15:58:28 http: TLS handshake error from 172.105.117.26:48426: no certificate available for '185.157.221.99'
2020/04/20 15:58:29 http: TLS handshake error from 172.105.117.26:49172: tls: client offered only unsupported versions: [302 301]
2020/04/20 15:58:29 http: TLS handshake error from 172.105.117.26:49960: tls: client offered only unsupported versions: [302 301]
2020/04/20 15:58:30 http: TLS handshake error from 172.105.117.26:50726: tls: client offered only unsupported versions: [301]
2020/04/20 15:58:31 http: TLS handshake error from 172.105.117.26:51574: tls: client offered only unsupported versions: [301]
2020/04/20 15:58:31 http: TLS handshake error from 172.105.117.26:52384: tls: client offered only unsupported versions: []
2020/04/20 15:58:32 http: TLS handshake error from 172.105.117.26:52960: tls: client offered only unsupported versions: []
2020/04/20 15:58:32 http: TLS handshake error from 172.105.117.26:53406: no certificate available for '185.157.221.99'
2020/04/20 15:58:33 http: TLS handshake error from 172.105.117.26:53864: no certificate available for '185.157.221.99'
2020/04/20 17:20:15 http: TLS handshake error from 172.104.242.173:48165: tls: first record does not look like a TLS handshake

[einarpersson]

Part 2

2020/04/20 17:42:15 http: TLS handshake error from 128.14.133.58:48518: no certificate available for '185.157.221.99'
2020/04/20 17:44:51 http: TLS handshake error from 137.226.113.21:32874: no certificate available for 'www.tullingelabs.se'
2020/04/20 18:49:01 http: TLS handshake error from 213.233.179.194:42702: no certificate available for '185.157.221.99'
2020/04/20 20:41:24 http: TLS handshake error from 185.202.2.57:1898: tls: first record does not look like a TLS handshake
2020/04/20 21:16:06 http: TLS handshake error from 146.88.240.20:34512: no certificate available for '185.157.221.99'
2020/04/20 21:26:10 http: TLS handshake error from 199.30.231.5:8205: EOF
2020/04/20 22:31:12 http: TLS handshake error from 137.226.113.27:53580: no certificate available for 'www.tullingelabs.se'
{"level":"info","ts":1587430447.117542,"logger":"tls","msg":"cleaned up storage units"}
2020/04/21 01:15:35 http: TLS handshake error from 51.178.78.154:37212: no certificate available for '185.157.221.99'
2020/04/21 01:23:56 http: TLS handshake error from 39.106.190.42:46000: no certificate available for '185.157.221.99'
2020/04/21 01:23:57 http: TLS handshake error from 39.106.190.42:46378: no certificate available for '185.157.221.99'
2020/04/21 01:26:09 http: TLS handshake error from 96.126.100.87:35932: no certificate available for '185.157.221.99'
2020/04/21 01:40:52 http: TLS handshake error from 74.82.47.5:60624: no certificate available for '185.157.221.99'
2020/04/21 03:18:10 http: TLS handshake error from 71.6.232.7:60012: no certificate available for '185.157.221.99'
2020/04/21 03:57:52 http: TLS handshake error from 171.67.71.243:48478: no certificate available for '185.157.221.99'
2020/04/21 04:53:14 http: TLS handshake error from 172.104.242.173:54165: tls: client offered only unsupported versions: [302 301]
2020/04/21 05:34:22 http: TLS handshake error from 198.108.66.176:27264: no certificate available for '185.157.221.99'
2020/04/21 07:32:06 http: TLS handshake error from 5.101.0.209:48758: no certificate available for '185.157.221.99'
2020/04/21 07:36:18 http: TLS handshake error from 162.243.132.88:35306: no certificate available for '185.157.221.99'
2020/04/21 07:41:53 http: TLS handshake error from 5.101.0.209:56526: no certificate available for '185.157.221.99'
2020/04/21 07:41:54 http: TLS handshake error from 5.101.0.209:60150: no certificate available for '185.157.221.99'
2020/04/21 07:42:05 http: TLS handshake error from 137.226.113.27:50592: no certificate available for 'www.tullingelabs.se'
2020/04/21 07:49:46 http: TLS handshake error from 5.101.0.209:43898: no certificate available for '185.157.221.99'
2020/04/21 08:40:04 http: TLS handshake error from 64.227.12.99:51897: tls: client offered only unsupported versions: [302 301]
2020/04/21 08:59:49 http: TLS handshake error from 198.108.66.96:12456: no certificate available for '185.157.221.99'
2020/04/21 11:29:41 http: TLS handshake error from 128.14.134.170:43040: no certificate available for '185.157.221.99'
2020/04/21 13:05:25 http: TLS handshake error from 45.13.93.82:43112: tls: first record does not look like a TLS handshake
2020/04/21 13:42:04 http: TLS handshake error from 137.226.113.27:42806: no certificate available for 'www.tullingelabs.se'
2020/04/21 13:55:05 http: TLS handshake error from 128.14.134.134:52588: no certificate available for '185.157.221.99'
2020/04/21 14:56:12 http: TLS handshake error from 103.107.143.98:48640: no certificate available for '185.157.221.99'
2020/04/21 15:19:32 http: TLS handshake error from 45.148.10.115:53824: no certificate available for '185.157.221.99'
2020/04/21 16:46:59 http: TLS handshake error from 146.88.240.13:38675: no certificate available for '185.157.221.99'
2020/04/21 16:47:47 http: TLS handshake error from 192.241.235.76:42214: no certificate available for '185.157.221.99'
2020/04/21 17:00:22 http: TLS handshake error from 5.101.0.209:57678: no certificate available for '185.157.221.99'
2020/04/21 17:02:05 http: TLS handshake error from 83.97.20.21:27307: no certificate available for '185.157.221.99'
2020/04/21 17:02:19 http: TLS handshake error from 83.97.20.21:9561: no certificate available for '185.157.221.99'
2020/04/21 17:02:32 http: TLS handshake error from 83.97.20.21:3044: no certificate available for '185.157.221.99'
2020/04/21 17:03:38 http: TLS handshake error from 83.97.20.21:37559: EOF
2020/04/21 17:03:42 http: TLS handshake error from 83.97.20.21:63347: no certificate available for '185.157.221.99'
2020/04/21 17:03:55 http: TLS handshake error from 83.97.20.21:39565: no certificate available for '185.157.221.99'
2020/04/21 17:04:09 http: TLS handshake error from 83.97.20.21:58429: no certificate available for '185.157.221.99'
2020/04/21 17:04:21 http: TLS handshake error from 83.97.20.21:46311: no certificate available for '185.157.221.99'
2020/04/21 17:04:35 http: TLS handshake error from 83.97.20.21:10770: tls: first record does not look like a TLS handshake
2020/04/21 17:25:14 http: TLS handshake error from 5.188.210.101:41337: unexpected EOF
2020/04/21 17:25:19 http: TLS handshake error from 5.188.210.101:15226: unexpected EOF
2020/04/21 17:25:25 http: TLS handshake error from 5.188.210.101:55805: unexpected EOF
2020/04/21 17:26:37 http: TLS handshake error from 5.188.210.101:14843: tls: first record does not look like a TLS handshake
2020/04/21 17:28:24 http: TLS handshake error from 5.188.210.101:62939: tls: first record does not look like a TLS handshake
2020/04/21 18:07:15 http: TLS handshake error from 171.67.71.98:46053: no certificate available for '185.157.221.99'
2020/04/21 18:14:21 http: TLS handshake error from 5.101.0.209:60638: no certificate available for '185.157.221.99'
2020/04/21 18:24:44 http: TLS handshake error from 162.243.130.80:44372: no certificate available for '185.157.221.99'
2020/04/21 21:23:16 http: TLS handshake error from 45.148.10.115:44520: no certificate available for '185.157.221.99'
2020/04/21 21:41:38 http: TLS handshake error from 64.225.107.87:40528: no certificate available for '185.157.221.99'
2020/04/21 21:41:44 http: TLS handshake error from 64.225.107.87:47318: no certificate available for '185.157.221.99'
2020/04/21 21:46:06 http: TLS handshake error from 223.166.32.171:32558: EOF
2020/04/21 21:46:32 http: TLS handshake error from 223.166.32.171:33546: no certificate available for '185.157.221.99'
2020/04/21 23:15:07 http: TLS handshake error from 213.32.122.82:32743: no certificate available for '185.157.221.99'
2020/04/21 23:15:07 http: TLS handshake error from 213.32.122.82:32179: no certificate available for '185.157.221.99'
2020/04/21 23:54:07 [INFO] Advancing OCSP staple for [tullingelabs.se] from 2020-04-25 11:00:00 +0000 UTC to 2020-04-28 11:00:00 +0000 UTC
2020/04/22 00:46:09 http: TLS handshake error from 137.226.113.21:34342: no certificate available for 'www.tullingelabs.se'
{"level":"info","ts":1587516847.1183965,"logger":"tls","msg":"cleaned up storage units"}
2020/04/22 01:07:07 http: TLS handshake error from 61.219.11.153:62261: tls: first record does not look like a TLS handshake
2020/04/22 02:54:07 [INFO] Advancing OCSP staple for [booking.tullingelabs.se] from 2020-04-25 14:00:00 +0000 UTC to 2020-04-28 14:00:00 +0000 UTC
2020/04/22 03:43:10 http: TLS handshake error from 74.82.47.5:42176: no certificate available for '185.157.221.99'
2020/04/22 03:56:20 http: TLS handshake error from 171.67.71.243:47968: no certificate available for '185.157.221.99'
2020/04/22 06:23:32 http: TLS handshake error from 138.246.253.15:33986: no certificate available for '185.157.221.99'
{"level":"info","ts":1587537941.1097248,"logger":"watcher","msg":"config file changed; reloading","config_file":"/etc/caddy/Caddyfile"}
{"level":"info","ts":1587537941.1108606,"msg":"using provided configuration","config_file":"/etc/caddy/Caddyfile","config_adapter":""}
{"level":"info","ts":1587537941.311167,"logger":"admin","msg":"admin endpoint started","address":"tcp/localhost:2019","enforce_origin":false,"origins":["localhost:2019","[::1]:2019","127.0.0.1:2019"]}
{"level":"info","ts":1587537941.3127728,"logger":"http","msg":"server is listening only on the HTTPS port but has no TLS connection policies; adding one to enable TLS","server_name":"srv0","https_port":443}
{"level":"info","ts":1587537941.3291843,"logger":"http","msg":"enabling automatic HTTP->HTTPS redirects","server_name":"srv0"}
2020/04/22 06:45:41 [INFO][cache:0xc0003c7f40] Started certificate maintenance routine
{"level":"info","ts":1587537941.3297515,"logger":"admin","msg":"stopped previous server"}
{"level":"info","ts":1587537941.3302448,"logger":"http","msg":"enabling automatic TLS certificate management","domains":["attendance.tullingelabs.se","booking.tullingelabs.se","tullingelabs.se"]}
2020/04/22 06:45:41 [INFO][cache:0xc00019b9f0] Stopped certificate maintenance routine
{"level":"info","ts":1587537941.5624468,"msg":"autosaved config","file":"/config/caddy/autosave.json"}
{"level":"info","ts":1587538033.1094909,"logger":"watcher","msg":"config file changed; reloading","config_file":"/etc/caddy/Caddyfile"}
{"level":"info","ts":1587538033.1101055,"msg":"using provided configuration","config_file":"/etc/caddy/Caddyfile","config_adapter":""}
{"level":"info","ts":1587538033.111471,"logger":"admin","msg":"admin endpoint started","address":"tcp/localhost:2019","enforce_origin":false,"origins":["localhost:2019","[::1]:2019","127.0.0.1:2019"]}
{"level":"info","ts":1587538033.1117818,"logger":"admin","msg":"stopped previous server"}
2020/04/22 06:47:13 [INFO][cache:0xc0002788c0] Started certificate maintenance routine
{"level":"info","ts":1587538033.1121714,"logger":"http","msg":"server is listening only on the HTTPS port but has no TLS connection policies; adding one to enable TLS","server_name":"srv0","https_port":443}
{"level":"info","ts":1587538033.1122217,"logger":"http","msg":"enabling automatic HTTP->HTTPS redirects","server_name":"srv0"}
{"level":"info","ts":1587538033.1126246,"logger":"http","msg":"enabling automatic TLS certificate management","domains":["booking.tullingelabs.se","tullingelabs.se","attendance.tullingelabs.se"]}
2020/04/22 06:47:13 [INFO][cache:0xc0003c7f40] Stopped certificate maintenance routine
{"level":"info","ts":1587538033.1153352,"msg":"autosaved config","file":"/config/caddy/autosave.json"}
2020/04/22 08:25:43 http: TLS handshake error from 198.108.66.208:14988: tls: client offered only unsupported versions: []
2020/04/22 08:45:57 http: TLS handshake error from 195.74.38.98:55696: no certificate available for 'www.tullingelabs.se'
2020/04/22 08:45:57 http: TLS handshake error from 195.74.38.98:55702: no certificate available for 'www.tullingelabs.se'
2020/04/22 08:45:57 http: TLS handshake error from 195.74.38.98:55708: no certificate available for 'www.tullingelabs.se'
2020/04/22 08:45:57 http: TLS handshake error from 195.74.38.98:55712: no certificate available for 'www.tullingelabs.se'
2020/04/22 08:45:57 http: TLS handshake error from 195.74.38.98:55716: no certificate available for 'www.tullingelabs.se'
2020/04/22 08:46:59 http: TLS handshake error from 192.71.40.14:33545: no certificate available for 'weather.tullingelabs.se'
{"level":"info","ts":1587545394.1096785,"logger":"watcher","msg":"config file changed; reloading","config_file":"/etc/caddy/Caddyfile"}
{"level":"info","ts":1587545394.1102495,"msg":"using provided configuration","config_file":"/etc/caddy/Caddyfile","config_adapter":""}
{"level":"info","ts":1587545394.1131244,"logger":"admin","msg":"admin endpoint started","address":"tcp/localhost:2019","enforce_origin":false,"origins":["localhost:2019","[::1]:2019","127.0.0.1:2019"]}
{"level":"info","ts":1587545394.1135328,"logger":"admin","msg":"stopped previous server"}
2020/04/22 08:49:54 [INFO][cache:0xc000782be0] Started certificate maintenance routine
{"level":"info","ts":1587545394.11443,"logger":"http","msg":"server is listening only on the HTTPS port but has no TLS connection policies; adding one to enable TLS","server_name":"srv0","https_port":443}
{"level":"info","ts":1587545394.1145492,"logger":"http","msg":"enabling automatic HTTP->HTTPS redirects","server_name":"srv0"}
{"level":"info","ts":1587545394.115025,"logger":"http","msg":"enabling automatic TLS certificate management","domains":["attendance.tullingelabs.se","booking.tullingelabs.se","weather.tullingelabs.se","tullingelabs.se"]}
2020/04/22 08:49:54 [INFO][cache:0xc0002788c0] Stopped certificate maintenance routine
{"level":"info","ts":1587545394.1190484,"msg":"autosaved config","file":"/config/caddy/autosave.json"}
2020/04/22 08:49:54 [INFO][weather.tullingelabs.se] Obtain certificate; acquiring lock...
2020/04/22 08:49:54 [INFO][weather.tullingelabs.se] Obtain: Lock acquired; proceeding...
2020/04/22 08:49:54 [INFO][weather.tullingelabs.se] Waiting on rate limiter...
2020/04/22 08:49:54 [INFO][weather.tullingelabs.se] Done waiting
2020/04/22 08:49:54 [INFO] [weather.tullingelabs.se] acme: Obtaining bundled SAN certificate given a CSR
2020/04/22 08:49:54 [INFO] retry due to: acme: error: 400 :: POST :: https://acme-v02.api.letsencrypt.org/acme/new-order :: urn:ietf:params:acme:error:badNonce :: JWS has an invalid anti-replay nonce: "0002hsIQNRvHOrkxunSX8YQyC_NtLsWT8x6A2Ar0bldVLJ4", url: 
2020/04/22 08:49:55 [INFO] [weather.tullingelabs.se] AuthURL: https://acme-v02.api.letsencrypt.org/acme/authz-v3/4109884048
2020/04/22 08:49:55 [INFO] [weather.tullingelabs.se] acme: Could not find solver for: tls-alpn-01
2020/04/22 08:49:55 [INFO] [weather.tullingelabs.se] acme: use http-01 solver
2020/04/22 08:49:55 [INFO] [weather.tullingelabs.se] acme: Trying to solve HTTP-01
2020/04/22 08:49:55 [INFO][weather.tullingelabs.se] Served key authentication (HTTP challenge)
2020/04/22 08:49:56 [INFO][weather.tullingelabs.se] Served key authentication (HTTP challenge)
2020/04/22 08:49:56 [INFO][weather.tullingelabs.se] Served key authentication (HTTP challenge)
2020/04/22 08:49:56 [INFO][weather.tullingelabs.se] Served key authentication (HTTP challenge)
2020/04/22 08:50:03 [INFO] [weather.tullingelabs.se] The server validated our request
2020/04/22 08:50:03 [INFO] [weather.tullingelabs.se] acme: Validations succeeded; requesting certificates
2020/04/22 08:50:06 [INFO] [weather.tullingelabs.se] Server responded with a certificate.
2020/04/22 08:50:06 [INFO][weather.tullingelabs.se] Certificate obtained successfully
2020/04/22 08:50:06 [INFO][weather.tullingelabs.se] Obtain: Releasing lock
2020/04/22 10:03:48 http: TLS handshake error from 162.243.129.4:56212: no certificate available for '185.157.221.99'
2020/04/22 10:17:47 http: TLS handshake error from 185.40.4.112:57521: tls: client offered only unsupported versions: [301]
2020/04/22 13:51:03 http: TLS handshake error from 92.118.160.57:36009: no certificate available for '185.157.221.99:443'
2020/04/22 14:04:56 http: TLS handshake error from 137.226.113.28:42458: no certificate available for 'www.tullingelabs.se'
2020/04/22 14:04:56 http: TLS handshake error from 137.226.113.28:42472: no certificate available for '185.157.221.99'
2020/04/22 14:06:36 http: TLS handshake error from 137.226.113.27:36578: no certificate available for 'www.tullingelabs.se'
2020/04/22 14:27:04 http: TLS handshake error from 51.15.43.205:41852: no certificate available for '185.157.221.99'
2020/04/22 15:22:06 http: TLS handshake error from 128.14.134.134:37436: no certificate available for '185.157.221.99'
2020/04/22 15:56:16 http: TLS handshake error from 107.189.11.78:42712: no certificate available for '185.157.221.99'
2020/04/22 16:14:29 http: TLS handshake error from 92.118.160.57:61348: tls: client offered only unsupported versions: [302 301]
2020/04/22 16:40:08 http: TLS handshake error from 172.104.242.173:44146: tls: client offered only unsupported versions: [302 301]
2020/04/22 16:40:42 http: TLS handshake error from 18.144.22.135:45994: no certificate available for '185.157.221.99'
2020/04/22 17:40:41 http: TLS handshake error from 92.118.160.49:57403: no certificate available for '185.157.221.99:443'
2020/04/22 18:07:11 http: TLS handshake error from 146.88.240.23:43550: no certificate available for '185.157.221.99'
{"level":"info","ts":1587588594.119013,"logger":"tls","msg":"cleaned up storage units"}
2020/04/22 21:47:53 http: TLS handshake error from 128.14.134.170:51660: no certificate available for '185.157.221.99'
2020/04/22 22:56:49 http: TLS handshake error from 128.199.220.9:33100: no certificate available for '185.157.221.99'
2020/04/23 00:21:15 http: TLS handshake error from 5.101.0.209:55662: no certificate available for '185.157.221.99'
2020/04/23 00:26:17 http: TLS handshake error from 5.101.0.209:57280: no certificate available for '185.157.221.99'
2020/04/23 00:49:54 [INFO] Advancing OCSP staple for [attendance.tullingelabs.se] from 2020-04-26 12:00:00 +0000 UTC to 2020-04-29 12:00:00 +0000 UTC
2020/04/23 01:48:52 http: TLS handshake error from 51.178.78.154:58918: no certificate available for '185.157.221.99'
2020/04/23 02:19:07 http: TLS handshake error from 61.219.11.153:61441: tls: first record does not look like a TLS handshake
2020/04/23 02:23:52 http: TLS handshake error from 5.101.0.209:53656: no certificate available for '185.157.221.99'
2020/04/23 02:23:52 http: TLS handshake error from 5.101.0.209:54456: no certificate available for '185.157.221.99'
2020/04/23 02:44:43 http: TLS handshake error from 103.114.104.123:51995: no certificate available for '185.157.221.99'
2020/04/23 02:44:44 http: TLS handshake error from 103.114.104.123:52137: no certificate available for '185.157.221.99'
2020/04/23 02:44:44 http: TLS handshake error from 103.114.104.123:52272: no certificate available for '185.157.221.99'
2020/04/23 02:44:45 http: TLS handshake error from 103.114.104.123:52420: no certificate available for '185.157.221.99'
2020/04/23 02:44:46 http: TLS handshake error from 103.114.104.123:52573: no certificate available for '185.157.221.99'
2020/04/23 02:44:46 http: TLS handshake error from 103.114.104.123:52730: no certificate available for '185.157.221.99'
2020/04/23 03:59:58 http: TLS handshake error from 171.67.71.243:47742: no certificate available for '185.157.221.99'
2020/04/23 04:20:30 http: TLS handshake error from 172.104.242.173:54113: tls: client offered only unsupported versions: [302 301]
2020/04/23 04:57:45 http: TLS handshake error from 216.218.206.68:4392: no certificate available for '185.157.221.99'
2020/04/23 07:11:17 http: TLS handshake error from 185.237.177.56:34449: no certificate available for '185.157.221.99'
2020/04/23 07:12:44 http: TLS handshake error from 162.243.129.20:33398: no certificate available for '185.157.221.99'
2020/04/23 09:30:11 http: TLS handshake error from 198.108.66.144:26538: no certificate available for '185.157.221.99'
2020/04/23 09:34:10 http: TLS handshake error from 45.33.80.76:45888: no certificate available for '185.157.221.99'
2020/04/23 11:14:07 http: TLS handshake error from 172.105.89.161:43408: no certificate available for '185.157.221.99'
2020/04/23 12:28:30 http: TLS handshake error from 94.102.50.150:56995: tls: client offered only unsupported versions: [302 301]
2020/04/23 12:33:19 http: TLS handshake error from 198.108.66.96:58888: no certificate available for '185.157.221.99'
2020/04/23 12:34:06 http: TLS handshake error from 45.91.226.239:37206: no certificate available for '185.157.221.99'
2020/04/23 13:37:02 http: TLS handshake error from 137.226.113.27:54960: no certificate available for 'www.tullingelabs.se'
2020/04/23 14:52:25 http: TLS handshake error from 38.134.115.15:47724: EOF
2020/04/23 17:33:39 http: TLS handshake error from 128.14.134.134:33540: no certificate available for '185.157.221.99'
{"level":"info","ts":1587664796.1095905,"logger":"watcher","msg":"config file changed; reloading","config_file":"/etc/caddy/Caddyfile"}
{"level":"info","ts":1587664796.1100836,"msg":"using provided configuration","config_file":"/etc/caddy/Caddyfile","config_adapter":""}
{"level":"info","ts":1587664796.1132622,"logger":"admin","msg":"admin endpoint started","address":"tcp/localhost:2019","enforce_origin":false,"origins":["localhost:2019","[::1]:2019","127.0.0.1:2019"]}
{"level":"info","ts":1587664796.1137035,"logger":"admin","msg":"stopped previous server"}
2020/04/23 17:59:56 [INFO][cache:0xc0001e4050] Started certificate maintenance routine
{"level":"info","ts":1587664796.1139488,"logger":"http","msg":"server is listening only on the HTTPS port but has no TLS connection policies; adding one to enable TLS","server_name":"srv0","https_port":443}
{"level":"info","ts":1587664796.1139832,"logger":"http","msg":"enabling automatic HTTP->HTTPS redirects","server_name":"srv0"}
{"level":"info","ts":1587664796.1139987,"logger":"http","msg":"enabling automatic HTTP->HTTPS redirects","server_name":"srv1"}
{"level":"info","ts":1587664796.117605,"logger":"tls","msg":"cleaned up storage units"}
{"level":"error","ts":1587664796.1179454,"logger":"watcher","msg":"applying latest config","config_file":"/etc/caddy/Caddyfile","error":"loading new config: http app module: start: tcp: listening on :8081: listen tcp :8081: bind: address already in use"}
{"level":"info","ts":1587664902.1096866,"logger":"watcher","msg":"config file changed; reloading","config_file":"/etc/caddy/Caddyfile"}
{"level":"info","ts":1587664902.1108336,"msg":"using provided configuration","config_file":"/etc/caddy/Caddyfile","config_adapter":""}
{"level":"info","ts":1587664902.1128986,"logger":"admin.api","msg":"config is unchanged"}
2020/04/23 18:26:49 http: TLS handshake error from 207.46.13.237:8138: no certificate available for 'booking.tullingelabs.se'
2020/04/23 18:26:49 http: TLS handshake error from 207.46.13.237:8213: tls: client offered only unsupported versions: [302 301]
2020/04/23 18:26:50 http: TLS handshake error from 207.46.13.237:8282: tls: client offered only unsupported versions: [301]
2020/04/23 18:26:50 http: TLS handshake error from 207.46.13.237:8357: EOF
{"level":"info","ts":1587666763.1097364,"logger":"watcher","msg":"config file changed; reloading","config_file":"/etc/caddy/Caddyfile"}
{"level":"info","ts":1587666763.1107955,"msg":"using provided configuration","config_file":"/etc/caddy/Caddyfile","config_adapter":""}
{"level":"info","ts":1587666763.112986,"logger":"admin","msg":"admin endpoint started","address":"tcp/localhost:2019","enforce_origin":false,"origins":["localhost:2019","[::1]:2019","127.0.0.1:2019"]}
{"level":"info","ts":1587666763.1134996,"logger":"admin","msg":"stopped previous server"}
{"level":"info","ts":1587666763.113718,"logger":"http","msg":"server is listening only on the HTTPS port but has no TLS connection policies; adding one to enable TLS","server_name":"srv0","https_port":443}
{"level":"info","ts":1587666763.1138117,"logger":"http","msg":"enabling automatic HTTP->HTTPS redirects","server_name":"srv0"}
2020/04/23 18:32:43 [INFO][cache:0xc0003c6eb0] Started certificate maintenance routine
{"level":"info","ts":1587666763.114187,"logger":"http","msg":"enabling automatic TLS certificate management","domains":["attendance.tullingelabs.se","booking.tullingelabs.se","weather.tullingelabs.se","tullingelabs.se"]}
2020/04/23 18:32:43 [INFO][cache:0xc000782be0] Stopped certificate maintenance routine
{"level":"info","ts":1587666763.5637667,"msg":"autosaved config","file":"/config/caddy/autosave.json"}
2020/04/23 19:27:58 http: TLS handshake error from 146.88.240.23:59294: no certificate available for '185.157.221.99'
2020/04/23 20:25:19 http: TLS handshake error from 128.14.134.170:37222: no certificate available for '185.157.221.99'
2020/04/23 21:05:20 http: TLS handshake error from 185.202.2.57:1158: tls: first record does not look like a TLS handshake
2020/04/23 21:52:07 http: TLS handshake error from 83.97.20.21:49148: no certificate available for '185.157.221.99'
2020/04/23 21:52:19 http: TLS handshake error from 83.97.20.21:44269: no certificate available for '185.157.221.99'
2020/04/23 21:52:31 http: TLS handshake error from 83.97.20.21:19359: no certificate available for '185.157.221.99'
2020/04/23 21:52:43 http: TLS handshake error from 83.97.20.21:45327: no certificate available for '185.157.221.99'
2020/04/23 21:52:55 http: TLS handshake error from 83.97.20.21:49693: no certificate available for '185.157.221.99'
2020/04/23 21:53:07 http: TLS handshake error from 83.97.20.21:59304: no certificate available for '185.157.221.99'
2020/04/23 21:53:19 http: TLS handshake error from 83.97.20.21:27476: no certificate available for '185.157.221.99'
2020/04/23 21:53:31 http: TLS handshake error from 83.97.20.21:60241: no certificate available for '185.157.221.99'
2020/04/23 21:53:42 http: TLS handshake error from 83.97.20.21:28618: tls: first record does not look like a TLS handshake
2020/04/23 22:18:27 http: TLS handshake error from 144.21.103.96:55367: no certificate available for '185.157.221.99'
2020/04/23 22:18:27 http: TLS handshake error from 144.21.103.96:55419: no certificate available for '185.157.221.99'
2020/04/23 22:18:27 http: TLS handshake error from 144.21.103.96:55442: no certificate available for '185.157.221.99'
2020/04/23 22:18:27 http: TLS handshake error from 144.21.103.96:55473: no certificate available for '185.157.221.99'
2020/04/23 22:18:27 http: TLS handshake error from 144.21.103.96:55494: no certificate available for '185.157.221.99'
2020/04/24 00:25:37 http: TLS handshake error from 172.104.242.173:48120: tls: first record does not look like a TLS handshake
2020/04/24 01:11:09 http: TLS handshake error from 94.242.26.158:56352: EOF
2020/04/24 01:11:09 http: TLS handshake error from 94.242.26.158:63995: no certificate available for '185.157.221.99'
2020/04/24 01:11:09 http: TLS handshake error from 94.242.26.158:64860: no certificate available for '185.157.221.99'
2020/04/24 01:11:09 http: TLS handshake error from 94.242.26.158:65185: no certificate available for '185.157.221.99'
2020/04/24 01:11:10 http: TLS handshake error from 94.242.26.158:65521: no certificate available for '185.157.221.99'
2020/04/24 01:11:10 http: TLS handshake error from 94.242.26.158:65506: no certificate available for '185.157.221.99'
2020/04/24 01:11:10 http: TLS handshake error from 94.242.26.158:49633: no certificate available for '185.157.221.99'
2020/04/24 01:23:40 http: TLS handshake error from 198.108.66.16:21348: no certificate available for '185.157.221.99'
2020/04/24 01:24:44 http: TLS handshake error from 158.101.196.14:60804: no certificate available for '185.157.221.99'
2020/04/24 01:24:44 http: TLS handshake error from 158.101.196.14:60878: no certificate available for '185.157.221.99'
2020/04/24 01:24:45 http: TLS handshake error from 158.101.196.14:60954: no certificate available for '185.157.221.99'
2020/04/24 01:24:45 http: TLS handshake error from 158.101.196.14:61018: no certificate available for '185.157.221.99'
2020/04/24 01:24:45 http: TLS handshake error from 158.101.196.14:61106: no certificate available for '185.157.221.99'
2020/04/24 01:24:45 http: TLS handshake error from 158.101.196.14:61188: no certificate available for '185.157.221.99'
2020/04/24 01:43:20 http: TLS handshake error from 165.227.220.53:52760: no certificate available for 'booking.tullingelabs.se'
2020/04/24 02:00:28 http: TLS handshake error from 84.216.183.212:43720: no certificate available for 'tullingelabs.se'
2020/04/24 02:20:07 http: TLS handshake error from 185.254.70.34:32271: no certificate available for 'www.tullingelabs.se'
2020/04/24 02:23:39 http: TLS handshake error from 208.93.152.17:21030: EOF
2020/04/24 02:23:40 http: TLS handshake error from 208.93.152.17:44801: tls: unsupported SSLv2 handshake received
2020/04/24 02:23:43 http: TLS handshake error from 208.93.152.17:60397: tls: client offered only unsupported versions: []
2020/04/24 02:23:44 http: TLS handshake error from 208.93.152.17:32642: tls: client offered only unsupported versions: [301]
2020/04/24 02:23:45 http: TLS handshake error from 208.93.152.17:33316: tls: client offered only unsupported versions: [302 301]
2020/04/24 02:23:54 http: TLS handshake error from 208.93.152.17:25964: no certificate available for '185.157.221.99'
2020/04/24 02:23:55 http: TLS handshake error from 208.93.152.17:23998: no certificate available for '185.157.221.99'
2020/04/24 02:40:40 http: TLS handshake error from 103.208.72.22:10093: no certificate available for '185.157.221.99'
2020/04/24 02:40:41 http: TLS handshake error from 103.208.72.22:31767: no certificate available for '185.157.221.99'
2020/04/24 02:40:41 http: TLS handshake error from 103.208.72.22:47329: no certificate available for '185.157.221.99'
2020/04/24 02:40:42 http: TLS handshake error from 103.208.72.22:29811: no certificate available for '185.157.221.99'
2020/04/24 02:40:42 http: TLS handshake error from 103.208.72.22:2188: no certificate available for '185.157.221.99'
2020/04/24 02:40:43 http: TLS handshake error from 103.208.72.22:17323: no certificate available for '185.157.221.99'
2020/04/24 02:49:41 http: TLS handshake error from 193.118.53.194:43284: no certificate available for '185.157.221.99'
2020/04/24 02:59:48 http: TLS handshake error from 196.52.43.86:62066: tls: client offered only unsupported versions: [302 301]
2020/04/24 03:30:35 http: TLS handshake error from 45.13.93.82:54406: tls: first record does not look like a TLS handshake
2020/04/24 03:51:46 http: TLS handshake error from 192.81.128.37:44886: no certificate available for '185.157.221.99'
2020/04/24 03:57:37 http: TLS handshake error from 171.67.71.243:34730: no certificate available for '185.157.221.99'
2020/04/24 04:29:31 http: TLS handshake error from 216.218.206.67:12760: no certificate available for '185.157.221.99'
2020/04/24 05:37:56 http: TLS handshake error from 66.249.73.238:59197: no certificate available for 'booking.tullingelabs.se'
2020/04/24 05:38:52 http: TLS handshake error from 66.249.73.238:59544: no certificate available for 'booking.tullingelabs.se'
2020/04/24 05:52:29 http: TLS handshake error from 92.244.29.246:60916: no certificate available for 'attendance.tullingelabs.se'
2020/04/24 05:57:27 http: TLS handshake error from 92.244.29.246:60960: no certificate available for 'booking.tullingelabs.se'
2020/04/24 05:57:28 http: TLS handshake error from 92.244.29.246:60964: no certificate available for 'booking.tullingelabs.se'
2020/04/24 06:19:23 http: TLS handshake error from 185.173.35.41:57721: no certificate available for '185.157.221.99:443'
[...MORE OF THE SAME]

[greenpau (Paul G.)]

@einarpersson , can you paste JSON config here (from management port)? I am interested to see what is in your tls_connection_policies and tls. For example here is mine.

         "tls_connection_policies": [
            {
              "certificate_selection": {
                "any_tag": [
                  "cert0"
                ]
              }
            }
          ]
        }
      }
    },
    "tls": {
      "certificates": {
        "load_files": [
          {
            "certificate": "/etc/caddy/tls/server.crt",
            "key": "/etc/caddy/tls/server.key",
            "tags": [
              "cert0"
            ]
          }
        ]
      }
    }

[einarpersson]

Where do I locate this JSON file? When I exec into the container, I find autosave.json which contains this:

{
    "apps": {
        "http": {
            "servers": {
                "srv0": {
                    "listen": [
                        ":443"
                    ],
                    "routes": [
                        {
                            "handle": [
                                {
                                    "handler": "subroute",
                                    "routes": [
                                        {
                                            "handle": [
                                                {
                                                    "body": "Welcome back later.",
                                                    "handler": "static_response"
                                                }
                                            ]
                                        }
                                    ]
                                }
                            ],
                            "match": [
                                {
                                    "host": [
                                        "attendance.tullingelabs.se"
                                    ]
                                }
                            ],
                            "terminal": true
                        },
                        {
                            "handle": [
                                {
                                    "handler": "subroute",
                                    "routes": [
                                        {
                                            "handle": [
                                                {
                                                    "body": "Service currently unavailable. Welcome back in 2021.",
                                                    "handler": "static_response"
                                                }
                                            ]
                                        }
                                    ]
                                }
                            ],
                            "match": [
                                {
                                    "host": [
                                        "booking.tullingelabs.se"
                                    ]
                                }
                            ],
                            "terminal": true
                        },
                        {
                            "handle": [
                                {
                                    "handler": "subroute",
                                    "routes": [
                                        {
                                            "handle": [
                                                {
                                                    "body": "It is sunny. Or is it?",
                                                    "handler": "static_response"
                                                }
                                            ]
                                        }
                                    ]
                                }
                            ],
                            "match": [
                                {
                                    "host": [
                                        "weather.tullingelabs.se"
                                    ]
                                }
                            ],
                            "terminal": true
                        },
                        {
                            "handle": [
                                {
                                    "handler": "subroute",
                                    "routes": [
                                        {
                                            "handle": [
                                                {
                                                    "body": "Welcome back later.",
                                                    "handler": "static_response"
                                                }
                                            ]
                                        }
                                    ]
                                }
                            ],
                            "match": [
                                {
                                    "host": [
                                        "tullingelabs.se"
                                    ]
                                }
                            ],
                            "terminal": true
                        }
                    ]
                }
            }
        }
    }
}

[francislavoie]

That’s probably what we’re looking for - for reference, you can use the API with curl to fetch the currently running config

API - Caddy Documentation

You can also use the caddy adapt command to convert your Caddyfile to JSON without running Caddy.

[matt (Matt Holt)]

Not really familiar with Docker but I can answer one part:

@einarpersson

I don’t even know if 2020/04/24 16:11:52 http: TLS handshake error from 61.153.110.83:45404: no certificate available for '185.157.221.99' has actually anything to do with the error I have, since it appears to have been happening over several days whereas the problem I’ve experienced only started last night.

That error just means that clients are accessing your server by IP address instead of by domain name, and Caddy doesn’t have a certificate for that IP address to present to the clients.

[einarpersson]

So that is just… bots…? scanning for ports? or what? I’m new into all this, so I’m happy to pick up any learnings along the way.

Thank you. My main problem still persists though

[matt (Matt Holt)]

@einarpersson

So that is just… bots…? scanning for ports? or what? I’m new into all this, so I’m happy to pick up any learnings along the way.

Yeah, usually some or all of the above. Pretty normal noise for the Internet, unfortunately.

@einarpersson

I did do a docker volume prune at some time which could have caused this, but I was under the impression that that command only removed volumes not used by any running containers. I don’t know what else I could have done to have caused it stopped working suddenly.

Regrettably, I don’t know anything about Docker so I can’t help with this question. Someone else probably can though!

Best bet would be to make sure it’s using the Caddyfile you think it’s using.

[einarpersson]

So this is weird: tullingelabs.se works but not the subdomains eg. weather.tullingelabs.se

I notice that in the config that is running below (curl ...) I don’t see anything about tls_connection_policies and tls @greenpau . But I have gotten the Caddyfile to work before, with subdomains. I thought that Caddy managed certificates automatically…? Just put the patterns in the Caddyfile and Caddy will reach out to let’s encrypt and do it’s thing. Do I have to do anything else?

{
  "apps": {
    "http": {
      "servers": {
        "srv0": {
          "listen": [
            ":443"
          ],
          "routes": [
            {
              "handle": [
                {
                  "handler": "subroute",
                  "routes": [
                    {
                      "handle": [
                        {
                          "body": "Welcome back later.",
                          "handler": "static_response"
                        }
                      ]
                    }
                  ]
                }
              ],
              "match": [
                {
                  "host": [
                    "attendance.tullingelabs.se"
                  ]
                }
              ],
              "terminal": true
            },
            {
              "handle": [
                {
                  "handler": "subroute",
                  "routes": [
                    {
                      "handle": [
                        {
                          "body": "Service currently unavailable. Welcome back in 2021.",
                          "handler": "static_response"
                        }
                      ]
                    }
                  ]
                }
              ],
              "match": [
                {
                  "host": [
                    "booking.tullingelabs.se"
                  ]
                }
              ],
              "terminal": true
            },
            {
              "handle": [
                {
                  "handler": "subroute",
                  "routes": [
                    {
                      "handle": [
                        {
                          "body": "It is sunny. Or is it?",
                          "handler": "static_response"
                        }
                      ]
                    }
                  ]
                }
              ],
              "match": [
                {
                  "host": [
                    "weather.tullingelabs.se"
                  ]
                }
              ],
              "terminal": true
            },
            {
              "handle": [
                {
                  "handler": "subroute",
                  "routes": [
                    {
                      "handle": [
                        {
                          "body": "Welcome back later.",
                          "handler": "static_response"
                        }
                      ]
                    }
                  ]
                }
              ],
              "match": [
                {
                  "host": [
                    "tullingelabs.se"
                  ]
                }
              ],
              "terminal": true
            }
          ]
        }
      }
    }
  }
}

[matt (Matt Holt)]

@einarpersson What is the problem exactly? Maybe help us reproduce the problem on our own machines (i.e. use curl, drop the docker stuff, simplify as much as possible, and show us an error message that you’re seeing vs. what you expect to have happen) and someone can help you out.

[einarpersson]

@matt Sure! I’ll try to clarify:

I’ve had a caddy docker container working just fine, using the Caddyfile specified in the first post. My first experience with Caddy. This, however, stopped working over a night. When I navigated to weather.tullingelabs.se I no longer saw my dummy hello-world-isch response “It is sunny. Or is it?” specified in the Caddyfile. Instead I get a ERR_SSL_PROTOCOL_ERROR. And I don’t know what happened.

I could of course just try and restart the container, or even remove it and all volumes (containing the certificates). And maybe it would work. But I wouldn’t learn anything, and I’d not feel confident going forward.

So that is why I’m asking for help in investigating the matter.

The current status is that tullingelabs.se seems to work, but none of the subdomains. Any ideas on how I can proceed to troubleshoot it?

[francislavoie]

What do your logs say?

Probably some thing like docker logs stoic_cannon based on your original post

[matt (Matt Holt)]

@einarpersson Hm, what you’re saying isn’t adding up with your logs.

“loading new config: http app module: start: tcp: listening on :8081: listen tcp :8081: bind: address already in use”

But I don’t see anywhere in your config that :8081 is used. Right now I’m not convinced anything is really wrong since there’s not enough information here about your setup, your config, and what actually happened.

Can you please explain the problem so we can reproduce it? As minimally as possible would be ideal (i.e. curl commands, no Docker, etc.) I think we just need more information.

Edit: What time did the subdomains stop working?

[einarpersson]

I understand the difficulty - but I really try to be as exhaustive as possible

The command I ran initially
docker run -d --network host -v caddy_data:/data -v caddy_config:/config -v $PWD/Caddyfile:/etc/caddy/Caddyfile --name stoic_cannon caddy caddy run --watch --config /etc/caddy/Caddyfile

Since then, I’ve updated the Caddyfile (mounted on the host) a couple of times. Everytime caddy has reloaded fine. I also posted the current caddyfile in the initial post:
Caddyfile

booking.tullingelabs.se {
    respond "Service currently unavailable. Welcome back in 2021."
}

weather.tullingelabs.se {
    respond "It is sunny. Or is it?"
}

attendance.tullingelabs.se {
    #reverse_proxy localhost:5000
    respond "Welcome back later."
}

tullingelabs.se {
    respond "Welcome back later."
}

The problem
On the morning of the 24th I discovered that routes were suddenly responding with ERR_SSL_PROTOCOL_ERROR. The evening before (23th) I had performed a docker volume prune as described in the first post, but it shouldn’t affect Caddy to my understanding. Other than that I cannot recollect anything that might have caused it.

The reason there is a :8081 in the logs is that somewhere along the way I temporarly fired up a docker container with mongo-express (a admin UI for mongodb) bypassing caddy.

I did a curl "http://localhost:2019/config/" | jq as requested and posted the results above.

Timeline
I cannot post the whole logfile since I’d hit the max number of characters, but if you just look at this portion which begins with Caddy acquiring certs successfully and ends with me not being able to go to the subdomain:

{"level":"info","ts":1587545394.115025,"logger":"http","msg":"enabling automatic TLS certificate management","domains":["attendance.tullingelabs.se","booking.tullingelabs.se","weather.tullingelabs.se","tullingelabs.se"]}
2020/04/22 08:49:54 [INFO][cache:0xc0002788c0] Stopped certificate maintenance routine
{"level":"info","ts":1587545394.1190484,"msg":"autosaved config","file":"/config/caddy/autosave.json"}
2020/04/22 08:49:54 [INFO][weather.tullingelabs.se] Obtain certificate; acquiring lock...
2020/04/22 08:49:54 [INFO][weather.tullingelabs.se] Obtain: Lock acquired; proceeding...
2020/04/22 08:49:54 [INFO][weather.tullingelabs.se] Waiting on rate limiter...
2020/04/22 08:49:54 [INFO][weather.tullingelabs.se] Done waiting
2020/04/22 08:49:54 [INFO] [weather.tullingelabs.se] acme: Obtaining bundled SAN certificate given a CSR
2020/04/22 08:49:54 [INFO] retry due to: acme: error: 400 :: POST :: https://acme-v02.api.letsencrypt.org/acme/new-order :: urn:ietf:params:acme:error:badNonce :: JWS has an invalid anti-replay nonce: "0002hsIQNRvHOrkxunSX8YQyC_NtLsWT8x6A2Ar0bldVLJ4", url:
2020/04/22 08:49:55 [INFO] [weather.tullingelabs.se] AuthURL: https://acme-v02.api.letsencrypt.org/acme/authz-v3/4109884048
2020/04/22 08:49:55 [INFO] [weather.tullingelabs.se] acme: Could not find solver for: tls-alpn-01
2020/04/22 08:49:55 [INFO] [weather.tullingelabs.se] acme: use http-01 solver
2020/04/22 08:49:55 [INFO] [weather.tullingelabs.se] acme: Trying to solve HTTP-01
2020/04/22 08:49:55 [INFO][weather.tullingelabs.se] Served key authentication (HTTP challenge)
2020/04/22 08:49:56 [INFO][weather.tullingelabs.se] Served key authentication (HTTP challenge)
2020/04/22 08:49:56 [INFO][weather.tullingelabs.se] Served key authentication (HTTP challenge)
2020/04/22 08:49:56 [INFO][weather.tullingelabs.se] Served key authentication (HTTP challenge)
2020/04/22 08:50:03 [INFO] [weather.tullingelabs.se] The server validated our request
2020/04/22 08:50:03 [INFO] [weather.tullingelabs.se] acme: Validations succeeded; requesting certificates
2020/04/22 08:50:06 [INFO] [weather.tullingelabs.se] Server responded with a certificate.
2020/04/22 08:50:06 [INFO][weather.tullingelabs.se] Certificate obtained successfully
2020/04/22 08:50:06 [INFO][weather.tullingelabs.se] Obtain: Releasing lock
2020/04/22 10:03:48 http: TLS handshake error from 162.243.129.4:56212: no certificate available for '185.157.221.99'
2020/04/22 10:17:47 http: TLS handshake error from 185.40.4.112:57521: tls: client offered only unsupported versions: [301]
2020/04/22 13:51:03 http: TLS handshake error from 92.118.160.57:36009: no certificate available for '185.157.221.99:443'
2020/04/22 14:04:56 http: TLS handshake error from 137.226.113.28:42458: no certificate available for 'www.tullingelabs.se'
2020/04/22 14:04:56 http: TLS handshake error from 137.226.113.28:42472: no certificate available for '185.157.221.99'
2020/04/22 14:06:36 http: TLS handshake error from 137.226.113.27:36578: no certificate available for 'www.tullingelabs.se'
2020/04/22 14:27:04 http: TLS handshake error from 51.15.43.205:41852: no certificate available for '185.157.221.99'
2020/04/22 15:22:06 http: TLS handshake error from 128.14.134.134:37436: no certificate available for '185.157.221.99'
2020/04/22 15:56:16 http: TLS handshake error from 107.189.11.78:42712: no certificate available for '185.157.221.99'
2020/04/22 16:14:29 http: TLS handshake error from 92.118.160.57:61348: tls: client offered only unsupported versions: [302 301]
2020/04/22 16:40:08 http: TLS handshake error from 172.104.242.173:44146: tls: client offered only unsupported versions: [302 301]
2020/04/22 16:40:42 http: TLS handshake error from 18.144.22.135:45994: no certificate available for '185.157.221.99'
2020/04/22 17:40:41 http: TLS handshake error from 92.118.160.49:57403: no certificate available for '185.157.221.99:443'
2020/04/22 18:07:11 http: TLS handshake error from 146.88.240.23:43550: no certificate available for '185.157.221.99'
{"level":"info","ts":1587588594.119013,"logger":"tls","msg":"cleaned up storage units"}
2020/04/22 21:47:53 http: TLS handshake error from 128.14.134.170:51660: no certificate available for '185.157.221.99'
2020/04/22 22:56:49 http: TLS handshake error from 128.199.220.9:33100: no certificate available for '185.157.221.99'
2020/04/23 00:21:15 http: TLS handshake error from 5.101.0.209:55662: no certificate available for '185.157.221.99'
2020/04/23 00:26:17 http: TLS handshake error from 5.101.0.209:57280: no certificate available for '185.157.221.99'
2020/04/23 00:49:54 [INFO] Advancing OCSP staple for [attendance.tullingelabs.se] from 2020-04-26 12:00:00 +0000 UTC to 2020-04-29 12:00:00 +0000 UTC
2020/04/23 01:48:52 http: TLS handshake error from 51.178.78.154:58918: no certificate available for '185.157.221.99'
2020/04/23 02:19:07 http: TLS handshake error from 61.219.11.153:61441: tls: first record does not look like a TLS handshake
2020/04/23 02:23:52 http: TLS handshake error from 5.101.0.209:53656: no certificate available for '185.157.221.99'
2020/04/23 02:23:52 http: TLS handshake error from 5.101.0.209:54456: no certificate available for '185.157.221.99'
2020/04/23 02:44:43 http: TLS handshake error from 103.114.104.123:51995: no certificate available for '185.157.221.99'
2020/04/23 02:44:44 http: TLS handshake error from 103.114.104.123:52137: no certificate available for '185.157.221.99'
2020/04/23 02:44:44 http: TLS handshake error from 103.114.104.123:52272: no certificate available for '185.157.221.99'
2020/04/23 02:44:45 http: TLS handshake error from 103.114.104.123:52420: no certificate available for '185.157.221.99'
2020/04/23 02:44:46 http: TLS handshake error from 103.114.104.123:52573: no certificate available for '185.157.221.99'
2020/04/23 02:44:46 http: TLS handshake error from 103.114.104.123:52730: no certificate available for '185.157.221.99'
2020/04/23 03:59:58 http: TLS handshake error from 171.67.71.243:47742: no certificate available for '185.157.221.99'
2020/04/23 04:20:30 http: TLS handshake error from 172.104.242.173:54113: tls: client offered only unsupported versions: [302 301]
2020/04/23 04:57:45 http: TLS handshake error from 216.218.206.68:4392: no certificate available for '185.157.221.99'
2020/04/23 07:11:17 http: TLS handshake error from 185.237.177.56:34449: no certificate available for '185.157.221.99'
2020/04/23 07:12:44 http: TLS handshake error from 162.243.129.20:33398: no certificate available for '185.157.221.99'
2020/04/23 09:30:11 http: TLS handshake error from 198.108.66.144:26538: no certificate available for '185.157.221.99'
2020/04/23 09:34:10 http: TLS handshake error from 45.33.80.76:45888: no certificate available for '185.157.221.99'
2020/04/23 11:14:07 http: TLS handshake error from 172.105.89.161:43408: no certificate available for '185.157.221.99'
2020/04/23 12:28:30 http: TLS handshake error from 94.102.50.150:56995: tls: client offered only unsupported versions: [302 301]
2020/04/23 12:33:19 http: TLS handshake error from 198.108.66.96:58888: no certificate available for '185.157.221.99'
2020/04/23 12:34:06 http: TLS handshake error from 45.91.226.239:37206: no certificate available for '185.157.221.99'
2020/04/23 13:37:02 http: TLS handshake error from 137.226.113.27:54960: no certificate available for 'www.tullingelabs.se'
2020/04/23 14:52:25 http: TLS handshake error from 38.134.115.15:47724: EOF
2020/04/23 17:33:39 http: TLS handshake error from 128.14.134.134:33540: no certificate available for '185.157.221.99'
{"level":"info","ts":1587664796.1095905,"logger":"watcher","msg":"config file changed; reloading","config_file":"/etc/caddy/Caddyfile"}
{"level":"info","ts":1587664796.1100836,"msg":"using provided configuration","config_file":"/etc/caddy/Caddyfile","config_adapter":""}
{"level":"info","ts":1587664796.1132622,"logger":"admin","msg":"admin endpoint started","address":"tcp/localhost:2019","enforce_origin":false,"origins":["localhost:2019","[::1]:2019","127.0.0.1:2019"]}
{"level":"info","ts":1587664796.1137035,"logger":"admin","msg":"stopped previous server"}
2020/04/23 17:59:56 [INFO][cache:0xc0001e4050] Started certificate maintenance routine
{"level":"info","ts":1587664796.1139488,"logger":"http","msg":"server is listening only on the HTTPS port but has no TLS connection policies; adding one to enable TLS","server_name":"srv0","https_port":443}
{"level":"info","ts":1587664796.1139832,"logger":"http","msg":"enabling automatic HTTP->HTTPS redirects","server_name":"srv0"}
{"level":"info","ts":1587664796.1139987,"logger":"http","msg":"enabling automatic HTTP->HTTPS redirects","server_name":"srv1"}
{"level":"info","ts":1587664796.117605,"logger":"tls","msg":"cleaned up storage units"}
{"level":"error","ts":1587664796.1179454,"logger":"watcher","msg":"applying latest config","config_file":"/etc/caddy/Caddyfile","error":"loading new config: http app module: start: tcp: listening on :8081: listen tcp :8081: bind: address already in use"}
{"level":"info","ts":1587664902.1096866,"logger":"watcher","msg":"config file changed; reloading","config_file":"/etc/caddy/Caddyfile"}
{"level":"info","ts":1587664902.1108336,"msg":"using provided configuration","config_file":"/etc/caddy/Caddyfile","config_adapter":""}
{"level":"info","ts":1587664902.1128986,"logger":"admin.api","msg":"config is unchanged"}
2020/04/23 18:26:49 http: TLS handshake error from 207.46.13.237:8138: no certificate available for 'booking.tullingelabs.se'
2020/04/23 18:26:49 http: TLS handshake error from 207.46.13.237:8213: tls: client offered only unsupported versions: [302 301]
2020/04/23 18:26:50 http: TLS handshake error from 207.46.13.237:8282: tls: client offered only unsupported versions: [301]
2020/04/23 18:26:50 http: TLS handshake error from 207.46.13.237:8357: EOF
{"level":"info","ts":1587666763.1097364,"logger":"watcher","msg":"config file changed; reloading","config_file":"/etc/caddy/Caddyfile"}
{"level":"info","ts":1587666763.1107955,"msg":"using provided configuration","config_file":"/etc/caddy/Caddyfile","config_adapter":""}
{"level":"info","ts":1587666763.112986,"logger":"admin","msg":"admin endpoint started","address":"tcp/localhost:2019","enforce_origin":false,"origins":["localhost:2019","[::1]:2019","127.0.0.1:2019"]}
{"level":"info","ts":1587666763.1134996,"logger":"admin","msg":"stopped previous server"}
{"level":"info","ts":1587666763.113718,"logger":"http","msg":"server is listening only on the HTTPS port but has no TLS connection policies; adding one to enable TLS","server_name":"srv0","https_port":443}
{"level":"info","ts":1587666763.1138117,"logger":"http","msg":"enabling automatic HTTP->HTTPS redirects","server_name":"srv0"}
2020/04/23 18:32:43 [INFO][cache:0xc0003c6eb0] Started certificate maintenance routine
{"level":"info","ts":1587666763.114187,"logger":"http","msg":"enabling automatic TLS certificate management","domains":["attendance.tullingelabs.se","booking.tullingelabs.se","weather.tullingelabs.se","tullingelabs.se"]}
2020/04/23 18:32:43 [INFO][cache:0xc000782be0] Stopped certificate maintenance routine
{"level":"info","ts":1587666763.5637667,"msg":"autosaved config","file":"/config/caddy/autosave.json"}
2020/04/23 19:27:58 http: TLS handshake error from 146.88.240.23:59294: no certificate available for '185.157.221.99'
2020/04/23 20:25:19 http: TLS handshake error from 128.14.134.170:37222: no certificate available for '185.157.221.99'
2020/04/23 21:05:20 http: TLS handshake error from 185.202.2.57:1158: tls: first record does not look like a TLS handshake
2020/04/23 21:52:07 http: TLS handshake error from 83.97.20.21:49148: no certificate available for '185.157.221.99'
2020/04/23 21:52:19 http: TLS handshake error from 83.97.20.21:44269: no certificate available for '185.157.221.99'
2020/04/23 21:52:31 http: TLS handshake error from 83.97.20.21:19359: no certificate available for '185.157.221.99'
2020/04/23 21:52:43 http: TLS handshake error from 83.97.20.21:45327: no certificate available for '185.157.221.99'
2020/04/23 21:52:55 http: TLS handshake error from 83.97.20.21:49693: no certificate available for '185.157.221.99'
2020/04/23 21:53:07 http: TLS handshake error from 83.97.20.21:59304: no certificate available for '185.157.221.99'
2020/04/23 21:53:19 http: TLS handshake error from 83.97.20.21:27476: no certificate available for '185.157.221.99'
2020/04/23 21:53:31 http: TLS handshake error from 83.97.20.21:60241: no certificate available for '185.157.221.99'
2020/04/23 21:53:42 http: TLS handshake error from 83.97.20.21:28618: tls: first record does not look like a TLS handshake
2020/04/23 22:18:27 http: TLS handshake error from 144.21.103.96:55367: no certificate available for '185.157.221.99'
2020/04/23 22:18:27 http: TLS handshake error from 144.21.103.96:55419: no certificate available for '185.157.221.99'
2020/04/23 22:18:27 http: TLS handshake error from 144.21.103.96:55442: no certificate available for '185.157.221.99'
2020/04/23 22:18:27 http: TLS handshake error from 144.21.103.96:55473: no certificate available for '185.157.221.99'
2020/04/23 22:18:27 http: TLS handshake error from 144.21.103.96:55494: no certificate available for '185.157.221.99'
2020/04/24 00:25:37 http: TLS handshake error from 172.104.242.173:48120: tls: first record does not look like a TLS handshake
2020/04/24 01:11:09 http: TLS handshake error from 94.242.26.158:56352: EOF
2020/04/24 01:11:09 http: TLS handshake error from 94.242.26.158:63995: no certificate available for '185.157.221.99'
2020/04/24 01:11:09 http: TLS handshake error from 94.242.26.158:64860: no certificate available for '185.157.221.99'
2020/04/24 01:11:09 http: TLS handshake error from 94.242.26.158:65185: no certificate available for '185.157.221.99'
2020/04/24 01:11:10 http: TLS handshake error from 94.242.26.158:65521: no certificate available for '185.157.221.99'
2020/04/24 01:11:10 http: TLS handshake error from 94.242.26.158:65506: no certificate available for '185.157.221.99'
2020/04/24 01:11:10 http: TLS handshake error from 94.242.26.158:49633: no certificate available for '185.157.221.99'
2020/04/24 01:23:40 http: TLS handshake error from 198.108.66.16:21348: no certificate available for '185.157.221.99'
2020/04/24 01:24:44 http: TLS handshake error from 158.101.196.14:60804: no certificate available for '185.157.221.99'
2020/04/24 01:24:44 http: TLS handshake error from 158.101.196.14:60878: no certificate available for '185.157.221.99'
2020/04/24 01:24:45 http: TLS handshake error from 158.101.196.14:60954: no certificate available for '185.157.221.99'
2020/04/24 01:24:45 http: TLS handshake error from 158.101.196.14:61018: no certificate available for '185.157.221.99'
2020/04/24 01:24:45 http: TLS handshake error from 158.101.196.14:61106: no certificate available for '185.157.221.99'
2020/04/24 01:24:45 http: TLS handshake error from 158.101.196.14:61188: no certificate available for '185.157.221.99'
2020/04/24 01:43:20 http: TLS handshake error from 165.227.220.53:52760: no certificate available for 'booking.tullingelabs.se'
2020/04/24 02:00:28 http: TLS handshake error from 84.216.183.212:43720: no certificate available for 'tullingelabs.se'
2020/04/24 02:20:07 http: TLS handshake error from 185.254.70.34:32271: no certificate available for 'www.tullingelabs.se'
2020/04/24 02:23:39 http: TLS handshake error from 208.93.152.17:21030: EOF
2020/04/24 02:23:40 http: TLS handshake error from 208.93.152.17:44801: tls: unsupported SSLv2 handshake received
2020/04/24 02:23:43 http: TLS handshake error from 208.93.152.17:60397: tls: client offered only unsupported versions: []
2020/04/24 02:23:44 http: TLS handshake error from 208.93.152.17:32642: tls: client offered only unsupported versions: [301]
2020/04/24 02:23:45 http: TLS handshake error from 208.93.152.17:33316: tls: client offered only unsupported versions: [302 301]
2020/04/24 02:23:54 http: TLS handshake error from 208.93.152.17:25964: no certificate available for '185.157.221.99'
2020/04/24 02:23:55 http: TLS handshake error from 208.93.152.17:23998: no certificate available for '185.157.221.99'
2020/04/24 02:40:40 http: TLS handshake error from 103.208.72.22:10093: no certificate available for '185.157.221.99'
2020/04/24 02:40:41 http: TLS handshake error from 103.208.72.22:31767: no certificate available for '185.157.221.99'
2020/04/24 02:40:41 http: TLS handshake error from 103.208.72.22:47329: no certificate available for '185.157.221.99'
2020/04/24 02:40:42 http: TLS handshake error from 103.208.72.22:29811: no certificate available for '185.157.221.99'
2020/04/24 02:40:42 http: TLS handshake error from 103.208.72.22:2188: no certificate available for '185.157.221.99'
2020/04/24 02:40:43 http: TLS handshake error from 103.208.72.22:17323: no certificate available for '185.157.221.99'
2020/04/24 02:49:41 http: TLS handshake error from 193.118.53.194:43284: no certificate available for '185.157.221.99'
2020/04/24 02:59:48 http: TLS handshake error from 196.52.43.86:62066: tls: client offered only unsupported versions: [302 301]
2020/04/24 03:30:35 http: TLS handshake error from 45.13.93.82:54406: tls: first record does not look like a TLS handshake
2020/04/24 03:51:46 http: TLS handshake error from 192.81.128.37:44886: no certificate available for '185.157.221.99'
2020/04/24 03:57:37 http: TLS handshake error from 171.67.71.243:34730: no certificate available for '185.157.221.99'
2020/04/24 04:29:31 http: TLS handshake error from 216.218.206.67:12760: no certificate available for '185.157.221.99'
2020/04/24 05:37:56 http: TLS handshake error from 66.249.73.238:59197: no certificate available for 'booking.tullingelabs.se'
2020/04/24 05:38:52 http: TLS handshake error from 66.249.73.238:59544: no certificate available for 'booking.tullingelabs.se'
2020/04/24 05:52:29 http: TLS handshake error from 92.244.29.246:60916: no certificate available for 'attendance.tullingelabs.se'
2020/04/24 05:57:27 http: TLS handshake error from 92.244.29.246:60960: no certificate available for 'booking.tullingelabs.se'
2020/04/24 05:57:28 http: TLS handshake error from 92.244.29.246:60964: no certificate available for 'booking.tullingelabs.se'
2020/04/24 06:19:23 http: TLS handshake error from 185.173.35.41:57721: no certificate available for '185.157.221.99:443'
{"level":"info","ts":1587709963.1203327,"logger":"tls","msg":"cleaned up storage units"}
2020/04/24 08:08:31 http: TLS handshake error from 192.99.167.179:50166: no certificate available for '185.157.221.99'
2020/04/24 08:36:41 http: TLS handshake error from 134.122.55.9:48402: no certificate available for '185.157.221.99'
2020/04/24 08:38:07 http: TLS handshake error from 162.243.128.180:42196: no certificate available for '185.157.221.99'
2020/04/24 09:29:32 http: TLS handshake error from 66.249.83.75:39042: no certificate available for 'weather.tullingelabs.se'
2020/04/24 09:40:15 http: TLS handshake error from 84.216.157.99:14340: no certificate available for 'weather.tullingelabs.se'

[matt (Matt Holt)]

@einarpersson

The reason there is a :8081 in the logs is that somewhere along the way I temporarly fired up a docker container with mongo-express (a admin UI for mongodb) bypassing caddy.

This alone makes me suspect that there’s more going on than we know; apparently multiple logs are getting dumped into this one place, and there’s more to this setup than is being revealed here. As of now, my current suspicion is that the Caddy instance that is managing certs for your subdomains (which it is certainly doing, as evidenced by your logs – it even shows OCSP staples being refreshed) is not the instance that is receiving requests.

Try to reproduce the problem without Docker, without MongoDB, without anything else that complicates the setup, that will be helpful!

[einarpersson]

What does advancing OCSP staple mean?

I guess I’ll just restart fresh, let it tick for a while without me doing anything and get back to you if something happens… Thanks for your time, even though we didn’t get to the bottom of it.

PS. I thought the Docker-image was officially supported now? Sure, it gives another layer of abstraction but… that is how I run things all the time

PS2. Should I add something like fail2ban to get my logs more readable in the future?

[matt (Matt Holt)]

@einarpersson

What does advancing OCSP staple mean?

It means that Caddy is managing and serving that certificate, and its OCSP staple is somewhat through its validity period (about halfway, if I recall correctly), so it is requesting a newer response from the OCSP responder.

@einarpersson

I guess I’ll just restart fresh, let it tick for a while without me doing anything and get back to you if something happens… Thanks for your time, even though we didn’t get to the bottom of it.

And thanks for the logs, they’re helpful – there’s just not enough info and the setup is too complicated to debug with the given information. And I don’t know anything about Docker. Maybe somebody else will know why Mongo is affecting Caddy, and where all the extra logs are coming from – maybe you have more than 1 Caddy instance or container running?

@einarpersson

PS. I thought the Docker-image was officially supported now? Sure, it gives another layer of abstraction but… that is how I run things all the time

It is, but Docker is also a footgun. Just like DNS…

@einarpersson

PS2. Should I add something like fail2ban to get my logs more readable in the future?

I don’t understand what fail2ban has to do with log output…

[Whitestrake (Matthew Fay)]

@matt

I don’t understand what fail2ban has to do with log output…

It can be configured to ban repeat offenders, potentially reducing the amount of log lines related to clients connecting for sites with no certificate.

Don’t think that’d be too helpful in this case, though, @einarpersson - that output is fine.

If you sudo pgrep caddy, how many instances do you get?

Can we get a docker ps -a too?

Do you make a habit of running your Docker containers on the host’s network stack?

[einarpersson]

Output from docker ps -a:

CONTAINER ID        IMAGE                                         COMMAND                  CREATED             STATUS              PORTS                    NAMES
18ec894203d9        tullingegymnasium/student_attendance:latest   "docker-entrypoint.s…"   5 days ago          Up 5 days           0.0.0.0:5000->5000/tcp   student_attendance_webapp_1
e07b7ebfa52f        mongo                                         "docker-entrypoint.s…"   5 days ago          Up 5 days           27017/tcp                student_attendance_mongo_1
87de632a3ff5        caddy                                         "caddy run --watch -…"   10 days ago         Up 10 days                                   stoic_cannon

Output from sudo pgrep caddy:

19598

@Whitestrake: No, I usually do not run containers on the host. This was an exception, as to make it easy to reverse-proxy by pointing to localhost:5000 or localhost:xyzw etc. for future containers. But I’ve learned that I can instead attach the caddy container to networks afterwards when necessary, will try that next time.