Cannot get/renew SSL certificates; using Cloudflare/DNS-01

Help
1

1. The problem I’m having:

Today all of my reverse-proxied services started throwing HTTPS certificate errors; sure enough the cert expired yesterday, May 20. Attempting to restart Caddy to refresh certificates did not work (much detail below).

2. Error messages and/or full log output:

Full log output is too long to post; will add it in a comment below.

{"level":"info","ts":1747855629.6430657,"logger":"tls.renew","msg":"renewing certificate","identifier":"*.joshuaochs.com","remaining":-51690.643053343}
{"level":"debug","ts":1747855629.6431751,"logger":"events","msg":"event","name":"cert_obtaining","id":"84b81ab1-88e1-45f0-8923-3e5fbb2d0e2e","origin":"tls","data":{"forced":false,"identifier":"*.joshuaochs.com","issuer":"acme-v02.api.letsencrypt.org-directory","remaining":-51690643053343,"renewal":true}}
{"level":"debug","ts":1747855629.6435802,"logger":"tls","msg":"created CSR","identifiers":["*.joshuaochs.com"],"san_dns_names":["*.joshuaochs.com"],"san_emails":[],"common_name":"","extra_extensions":0}
{"level":"debug","ts":1747855629.645532,"logger":"tls.issuance.acme","msg":"using existing ACME account because key found in storage associated with email","email":"caddy@zerossl.com","ca":"https://acme-staging-v02.api.letsencrypt.org/directory"}
{"level":"info","ts":1747855629.6456578,"logger":"tls.issuance.acme","msg":"using ACME account","account_id":"https://acme-staging-v02.api.letsencrypt.org/acme/acct/196097374","account_contact":["mailto:caddy@zerossl.com"]}
{"level":"debug","ts":1747855629.645709,"msg":"creating order","account":"https://acme-staging-v02.api.letsencrypt.org/acme/acct/196097374","identifiers":["*.joshuaochs.com"]}
{"level":"debug","ts":1747855629.7128122,"msg":"http request","method":"HEAD","url":"https://acme-staging-v02.api.letsencrypt.org/acme/new-nonce","headers":{"User-Agent":["Caddy/2.10.0 CertMagic acmez (linux; amd64)"]},"response_headers":{"Cache-Control":["public, max-age=0, no-cache"],"Date":["Wed, 21 May 2025 19:27:09 GMT"],"Link":["<https://acme-staging-v02.api.letsencrypt.org/directory>;rel=\"index\""],"Replay-Nonce":["1jOOXM0FYWL6457b7Lxpo7TXol8OhsHFHt9HbJmoJT0WHNE1yAs"],"Server":["nginx"],"Strict-Transport-Security":["max-age=604800"],"X-Frame-Options":["DENY"]},"status_code":200}
{"level":"debug","ts":1747855629.8170385,"msg":"http request","method":"POST","url":"https://acme-staging-v02.api.letsencrypt.org/acme/new-order","headers":{"Content-Type":["application/jose+json"],"User-Agent":["Caddy/2.10.0 CertMagic acmez (linux; amd64)"]},"response_headers":{"Boulder-Requester":["196097374"],"Cache-Control":["public, max-age=0, no-cache"],"Content-Length":["362"],"Content-Type":["application/json"],"Date":["Wed, 21 May 2025 19:27:09 GMT"],"Link":["<https://acme-staging-v02.api.letsencrypt.org/directory>;rel=\"index\""],"Location":["https://acme-staging-v02.api.letsencrypt.org/acme/order/196097374/24795367464"],"Replay-Nonce":["1jOOXM0FVVfkYDkAGU-tFyRLdCJHVwAy24Z1kAzUzEctL1UELFM"],"Server":["nginx"],"Strict-Transport-Security":["max-age=604800"],"X-Frame-Options":["DENY"]},"status_code":201}
{"level":"debug","ts":1747855629.8840506,"msg":"http request","method":"POST","url":"https://acme-staging-v02.api.letsencrypt.org/acme/authz/196097374/17506087954","headers":{"Content-Type":["application/jose+json"],"User-Agent":["Caddy/2.10.0 CertMagic acmez (linux; amd64)"]},"response_headers":{"Boulder-Requester":["196097374"],"Cache-Control":["public, max-age=0, no-cache"],"Content-Length":["402"],"Content-Type":["application/json"],"Date":["Wed, 21 May 2025 19:27:09 GMT"],"Link":["<https://acme-staging-v02.api.letsencrypt.org/directory>;rel=\"index\""],"Replay-Nonce":["Yo1slBcdMleIC3giDWWePJKUScNXBjavBX0FMRw-FmA3mBRIzUU"],"Server":["nginx"],"Strict-Transport-Security":["max-age=604800"],"X-Frame-Options":["DENY"]},"status_code":200}
{"level":"info","ts":1747855629.8843112,"msg":"trying to solve challenge","identifier":"*.joshuaochs.com","challenge_type":"dns-01","ca":"https://acme-staging-v02.api.letsencrypt.org/directory"}
{"level":"debug","ts":1747855630.1192534,"msg":"waiting for solver before continuing","identifier":"*.joshuaochs.com","challenge_type":"dns-01"}
{"level":"debug","ts":1747855632.1341867,"msg":"done waiting for solver","identifier":"*.joshuaochs.com","challenge_type":"dns-01"}
{"level":"debug","ts":1747855632.5886219,"msg":"http request","method":"POST","url":"https://acme-staging-v02.api.letsencrypt.org/acme/authz/196097374/17506087954","headers":{"Content-Type":["application/jose+json"],"User-Agent":["Caddy/2.10.0 CertMagic acmez (linux; amd64)"]},"response_headers":{"Boulder-Requester":["196097374"],"Cache-Control":["public, max-age=0, no-cache"],"Content-Length":["406"],"Content-Type":["application/json"],"Date":["Wed, 21 May 2025 19:27:12 GMT"],"Link":["<https://acme-staging-v02.api.letsencrypt.org/directory>;rel=\"index\""],"Replay-Nonce":["Yo1slBcdYzeiBikkLfZWvGsM3XDR6YXOMiSdIQ6oD2cXcLXesSE"],"Server":["nginx"],"Strict-Transport-Security":["max-age=604800"],"X-Frame-Options":["DENY"]},"status_code":200}
{"level":"error","ts":1747855632.5890744,"logger":"tls.renew","msg":"could not get certificate from issuer","identifier":"*.joshuaochs.com","issuer":"acme-v02.api.letsencrypt.org-directory","error":"[*.joshuaochs.com] solving challenges: waiting for solver certmagic.solverWrapper to be ready: checking DNS propagation of \"_acme-challenge.joshuaochs.com.\" (relative=_acme-challenge zone=joshuaochs.com. resolvers=[127.0.0.11:53]): querying authoritative nameservers: read udp 172.30.29.2:40612->172.64.33.194:53: read: connection refused (order=https://acme-staging-v02.api.letsencrypt.org/acme/order/196097374/24795367464) (ca=https://acme-staging-v02.api.letsencrypt.org/directory)"}
{"level":"debug","ts":1747855632.5891614,"logger":"events","msg":"event","name":"cert_failed","id":"2d4bc3a4-0331-44fb-bd32-187ab25f346b","origin":"tls","data":{"error":{},"identifier":"*.joshuaochs.com","issuers":["acme-v02.api.letsencrypt.org-directory"],"remaining":-51690643053343,"renewal":true}}
{"level":"error","ts":1747855632.5892582,"logger":"tls.renew","msg":"will retry","error":"[*.joshuaochs.com] Renew: [*.joshuaochs.com] solving challenges: waiting for solver certmagic.solverWrapper to be ready: checking DNS propagation of \"_acme-challenge.joshuaochs.com.\" (relative=_acme-challenge zone=joshuaochs.com. resolvers=[127.0.0.11:53]): querying authoritative nameservers: read udp 172.30.29.2:40612->172.64.33.194:53: read: connection refused (order=https://acme-staging-v02.api.letsencrypt.org/acme/order/196097374/24795367464) (ca=https://acme-staging-v02.api.letsencrypt.org/directory)","attempt":2,"retrying_in":120,"elapsed":77.539273015,"max_duration":2592000}

3. Caddy version:

v2.10.0 h1:fonubSaQKF1YANl8TXqGcn4IbIRUDdfAkpcsfI/vX5U=

4. How I installed and ran Caddy:

Docker Compose; custom image to include DNS-01 plugin

a. System environment:

Linux docker 6.1.0-33-amd64 #1 SMP PREEMPT_DYNAMIC Debian 6.1.133-1 (2025-04-10) x86_64 GNU/Linux
Docker version 25.0.4, build 1a576c5

b. Command:

docker compose up -d && docker logs -f caddy

c. Service/unit/compose file:

Dockerfile

FROM caddy:builder AS builder
RUN caddy-builder \
    github.com/caddy-dns/cloudflare
FROM caddy:latest
COPY --from=builder /usr/bin/caddy /usr/bin/caddy

docker-compose.yml

services:
  caddy:
    build: .
    image: diamondsw/caddy
    container_name: caddy
    restart: unless-stopped
    ports:
      - "80:80"
      - "443:443"
      - "443:443/udp"
    environment:
      CLOUDFLARE_API_TOKEN: [REDACTED]      
    volumes:
      - ./caddyfile:/etc/caddy/Caddyfile
      - ./data:/data
      - ./config:/config

d. My complete Caddy config:

(basic-auth) {
	basicauth {
		[REDACTED]
	}
}

*.joshuaochs.com, joshuaochs.com {
	tls {
		dns cloudflare {env.CLOUDFLARE_API_TOKEN}
	}

	@microbin host microbin.joshuaochs.com
	reverse_proxy @microbin http://docker.homelab.joshuaochs.com:8082

	@tools host tools.joshuaochs.com
	reverse_proxy @tools http://docker.homelab.joshuaochs.com:8009

	@translate host translate.joshuaochs.com
	reverse_proxy @translate http://docker.homelab.joshuaochs.com:4445

	@vault host vault.joshuaochs.com
	reverse_proxy @vault http://docker.homelab.joshuaochs.com:8001

	@whoogle host whoogle.joshuaochs.com
	reverse_proxy @whoogle http://docker.homelab.joshuaochs.com:5000

	@wikipedia host wikipedia.joshuaochs.com
	reverse_proxy @wikipedia http://docker.homelab.joshuaochs.com:8081

	@chatgpt host chatgpt.joshuaochs.com
	reverse_proxy @chatgpt http://gpu.homelab.joshuaochs.com:3000
	@chatapi host chatapi.joshuaochs.com
	reverse_proxy @chatapi http://gpu.homelab.joshuaochs.com:4000

	@rss host rss.joshuaochs.com
	reverse_proxy @rss http://docker.homelab.joshuaochs.com:88

	@cal host cal.joshuaochs.com
	handle @cal {
		#import basic-auth
		reverse_proxy http://docker.homelab.joshuaochs.com:81
	}

	@snapdrop host snapdrop.joshuaochs.com
	handle @snapdrop {
		request_body {
			max_size 1GB
		}
		reverse_proxy https://docker.homelab.joshuaochs.com:3443 {
			transport http {
				tls_insecure_skip_verify
			}
		}
	}

	@unifi host unifi.joshuaochs.com
	handle @unifi {
		reverse_proxy /inform docker.homelab.joshuaochs.com:8080
		reverse_proxy https://docker.homelab.joshuaochs.com:8443 {
			header_up -Authorization
			transport http {
				tls
				tls_insecure_skip_verify
			}
		}
	}
}

5. Links to relevant resources:

Before posting, I checked the following common issues:

  1. Port forwarding in my router - verified port 80 and 443 are properly forwarded to my Docker host, 10.0.1.150, and confirmed in compose file that these are the ports being used.

  2. Verified Cloudflare API token is valid:

jochs@docker caddy-dns $ curl -s "https://api.cloudflare.com/client/v4/user/tokens/verify" --header "Authorization: Bearer [REDACTED]" | jq .

{
  "result": {
    "id": "fe1cbbd6e2ff06e4841a8d1e21d43e5c",
    "status": "active"
  },
  "success": true,
  "errors": [],
  "messages": [
    {
      "code": 10000,
      "message": "This API Token is valid and active",
      "type": null
    }
  ]
}
  1. Verified DNS resolution is working inside the container:
jochs@docker ~ $ docker exec -it caddy sh
/srv # ping acme-v02.api.letsencrypt.org
PING acme-v02.api.letsencrypt.org (172.65.32.248): 56 data bytes
64 bytes from 172.65.32.248: seq=0 ttl=56 time=13.911 ms
64 bytes from 172.65.32.248: seq=1 ttl=56 time=12.400 ms
^C
--- acme-v02.api.letsencrypt.org ping statistics ---
2 packets transmitted, 2 packets received, 0% packet loss
round-trip min/avg/max = 12.400/13.155/13.911 ms
/srv #

At this point everything on my end looks correct - and other than rebuilding the image after problems started, I have not made any configuration changes in some time.

Happy to check anything else that may be relevant.

2

Preceding log output:

{"level":"info","ts":1747855510.5363097,"msg":"maxprocs: Leaving GOMAXPROCS=8: CPU quota undefined"}
{"level":"info","ts":1747855510.5374823,"msg":"GOMEMLIMIT is updated","package":"github.com/KimMachineGun/automemlimit/memlimit","GOMEMLIMIT":30289261363,"previous":9223372036854775807}
{"level":"info","ts":1747855510.5380926,"msg":"using config from file","file":"/etc/caddy/Caddyfile"}
{"level":"warn","ts":1747855510.5437,"logger":"config.adapter.caddyfile","msg":"the 'basicauth' directive is deprecated, please use 'basic_auth' instead!"}
{"level":"warn","ts":1747855510.54431,"logger":"config.adapter.caddyfile","msg":"the 'basicauth' directive is deprecated, please use 'basic_auth' instead!"}
{"level":"info","ts":1747855510.5475507,"msg":"adapted config to JSON","adapter":"caddyfile"}
{"level":"info","ts":1747855510.5521019,"logger":"admin","msg":"admin endpoint started","address":"localhost:2019","enforce_origin":false,"origins":["//localhost:2019","//[::1]:2019","//127.0.0.1:2019"]}
{"level":"info","ts":1747855510.5535975,"logger":"http.auto_https","msg":"server is listening only on the HTTPS port but has no TLS connection policies; adding one to enable TLS","server_name":"srv0","https_port":443}
{"level":"info","ts":1747855510.5536778,"logger":"http.auto_https","msg":"enabling automatic HTTP->HTTPS redirects","server_name":"srv0"}
{"level":"debug","ts":1747855510.5537484,"logger":"http.auto_https","msg":"adjusted config","tls":{"automation":{"policies":[{"subjects":["*.joshuaochs.com","joshuaochs.com"]},{}]}},"http":{"servers":{"remaining_auto_https_redirects":{"listen":[":80"],"routes":[{},{}]},"srv0":{"listen":[":443"],"routes":[{"handle":[{"handler":"subroute","routes":[{"group":"group7","handle":[{"handler":"subroute","routes":[{"handle":[{"handler":"reverse_proxy","transport":{"protocol":"http","tls":{}},"upstreams":[{"dial":"docker.homelab.joshuaochs.com:9091"}]}]}]}],"match":[{"host":["transmission.joshuaochs.com"]}]},{"group":"group7","handle":[{"handler":"subroute","routes":[{"handle":[{"handler":"reverse_proxy","upstreams":[{"dial":"docker.homelab.joshuaochs.com:81"}]}]}]}],"match":[{"host":["cal.joshuaochs.com"]}]},{"group":"group7","handle":[{"handler":"subroute","routes":[{"handle":[{"handler":"request_body","max_size":1000000000},{"handler":"reverse_proxy","transport":{"protocol":"http","tls":{"insecure_skip_verify":true}},"upstreams":[{"dial":"docker.homelab.joshuaochs.com:3443"}]}]}]}],"match":[{"host":["snapdrop.joshuaochs.com"]}]},{"group":"group7","handle":[{"handler":"subroute","routes":[{"handle":[{"handler":"static_response","headers":{"Location":["http://booru.joshuaochs.com{http.request.uri}"]},"status_code":301}]}]}],"match":[{"host":["homebooru.joshuaochs.com"]}]},{"group":"group7","handle":[{"handler":"subroute","routes":[{"handle":[{"handler":"request_body","max_size":1000000000},{"handler":"authentication","providers":{"http_basic":{"accounts":[{"password":"$2a$14$iu/RQDKMQSzmaz.Lh2PRHO2w4qiLz3kloPA7ewUtJEdA5nPWZLKxe","username":"jochs"}],"hash":{"algorithm":"bcrypt"},"hash_cache":{}}}},{"handler":"reverse_proxy","upstreams":[{"dial":"docker.homelab.joshuaochs.com:8008"}]}]}]}],"match":[{"host":["booru.joshuaochs.com"]}]},{"group":"group7","handle":[{"handler":"subroute","routes":[{"handle":[{"handler":"request_body","max_size":1000000000},{"handler":"authentication","providers":{"http_basic":{"accounts":[{"password":"$2a$14$iu/RQDKMQSzmaz.Lh2PRHO2w4qiLz3kloPA7ewUtJEdA5nPWZLKxe","username":"jochs"}],"hash":{"algorithm":"bcrypt"},"hash_cache":{}}}},{"handler":"reverse_proxy","upstreams":[{"dial":"docker.homelab.joshuaochs.com:8018"}]}]}]}],"match":[{"host":["manga.joshuaochs.com"]}]},{"group":"group7","handle":[{"handler":"subroute","routes":[{"handle":[{"handler":"reverse_proxy","upstreams":[{"dial":"docker.homelab.joshuaochs.com:8080"}]}],"match":[{"path":["/inform"]}]},{"handle":[{"handler":"reverse_proxy","headers":{"request":{"delete":["Authorization"]}},"transport":{"protocol":"http","tls":{"insecure_skip_verify":true}},"upstreams":[{"dial":"docker.homelab.joshuaochs.com:8443"}]}]}]}],"match":[{"host":["unifi.joshuaochs.com"]}]},{"handle":[{"handler":"reverse_proxy","upstreams":[{"dial":"docker.homelab.joshuaochs.com:8082"}]}],"match":[{"host":["microbin.joshuaochs.com"]}]},{"handle":[{"handler":"reverse_proxy","upstreams":[{"dial":"docker.homelab.joshuaochs.com:8009"}]}],"match":[{"host":["tools.joshuaochs.com"]}]},{"handle":[{"handler":"reverse_proxy","upstreams":[{"dial":"docker.homelab.joshuaochs.com:4445"}]}],"match":[{"host":["translate.joshuaochs.com"]}]},{"handle":[{"handler":"reverse_proxy","upstreams":[{"dial":"docker.homelab.joshuaochs.com:8001"}]}],"match":[{"host":["vault.joshuaochs.com"]}]},{"handle":[{"handler":"reverse_proxy","upstreams":[{"dial":"docker.homelab.joshuaochs.com:5000"}]}],"match":[{"host":["whoogle.joshuaochs.com"]}]},{"handle":[{"handler":"reverse_proxy","upstreams":[{"dial":"docker.homelab.joshuaochs.com:8081"}]}],"match":[{"host":["wikipedia.joshuaochs.com"]}]},{"handle":[{"handler":"reverse_proxy","upstreams":[{"dial":"gpu.homelab.joshuaochs.com:3000"}]}],"match":[{"host":["chatgpt.joshuaochs.com"]}]},{"handle":[{"handler":"reverse_proxy","upstreams":[{"dial":"gpu.homelab.joshuaochs.com:4000"}]}],"match":[{"host":["chatapi.joshuaochs.com"]}]},{"handle":[{"handler":"reverse_proxy","upstreams":[{"dial":"docker.homelab.joshuaochs.com:88"}]}],"match":[{"host":["rss.joshuaochs.com"]}]}]}],"terminal":true}],"tls_connection_policies":[{}],"automatic_https":{}}}}}
{"level":"info","ts":1747855510.5591116,"logger":"tls.cache.maintenance","msg":"started background certificate maintenance","cache":"0xc00061ab80"}
{"level":"debug","ts":1747855510.563585,"logger":"http","msg":"starting server loop","address":"[::]:443","tls":true,"http3":false}
{"level":"info","ts":1747855510.563663,"logger":"http","msg":"enabling HTTP/3 listener","addr":":443"}
{"level":"info","ts":1747855510.5639627,"msg":"failed to sufficiently increase receive buffer size (was: 208 kiB, wanted: 7168 kiB, got: 416 kiB). See https://github.com/quic-go/quic-go/wiki/UDP-Buffer-Sizes for details."}
{"level":"info","ts":1747855510.5649269,"logger":"http.log","msg":"server running","name":"srv0","protocols":["h1","h2","h3"]}
{"level":"debug","ts":1747855510.5651305,"logger":"http","msg":"starting server loop","address":"[::]:80","tls":false,"http3":false}
{"level":"warn","ts":1747855510.565174,"logger":"http","msg":"HTTP/2 skipped because it requires TLS","network":"tcp","addr":":80"}
{"level":"warn","ts":1747855510.5651867,"logger":"http","msg":"HTTP/3 skipped because it requires TLS","network":"tcp","addr":":80"}
{"level":"info","ts":1747855510.5651987,"logger":"http.log","msg":"server running","name":"remaining_auto_https_redirects","protocols":["h1","h2","h3"]}
{"level":"info","ts":1747855510.5654805,"logger":"http","msg":"enabling automatic TLS certificate management","domains":["*.joshuaochs.com","joshuaochs.com"]}
{"level":"info","ts":1747855510.567348,"logger":"tls","msg":"storage cleaning happened too recently; skipping for now","storage":"FileStorage:/data/caddy","instance":"a1627015-e86e-4482-a840-56b2d42e5bca","try_again":1747941910.5673418,"try_again_in":86399.99999752}
{"level":"info","ts":1747855510.5675952,"logger":"tls","msg":"finished cleaning storage units"}
{"level":"warn","ts":1747855510.6648984,"logger":"tls","msg":"stapling OCSP","error":"no OCSP stapling for [joshuaochs.com]: parsing OCSP response: ocsp: error from server: unauthorized","identifiers":["joshuaochs.com"]}
{"level":"debug","ts":1747855510.6655247,"logger":"tls.cache","msg":"added certificate to cache","subjects":["joshuaochs.com"],"expiration":1747803939,"managed":true,"issuer_key":"acme-v02.api.letsencrypt.org-directory","hash":"838717196645677770c5754c979f16e1fd4716ecb105b6bc278f8b8bff2aadc3","cache_size":1,"cache_capacity":10000}
{"level":"debug","ts":1747855510.6657088,"logger":"events","msg":"event","name":"cached_managed_cert","id":"8edf957c-1fdf-443e-bc05-21ffe0f1a9fa","origin":"tls","data":{"sans":["joshuaochs.com"]}}
{"level":"info","ts":1747855510.6662314,"logger":"tls","msg":"certificate is in configured renewal window based on expiration date","subjects":["joshuaochs.com"],"expiration":1747803939,"ari_cert_id":"","next_ari_update":null,"renew_check_interval":600,"window_start":-6795364578.8713455,"window_end":-6795364578.8713455,"remaining":-51571.66622907}
{"level":"info","ts":1747855510.6695795,"logger":"tls.renew","msg":"acquiring lock","identifier":"joshuaochs.com"}
{"level":"info","ts":1747855510.6711946,"logger":"tls.renew","msg":"lock acquired","identifier":"joshuaochs.com"}
{"level":"info","ts":1747855510.6721807,"logger":"tls.renew","msg":"renewing certificate","identifier":"joshuaochs.com","remaining":-51571.672172525}
{"level":"debug","ts":1747855510.6723185,"logger":"events","msg":"event","name":"cert_obtaining","id":"b456c54a-37d0-406f-ba98-0e2166cdb5b9","origin":"tls","data":{"forced":false,"identifier":"joshuaochs.com","issuer":"acme-v02.api.letsencrypt.org-directory","remaining":-51571672172525,"renewal":true}}
{"level":"debug","ts":1747855510.6726995,"logger":"tls","msg":"created CSR","identifiers":["joshuaochs.com"],"san_dns_names":["joshuaochs.com"],"san_emails":[],"common_name":"","extra_extensions":0}
{"level":"debug","ts":1747855510.6749809,"logger":"tls.issuance.acme","msg":"using existing ACME account because key found in storage associated with email","email":"caddy@zerossl.com","ca":"https://acme-v02.api.letsencrypt.org/directory"}
{"level":"debug","ts":1747855510.6757894,"logger":"tls.issuance.acme","msg":"using existing ACME account because key found in storage associated with email","email":"caddy@zerossl.com","ca":"https://acme-v02.api.letsencrypt.org/directory"}
{"level":"info","ts":1747855510.6759117,"logger":"tls.issuance.acme","msg":"waiting on internal rate limiter","identifiers":["joshuaochs.com"],"ca":"https://acme-v02.api.letsencrypt.org/directory","account":"caddy@zerossl.com"}
{"level":"info","ts":1747855510.6759386,"logger":"tls.issuance.acme","msg":"done waiting on internal rate limiter","identifiers":["joshuaochs.com"],"ca":"https://acme-v02.api.letsencrypt.org/directory","account":"caddy@zerossl.com"}
{"level":"info","ts":1747855510.6761272,"logger":"tls.issuance.acme","msg":"using ACME account","account_id":"https://acme-v02.api.letsencrypt.org/acme/acct/1090263897","account_contact":["mailto:caddy@zerossl.com"]}
{"level":"warn","ts":1747855510.7477715,"logger":"tls","msg":"stapling OCSP","error":"no OCSP stapling for [*.joshuaochs.com]: parsing OCSP response: ocsp: error from server: unauthorized","identifiers":["*.joshuaochs.com"]}
{"level":"debug","ts":1747855510.7478569,"logger":"tls.cache","msg":"added certificate to cache","subjects":["*.joshuaochs.com"],"expiration":1747803939,"managed":true,"issuer_key":"acme-v02.api.letsencrypt.org-directory","hash":"53f91cc34dda7562b93a1ba32dac21c0f30e13a3479679fdef52561ae0920ef8","cache_size":2,"cache_capacity":10000}
{"level":"debug","ts":1747855510.747917,"logger":"events","msg":"event","name":"cached_managed_cert","id":"28e52e43-f433-4b47-90ae-6703d4c060d3","origin":"tls","data":{"sans":["*.joshuaochs.com"]}}
{"level":"debug","ts":1747855510.7480483,"logger":"events","msg":"event","name":"started","id":"2d3bb29a-d24e-4823-8ad7-84feab4caf25","origin":"","data":null}
{"level":"info","ts":1747855510.7482073,"logger":"tls","msg":"certificate is in configured renewal window based on expiration date","subjects":["*.joshuaochs.com"],"expiration":1747803939,"ari_cert_id":"","next_ari_update":null,"renew_check_interval":600,"window_start":-6795364578.8713455,"window_end":-6795364578.8713455,"remaining":-51571.748205341}
{"level":"info","ts":1747855510.7488534,"msg":"autosaved config (load with --resume flag)","file":"/config/caddy/autosave.json"}
{"level":"info","ts":1747855510.7488914,"msg":"serving initial configuration"}
{"level":"info","ts":1747855510.7507086,"logger":"tls.renew","msg":"acquiring lock","identifier":"*.joshuaochs.com"}
{"level":"info","ts":1747855510.7525642,"logger":"tls.renew","msg":"lock acquired","identifier":"*.joshuaochs.com"}
{"level":"info","ts":1747855510.7535658,"logger":"tls.renew","msg":"renewing certificate","identifier":"*.joshuaochs.com","remaining":-51571.753557501}
{"level":"debug","ts":1747855510.7536561,"logger":"events","msg":"event","name":"cert_obtaining","id":"8a6b60e8-6d8f-493c-a375-6fe106bec078","origin":"tls","data":{"forced":false,"identifier":"*.joshuaochs.com","issuer":"acme-v02.api.letsencrypt.org-directory","remaining":-51571753557501,"renewal":true}}
{"level":"debug","ts":1747855510.7539728,"logger":"tls","msg":"created CSR","identifiers":["*.joshuaochs.com"],"san_dns_names":["*.joshuaochs.com"],"san_emails":[],"common_name":"","extra_extensions":0}
{"level":"debug","ts":1747855510.7557263,"logger":"tls.issuance.acme","msg":"using existing ACME account because key found in storage associated with email","email":"caddy@zerossl.com","ca":"https://acme-v02.api.letsencrypt.org/directory"}
{"level":"info","ts":1747855510.755788,"logger":"tls.issuance.acme","msg":"waiting on internal rate limiter","identifiers":["*.joshuaochs.com"],"ca":"https://acme-v02.api.letsencrypt.org/directory","account":"caddy@zerossl.com"}
{"level":"info","ts":1747855510.7558208,"logger":"tls.issuance.acme","msg":"done waiting on internal rate limiter","identifiers":["*.joshuaochs.com"],"ca":"https://acme-v02.api.letsencrypt.org/directory","account":"caddy@zerossl.com"}
{"level":"info","ts":1747855510.7561076,"logger":"tls.issuance.acme","msg":"using ACME account","account_id":"https://acme-v02.api.letsencrypt.org/acme/acct/1090263897","account_contact":["mailto:caddy@zerossl.com"]}
{"level":"debug","ts":1747855510.798726,"msg":"http request","method":"GET","url":"https://acme-v02.api.letsencrypt.org/directory","headers":{"User-Agent":["Caddy/2.10.0 CertMagic acmez (linux; amd64)"]},"response_headers":{"Cache-Control":["public, max-age=0, no-cache"],"Content-Length":["1012"],"Content-Type":["application/json"],"Date":["Wed, 21 May 2025 19:25:10 GMT"],"Server":["nginx"],"Strict-Transport-Security":["max-age=604800"],"X-Frame-Options":["DENY"]},"status_code":200}
{"level":"debug","ts":1747855510.800328,"msg":"creating order","account":"https://acme-v02.api.letsencrypt.org/acme/acct/1090263897","identifiers":["*.joshuaochs.com"]}
{"level":"debug","ts":1747855510.8008173,"msg":"creating order","account":"https://acme-v02.api.letsencrypt.org/acme/acct/1090263897","identifiers":["joshuaochs.com"]}
{"level":"debug","ts":1747855510.8428214,"msg":"http request","method":"HEAD","url":"https://acme-v02.api.letsencrypt.org/acme/new-nonce","headers":{"User-Agent":["Caddy/2.10.0 CertMagic acmez (linux; amd64)"]},"response_headers":{"Cache-Control":["public, max-age=0, no-cache"],"Date":["Wed, 21 May 2025 19:25:10 GMT"],"Link":["<https://acme-v02.api.letsencrypt.org/directory>;rel=\"index\""],"Replay-Nonce":["NySGY0K7AemVSnQ3mVw9teQ3Ot3UAwbqkMiZoneNS1pZlO0gLGI"],"Server":["nginx"],"Strict-Transport-Security":["max-age=604800"],"X-Frame-Options":["DENY"]},"status_code":200}
{"level":"debug","ts":1747855510.8929267,"msg":"http request","method":"HEAD","url":"https://acme-v02.api.letsencrypt.org/acme/new-nonce","headers":{"User-Agent":["Caddy/2.10.0 CertMagic acmez (linux; amd64)"]},"response_headers":{"Cache-Control":["public, max-age=0, no-cache"],"Date":["Wed, 21 May 2025 19:25:10 GMT"],"Link":["<https://acme-v02.api.letsencrypt.org/directory>;rel=\"index\""],"Replay-Nonce":["NySGY0K7t2NhMmCi9HWos5bBlTs7B4S3ydJoIo_iy08gCI43Eug"],"Server":["nginx"],"Strict-Transport-Security":["max-age=604800"],"X-Frame-Options":["DENY"]},"status_code":200}
{"level":"debug","ts":1747855511.128187,"msg":"http request","method":"POST","url":"https://acme-v02.api.letsencrypt.org/acme/new-order","headers":{"Content-Type":["application/jose+json"],"User-Agent":["Caddy/2.10.0 CertMagic acmez (linux; amd64)"]},"response_headers":{"Boulder-Requester":["1090263897"],"Cache-Control":["public, max-age=0, no-cache"],"Content-Length":["350"],"Content-Type":["application/json"],"Date":["Wed, 21 May 2025 19:25:11 GMT"],"Link":["<https://acme-v02.api.letsencrypt.org/directory>;rel=\"index\""],"Location":["https://acme-v02.api.letsencrypt.org/acme/order/1090263897/386379739917"],"Replay-Nonce":["NySGY0K7NR6jJ6JV-APEqV8WLfhBSG0Gr6IO2EUmD8f3_agZd5g"],"Server":["nginx"],"Strict-Transport-Security":["max-age=604800"],"X-Frame-Options":["DENY"]},"status_code":201}
{"level":"debug","ts":1747855511.1309538,"msg":"http request","method":"POST","url":"https://acme-v02.api.letsencrypt.org/acme/new-order","headers":{"Content-Type":["application/jose+json"],"User-Agent":["Caddy/2.10.0 CertMagic acmez (linux; amd64)"]},"response_headers":{"Boulder-Requester":["1090263897"],"Cache-Control":["public, max-age=0, no-cache"],"Content-Length":["348"],"Content-Type":["application/json"],"Date":["Wed, 21 May 2025 19:25:11 GMT"],"Link":["<https://acme-v02.api.letsencrypt.org/directory>;rel=\"index\""],"Location":["https://acme-v02.api.letsencrypt.org/acme/order/1090263897/386379740257"],"Replay-Nonce":["NySGY0K7CoRnBUt8t8bycCRtRPjiKxwJAlGy0X_Z-ZHK1aVkMuw"],"Server":["nginx"],"Strict-Transport-Security":["max-age=604800"],"X-Frame-Options":["DENY"]},"status_code":201}
{"level":"debug","ts":1747855511.174806,"msg":"http request","method":"POST","url":"https://acme-v02.api.letsencrypt.org/acme/authz/1090263897/523783957757","headers":{"Content-Type":["application/jose+json"],"User-Agent":["Caddy/2.10.0 CertMagic acmez (linux; amd64)"]},"response_headers":{"Boulder-Requester":["1090263897"],"Cache-Control":["public, max-age=0, no-cache"],"Content-Length":["396"],"Content-Type":["application/json"],"Date":["Wed, 21 May 2025 19:25:11 GMT"],"Link":["<https://acme-v02.api.letsencrypt.org/directory>;rel=\"index\""],"Replay-Nonce":["NySGY0K7IqkUh3W4s1NGnZDQzJLB6SalY4tGFNM0V9X_RDvKr7I"],"Server":["nginx"],"Strict-Transport-Security":["max-age=604800"],"X-Frame-Options":["DENY"]},"status_code":200}
{"level":"info","ts":1747855511.1752958,"msg":"trying to solve challenge","identifier":"*.joshuaochs.com","challenge_type":"dns-01","ca":"https://acme-v02.api.letsencrypt.org/directory"}
{"level":"debug","ts":1747855511.1985388,"msg":"http request","method":"POST","url":"https://acme-v02.api.letsencrypt.org/acme/authz/1090263897/523783958157","headers":{"Content-Type":["application/jose+json"],"User-Agent":["Caddy/2.10.0 CertMagic acmez (linux; amd64)"]},"response_headers":{"Boulder-Requester":["1090263897"],"Cache-Control":["public, max-age=0, no-cache"],"Content-Length":["822"],"Content-Type":["application/json"],"Date":["Wed, 21 May 2025 19:25:11 GMT"],"Link":["<https://acme-v02.api.letsencrypt.org/directory>;rel=\"index\""],"Replay-Nonce":["4zqsHs_SiX2HWxMqVHtwFqrCm1pl0-oad-jLOaNmLHO1hCjVWMg"],"Server":["nginx"],"Strict-Transport-Security":["max-age=604800"],"X-Frame-Options":["DENY"]},"status_code":200}
{"level":"info","ts":1747855511.1989434,"msg":"trying to solve challenge","identifier":"joshuaochs.com","challenge_type":"dns-01","ca":"https://acme-v02.api.letsencrypt.org/directory"}
{"level":"debug","ts":1747855511.7439966,"msg":"waiting for solver before continuing","identifier":"joshuaochs.com","challenge_type":"dns-01"}
{"level":"debug","ts":1747855511.7779653,"msg":"waiting for solver before continuing","identifier":"*.joshuaochs.com","challenge_type":"dns-01"}
{"level":"debug","ts":1747855513.7537937,"msg":"done waiting for solver","identifier":"joshuaochs.com","challenge_type":"dns-01"}
{"level":"debug","ts":1747855514.2047212,"msg":"http request","method":"POST","url":"https://acme-v02.api.letsencrypt.org/acme/authz/1090263897/523783958157","headers":{"Content-Type":["application/jose+json"],"User-Agent":["Caddy/2.10.0 CertMagic acmez (linux; amd64)"]},"response_headers":{"Boulder-Requester":["1090263897"],"Cache-Control":["public, max-age=0, no-cache"],"Content-Length":["826"],"Content-Type":["application/json"],"Date":["Wed, 21 May 2025 19:25:14 GMT"],"Link":["<https://acme-v02.api.letsencrypt.org/directory>;rel=\"index\""],"Replay-Nonce":["4zqsHs_S_sLkIYJvtCw-d1FtrCDJFMJLcodaN74CCGvspo7ElAk"],"Server":["nginx"],"Strict-Transport-Security":["max-age=604800"],"X-Frame-Options":["DENY"]},"status_code":200}
{"level":"error","ts":1747855514.20507,"logger":"tls.renew","msg":"could not get certificate from issuer","identifier":"joshuaochs.com","issuer":"acme-v02.api.letsencrypt.org-directory","error":"[joshuaochs.com] solving challenges: waiting for solver certmagic.solverWrapper to be ready: checking DNS propagation of \"_acme-challenge.joshuaochs.com.\" (relative=_acme-challenge zone=joshuaochs.com. resolvers=[127.0.0.11:53]): querying authoritative nameservers: read udp 172.30.29.2:40957->172.64.33.194:53: read: connection refused (order=https://acme-v02.api.letsencrypt.org/acme/order/1090263897/386379740257) (ca=https://acme-v02.api.letsencrypt.org/directory)"}
{"level":"debug","ts":1747855514.2051535,"logger":"events","msg":"event","name":"cert_failed","id":"b200254c-daf8-4dae-8eb6-47797e7dff51","origin":"tls","data":{"error":{},"identifier":"joshuaochs.com","issuers":["acme-v02.api.letsencrypt.org-directory"],"remaining":-51571672172525,"renewal":true}}
{"level":"error","ts":1747855514.2052424,"logger":"tls.renew","msg":"will retry","error":"[joshuaochs.com] Renew: [joshuaochs.com] solving challenges: waiting for solver certmagic.solverWrapper to be ready: checking DNS propagation of \"_acme-challenge.joshuaochs.com.\" (relative=_acme-challenge zone=joshuaochs.com. resolvers=[127.0.0.11:53]): querying authoritative nameservers: read udp 172.30.29.2:40957->172.64.33.194:53: read: connection refused (order=https://acme-v02.api.letsencrypt.org/acme/order/1090263897/386379740257) (ca=https://acme-v02.api.letsencrypt.org/directory)","attempt":1,"retrying_in":60,"elapsed":3.53401141,"max_duration":2592000}
{"level":"debug","ts":1747855525.8454506,"msg":"done waiting for solver","identifier":"*.joshuaochs.com","challenge_type":"dns-01"}
{"level":"debug","ts":1747855526.3180687,"msg":"http request","method":"POST","url":"https://acme-v02.api.letsencrypt.org/acme/authz/1090263897/523783957757","headers":{"Content-Type":["application/jose+json"],"User-Agent":["Caddy/2.10.0 CertMagic acmez (linux; amd64)"]},"response_headers":{"Boulder-Requester":["1090263897"],"Cache-Control":["public, max-age=0, no-cache"],"Content-Length":["400"],"Content-Type":["application/json"],"Date":["Wed, 21 May 2025 19:25:26 GMT"],"Link":["<https://acme-v02.api.letsencrypt.org/directory>;rel=\"index\""],"Replay-Nonce":["NySGY0K7JUTWZXP94t3fsAB6fEnbBDy26-l_C9VXDLGlgVdeJTg"],"Server":["nginx"],"Strict-Transport-Security":["max-age=604800"],"X-Frame-Options":["DENY"]},"status_code":200}
{"level":"error","ts":1747855526.3186328,"logger":"tls.renew","msg":"could not get certificate from issuer","identifier":"*.joshuaochs.com","issuer":"acme-v02.api.letsencrypt.org-directory","error":"[*.joshuaochs.com] solving challenges: waiting for solver certmagic.solverWrapper to be ready: checking DNS propagation of \"_acme-challenge.joshuaochs.com.\" (relative=_acme-challenge zone=joshuaochs.com. resolvers=[127.0.0.11:53]): querying authoritative nameservers: dial tcp 108.162.192.174:53: connect: connection refused (order=https://acme-v02.api.letsencrypt.org/acme/order/1090263897/386379739917) (ca=https://acme-v02.api.letsencrypt.org/directory)"}
{"level":"debug","ts":1747855526.3187194,"logger":"events","msg":"event","name":"cert_failed","id":"71ec30ea-e109-44f5-b61d-714db8654a0d","origin":"tls","data":{"error":{},"identifier":"*.joshuaochs.com","issuers":["acme-v02.api.letsencrypt.org-directory"],"remaining":-51571753557501,"renewal":true}}
{"level":"error","ts":1747855526.3188126,"logger":"tls.renew","msg":"will retry","error":"[*.joshuaochs.com] Renew: [*.joshuaochs.com] solving challenges: waiting for solver certmagic.solverWrapper to be ready: checking DNS propagation of \"_acme-challenge.joshuaochs.com.\" (relative=_acme-challenge zone=joshuaochs.com. resolvers=[127.0.0.11:53]): querying authoritative nameservers: dial tcp 108.162.192.174:53: connect: connection refused (order=https://acme-v02.api.letsencrypt.org/acme/order/1090263897/386379739917) (ca=https://acme-v02.api.letsencrypt.org/directory)","attempt":1,"retrying_in":60,"elapsed":15.566185179,"max_duration":2592000}
{"level":"info","ts":1747855548.5614734,"msg":"shutting down apps, then terminating","signal":"SIGTERM"}
{"level":"warn","ts":1747855548.5615616,"msg":"exiting; byeee!! 👋","signal":"SIGTERM"}
{"level":"debug","ts":1747855548.561746,"logger":"events","msg":"event","name":"stopping","id":"f522f68e-90e9-4bb7-9f82-52d446895dd8","origin":"","data":null}
{"level":"info","ts":1747855548.5617971,"logger":"http","msg":"servers shutting down with eternal grace period"}
{"level":"info","ts":1747855548.5627606,"logger":"tls.renew","msg":"releasing lock","identifier":"joshuaochs.com"}
{"level":"info","ts":1747855548.5629582,"logger":"tls.renew","msg":"releasing lock","identifier":"*.joshuaochs.com"}
{"level":"error","ts":1747855548.5639136,"logger":"tls.renew","msg":"unable to unlock","identifier":"joshuaochs.com","lock_key":"issue_cert_joshuaochs.com","error":"remove /data/caddy/locks/issue_cert_joshuaochs.com.lock: no such file or directory"}
{"level":"error","ts":1747855548.5639813,"logger":"tls","msg":"job failed","error":"joshuaochs.com: renewing certificate: context canceled"}
{"level":"error","ts":1747855548.563957,"msg":"unable to clean up lock in storage backend","signal":"SIGTERM","storage":"FileStorage:/data/caddy","lock_key":"issue_cert_*.joshuaochs.com","error":"remove /data/caddy/locks/issue_cert_wildcard_.joshuaochs.com.lock: no such file or directory"}
{"level":"info","ts":1747855548.5642045,"logger":"admin","msg":"stopped previous server","address":"localhost:2019"}
{"level":"info","ts":1747855548.5642557,"msg":"shutdown complete","signal":"SIGTERM","exit_code":0}
{"level":"info","ts":1747855554.9466841,"msg":"maxprocs: Leaving GOMAXPROCS=8: CPU quota undefined"}
{"level":"info","ts":1747855554.947622,"msg":"GOMEMLIMIT is updated","package":"github.com/KimMachineGun/automemlimit/memlimit","GOMEMLIMIT":30289261363,"previous":9223372036854775807}
{"level":"info","ts":1747855554.9478374,"msg":"using config from file","file":"/etc/caddy/Caddyfile"}
{"level":"warn","ts":1747855554.9540567,"logger":"config.adapter.caddyfile","msg":"the 'basicauth' directive is deprecated, please use 'basic_auth' instead!"}
{"level":"warn","ts":1747855554.954837,"logger":"config.adapter.caddyfile","msg":"the 'basicauth' directive is deprecated, please use 'basic_auth' instead!"}
{"level":"info","ts":1747855554.9584875,"msg":"adapted config to JSON","adapter":"caddyfile"}
{"level":"info","ts":1747855554.9636338,"logger":"admin","msg":"admin endpoint started","address":"localhost:2019","enforce_origin":false,"origins":["//localhost:2019","//[::1]:2019","//127.0.0.1:2019"]}
{"level":"info","ts":1747855554.9647424,"logger":"http.auto_https","msg":"server is listening only on the HTTPS port but has no TLS connection policies; adding one to enable TLS","server_name":"srv0","https_port":443}
{"level":"info","ts":1747855554.9648025,"logger":"http.auto_https","msg":"enabling automatic HTTP->HTTPS redirects","server_name":"srv0"}
{"level":"info","ts":1747855554.9649763,"logger":"tls.cache.maintenance","msg":"started background certificate maintenance","cache":"0xc000686680"}
{"level":"debug","ts":1747855554.9648688,"logger":"http.auto_https","msg":"adjusted config","tls":{"automation":{"policies":[{"subjects":["*.joshuaochs.com","joshuaochs.com"]},{}]}},"http":{"servers":{"remaining_auto_https_redirects":{"listen":[":80"],"routes":[{},{}]},"srv0":{"listen":[":443"],"routes":[{"handle":[{"handler":"subroute","routes":[{"group":"group7","handle":[{"handler":"subroute","routes":[{"handle":[{"handler":"reverse_proxy","transport":{"protocol":"http","tls":{}},"upstreams":[{"dial":"docker.homelab.joshuaochs.com:9091"}]}]}]}],"match":[{"host":["transmission.joshuaochs.com"]}]},{"group":"group7","handle":[{"handler":"subroute","routes":[{"handle":[{"handler":"reverse_proxy","upstreams":[{"dial":"docker.homelab.joshuaochs.com:81"}]}]}]}],"match":[{"host":["cal.joshuaochs.com"]}]},{"group":"group7","handle":[{"handler":"subroute","routes":[{"handle":[{"handler":"request_body","max_size":1000000000},{"handler":"reverse_proxy","transport":{"protocol":"http","tls":{"insecure_skip_verify":true}},"upstreams":[{"dial":"docker.homelab.joshuaochs.com:3443"}]}]}]}],"match":[{"host":["snapdrop.joshuaochs.com"]}]},{"group":"group7","handle":[{"handler":"subroute","routes":[{"handle":[{"handler":"static_response","headers":{"Location":["http://booru.joshuaochs.com{http.request.uri}"]},"status_code":301}]}]}],"match":[{"host":["homebooru.joshuaochs.com"]}]},{"group":"group7","handle":[{"handler":"subroute","routes":[{"handle":[{"handler":"request_body","max_size":1000000000},{"handler":"authentication","providers":{"http_basic":{"accounts":[{"password":"$2a$14$iu/RQDKMQSzmaz.Lh2PRHO2w4qiLz3kloPA7ewUtJEdA5nPWZLKxe","username":"jochs"}],"hash":{"algorithm":"bcrypt"},"hash_cache":{}}}},{"handler":"reverse_proxy","upstreams":[{"dial":"docker.homelab.joshuaochs.com:8008"}]}]}]}],"match":[{"host":["booru.joshuaochs.com"]}]},{"group":"group7","handle":[{"handler":"subroute","routes":[{"handle":[{"handler":"request_body","max_size":1000000000},{"handler":"authentication","providers":{"http_basic":{"accounts":[{"password":"$2a$14$iu/RQDKMQSzmaz.Lh2PRHO2w4qiLz3kloPA7ewUtJEdA5nPWZLKxe","username":"jochs"}],"hash":{"algorithm":"bcrypt"},"hash_cache":{}}}},{"handler":"reverse_proxy","upstreams":[{"dial":"docker.homelab.joshuaochs.com:8018"}]}]}]}],"match":[{"host":["manga.joshuaochs.com"]}]},{"group":"group7","handle":[{"handler":"subroute","routes":[{"handle":[{"handler":"reverse_proxy","upstreams":[{"dial":"docker.homelab.joshuaochs.com:8080"}]}],"match":[{"path":["/inform"]}]},{"handle":[{"handler":"reverse_proxy","headers":{"request":{"delete":["Authorization"]}},"transport":{"protocol":"http","tls":{"insecure_skip_verify":true}},"upstreams":[{"dial":"docker.homelab.joshuaochs.com:8443"}]}]}]}],"match":[{"host":["unifi.joshuaochs.com"]}]},{"handle":[{"handler":"reverse_proxy","upstreams":[{"dial":"docker.homelab.joshuaochs.com:8082"}]}],"match":[{"host":["microbin.joshuaochs.com"]}]},{"handle":[{"handler":"reverse_proxy","upstreams":[{"dial":"docker.homelab.joshuaochs.com:8009"}]}],"match":[{"host":["tools.joshuaochs.com"]}]},{"handle":[{"handler":"reverse_proxy","upstreams":[{"dial":"docker.homelab.joshuaochs.com:4445"}]}],"match":[{"host":["translate.joshuaochs.com"]}]},{"handle":[{"handler":"reverse_proxy","upstreams":[{"dial":"docker.homelab.joshuaochs.com:8001"}]}],"match":[{"host":["vault.joshuaochs.com"]}]},{"handle":[{"handler":"reverse_proxy","upstreams":[{"dial":"docker.homelab.joshuaochs.com:5000"}]}],"match":[{"host":["whoogle.joshuaochs.com"]}]},{"handle":[{"handler":"reverse_proxy","upstreams":[{"dial":"docker.homelab.joshuaochs.com:8081"}]}],"match":[{"host":["wikipedia.joshuaochs.com"]}]},{"handle":[{"handler":"reverse_proxy","upstreams":[{"dial":"gpu.homelab.joshuaochs.com:3000"}]}],"match":[{"host":["chatgpt.joshuaochs.com"]}]},{"handle":[{"handler":"reverse_proxy","upstreams":[{"dial":"gpu.homelab.joshuaochs.com:4000"}]}],"match":[{"host":["chatapi.joshuaochs.com"]}]},{"handle":[{"handler":"reverse_proxy","upstreams":[{"dial":"docker.homelab.joshuaochs.com:88"}]}],"match":[{"host":["rss.joshuaochs.com"]}]}]}],"terminal":true}],"tls_connection_policies":[{}],"automatic_https":{}}}}}
{"level":"debug","ts":1747855554.9778419,"logger":"http","msg":"starting server loop","address":"[::]:443","tls":true,"http3":false}
{"level":"info","ts":1747855554.9779391,"logger":"http","msg":"enabling HTTP/3 listener","addr":":443"}
{"level":"info","ts":1747855554.9783242,"msg":"failed to sufficiently increase receive buffer size (was: 208 kiB, wanted: 7168 kiB, got: 416 kiB). See https://github.com/quic-go/quic-go/wiki/UDP-Buffer-Sizes for details."}
{"level":"info","ts":1747855554.9792023,"logger":"http.log","msg":"server running","name":"srv0","protocols":["h1","h2","h3"]}
{"level":"debug","ts":1747855554.9798627,"logger":"http","msg":"starting server loop","address":"[::]:80","tls":false,"http3":false}
{"level":"warn","ts":1747855554.9799995,"logger":"http","msg":"HTTP/2 skipped because it requires TLS","network":"tcp","addr":":80"}
{"level":"warn","ts":1747855554.9800236,"logger":"http","msg":"HTTP/3 skipped because it requires TLS","network":"tcp","addr":":80"}
{"level":"info","ts":1747855554.9800365,"logger":"http.log","msg":"server running","name":"remaining_auto_https_redirects","protocols":["h1","h2","h3"]}
{"level":"info","ts":1747855554.9801295,"logger":"http","msg":"enabling automatic TLS certificate management","domains":["joshuaochs.com","*.joshuaochs.com"]}
{"level":"warn","ts":1747855555.0416827,"logger":"tls","msg":"stapling OCSP","error":"no OCSP stapling for [*.joshuaochs.com]: parsing OCSP response: ocsp: error from server: unauthorized","identifiers":["*.joshuaochs.com"]}
{"level":"debug","ts":1747855555.042337,"logger":"tls.cache","msg":"added certificate to cache","subjects":["*.joshuaochs.com"],"expiration":1747803939,"managed":true,"issuer_key":"acme-v02.api.letsencrypt.org-directory","hash":"53f91cc34dda7562b93a1ba32dac21c0f30e13a3479679fdef52561ae0920ef8","cache_size":1,"cache_capacity":10000}
{"level":"debug","ts":1747855555.0424476,"logger":"events","msg":"event","name":"cached_managed_cert","id":"4b83e424-8951-458d-b320-7f781298acd6","origin":"tls","data":{"sans":["*.joshuaochs.com"]}}
{"level":"info","ts":1747855555.0431507,"logger":"tls","msg":"certificate is in configured renewal window based on expiration date","subjects":["*.joshuaochs.com"],"expiration":1747803939,"ari_cert_id":"","next_ari_update":null,"renew_check_interval":600,"window_start":-6795364578.8713455,"window_end":-6795364578.8713455,"remaining":-51616.043148208}
{"level":"info","ts":1747855555.0483167,"logger":"tls.renew","msg":"acquiring lock","identifier":"*.joshuaochs.com"}
{"level":"info","ts":1747855555.0499198,"logger":"tls.renew","msg":"lock acquired","identifier":"*.joshuaochs.com"}
{"level":"info","ts":1747855555.050884,"logger":"tls.renew","msg":"renewing certificate","identifier":"*.joshuaochs.com","remaining":-51616.050876162}
{"level":"debug","ts":1747855555.0511332,"logger":"events","msg":"event","name":"cert_obtaining","id":"9fdf464f-6b12-4e63-8371-7fc6978b795d","origin":"tls","data":{"forced":false,"identifier":"*.joshuaochs.com","issuer":"acme-v02.api.letsencrypt.org-directory","remaining":-51616050876162,"renewal":true}}
{"level":"debug","ts":1747855555.0518055,"logger":"tls","msg":"created CSR","identifiers":["*.joshuaochs.com"],"san_dns_names":["*.joshuaochs.com"],"san_emails":[],"common_name":"","extra_extensions":0}
{"level":"debug","ts":1747855555.054718,"logger":"tls.issuance.acme","msg":"using existing ACME account because key found in storage associated with email","email":"caddy@zerossl.com","ca":"https://acme-v02.api.letsencrypt.org/directory"}
{"level":"debug","ts":1747855555.0553946,"logger":"tls.issuance.acme","msg":"using existing ACME account because key found in storage associated with email","email":"caddy@zerossl.com","ca":"https://acme-v02.api.letsencrypt.org/directory"}
{"level":"info","ts":1747855555.0555089,"logger":"tls.issuance.acme","msg":"waiting on internal rate limiter","identifiers":["*.joshuaochs.com"],"ca":"https://acme-v02.api.letsencrypt.org/directory","account":"caddy@zerossl.com"}
{"level":"info","ts":1747855555.0556052,"logger":"tls.issuance.acme","msg":"done waiting on internal rate limiter","identifiers":["*.joshuaochs.com"],"ca":"https://acme-v02.api.letsencrypt.org/directory","account":"caddy@zerossl.com"}
{"level":"info","ts":1747855555.055716,"logger":"tls.issuance.acme","msg":"using ACME account","account_id":"https://acme-v02.api.letsencrypt.org/acme/acct/1090263897","account_contact":["mailto:caddy@zerossl.com"]}
{"level":"warn","ts":1747855555.0676498,"logger":"tls","msg":"stapling OCSP","error":"no OCSP stapling for [joshuaochs.com]: parsing OCSP response: ocsp: error from server: unauthorized","identifiers":["joshuaochs.com"]}
{"level":"debug","ts":1747855555.0678358,"logger":"tls.cache","msg":"added certificate to cache","subjects":["joshuaochs.com"],"expiration":1747803939,"managed":true,"issuer_key":"acme-v02.api.letsencrypt.org-directory","hash":"838717196645677770c5754c979f16e1fd4716ecb105b6bc278f8b8bff2aadc3","cache_size":2,"cache_capacity":10000}
{"level":"debug","ts":1747855555.0679522,"logger":"events","msg":"event","name":"cached_managed_cert","id":"690827b0-6267-4be0-85e7-6a0bee6c0dcf","origin":"tls","data":{"sans":["joshuaochs.com"]}}
{"level":"debug","ts":1747855555.068324,"logger":"events","msg":"event","name":"started","id":"9a7a372b-c321-48a0-bd33-1806d869d453","origin":"","data":null}
{"level":"info","ts":1747855555.0683568,"logger":"tls","msg":"certificate is in configured renewal window based on expiration date","subjects":["joshuaochs.com"],"expiration":1747803939,"ari_cert_id":"","next_ari_update":null,"renew_check_interval":600,"window_start":-6795364578.8713455,"window_end":-6795364578.8713455,"remaining":-51616.068355096}
{"level":"info","ts":1747855555.0691552,"msg":"autosaved config (load with --resume flag)","file":"/config/caddy/autosave.json"}
{"level":"info","ts":1747855555.069184,"msg":"serving initial configuration"}
{"level":"info","ts":1747855555.071444,"logger":"tls.renew","msg":"acquiring lock","identifier":"joshuaochs.com"}
{"level":"info","ts":1747855555.072015,"logger":"tls","msg":"storage cleaning happened too recently; skipping for now","storage":"FileStorage:/data/caddy","instance":"a1627015-e86e-4482-a840-56b2d42e5bca","try_again":1747941955.0720115,"try_again_in":86399.999998846}
{"level":"info","ts":1747855555.0727544,"logger":"tls","msg":"finished cleaning storage units"}
{"level":"info","ts":1747855555.0729544,"logger":"tls.renew","msg":"lock acquired","identifier":"joshuaochs.com"}
{"level":"info","ts":1747855555.0737112,"logger":"tls.renew","msg":"renewing certificate","identifier":"joshuaochs.com","remaining":-51616.073703878}
{"level":"debug","ts":1747855555.0737875,"logger":"events","msg":"event","name":"cert_obtaining","id":"ee76e09d-84aa-4905-a73a-17b749eff73e","origin":"tls","data":{"forced":false,"identifier":"joshuaochs.com","issuer":"acme-v02.api.letsencrypt.org-directory","remaining":-51616073703878,"renewal":true}}
{"level":"debug","ts":1747855555.0739796,"logger":"tls","msg":"created CSR","identifiers":["joshuaochs.com"],"san_dns_names":["joshuaochs.com"],"san_emails":[],"common_name":"","extra_extensions":0}
{"level":"debug","ts":1747855555.0752943,"logger":"tls.issuance.acme","msg":"using existing ACME account because key found in storage associated with email","email":"caddy@zerossl.com","ca":"https://acme-v02.api.letsencrypt.org/directory"}
{"level":"info","ts":1747855555.0753546,"logger":"tls.issuance.acme","msg":"waiting on internal rate limiter","identifiers":["joshuaochs.com"],"ca":"https://acme-v02.api.letsencrypt.org/directory","account":"caddy@zerossl.com"}
{"level":"info","ts":1747855555.0753994,"logger":"tls.issuance.acme","msg":"done waiting on internal rate limiter","identifiers":["joshuaochs.com"],"ca":"https://acme-v02.api.letsencrypt.org/directory","account":"caddy@zerossl.com"}
{"level":"info","ts":1747855555.0754383,"logger":"tls.issuance.acme","msg":"using ACME account","account_id":"https://acme-v02.api.letsencrypt.org/acme/acct/1090263897","account_contact":["mailto:caddy@zerossl.com"]}
{"level":"debug","ts":1747855555.2420418,"msg":"http request","method":"GET","url":"https://acme-v02.api.letsencrypt.org/directory","headers":{"User-Agent":["Caddy/2.10.0 CertMagic acmez (linux; amd64)"]},"response_headers":{"Cache-Control":["public, max-age=0, no-cache"],"Content-Length":["1012"],"Content-Type":["application/json"],"Date":["Wed, 21 May 2025 19:25:55 GMT"],"Server":["nginx"],"Strict-Transport-Security":["max-age=604800"],"X-Frame-Options":["DENY"]},"status_code":200}
{"level":"debug","ts":1747855555.242667,"msg":"creating order","account":"https://acme-v02.api.letsencrypt.org/acme/acct/1090263897","identifiers":["*.joshuaochs.com"]}
{"level":"debug","ts":1747855555.2430038,"msg":"creating order","account":"https://acme-v02.api.letsencrypt.org/acme/acct/1090263897","identifiers":["joshuaochs.com"]}
{"level":"debug","ts":1747855555.3016253,"msg":"http request","method":"HEAD","url":"https://acme-v02.api.letsencrypt.org/acme/new-nonce","headers":{"User-Agent":["Caddy/2.10.0 CertMagic acmez (linux; amd64)"]},"response_headers":{"Cache-Control":["public, max-age=0, no-cache"],"Date":["Wed, 21 May 2025 19:25:55 GMT"],"Link":["<https://acme-v02.api.letsencrypt.org/directory>;rel=\"index\""],"Replay-Nonce":["NeMLvpFRJpPCbGbhqVDqLF72RK0CixhISHIjvPMg5ecChPEw0-o"],"Server":["nginx"],"Strict-Transport-Security":["max-age=604800"],"X-Frame-Options":["DENY"]},"status_code":200}
{"level":"debug","ts":1747855555.3519669,"msg":"http request","method":"HEAD","url":"https://acme-v02.api.letsencrypt.org/acme/new-nonce","headers":{"User-Agent":["Caddy/2.10.0 CertMagic acmez (linux; amd64)"]},"response_headers":{"Cache-Control":["public, max-age=0, no-cache"],"Date":["Wed, 21 May 2025 19:25:55 GMT"],"Link":["<https://acme-v02.api.letsencrypt.org/directory>;rel=\"index\""],"Replay-Nonce":["yVRbjWxmRTzG-bkmjbUoiE_H1i-TSS_sOfk6llVib2QqVP5QGDg"],"Server":["nginx"],"Strict-Transport-Security":["max-age=604800"],"X-Frame-Options":["DENY"]},"status_code":200}
{"level":"debug","ts":1747855555.477041,"msg":"http request","method":"POST","url":"https://acme-v02.api.letsencrypt.org/acme/new-order","headers":{"Content-Type":["application/jose+json"],"User-Agent":["Caddy/2.10.0 CertMagic acmez (linux; amd64)"]},"response_headers":{"Boulder-Requester":["1090263897"],"Cache-Control":["public, max-age=0, no-cache"],"Content-Length":["350"],"Content-Type":["application/json"],"Date":["Wed, 21 May 2025 19:25:55 GMT"],"Link":["<https://acme-v02.api.letsencrypt.org/directory>;rel=\"index\""],"Location":["https://acme-v02.api.letsencrypt.org/acme/order/1090263897/386379905007"],"Replay-Nonce":["NeMLvpFRxpUpL0b_cTYsGFya_fgpPi6-JZwMN2b_e7f0AMVgSg0"],"Server":["nginx"],"Strict-Transport-Security":["max-age=604800"],"X-Frame-Options":["DENY"]},"status_code":201}
{"level":"debug","ts":1747855555.482055,"msg":"http request","method":"POST","url":"https://acme-v02.api.letsencrypt.org/acme/new-order","headers":{"Content-Type":["application/jose+json"],"User-Agent":["Caddy/2.10.0 CertMagic acmez (linux; amd64)"]},"response_headers":{"Boulder-Requester":["1090263897"],"Cache-Control":["public, max-age=0, no-cache"],"Content-Length":["348"],"Content-Type":["application/json"],"Date":["Wed, 21 May 2025 19:25:55 GMT"],"Link":["<https://acme-v02.api.letsencrypt.org/directory>;rel=\"index\""],"Location":["https://acme-v02.api.letsencrypt.org/acme/order/1090263897/386379905387"],"Replay-Nonce":["NeMLvpFRDS5ek2KC0yCJYry6kW5byPqvcPCsSULVTyNID0QV4AE"],"Server":["nginx"],"Strict-Transport-Security":["max-age=604800"],"X-Frame-Options":["DENY"]},"status_code":201}
{"level":"debug","ts":1747855555.5403235,"msg":"http request","method":"POST","url":"https://acme-v02.api.letsencrypt.org/acme/authz/1090263897/523784208237","headers":{"Content-Type":["application/jose+json"],"User-Agent":["Caddy/2.10.0 CertMagic acmez (linux; amd64)"]},"response_headers":{"Boulder-Requester":["1090263897"],"Cache-Control":["public, max-age=0, no-cache"],"Content-Length":["396"],"Content-Type":["application/json"],"Date":["Wed, 21 May 2025 19:25:55 GMT"],"Link":["<https://acme-v02.api.letsencrypt.org/directory>;rel=\"index\""],"Replay-Nonce":["yVRbjWxmD8MGUCmr0E5g0FmZB2BRuzfi_sHQoI17S5kgb1RoG-M"],"Server":["nginx"],"Strict-Transport-Security":["max-age=604800"],"X-Frame-Options":["DENY"]},"status_code":200}
{"level":"info","ts":1747855555.5408928,"msg":"trying to solve challenge","identifier":"*.joshuaochs.com","challenge_type":"dns-01","ca":"https://acme-v02.api.letsencrypt.org/directory"}
{"level":"debug","ts":1747855555.5561748,"msg":"http request","method":"POST","url":"https://acme-v02.api.letsencrypt.org/acme/authz/1090263897/523784208757","headers":{"Content-Type":["application/jose+json"],"User-Agent":["Caddy/2.10.0 CertMagic acmez (linux; amd64)"]},"response_headers":{"Boulder-Requester":["1090263897"],"Cache-Control":["public, max-age=0, no-cache"],"Content-Length":["822"],"Content-Type":["application/json"],"Date":["Wed, 21 May 2025 19:25:55 GMT"],"Link":["<https://acme-v02.api.letsencrypt.org/directory>;rel=\"index\""],"Replay-Nonce":["yVRbjWxmOeTgI9MXqCCFXULxbJAo-CYw7UybreP2ycS_Y1K5gRI"],"Server":["nginx"],"Strict-Transport-Security":["max-age=604800"],"X-Frame-Options":["DENY"]},"status_code":200}
{"level":"info","ts":1747855555.5565162,"msg":"trying to solve challenge","identifier":"joshuaochs.com","challenge_type":"dns-01","ca":"https://acme-v02.api.letsencrypt.org/directory"}
{"level":"debug","ts":1747855556.1092632,"msg":"waiting for solver before continuing","identifier":"joshuaochs.com","challenge_type":"dns-01"}
{"level":"debug","ts":1747855556.1183121,"msg":"waiting for solver before continuing","identifier":"*.joshuaochs.com","challenge_type":"dns-01"}
{"level":"debug","ts":1747855558.121076,"msg":"done waiting for solver","identifier":"joshuaochs.com","challenge_type":"dns-01"}
{"level":"debug","ts":1747855558.656963,"msg":"http request","method":"POST","url":"https://acme-v02.api.letsencrypt.org/acme/authz/1090263897/523784208757","headers":{"Content-Type":["application/jose+json"],"User-Agent":["Caddy/2.10.0 CertMagic acmez (linux; amd64)"]},"response_headers":{"Boulder-Requester":["1090263897"],"Cache-Control":["public, max-age=0, no-cache"],"Content-Length":["826"],"Content-Type":["application/json"],"Date":["Wed, 21 May 2025 19:25:58 GMT"],"Link":["<https://acme-v02.api.letsencrypt.org/directory>;rel=\"index\""],"Replay-Nonce":["NeMLvpFRL1XjMOBY9OJ6nwi2IonBfN9AJz2q6t-nG2d3EnHx_uY"],"Server":["nginx"],"Strict-Transport-Security":["max-age=604800"],"X-Frame-Options":["DENY"]},"status_code":200}
{"level":"error","ts":1747855558.6573281,"logger":"tls.renew","msg":"could not get certificate from issuer","identifier":"joshuaochs.com","issuer":"acme-v02.api.letsencrypt.org-directory","error":"[joshuaochs.com] solving challenges: waiting for solver certmagic.solverWrapper to be ready: checking DNS propagation of \"_acme-challenge.joshuaochs.com.\" (relative=_acme-challenge zone=joshuaochs.com. resolvers=[127.0.0.11:53]): querying authoritative nameservers: read udp 172.30.29.2:58441->172.64.32.174:53: read: connection refused (order=https://acme-v02.api.letsencrypt.org/acme/order/1090263897/386379905387) (ca=https://acme-v02.api.letsencrypt.org/directory)"}
{"level":"debug","ts":1747855558.657458,"logger":"events","msg":"event","name":"cert_failed","id":"5d4619f7-5dc0-44dd-87f6-3b16f5f31625","origin":"tls","data":{"error":{},"identifier":"joshuaochs.com","issuers":["acme-v02.api.letsencrypt.org-directory"],"remaining":-51616073703878,"renewal":true}}
{"level":"error","ts":1747855558.6575835,"logger":"tls.renew","msg":"will retry","error":"[joshuaochs.com] Renew: [joshuaochs.com] solving challenges: waiting for solver certmagic.solverWrapper to be ready: checking DNS propagation of \"_acme-challenge.joshuaochs.com.\" (relative=_acme-challenge zone=joshuaochs.com. resolvers=[127.0.0.11:53]): querying authoritative nameservers: read udp 172.30.29.2:58441->172.64.32.174:53: read: connection refused (order=https://acme-v02.api.letsencrypt.org/acme/order/1090263897/386379905387) (ca=https://acme-v02.api.letsencrypt.org/directory)","attempt":1,"retrying_in":60,"elapsed":3.584584859,"max_duration":2592000}
{"level":"debug","ts":1747855569.1414702,"msg":"done waiting for solver","identifier":"*.joshuaochs.com","challenge_type":"dns-01"}
{"level":"debug","ts":1747855569.637854,"msg":"http request","method":"POST","url":"https://acme-v02.api.letsencrypt.org/acme/authz/1090263897/523784208237","headers":{"Content-Type":["application/jose+json"],"User-Agent":["Caddy/2.10.0 CertMagic acmez (linux; amd64)"]},"response_headers":{"Boulder-Requester":["1090263897"],"Cache-Control":["public, max-age=0, no-cache"],"Content-Length":["400"],"Content-Type":["application/json"],"Date":["Wed, 21 May 2025 19:26:09 GMT"],"Link":["<https://acme-v02.api.letsencrypt.org/directory>;rel=\"index\""],"Replay-Nonce":["NeMLvpFRDGoPsbpLFUMYNcHA234oqD4UwqcWwKkxYDutc5i_jsY"],"Server":["nginx"],"Strict-Transport-Security":["max-age=604800"],"X-Frame-Options":["DENY"]},"status_code":200}
{"level":"error","ts":1747855569.6383548,"logger":"tls.renew","msg":"could not get certificate from issuer","identifier":"*.joshuaochs.com","issuer":"acme-v02.api.letsencrypt.org-directory","error":"[*.joshuaochs.com] solving challenges: waiting for solver certmagic.solverWrapper to be ready: checking DNS propagation of \"_acme-challenge.joshuaochs.com.\" (relative=_acme-challenge zone=joshuaochs.com. resolvers=[127.0.0.11:53]): querying authoritative nameservers: dial tcp 173.245.59.194:53: connect: connection refused (order=https://acme-v02.api.letsencrypt.org/acme/order/1090263897/386379905007) (ca=https://acme-v02.api.letsencrypt.org/directory)"}
{"level":"debug","ts":1747855569.6384387,"logger":"events","msg":"event","name":"cert_failed","id":"63abe377-3177-4c6d-9423-b8278f41e2c0","origin":"tls","data":{"error":{},"identifier":"*.joshuaochs.com","issuers":["acme-v02.api.letsencrypt.org-directory"],"remaining":-51616050876162,"renewal":true}}
{"level":"error","ts":1747855569.6385481,"logger":"tls.renew","msg":"will retry","error":"[*.joshuaochs.com] Renew: [*.joshuaochs.com] solving challenges: waiting for solver certmagic.solverWrapper to be ready: checking DNS propagation of \"_acme-challenge.joshuaochs.com.\" (relative=_acme-challenge zone=joshuaochs.com. resolvers=[127.0.0.11:53]): querying authoritative nameservers: dial tcp 173.245.59.194:53: connect: connection refused (order=https://acme-v02.api.letsencrypt.org/acme/order/1090263897/386379905007) (ca=https://acme-v02.api.letsencrypt.org/directory)","attempt":1,"retrying_in":60,"elapsed":14.588592423,"max_duration":2592000}
{"level":"debug","ts":1747855574.86239,"logger":"events","msg":"event","name":"tls_get_certificate","id":"b1554329-2f46-4e9c-b530-5f6b29cc06f5","origin":"tls","data":{"client_hello":{"CipherSuites":[4865,4866,4867,49195,49196,49199,49200,49171,49192,156,157,47,53,10],"ServerName":"social.joshuaochs.com","SupportedCurves":[29,23,24,25,4588,25497,65074],"SupportedPoints":"AA==","SignatureSchemes":[1027,2052,1025,1283,2053,1281,2054,1537,513,1539],"SupportedProtos":["h2","http/1.1"],"SupportedVersions":[772,771,770,769],"RemoteAddr":{"IP":"172.69.150.77","Port":20438,"Zone":""},"LocalAddr":{"IP":"172.30.29.2","Port":443,"Zone":""}}}}
{"level":"debug","ts":1747855574.8627396,"logger":"tls.handshake","msg":"no matching certificates and no custom selection logic","identifier":"social.joshuaochs.com"}
{"level":"debug","ts":1747855574.8627946,"logger":"tls.handshake","msg":"choosing certificate","identifier":"*.joshuaochs.com","num_choices":1}
{"level":"debug","ts":1747855574.862818,"logger":"tls.handshake","msg":"default certificate selection results","identifier":"*.joshuaochs.com","subjects":["*.joshuaochs.com"],"managed":true,"issuer_key":"acme-v02.api.letsencrypt.org-directory","hash":"53f91cc34dda7562b93a1ba32dac21c0f30e13a3479679fdef52561ae0920ef8"}
{"level":"debug","ts":1747855574.8628428,"logger":"tls.handshake","msg":"matched certificate in cache","remote_ip":"172.69.150.77","remote_port":"20438","subjects":["*.joshuaochs.com"],"managed":true,"expiration":1747803939,"hash":"53f91cc34dda7562b93a1ba32dac21c0f30e13a3479679fdef52561ae0920ef8"}
{"level":"debug","ts":1747855574.989067,"logger":"http.stdlib","msg":"http: TLS handshake error from 172.69.150.77:20438: remote error: tls: expired certificate"}
{"level":"info","ts":1747855618.6613762,"logger":"tls.renew","msg":"renewing certificate","identifier":"joshuaochs.com","remaining":-51679.661363509}
{"level":"debug","ts":1747855618.6615307,"logger":"events","msg":"event","name":"cert_obtaining","id":"0838b5f2-0b7a-40ad-842a-a6ddd8a1b0d6","origin":"tls","data":{"forced":false,"identifier":"joshuaochs.com","issuer":"acme-v02.api.letsencrypt.org-directory","remaining":-51679661363509,"renewal":true}}
{"level":"debug","ts":1747855618.6618404,"logger":"tls","msg":"created CSR","identifiers":["joshuaochs.com"],"san_dns_names":["joshuaochs.com"],"san_emails":[],"common_name":"","extra_extensions":0}
{"level":"debug","ts":1747855618.6648133,"logger":"tls.issuance.acme","msg":"using existing ACME account because key found in storage associated with email","email":"caddy@zerossl.com","ca":"https://acme-staging-v02.api.letsencrypt.org/directory"}
{"level":"info","ts":1747855618.6648986,"logger":"tls.issuance.acme","msg":"using ACME account","account_id":"https://acme-staging-v02.api.letsencrypt.org/acme/acct/196097374","account_contact":["mailto:caddy@zerossl.com"]}
{"level":"debug","ts":1747855618.8521442,"msg":"http request","method":"GET","url":"https://acme-staging-v02.api.letsencrypt.org/directory","headers":{"User-Agent":["Caddy/2.10.0 CertMagic acmez (linux; amd64)"]},"response_headers":{"Cache-Control":["public, max-age=0, no-cache"],"Content-Length":["1086"],"Content-Type":["application/json"],"Date":["Wed, 21 May 2025 19:26:58 GMT"],"Server":["nginx"],"Strict-Transport-Security":["max-age=604800"],"X-Frame-Options":["DENY"]},"status_code":200}
{"level":"debug","ts":1747855618.8524923,"msg":"creating order","account":"https://acme-staging-v02.api.letsencrypt.org/acme/acct/196097374","identifiers":["joshuaochs.com"]}
{"level":"debug","ts":1747855618.909933,"msg":"http request","method":"HEAD","url":"https://acme-staging-v02.api.letsencrypt.org/acme/new-nonce","headers":{"User-Agent":["Caddy/2.10.0 CertMagic acmez (linux; amd64)"]},"response_headers":{"Cache-Control":["public, max-age=0, no-cache"],"Date":["Wed, 21 May 2025 19:26:58 GMT"],"Link":["<https://acme-staging-v02.api.letsencrypt.org/directory>;rel=\"index\""],"Replay-Nonce":["1jOOXM0FAzJDqGPpQNCnEfeMxRdewRYmzAGqjnLYRubJeLWRyhE"],"Server":["nginx"],"Strict-Transport-Security":["max-age=604800"],"X-Frame-Options":["DENY"]},"status_code":200}
{"level":"debug","ts":1747855619.010614,"msg":"http request","method":"POST","url":"https://acme-staging-v02.api.letsencrypt.org/acme/new-order","headers":{"Content-Type":["application/jose+json"],"User-Agent":["Caddy/2.10.0 CertMagic acmez (linux; amd64)"]},"response_headers":{"Boulder-Requester":["196097374"],"Cache-Control":["public, max-age=0, no-cache"],"Content-Length":["360"],"Content-Type":["application/json"],"Date":["Wed, 21 May 2025 19:26:58 GMT"],"Link":["<https://acme-staging-v02.api.letsencrypt.org/directory>;rel=\"index\""],"Location":["https://acme-staging-v02.api.letsencrypt.org/acme/order/196097374/24795364404"],"Replay-Nonce":["Yo1slBcdLOF_P0Y9CzzmWcfWotSEY9Pp-JnefsiaNjE4f-Kx6Zw"],"Server":["nginx"],"Strict-Transport-Security":["max-age=604800"],"X-Frame-Options":["DENY"]},"status_code":201}
{"level":"debug","ts":1747855619.080055,"msg":"http request","method":"POST","url":"https://acme-staging-v02.api.letsencrypt.org/acme/authz/196097374/17506084884","headers":{"Content-Type":["application/jose+json"],"User-Agent":["Caddy/2.10.0 CertMagic acmez (linux; amd64)"]},"response_headers":{"Boulder-Requester":["196097374"],"Cache-Control":["public, max-age=0, no-cache"],"Content-Length":["840"],"Content-Type":["application/json"],"Date":["Wed, 21 May 2025 19:26:59 GMT"],"Link":["<https://acme-staging-v02.api.letsencrypt.org/directory>;rel=\"index\""],"Replay-Nonce":["Yo1slBcd27KEraxiQcforVU8yAl1ehLB1WBdmw-EH-XLvd2iQO4"],"Server":["nginx"],"Strict-Transport-Security":["max-age=604800"],"X-Frame-Options":["DENY"]},"status_code":200}
{"level":"info","ts":1747855619.080385,"msg":"trying to solve challenge","identifier":"joshuaochs.com","challenge_type":"dns-01","ca":"https://acme-staging-v02.api.letsencrypt.org/directory"}
{"level":"debug","ts":1747855619.3709352,"msg":"waiting for solver before continuing","identifier":"joshuaochs.com","challenge_type":"dns-01"}
{"level":"debug","ts":1747855621.3822887,"msg":"done waiting for solver","identifier":"joshuaochs.com","challenge_type":"dns-01"}
{"level":"debug","ts":1747855621.8349202,"msg":"http request","method":"POST","url":"https://acme-staging-v02.api.letsencrypt.org/acme/authz/196097374/17506084884","headers":{"Content-Type":["application/jose+json"],"User-Agent":["Caddy/2.10.0 CertMagic acmez (linux; amd64)"]},"response_headers":{"Boulder-Requester":["196097374"],"Cache-Control":["public, max-age=0, no-cache"],"Content-Length":["844"],"Content-Type":["application/json"],"Date":["Wed, 21 May 2025 19:27:01 GMT"],"Link":["<https://acme-staging-v02.api.letsencrypt.org/directory>;rel=\"index\""],"Replay-Nonce":["Yo1slBcdYkwiBKv_a2o04c1pR9VwJf5hzYxm5k-Leh4LrFFK5bU"],"Server":["nginx"],"Strict-Transport-Security":["max-age=604800"],"X-Frame-Options":["DENY"]},"status_code":200}
{"level":"error","ts":1747855621.8352947,"logger":"tls.renew","msg":"could not get certificate from issuer","identifier":"joshuaochs.com","issuer":"acme-v02.api.letsencrypt.org-directory","error":"[joshuaochs.com] solving challenges: waiting for solver certmagic.solverWrapper to be ready: checking DNS propagation of \"_acme-challenge.joshuaochs.com.\" (relative=_acme-challenge zone=joshuaochs.com. resolvers=[127.0.0.11:53]): querying authoritative nameservers: read udp 172.30.29.2:38375->173.245.58.174:53: read: connection refused (order=https://acme-staging-v02.api.letsencrypt.org/acme/order/196097374/24795364404) (ca=https://acme-staging-v02.api.letsencrypt.org/directory)"}
{"level":"debug","ts":1747855621.8353994,"logger":"events","msg":"event","name":"cert_failed","id":"57a31a77-c746-4b4e-b66d-5a9dab061003","origin":"tls","data":{"error":{},"identifier":"joshuaochs.com","issuers":["acme-v02.api.letsencrypt.org-directory"],"remaining":-51679661363509,"renewal":true}}
{"level":"error","ts":1747855621.8354673,"logger":"tls.renew","msg":"will retry","error":"[joshuaochs.com] Renew: [joshuaochs.com] solving challenges: waiting for solver certmagic.solverWrapper to be ready: checking DNS propagation of \"_acme-challenge.joshuaochs.com.\" (relative=_acme-challenge zone=joshuaochs.com. resolvers=[127.0.0.11:53]): querying authoritative nameservers: read udp 172.30.29.2:38375->173.245.58.174:53: read: connection refused (order=https://acme-staging-v02.api.letsencrypt.org/acme/order/196097374/24795364404) (ca=https://acme-staging-v02.api.letsencrypt.org/directory)","attempt":2,"retrying_in":120,"elapsed":66.762468971,"max_duration":2592000}
3

Here’s the issue. The DNS server in your environment is unavailable, breaking Caddy’s validation

4

I found the issue, but it raises more questions (and would have replied already, but forum post rate limits are nuts). The key appears to be this line:

{"level":"error","ts":1747855514.20507,"logger":"tls.renew","msg":"could not get certificate from issuer","identifier":"joshuaochs.com","issuer":"acme-v02.api.letsencrypt.org-directory","error":"[joshuaochs.com] solving challenges: waiting for solver certmagic.solverWrapper to be ready: checking DNS propagation of \"_acme-challenge.joshuaochs.com.\" (relative=_acme-challenge zone=joshuaochs.com. resolvers=[127.0.0.11:53]): querying authoritative nameservers: read udp 172.30.29.2:40957->172.64.33.194:53: read: connection refused (order=https://acme-v02.api.letsencrypt.org/acme/order/1090263897/386379740257) (ca=https://acme-v02.api.letsencrypt.org/directory)"}

Rather than using the host’s DNS resolver, it’s trying to send a direct DNS query to Cloudflare at 172.64.33.194 - I assume this is the DNS-01 check. However, I have all DNS blocked out of my network, other than via my own local DNS resolvers (for internal DNS, filtering/ad blocking, etc). That way services, malware, kids, etc can’t use their own DNS settings to bypass my filters and logging.

For the moment I’ve allowed my Docker host to send outbound DNS queries bypassing my servers. I’d really like to tighten this down though - there’s a lot of software on there, and I don’t particularly want to let anything else start bypassing my DNS. Is there a fixed list of addresses it’s going to check? Is there any reasonable way to send this via my own resolver, or is it just not going to work due to TXT-record propagation delays?

Thankful for any ideas.

(OT: the post length limits are non-sensical given the insistence on full debug logs - easily blowing past 64K - and the post frequency limits are likewise very restrictive. An hour between replies - really?)

5

Sorry, We had aggressive spammers for a long time, and the rate limit helped control them. We may loosen them later once we’re confident and comfortable with the control of spammers.

Now…

You can use the resolvers subdirective of tls to specify your own DNS server.

6

I saw that - along with the many warnings about using the tls directives. :slight_smile: Since this is presumably part of the DNS-01 verification, I’m guessing I should (reluctantly) leave it alone. After all, if my personal resolver forwards to Google’s DNS and Cloudflare hasn’t yet propagated there - I assume that would break things.

I get why this is going straight to Cloudflare (or for that matter, whatever DNS provider you’re it’s issuing the challenge via) - set the record, check for whether it happened. I noticed that it’s headed straight to a particular IP with Cloudflare, but not the usual 1.1.1.1 public resolver. Is that something hard-coded that I could add to firewall rules, or something that it’s being told by Cloudflare on the fly?

7

We look up the SOA of the FQDN to find the zone. Once know the zone, we find the NS of the zone, and query that directly for an authoritative answer. Propagation through other DNS servers around the globe might take time. We only care about the authoritative server.

If course the above is only done if the user hasn’t specified resolvers.

8

Hmmm - such a programmatic approach may yield a stable enough result that I could make a rule that simply assumes the answer. (Or I could be creating a well-hidden footgun.)

My main concern with using resolvers that if I try to push this through my DNS with the resolvers directive, then when it checks for the TXT record, it may not have propagated beyond Cloudflare yet. After all, the goal of the request is not to resolve the domain name to the IP address, but to check for the existence of the TXT record set as part of the DNS-01 challenge. A change that may run into propagation delays outside its home network.

For instance, say I use resolvers to force it via my DNS. Request goes to my server, which sees it’s for something not part of my local network, so it forwards upstream. Due to load balancing different resolvers (avoid profiling of requests) it happens to land at Google’s 8.8.8.8. Now it’s looking for the DNS TXT record that was set via the Cloudflare API, which Cloudflare knows about, but hasn’t propagated to Google yet. What happens now? I assume the ACME process for getting the cert fails. Which means using “resolvers” would generally break DNS-01 challenges, yes?

Please don’t misunderstand - I fully realize this is a problem that’s due to my setup and nothing for the Caddy devs to address. I’m just talking through it now to see what my options are for keeping my network the way I want it while using Caddy as normally as possible.

I have to assume this will affect any ACME client (e.g. certbot) the same way. Hmmm.

(ARGH: “An error occurred: You’re replying a bit too quickly. Please wait 12 9 5 minutes 40 seconds before trying again.” Eh, meant I could do a lot of editing.)

9

You can disable the propagation checks. See propagation_timeout and propagation_delay.

10

For anyone else’s future reference, this appears to work fine (implemented the suggestions above), allowing the DNS-01 challenge to be sent through my local DNS and avoid my network’s firewall blocking “unauthorized” outbound DNS.

*.joshuaochs.com, joshuaochs.com {
	tls {
		dns cloudflare {env.CLOUDFLARE_API_TOKEN}
		propagation_timeout -1
		resolvers 10.0.1.2 10.0.1.4
	}

On my network, 10.0.1.2 and 10.0.1.4 are my DNS servers. Seems like it’s all good! :+1:

11

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.