ECH using incorrect resolver

1. The problem I’m having:

ECH cannot find SOA record because it queries the wrong resolver (PiHole). I have split DNS so when it comes to getting certs with the DNS challenge, I use the option to override the resolver to “1.1.1.1” instead of the system default which is a local PiHole. For ECH, I do not see an option for override the resolver but it also seems that the Json Config Structure isn’t fully updated for ECH options yet. My question is does any option exist and if not would it be implemented at some point? Or is there a better way to handle DNS that I could be pushed into the direction of? The current solution I’m using is setting the DNS option in the quadlet, restarting the container long enough for it to set the ECH config, then reverting the DNS option.

Any help would be greatly appreciated!

2. Error messages and/or full log output:

caddy[468167]: {"level":"error","ts":1778723370.8787408,"logger":"tls.ech","msg":"failed to publish ECH configuration list","publisher":"dns.providers.cloudflare","domain":"moonmoondomain.com","config_ids":[128,153,237],"error":"could not determine zone for domain: could not find the start of authority for moonmoondomain.com.: NOERROR (domain=moonmoondomain.com nameservers=[10.89.1.1:53 10.89.2.1:53 10.89.3.1:53])"}

3. Caddy version:

Caddy v2.11.3

4. How I installed and ran Caddy:

Built caddy v2.11.3 image with

  • mholt/caddy-l4
  • caddy-dns/cloudflare
  • abiosoft/caddy-yaml
  • abiosoft/caddy-json-schema

ran using Podman

a. System environment:

Debian on a Raspberry Pi 4. Caddy is managed using a Podman Quadlet with socket activation.

b. Command:

N/A?

c. Service/unit/compose file:

[Unit]
Description=Caddy Container

[Container]
Image=localhost/caddy:2.11.3

ContainerName=caddy

Network=caddy
Network=authentik
Network=netbird

#DNS=1.1.1.1

Exec=caddy run --config /etc/caddy/caddy.yaml --adapter yaml
ReloadCmd=caddy reload --config /etc/caddy/caddy.yaml --adapter yaml

Volume=caddy-data:/data
Volume=caddy-config:/config
Volume=caddy-file:/etc/caddy
Volume=/var/log/caddy:/var/log/caddy
Volume=/var/www:/var/www:ro

d. My complete Caddy config:

logging:
  logs:
    default:
      exclude: [http.log.access.crowdsec]
    crowdsec:
      include: [http.log.access.crowdsec]
      writer:
        output: file
        filename: /var/log/caddy/access.log
apps: 
  tls:
    dns:
      name: cloudflare
      api_token: "###############"
    encrypted_client_hello:
      configs: [public_name: ech.moonmoondomain.com]
    automation:
      policies:
        - issuers:
          - module: acme
            email: ###############
            challenges:
              dns:
                provider:
                  name: cloudflare
                  api_token: "###############"
                resolvers: [1.1.1.1]
  http:
    servers:
      redirect:
        listen: [fd/3]
        listen_protocols: [[h1]]
        routes:
        - handle:
          - handler: static_response
            headers: {Location: ["https://{http.request.host}{http.request.uri}"]}
            status_code: "302"
      moonmoon:
        listen: [fd/4, fdgram/5]
        listen_protocols: [[h1, h2], [h3]]
        tls_connection_policies: [protocol_min: tls1.3]
        automatic_https: {disable_redirects: true}
        logs: {default_logger_name: crowdsec}
        routes:
        - match: [host: [moonmoondomain.com]] # currently down
          handle:
          - handler: encode
            encodings: {gzip: {}, zstd: {}}
            prefer: [zstd, gzip]
          - handler: headers
            response:
              require:
                headers:
                  Strict-Transport-Security: [null]
                  Permissions-Policy: [null]
                  Content-Security-Policy: [null]
                  X-Content-Type-Options: [null]
                  X-Frame-Options: [null]
                  X-Robots-Tag: [null]
              set:
                Strict-Transport-Security: [max-age=31536000; includeSubdomains; preload]
                Permissions-Policy: [interest-cohort=()]
                Content-Security-Policy: [upgrade-insecure-requests; frame-ancestors 'self'; form-action 'self']
                X-Content-Type-Options: [nosniff]
                X-Frame-Options: [DENY]
                X-Robots-Tag: [noindex, nofollow]
          - handler: subroute
            routes:
              - handle: # public
                  - handler: subroute
                    routes:
                      - handle: # public redirects
                          - handler: subroute
                            routes:
                              - match: [path: [/5etools]]
                                handle:
                                - handler: static_response
                                  headers: {Location: [/5etools/]}
                                  status_code: "302"
                              - match: [path: [/dnd]]
                                handle:
                                - handler: static_response
                                  headers: {Location: [/dnd/join]}
                                  status_code: "302"
                      - handle: # public responses
                          - handler: subroute
                            routes:
                              - match: [{path: [/5etools/*]}]
                                handle:
                                - handler: file_server
                                  root: /var/www/html
                              - match: [{path: [/, /static/*]}]
                                handle:
                                - handler: reverse_proxy
                                  upstreams: [dial: host.containers.internal:8000]        
        - match: [host: ["*.moonmoondomain.com"]]
          handle:
          - handler: encode
            encodings: {gzip: {}, zstd: {}}
            prefer: [zstd, gzip]
          - handler: headers
            response:
              require:
                headers:
                  Strict-Transport-Security: [null]
                  Permissions-Policy: [null]
                  Content-Security-Policy: [null]
                  X-Content-Type-Options: [null]
                  X-Frame-Options: [null]
                  X-Robots-Tag: [null]
              set:
                Strict-Transport-Security: [max-age=31536000; includeSubdomains; preload]
                Permissions-Policy: [interest-cohort=()]
                Content-Security-Policy: [upgrade-insecure-requests; frame-ancestors 'self'; form-action 'self']
                X-Content-Type-Options: [nosniff]
                X-Frame-Options: [DENY]
                X-Robots-Tag: [noindex, nofollow]
          - handler: subroute
            routes:   
            - match: [host: [authentik.moonmoondomain.com]]
              handle:
              - handler: subroute
                routes:
                - match: [remote_ip: {ranges: ["192.168.0.0/16", "172.16.0.0/12", "10.0.0.0/8", "127.0.0.1/8", "fd00::/8", "::1"]}] # private
                  handle:
                  - handler: reverse_proxy
                    transport: {protocol: http}
                    upstreams: [dial: authentik:9000]
                    headers:
                      response:
                        set:
                          Access-Control-Allow-Origin: [https://netbird.moonmoondomain.com]
                          Content-Security-Policy: [upgrade-insecure-requests; frame-ancestors https://netbird.moonmoondomain.com; form-action 'self']
            - match: [host: [forgejo.moonmoondomain.com]]
              handle:
              - handler: subroute
                routes:
                - match: [remote_ip: {ranges: ["192.168.0.0/16", "172.16.0.0/12", "10.0.0.0/8", "127.0.0.1/8", "fd00::/8", "::1"]}] # private
                  handle:
                  - handler: reverse_proxy
                    transport: {protocol: http,}
                    upstreams: [dial: moonmoonserver.moonmoondomain.com:3000]            
            - match: [host: [crafty.moonmoondomain.com]] # currently down
              handle:
              - handler: subroute
                routes:
                - match: [remote_ip: {ranges: ["192.168.0.0/16", "172.16.0.0/12", "10.0.0.0/8", "127.0.0.1/8", "fd00::/8", "::1"]}] # private
                  handle:
                  - handler: reverse_proxy
                    transport: {protocol: http, tls: {}}
                    upstreams: [dial: moonmoonserver.moonmoondomain.com:25550]
                - match: [path: [/status]]
                  handle:
                  - handler: reverse_proxy
                    transport: {protocol: http, tls: {}}
                    upstreams: [dial: moonmoonserver.moonmoondomain.com:25550]
            - match: [host: [netbird.moonmoondomain.com]]
              handle:
              - handler: subroute
                routes:
                - match: [header: {Content-Type: [application/grpc*]}]
                  handle:
                  - handler: reverse_proxy
                    transport: {protocol: http, versions: [h2c, "2"]}
                    upstreams: [dial: netbird-server:80]
                - match: [path: [/relay*, /ws-proxy/*, /api/*, /oauth2/*]]
                  handle:
                    - handler: reverse_proxy
                      transport: {protocol: http}
                      upstreams: [dial: netbird-server:80]
                - match: [path: [/*]]
                  handle:
                  - handler: reverse_proxy
                    transport: {protocol: http}
                    upstreams: [dial: netbird-dashboard:80]
            - match: [host: [norish.moonmoondomain.com]]
              handle:
              - handler: reverse_proxy
                transport: {protocol: http}
                upstreams: [dial: moonmoonserver.moonmoondomain.com:8201]
            - match: [host: [freshrss.moonmoondomain.com]]
              handle:
              - handler: subroute
                routes:
                - match: [remote_ip: {ranges: ["192.168.0.0/16", "172.16.0.0/12", "10.0.0.0/8", "127.0.0.1/8", "fd00::/8", "::1"]}] # private
                  handle:
                  - handler: reverse_proxy
                    transport: {protocol: http}
                    upstreams: [dial: moonmoonserver.moonmoondomain.com:8099]
        - handle: # unhandled anything
          - handler: static_response
            abort: true
  layer4:
    servers:
      authentik-ldap-proxy:
        listen: [fd/6]
        routes:
        - handle:
          - handler: proxy
            upstreams: [dial: [authentik:3389]]
      forgejo-ssh-proxy:
        listen: [fd/7]
        routes:
        - match: [ssh: {}]
          handle:
          - handler: proxy
            upstreams: [dial: [moonmoonserver.moonmoondomain.com:3001]]
      

5. Links to relevant resources:

Well, this line of the code in ech.go might explain it: :laughing:

// TODO: we could make resolvers configurable

So, we will need to do this I guess! Would you be interested in submitting a PR?