Letsencrypt rate limiting certs for a single device hosting multiple domains on various IPs

Help
1

1. Caddy version (caddy version):

v2.4.5 h1:P1mRs6V2cMcagSPn+NWpD+OEYUYLIf6ecOa48cFGeUg=

2. How I run Caddy:

a. System environment:

Fedora 34, docker:

FROM caddy:builder AS builder
RUN caddy-builder github.com/caddy-dns/cloudflare
FROM caddy
COPY --from=builder /usr/bin/caddy /usr/bin/caddy

b. Command:

docker-compose up

c. Service/unit/compose file:

  caddy:
    image: "cloudflare-caddy"
    restart: unless-stopped
    container_name: "caddy"
    network_mode: "host"
    environment:
      - CLOUDFLARE_API_KEY=my_secret_key
    volumes:
      - /home/foo/caddy/Caddyfile:/srv/Caddyfile:ro
    command: ["caddy", "run"]

d. My complete Caddyfile or JSON config:

(blacklists) {
        @public {
                remote_ip REDACTED_IP/24
        }
}

a1.zwimer.com {
        import blacklists
        tls zwimer@gmail.com {
                dns cloudflare {env.CLOUDFLARE_API_KEY}
        }
        handle @public {
                respond "Access denied" 403 {
                        close
                }
        }
        reverse_proxy localhost:8000
}

a2.zwimer.com {
        import blacklists
        tls zwimer@gmail.com {
                dns cloudflare {env.CLOUDFLARE_API_KEY}
        }
        handle @public {
                respond "Access denied" 403 {
                        close
                }
        }
        reverse_proxy localhost:8000
}

a3.zwimer.com {
        import blacklists
        tls zwimer@gmail.com {
                dns cloudflare {env.CLOUDFLARE_API_KEY}
        }
        handle @public {
                respond "Access denied" 403 {
                        close
                }
        }
        reverse_proxy localhost:8000
}

a4.zwimer.com {
        import blacklists
        tls zwimer@gmail.com {
                dns cloudflare {env.CLOUDFLARE_API_KEY}
        }
        handle @public {
                respond "Access denied" 403 {
                        close
                }
        }
        reverse_proxy localhost:8000
}

a5.zwimer.com {
        import blacklists
        tls zwimer@gmail.com {
                dns cloudflare {env.CLOUDFLARE_API_KEY}
        }
        handle @public {
                respond "Access denied" 403 {
                        close
                }
        }
        reverse_proxy localhost:8000
}

a6.zwimer.com {
        import blacklists
        tls zwimer@gmail.com {
                dns cloudflare {env.CLOUDFLARE_API_KEY}
        }
        handle @public {
                respond "Access denied" 403 {
                        close
                }
        }
        reverse_proxy localhost:8000
}

a7.zwimer.com {
        import blacklists
        tls zwimer@gmail.com {
                dns cloudflare {env.CLOUDFLARE_API_KEY}
        }
        handle @public {
                respond "Access denied" 403 {
                        close
                }
        }
        reverse_proxy localhost:8000
}

a8.zwimer.com {
        import blacklists
        tls zwimer@gmail.com {
                dns cloudflare {env.CLOUDFLARE_API_KEY}
        }
        handle @public {
                respond "Access denied" 403 {
                        close
                }
        }
        reverse_proxy localhost:8000
}

a9.zwimer.com {
        import blacklists
        tls zwimer@gmail.com {
                dns cloudflare {env.CLOUDFLARE_API_KEY}
        }
        handle @public {
                respond "Access denied" 403 {
                        close
                }
        }
        reverse_proxy localhost:8000
}

a10.zwimer.com {
        import blacklists
        tls zwimer@gmail.com {
                dns cloudflare {env.CLOUDFLARE_API_KEY}
        }
        handle @public {
                respond "Access denied" 403 {
                        close
                }
        }
        reverse_proxy localhost:8000
}

a11.zwimer.com {
        import blacklists
        tls zwimer@gmail.com {
                dns cloudflare {env.CLOUDFLARE_API_KEY}
        }
        handle @public {
                respond "Access denied" 403 {
                        close
                }
        }
        reverse_proxy localhost:8000
}

a12.zwimer.com {
        import blacklists
        tls zwimer@gmail.com {
                dns cloudflare {env.CLOUDFLARE_API_KEY}
        }
        handle @public {
                respond "Access denied" 403 {
                        close
                }
        }
        reverse_proxy localhost:8000
}

There are about 15 URLs (other than raw.zwimer.com) with exactly the same configuration but with a different ports on the reverse_proxy localhost:8000 line.

3. The problem I’m having:

Why the problem is happening is partly understood! The problem: Let’s Encrypt is rejecting these due to rate limiting, since I have more than 10 domains.

However, key information here is that cloudflare maps these various domains to different IP addresses. That is, a1.zwimer.com might map to 10.20.147.10 and a2.zwimer.com might map to 10.20.147.11, etc. No individual IP address has more than 4 domains associated with it; which is less than 10.

My entire building is natted, plus my router nat’s atop that so I am double natted; I cannot control my public IP. Instead I have put my server on a ZeroTier VPN; that is, zerotier has assigned my server the IPs that a1.zwimer.com etc map to (i.e. 10.20.147.10, 10.20.147.11, etc).

Given that each IP has only a few (sub)domains associated with it, my assumption is that caddy does ip a | grep addr | head -n 1 to get the IP address it submits to lets encrypt or something rather than nslookup a1.zwimer.com. Either that or it gets the public IP.

I’m not entirely sure what the problem is nor the solution from a technical standpoint; I’d like to avoid letsencrypt’s rate limit. These services are not all related, they simply happen to be put on the same device as I only have one server. Is the only solution to put them on multiple servers or is that also not a solution because the public IP would be the same for both?

4. Error messages and/or full log output:

caddy            | {"level":"info","ts":1643430894.4619663,"msg":"using adjacent Caddyfile"}
caddy            | {"level":"info","ts":1643430894.4659402,"logger":"admin","msg":"admin endpoint started","address":"tcp/localhost:2019","enforce_origin":false,"origins":["localhost:2019","[::1]:2019","127.0.0.1:2019"]}
caddy            | {"level":"info","ts":1643430894.4661772,"logger":"tls.cache.maintenance","msg":"started background certificate maintenance","cache":"0xc00043ec40"}
caddy            | {"level":"info","ts":1643430894.4661944,"logger":"http","msg":"server is listening only on the HTTPS port but has no TLS connection policies; adding one to enable TLS","server_name":"srv0","https_port":443}
caddy            | {"level":"info","ts":1643430894.466214,"logger":"http","msg":"enabling automatic HTTP->HTTPS redirects","server_name":"srv0"}
caddy            | {"level":"info","ts":1643430894.467093,"logger":"http","msg":"enabling automatic TLS certificate management","domains":["a1.zwimer.com","a2.zwimer.com","a3.zwimer.com","a4.zwimer.com","a5.zwimer.com","a6.zwimer.com","a7.zwimer.com","a8.zwimer.com","a9.zwimer.com","a10.zwimer.com","a11.zwimer.com","a12.zwimer.com"]}
caddy            | {"level":"info","ts":1643430894.4671152,"logger":"tls","msg":"cleaning storage unit","description":"FileStorage:/data/caddy"}
caddy            | {"level":"info","ts":1643430894.4671524,"logger":"tls","msg":"finished cleaning storage units"}
caddy            | {"level":"info","ts":1643430894.4677985,"logger":"tls.obtain","msg":"acquiring lock","identifier":"a1.rpoxy.zwimer.com"}
caddy            | {"level":"info","ts":1643430894.4677996,"logger":"tls.obtain","msg":"acquiring lock","identifier":"a2.rpoxy.zwimer.com"}
caddy            | {"level":"info","ts":1643430894.4678012,"logger":"tls.obtain","msg":"acquiring lock","identifier":"a3.rpoxy.zwimer.com"}
caddy            | {"level":"info","ts":1643430894.4678022,"logger":"tls.obtain","msg":"acquiring lock","identifier":"a4.rpoxy.zwimer.com"}
caddy            | {"level":"info","ts":1643430894.4678037,"logger":"tls.obtain","msg":"acquiring lock","identifier":"a5.rpoxy.zwimer.com"}
caddy            | {"level":"info","ts":1643430894.4678042,"logger":"tls.obtain","msg":"acquiring lock","identifier":"a6.rpoxy.zwimer.com"}
caddy            | {"level":"info","ts":1643430894.4678059,"logger":"tls.obtain","msg":"acquiring lock","identifier":"a7.rpoxy.zwimer.com"}
caddy            | {"level":"info","ts":1643430894.4678074,"logger":"tls.obtain","msg":"acquiring lock","identifier":"a8.rpoxy.zwimer.com"}
caddy            | {"level":"info","ts":1643430894.4678093,"logger":"tls.obtain","msg":"acquiring lock","identifier":"a9.rpoxy.zwimer.com"}
caddy            | {"level":"info","ts":1643430894.4678124,"logger":"tls.obtain","msg":"acquiring lock","identifier":"a10.rpoxy.zwimer.com"}
caddy            | {"level":"info","ts":1643430894.4678149,"logger":"tls.obtain","msg":"acquiring lock","identifier":"a11.rpoxy.zwimer.com"}
caddy            | {"level":"info","ts":1643430894.4678165,"logger":"tls.obtain","msg":"acquiring lock","identifier":"a12.rpoxy.zwimer.com"}
caddy            | {"level":"info","ts":1643430894.4686415,"msg":"autosaved config (load with --resume flag)","file":"/config/caddy/autosave.json"}
caddy            | {"level":"info","ts":1643430894.4686506,"msg":"serving initial configuration"}
caddy            | {"level":"info","ts":1643430894.4713676,"logger":"tls.obtain","msg":"lock acquired","identifier":"a1.zwimer.com"}
caddy            | {"level":"error","ts":1643430894.717279,"logger":"tls.obtain","msg":"could not get certificate from issuer","identifier":"a1.zwimer.com","issuer":"acme-v02.api.letsencrypt.org-directory","error":"HTTP 429 urn:ietf:params:acme
:error:rateLimited - Error creating new account :: too many registrations for this IP: see https://letsencrypt.org/docs/rate-limits/"}

5. What I already tried:

  1. I read through caddy’s docs on reverse proxies
  2. I’ve tried changing the VPN IPs (and dns records) to see if that made a difference (it did not)
  3. I read through lets encrypt’s docs about rate limiting and how to avoid it
  4. 2 hours worth of googling to see if this is something others have encountered.
2

I’m not seeing where you’re persisting Caddy’s storage here. You should have a volume for /data, otherwise every time your remove the containers and spin it up again, it’ll have to get fresh certs. That means you’ll hit rate limits pretty fast, depending on how often you restart the containers.

Please review the docs on Docker.

Caddy doesn’t tell the ACME CAs the IP address, it just tells them the domain it wants to issue a cert for. Then the ACME CA will resolve the DNS using public DNS servers.

3

This is a very good point! Thank you, I missed that; I’ll be fixing that now! :slight_smile:

Unfortunately, it does not answer my confusion. a1.zwimer.com spat back this error right after being given a fresh IP which means at least that domain is not having its cert generated with the IP that DNS gives but rather either its public IP or a different IP on the machine. :frowning:

4

This did however allow me to get past the rate limiting by waiting out the time though. Thanks again! :grinning:

1 Like
5

This topic was automatically closed after 30 days. New replies are no longer allowed.