Trying to understand why Let’s Encrypt rate limited me

Help
1

Hi,

I’m running Caddy 0.11 in a Docker container on an Amazon EC2 instance. Everything was working great until I made a few tweaks to my site, pushing and pulling the Docker image and restarting the container several times within a few minutes. At which point I hit the rate limit error below from Let’s Encrypt:

registration error: acme: error: 429 :: POST :: https://acme-v02.api.letsencrypt.org/acme/new-acct :: urn:ietf:params:acme:error:rateLimited :: Error creating new account :: too many registrations for this IP: see Rate Limits - Let's Encrypt

I’m running the Docker container with the Let’s Encrypt files mounted in a volume using this command: docker run -d -p 80:80 -p 443:443 -v /home/ec2-user/dotcaddy:/.caddy:rw --name caddy --ulimit nofile=8192:8192 --restart always myrepo/mywebsite.com

This is what my Dockerfile looks like:

FROM scratch
COPY rootfs /
COPY Caddyfile /var/www/html/
COPY public/ /var/www/html/public/
EXPOSE 443 2015
WORKDIR /var/www/html
ENTRYPOINT ["/bin/caddy", "-agree”]

rootfs just contains the Caddy executable in /bin and the SSL root certificates in /etc/ssl/certs/ca-certificates.crt

This is my Caddyfile:

www.mywebsite.com {
    redir https://mywebsite.com{uri}
}

mywebsite.com {
    errors {
        404 404.html
    }
    
    expires {
        match .css$ 7d
        match .js$ 7d
        match .png$ 7d
        match .svg$ 7d
    }

    ext .html .htm
    gzip
    root /var/www/html/public
    tls myemail@mywebsite.com

    header / {
        -Server
        Strict-Transport-Security "max-age=31536000;"
        X-XSS-Protection "1; mode=block"
        X-Content-Type-Options "nosniff"
        X-Frame-Options "DENY"
    }
}

I don’t understand why Caddy is trying to register my site with Let’s Encrypt every time I restart the Docker container. How can I avoid this error in future?

Thanks,

John

2

What is actually contained within your rootfs? Obviously you must have the caddy binary in there because it’s not being added from anywhere else but what else is included?

If I were you I would check if caddy has stored the certs inside /var/www/html/.caddy. I checked the documentation and came to these conclusions:

  • You are not setting $CADDYPATH anywhere so caddy won’t use a path based on that.
  • In that case caddy will try to use $HOME/.caddy. The only way I see this generating the path /.caddy is if your rootfs contains a user (I guess it has to be root because you aren’t using USER) whose home directory is /. I’m gonna say this isn’t true because it should work if it were.
  • So I think the options are either that you are defining a home directory for the root user (other than /) or caddy will fallback to the final behaviour which is creating .caddy in the current working directory (which in your case will be /var/www/html due to the WORKDIR line).

In any case I would suggest you configure $CADDYPATH to explicitly tell caddy where you want the certs to be. That always feels the most clear to me anyway.

EDIT: To test a new config to fix your issue you could consider using the -ca option to use the staging endpoint:

caddy -ca https://acme-staging-v02.api.letsencrypt.org/directory ...

You can also define this in the Caddyfile which might suit you better. This gets you non-trusted certs but worth using to test new changes & while you are rate-limited for production certs anyway.

1 Like
3

You could be experiencing this bug, which has been fixed on master: https://github.com/mholt/caddy/issues/2400

4

Thanks I’ll take a look. I’ll have to temporarily change my Docker image to explore the filesystem because as I’m using the scratch base image it doesn’t have a shell!

5

Thanks Matt, it seems likely it’s that bug because I’m seeing exactly the same behaviour.

6

OK, it looks like there are three things going on here. My Docker run command had a couple of errors. I was publishing the container port as port 80, but Caddy was listening on the default port 2015. I was also mounting my dotcaddy volume in /.caddy rather than root’s home directory, as Joel pointed out. So the corrected command looks like this:

docker run -d -p 80:2015 -p 443:443 -v /home/ec2-user/dotcaddy:/root/.caddy:rw --name caddy --ulimit nofile=8192:8192 --restart always myrepo/mywebsite.com

The third issue is that I’m hitting the bug Matt mentioned, because I can see from the logs that Caddy is requesting certificates from Let’s Encrypt every time the Docker container is started, rather than using its local copies mounted in the container.

Thanks for your help guys.

1 Like
7

@johntopley If you could, please try building Caddy from https://github.com/mholt/caddy/pull/2452 and help verify that this works for your use cases.

8

@matt Happy to. How do I compile in the http.expires plugin?

9

Just add a single line to run.go: https://github.com/mholt/caddy/wiki/Plugging-in-Plugins-Yourself

10

Thanks Matt. I should hopefully get time to try this tonight and report back.

11

I’m not sure what behaviour you’re expecting, but I started my Docker container using my Caddy binary built from the sni-fixes branch as requested. Then I stopped the container and restarted it. Both times the logs indicate that the server validated the request and that the server responded with a certificate. I was expecting the locally stored certificate to be used the second time.

12

Can you please post the full log output here? (no redactions, please)

13

Sure.

$ docker logs caddy
Activating privacy features... 

Your sites will be served over HTTPS automatically using Let's Encrypt.
By continuing, you agree to the Let's Encrypt Subscriber Agreement at:
  https://letsencrypt.org/documents/LE-SA-v1.2-November-15-2017.pdf
Please enter your email address to signify agreement and to be notified
in case of issues. You can leave it blank, but we don't recommend it.
  Email address: 2019/02/04 18:04:35 [INFO] [www.johntopley.com] acme: Obtaining bundled SAN certificate
2019/02/04 18:04:36 [INFO] [www.johntopley.com] AuthURL: https://acme-v02.api.letsencrypt.org/acme/authz/8iXlD_77ScOX94iQphcm3DVCxGHKvrKNs8fzAPZYFzw
2019/02/04 18:04:36 [INFO] [www.johntopley.com] acme: use tls-alpn-01 solver
2019/02/04 18:04:36 [INFO] [www.johntopley.com] acme: Trying to solve TLS-ALPN-01
2019/02/04 18:04:41 [INFO] [www.johntopley.com] The server validated our request
2019/02/04 18:04:41 [INFO] [www.johntopley.com] acme: Validations succeeded; requesting certificates
2019/02/04 18:04:43 [INFO] [www.johntopley.com] Server responded with a certificate.


Your sites will be served over HTTPS automatically using Let's Encrypt.
By continuing, you agree to the Let's Encrypt Subscriber Agreement at:
  https://letsencrypt.org/documents/LE-SA-v1.2-November-15-2017.pdf
Please enter your email address to signify agreement and to be notified
in case of issues. You can leave it blank, but we don't recommend it.
  Email address: 2019/02/04 18:04:44 [INFO] [mail.johntopley.com] acme: Obtaining bundled SAN certificate
2019/02/04 18:04:45 [INFO] [mail.johntopley.com] AuthURL: https://acme-v02.api.letsencrypt.org/acme/authz/_GM5PwhkmkrCqpfzgvViIu-QaC0WDchxnow7kVSJU1E
2019/02/04 18:04:45 [INFO] [mail.johntopley.com] acme: use tls-alpn-01 solver
2019/02/04 18:04:45 [INFO] [mail.johntopley.com] acme: Trying to solve TLS-ALPN-01
2019/02/04 18:04:51 [INFO] [mail.johntopley.com] The server validated our request
2019/02/04 18:04:51 [INFO] [mail.johntopley.com] acme: Validations succeeded; requesting certificates
2019/02/04 18:04:52 [INFO] [mail.johntopley.com] Server responded with a certificate.
2019/02/04 18:04:52 [INFO] acme: Registering account for xxxx@johntopley.com
2019/02/04 18:04:53 [INFO] [johntopley.com] acme: Obtaining bundled SAN certificate
2019/02/04 18:04:53 [INFO] [johntopley.com] AuthURL: https://acme-v02.api.letsencrypt.org/acme/authz/PBlBGphCn_etxlvcdz8J2UYZb53OGFhHZeaLDoN3P-M
2019/02/04 18:04:53 [INFO] [johntopley.com] acme: use tls-alpn-01 solver
2019/02/04 18:04:53 [INFO] [johntopley.com] acme: Trying to solve TLS-ALPN-01
2019/02/04 18:04:59 [INFO] [johntopley.com] The server validated our request
2019/02/04 18:04:59 [INFO] [johntopley.com] acme: Validations succeeded; requesting certificates
2019/02/04 18:05:01 [INFO] [johntopley.com] Server responded with a certificate.
done.
https://www.johntopley.com
https://mail.johntopley.com
https://johntopley.com
http://www.johntopley.com
http://mail.johntopley.com
http://johntopley.com

$ docker stop caddy && docker rm caddy
caddy
caddy
$ docker run -d -p 80:2015 -p 443:443 -v /home/ec2-user/dotcaddy:/root/.caddy:rw --name caddy --ulimit nofile=8192:8192 --restart always johntopley/johntopley.com:sni-fixes

$ docker logs caddy
Activating privacy features... 

Your sites will be served over HTTPS automatically using Let's Encrypt.
By continuing, you agree to the Let's Encrypt Subscriber Agreement at:
  https://letsencrypt.org/documents/LE-SA-v1.2-November-15-2017.pdf
Please enter your email address to signify agreement and to be notified
in case of issues. You can leave it blank, but we don't recommend it.
  Email address: 2019/02/04 19:29:00 [INFO] [www.johntopley.com] acme: Obtaining bundled SAN certificate
2019/02/04 19:29:00 [INFO] [www.johntopley.com] AuthURL: https://acme-v02.api.letsencrypt.org/acme/authz/q026fAyGHnOYgx3Whmvbd3Do8WeSL27v_Yh0kYZQJLY
2019/02/04 19:29:00 [INFO] [www.johntopley.com] acme: use tls-alpn-01 solver
2019/02/04 19:29:00 [INFO] [www.johntopley.com] acme: Trying to solve TLS-ALPN-01
2019/02/04 19:29:07 [INFO] [www.johntopley.com] The server validated our request
2019/02/04 19:29:07 [INFO] [www.johntopley.com] acme: Validations succeeded; requesting certificates
2019/02/04 19:29:08 [INFO] [www.johntopley.com] Server responded with a certificate.


Your sites will be served over HTTPS automatically using Let's Encrypt.
By continuing, you agree to the Let's Encrypt Subscriber Agreement at:
  https://letsencrypt.org/documents/LE-SA-v1.2-November-15-2017.pdf
Please enter your email address to signify agreement and to be notified
in case of issues. You can leave it blank, but we don't recommend it.
  Email address: 2019/02/04 19:29:09 [INFO] [mail.johntopley.com] acme: Obtaining bundled SAN certificate
2019/02/04 19:29:10 [INFO] [mail.johntopley.com] AuthURL: https://acme-v02.api.letsencrypt.org/acme/authz/C-xBrH6cq_oMVLOfKTNNUAO2skPsoQlXBiyytY__XPo
2019/02/04 19:29:10 [INFO] [mail.johntopley.com] acme: use tls-alpn-01 solver
2019/02/04 19:29:10 [INFO] [mail.johntopley.com] acme: Trying to solve TLS-ALPN-01
2019/02/04 19:29:15 [INFO] [mail.johntopley.com] The server validated our request
2019/02/04 19:29:15 [INFO] [mail.johntopley.com] acme: Validations succeeded; requesting certificates
2019/02/04 19:29:17 [INFO] [mail.johntopley.com] Server responded with a certificate.
2019/02/04 19:29:17 [INFO] acme: Registering account for xxxx@johntopley.com
2019/02/04 19:29:18 [INFO] [johntopley.com] acme: Obtaining bundled SAN certificate
2019/02/04 19:29:18 [INFO] [johntopley.com] AuthURL: https://acme-v02.api.letsencrypt.org/acme/authz/y7K7bVZmiDhJ49fuuOHzdWA72x7Af2ncMbneAThKtp0
2019/02/04 19:29:18 [INFO] [johntopley.com] acme: use tls-alpn-01 solver
2019/02/04 19:29:18 [INFO] [johntopley.com] acme: Trying to solve TLS-ALPN-01
2019/02/04 19:29:24 [INFO] [johntopley.com] The server validated our request
2019/02/04 19:29:24 [INFO] [johntopley.com] acme: Validations succeeded; requesting certificates
2019/02/04 19:29:26 [INFO] [johntopley.com] Server responded with a certificate.
done.
https://www.johntopley.com
https://mail.johntopley.com
https://johntopley.com
http://www.johntopley.com
http://mail.johntopley.com
http://johntopley.com
14

If you could verify that, after using docker rm and then when the new container starts up, the certificates still exist in $CADDYPATH ($HOME/.caddy by default), that would be good. Right now it looks like your certificates are getting destroyed when you delete the container.

I would also be sure to follow @joelnb’s advice above to use the staging endpoint before you rate limit yourself again.

15

It looks like they are bind mounting to /root/.caddy just fine.

That said, I’ve just gotten around to building this myself on Ubuntu v18.04 running Docker v18.09.1 with Compose v1.23.2.

I built Caddy v0.11.2 +73d3146 with the following Dockerfile:

Dockerfile 
FROM golang:alpine

RUN apk add --no-cache git
RUN go get \
  github.com/abiosoft/caddy-git \
  github.com/caddyserver/builds \
  github.com/mholt/caddy

WORKDIR /go/src/github.com/mholt/caddy
RUN git checkout sni-fixes

RUN sed '/This is where other plugins get plugged in (imported)/ a 	_ "github.com/abiosoft/caddy-git"' -i /go/src/github.com/mholt/caddy/caddy/caddymain/run.go

WORKDIR /go/src/github.com/mholt/caddy/caddy
RUN go run build.go

FROM alpine:3.8

RUN apk add --no-cache ca-certificates

COPY --from=0 /go/src/github.com/mholt/caddy/caddy/caddy /usr/bin/caddy
RUN /usr/bin/caddy -version
RUN /usr/bin/caddy -plugins

WORKDIR /srv
RUN echo "Caddy is running!" > index.txt

ENTRYPOINT ["/usr/bin/caddy"]
CMD ["-conf", "/etc/Caddyfile", "-log", "stdout", "-agree"]

Ran the container with the following Compose file:

docker-compose.yml 
version: '3'
services:

  caddy:
    image: caddy-sni-fixes
    command: ["-log", "stdout", "-agree",
      "-email", "letsencrypt@whitestrake.net",
      "-conf", "/etc/Caddyfile"]
    ports:
      - 80:80
      - 443:443
    volumes:
      - ./caddy/Caddyfile:/etc/Caddyfile
      - ./caddy/certificates:/root/.caddy
      - ./caddy/srv:/srv

I saw the following output on first run (dc is just an alias for docker-compose):

whitestrake in ~ at kairos is 🐳 v18.09.1 took 12s 485ms
➜ dc up caddy
Recreating whitestrake_caddy_1 ... done
Attaching to whitestrake_caddy_1
caddy_1  | Activating privacy features... 2019/02/05 02:13:06 [INFO][FileStorage:/root/.caddy] Started certificate maintenance routine
caddy_1  | 2019/02/05 02:13:07 [INFO] acme: Registering account for letsencrypt@whitestrake.net
caddy_1  | 2019/02/05 02:13:07 [INFO] [kairos.whitestrake.net] acme: Obtaining bundled SAN certificate
caddy_1  | 2019/02/05 02:13:08 [INFO] [kairos.whitestrake.net] AuthURL: https://acme-v02.api.letsencrypt.org/acme/authz/sYLkPP_NQIgUI4JPsDxZIbFPEJw_CHj7MvJVIs0WQT8
caddy_1  | 2019/02/05 02:13:08 [INFO] [kairos.whitestrake.net] acme: use tls-alpn-01 solver
caddy_1  | 2019/02/05 02:13:08 [INFO] [kairos.whitestrake.net] acme: Trying to solve TLS-ALPN-01
caddy_1  | 2019/02/05 02:13:14 [INFO] [kairos.whitestrake.net] The server validated our request
caddy_1  | 2019/02/05 02:13:14 [INFO] [kairos.whitestrake.net] acme: Validations succeeded; requesting certificates
caddy_1  | 2019/02/05 02:13:16 [INFO] [kairos.whitestrake.net] Server responded with a certificate.
caddy_1  | done.
caddy_1  | https://kairos.whitestrake.net
caddy_1  | 2019/02/05 02:13:16 https://kairos.whitestrake.net
caddy_1  | http://kairos.whitestrake.net
caddy_1  | 2019/02/05 02:13:16 http://kairos.whitestrake.net
^CGracefully stopping... (press Ctrl+C again to force)
Stopping whitestrake_caddy_1   ... done

And the following output on the second run:

whitestrake in ~ at kairos is 🐳 v18.09.1
➜ dc up caddy
Creating whitestrake_caddy_1 ... done
Attaching to whitestrake_caddy_1
caddy_1  | Activating privacy features... done.
caddy_1  | https://kairos.whitestrake.net
caddy_1  | 2019/02/05 02:24:33 https://kairos.whitestrake.net
caddy_1  | http://kairos.whitestrake.net
caddy_1  | 2019/02/05 02:24:33 http://kairos.whitestrake.net
caddy_1  | 2019/02/05 02:24:33 [INFO][FileStorage:/root/.caddy] Started certificate maintenance routine
caddy_1  | 2019/02/05 02:24:35 [NOTICE] Sending telemetry: we were too early; waiting 51m0.630262303s before trying again

In conclusion: Looks like this build is fully functional.

1 Like
16

I changed my Docker base image from scratch to golang-alpine, rebuilt the image and redeployed the container so I could get a shell for the purposes of inspecting the filesystem. I can confirm that my /home/ec2-user/dotcaddy volume is correctly mounted at /root/.caddy within the running container. The certificates under acme/acme-v02.api.letsencrypt.org/sites are dated 26th January, so nothing’s getting deleted.

Interestingly I got different log output when running this golang-alpine version:

Activating privacy features... 

Your sites will be served over HTTPS automatically using Let's Encrypt.
By continuing, you agree to the Let's Encrypt Subscriber Agreement at:
  https://letsencrypt.org/documents/LE-SA-v1.2-November-15-2017.pdf
Please enter your email address to signify agreement and to be notified
in case of issues. You can leave it blank, but we don't recommend it.
  Email address: 

Your sites will be served over HTTPS automatically using Let's Encrypt.
By continuing, you agree to the Let's Encrypt Subscriber Agreement at:
  https://letsencrypt.org/documents/LE-SA-v1.2-November-15-2017.pdf
Please enter your email address to signify agreement and to be notified
in case of issues. You can leave it blank, but we don't recommend it.
  Email address: done.
https://www.johntopley.com
https://mail.johntopley.com
https://johntopley.com
http://www.johntopley.com
http://mail.johntopley.com
http://johntopley.com

My understanding of the above output is that it’s using the cached certificates rather than requesting new ones. I don’t understand why changing the Docker base image would affect this.

17

That’s the correct inference. I couldn’t tell you why scratch didn’t want to play nicely, though. Was Caddy running as root inside the scratch version?

P.S. You can use the -agree and -email youremail@example.com flags to avoid the interactive prompt for your email on a fresh startup.

18

I know this is an old thread now, but for the sake of completeness what I’ve discovered is that when using scratch as the base Docker image the bind mount wasn’t working and my certificates weren’t getting cached. At least not on Amazon Linux. Changing the base image to alpine fixed that.

1 Like