How to reverse proxy an express.js server?

1. Caddy version (caddy version):

Not sure which exact version, but I know it’s Caddy 1.x running inside a docker container (if someone can tell me how to get the version from the container I’ll go get it)

2. How I run Caddy:

a. System environment:

Ubuntu 14.04, running inside a Docker container

b. Command:

docker start caddy

c. Service/unit/compose file:

Using the abiosoft/caddy image (old version as I haven’t updated in a long time)

d. My complete Caddyfile or JSON config:

blog.rooday.com {
        gzip
        proxy / 172.17.0.2:2368 {
                transparent
        }
}

smmry.rooday.com {
        gzip
        proxy / 127.0.0.1:8080 {
                transparent
        }
}

3. The problem I’m having:

So I have caddy 1.x running inside a docker container, and I have a ghost blog running in a docker container (the blog subdomain). So far they have been working together just fine. I recently added a new project to this server, a node.js/express.js server that’s running under PM2, exposed on port 8080. I want to reverse proxy smmry.rooday.com to this express.js server, but it keeps failing the letsencrypt step to setup SSL and I’m not sure what else I need to do. Here’s my server code:

const express = require("express");
const bodyParser = require("body-parser");
const favicon = require("serve-favicon");
const path = require("path");
const app = express();
const axios = require("axios");
const qs = require("qs");

app.use(bodyParser.json());
app.use(express.static(path.join(__dirname, "build")));
app.use(favicon(path.join(__dirname, "build", "favicon.ico")));

app.post("/smmry", (req, res) => {
  if (req.body.url) {
    axios
      .get(
        `https://api.smmry.com/&SM_API_KEY=${req.body.apiKey}&SM_LENGTH=${req.body.lines}&SM_WITH_BREAK&SM_URL=${req.body.url}`
      )
      .then((smmry) => res.json(smmry.data));
  } else {
    axios
      .post(
        `https://api.smmry.com/&SM_API_KEY=${req.body.apiKey}&SM_LENGTH=${req.body.lines}&SM_WITH_BREAK`,
        qs.stringify({
          sm_api_input: req.body.text,
        }),
        {
          headers: {
            Expect: "100-continue",
            "Content-Type": "application/x-www-form-urlencoded",
          },
        }
      )
      .then((smmry) => res.json(smmry.data));
  }
});

app.get("/", (req, res) => {
  res.sendFile(path.join(__dirname, "build", "index.html"));
});

const port = process.env.PORT || 8080;
app.listen(port, () => {
  console.log(`Listening on port ${port}`);
});

4. Error messages and/or full log output:

Activating privacy features... 2020/11/14 23:48:10 [INFO][smmry.rooday.com] acme: Obtaining bundled SAN certificate
2020/11/14 23:48:10 [smmry.rooday.com] failed to obtain certificate: acme: Error 429 - urn:ietf:params:acme:error:rateLimited - Error creating new order :: too many failed authorizations recently: see https://letsencrypt.org/docs/rate-limits/
exit status 1
Activating privacy features... 2020/11/14 23:48:14 [INFO][smmry.rooday.com] acme: Obtaining bundled SAN certificate
2020/11/14 23:48:14 [smmry.rooday.com] failed to obtain certificate: acme: Error 429 - urn:ietf:params:acme:error:rateLimited - Error creating new order :: too many failed authorizations recently: see https://letsencrypt.org/docs/rate-limits/
exit status 1
Activating privacy features... 2020/11/14 23:48:21 [INFO][smmry.rooday.com] acme: Obtaining bundled SAN certificate
2020/11/14 23:48:21 [smmry.rooday.com] failed to obtain certificate: acme: Error 429 - urn:ietf:params:acme:error:rateLimited - Error creating new order :: too many failed authorizations recently: see https://letsencrypt.org/docs/rate-limits/
exit status 1
Activating privacy features... 2020/11/14 23:48:34 [INFO][smmry.rooday.com] acme: Obtaining bundled SAN certificate
2020/11/14 23:48:34 [smmry.rooday.com] failed to obtain certificate: acme: Error 429 - urn:ietf:params:acme:error:rateLimited - Error creating new order :: too many failed authorizations recently: see https://letsencrypt.org/docs/rate-limits/
exit status 1

5. What I already tried:

At first I thought the issue was that I ran caddy before my DNS for smmry.rooday.com (cloudflare) finished propagating, so I waited a few hours and came back. When I do nslookup I can confirm that the DNS is cloudflare (it’s the same IP for blog.rooday.com), so the domain is definitely live. I feel like the issue has to do with caddy being inside docker and my server being run as a system process with PM2 (this would explain why the blog works since it’s also under docker). Does anyone know how to get caddy to work in this scenario?

You can get your caddy version by running:

docker exec <your caddy container's name> caddy --version

The error message in your logs says it all:

2020/11/14 23:48:34 [smmry.rooday.com] failed to obtain certificate: acme: Error 429 - urn:ietf:params:acme:error:rateLimited - Error creating new order :: too many failed authorizations recently: see https://letsencrypt.org/docs/rate-limits/

Caddy is telling you that it failed to request a certificate because it is being rate limited by the Lets Encrypt API. This usually happens to me when I have been working on setting up a new domain, and the first couple attempts to setup the domain / subdomain don’t work. The entire time I’m setting up the domain, and it’s not working, Caddy is sitting there requesting a new certificate every few seconds. So eventually you’re bound to get rate limited.

Unfortunately all you can do here is wait until they decide to un rate limit you. It shouldn’t be more than a day. However in order for them to stop rate limiting you, you must stop Caddy from continuously requesting certificates. So disable SSL for that subdomain in the meantime.

In the future you can avoid this happening by setting Caddy to use the staging Lets Encrypt environment when setting new subdomains up in the future.

1 Like

Yeah it’s hitting the rate limit because something in the acme challenge is failing, although I’m not sure how to fix that (although you saying this does point out to me that I forgot to include that part in the logs).

Here’s the output of the command for version:

➜  ~ docker exec caddy caddy --version
Caddy 0.11.0 (unofficial)

And here’s more of the log that shows the real issue (my b that I forgot this initially).

Activating privacy features... 2020/11/16 00:34:55 [INFO][smmry.rooday.com] acme: Obtaining bundled SAN certificate
2020/11/16 00:34:55 [INFO][smmry.rooday.com] AuthURL: https://acme-v02.api.letsencrypt.org/acme/authz-v3/8633286051
2020/11/16 00:34:55 [INFO][smmry.rooday.com] acme: Trying to solve HTTP-01
2020/11/16 00:35:00 [smmry.rooday.com] failed to get certificate: acme: Error 403 - urn:ietf:params:acme:error:unauthorized - Invalid response from https://smmry.rooday.com/.well-known/acme-challenge/oOqjlZL3yD8SnzsW9J9S4-5db1rLpwXutO-c1uEcTzw [2606:4700:3032::ac43:b446]: "<!DOCTYPE html>\n<!--[if lt IE 7]> <html class=\"no-js ie6 oldie\" lang=\"en-US\"> <![endif]-->\n<!--[if IE 7]>    <html class=\"no-js "
exit status 1
Activating privacy features... 2020/11/16 00:35:01 [INFO][smmry.rooday.com] acme: Obtaining bundled SAN certificate
2020/11/16 00:35:03 [INFO][smmry.rooday.com] AuthURL: https://acme-v02.api.letsencrypt.org/acme/authz-v3/8633288096
2020/11/16 00:35:03 [INFO][smmry.rooday.com] acme: Trying to solve HTTP-01
2020/11/16 00:35:08 [smmry.rooday.com] failed to get certificate: acme: Error 403 - urn:ietf:params:acme:error:unauthorized - Invalid response from https://smmry.rooday.com/.well-known/acme-challenge/pdta_h5MGhQtvZIPijqENmSgFHM9gdpG8L-8gI_pwW8 [2606:4700:3032::ac43:b446]: "<!DOCTYPE html>\n<!--[if lt IE 7]> <html class=\"no-js ie6 oldie\" lang=\"en-US\"> <![endif]-->\n<!--[if IE 7]>    <html class=\"no-js "
exit status 1
Activating privacy features... 2020/11/16 00:35:09 [INFO][smmry.rooday.com] acme: Obtaining bundled SAN certificate
2020/11/16 00:35:10 [INFO][smmry.rooday.com] AuthURL: https://acme-v02.api.letsencrypt.org/acme/authz-v3/8633291969
2020/11/16 00:35:10 [INFO][smmry.rooday.com] acme: Trying to solve HTTP-01
2020/11/16 00:35:15 [smmry.rooday.com] failed to get certificate: acme: Error 403 - urn:ietf:params:acme:error:unauthorized - Invalid response from https://smmry.rooday.com/.well-known/acme-challenge/mZLS_-DytdU6Lw67n8DRq4JqYnMQiEPhYQIAUV8Ghy8 [2606:4700:3036::6818:68ac]: "<!DOCTYPE html>\n<!--[if lt IE 7]> <html class=\"no-js ie6 oldie\" lang=\"en-US\"> <![endif]-->\n<!--[if IE 7]>    <html class=\"no-js "
exit status 1
Activating privacy features... 2020/11/16 00:35:16 [INFO][smmry.rooday.com] acme: Obtaining bundled SAN certificate
2020/11/16 00:35:16 [INFO][smmry.rooday.com] AuthURL: https://acme-v02.api.letsencrypt.org/acme/authz-v3/8633294213
2020/11/16 00:35:16 [INFO][smmry.rooday.com] acme: Trying to solve HTTP-01
2020/11/16 00:35:21 [smmry.rooday.com] failed to get certificate: acme: Error 403 - urn:ietf:params:acme:error:unauthorized - Invalid response from https://smmry.rooday.com/.well-known/acme-challenge/opreWjibhkB4pisqb8GX-8O8YI7Z3Gvlw4UUiiayFkc [2606:4700:3036::6818:68ac]: "<!DOCTYPE html>\n<!--[if lt IE 7]> <html class=\"no-js ie6 oldie\" lang=\"en-US\"> <![endif]-->\n<!--[if IE 7]>    <html class=\"no-js "
exit status 1
Activating privacy features... 2020/11/16 00:35:23 [INFO][smmry.rooday.com] acme: Obtaining bundled SAN certificate
2020/11/16 00:35:23 [INFO][smmry.rooday.com] AuthURL: https://acme-v02.api.letsencrypt.org/acme/authz-v3/8633296171
2020/11/16 00:35:23 [INFO][smmry.rooday.com] acme: Trying to solve HTTP-01
2020/11/16 00:35:28 [smmry.rooday.com] failed to get certificate: acme: Error 403 - urn:ietf:params:acme:error:unauthorized - Invalid response from https://smmry.rooday.com/.well-known/acme-challenge/jGfyovRr7fsyz6eqI8nYmOuSZ9tY8VAqPvxHmJ83hXM [2606:4700:3032::ac43:b446]: "<!DOCTYPE html>\n<!--[if lt IE 7]> <html class=\"no-js ie6 oldie\" lang=\"en-US\"> <![endif]-->\n<!--[if IE 7]>    <html class=\"no-js "
exit status 1
Activating privacy features... 2020/11/16 00:35:30 [INFO][smmry.rooday.com] acme: Obtaining bundled SAN certificate
2020/11/16 00:35:31 [smmry.rooday.com] failed to obtain certificate: acme: Error 429 - urn:ietf:params:acme:error:rateLimited - Error creating new order :: too many failed authorizations recently: see https://letsencrypt.org/docs/rate-limits/
exit status 1

Basically it’s saying invalid response from https://smmry.rooday.com/.well-known/acme-challenge/jGfyovRr7fsyz6eqI8nYmOuSZ9tY8VAqPvxHmJ83hXM, which is an issue I’ve never ran into before when reverse proxying apps inside docker containers. Is this something where express.js needs to be told to serve that route, or is it a firewall, docker network issue since caddy is in a container and this app isn’t?

Caddy v1 is now EOL, please upgrade to Caddy v2.

You can find the official docker image here: Docker Hub

Please read through the instructions there (don’t forget to use volumes as described).

Caddy v2 has significantly improved renewal logic to avoid hitting rate limits (as long as the storage is properly persisted).

You can find the upgrade guide here:

This topic was automatically closed after 30 days. New replies are no longer allowed.