AWS CloudFront + TLS-ALPN/HTTP Challenges

Help
1

In follow-up to my previous thread, I’ve made changes to my deployment that allow TLS-ALPN-01 and HTTP-01 (in lieu of DNS-01 which takes too long).

When accessing sites directly (without CloudFront), ALPN works fantastic, and a cert is issued typically in < 5s.

However, since ALPN is not supported with CloudFront, I also enabled HTTP-01, but I can’t seem to quite get it to work–the request from ACME seems to go all the way to the backend service, rather than having Caddy respond directly.

Note that the CF distribution has HTTP/2 + HTTP/3 enabled, and is passing both 80+443 ports to Caddy.

Caddy Logs

May 19 08:32:42  2023/05/19 15:32:42.953        INFO        tls.on_demand        obtaining new certificate        {"remote_ip": "172.16.45.16", "remote_port": "59450", "server_name": "prod4.localdev.mywordpress.io"}
May 19 08:32:45  2023/05/19 15:32:45.245        INFO        tls.obtain        acquiring lock        {"identifier": "prod4.localdev.mywordpress.io"}
May 19 08:32:45  2023/05/19 15:32:45.359        INFO        tls.obtain        lock acquired        {"identifier": "prod4.localdev.mywordpress.io"}
May 19 08:32:45  2023/05/19 15:32:45.360        INFO        tls.obtain        obtaining certificate        {"identifier": "prod4.localdev.mywordpress.io"}
May 19 08:32:45  2023/05/19 15:32:45.364        INFO        http        waiting on internal rate limiter        {"identifiers": ["prod4.localdev.mywordpress.io"], "ca": "https://acme-v02.api.letsencrypt.org/directory", "account": "info@mywordpress.io"}
May 19 08:32:45  2023/05/19 15:32:45.364        INFO        http        done waiting on internal rate limiter        {"identifiers": ["prod4.localdev.mywordpress.io"], "ca": "https://acme-v02.api.letsencrypt.org/directory", "account": "info@mywordpress.io"}
May 19 08:32:45  2023/05/19 15:32:45.853        INFO        http.acme_client        trying to solve challenge        {"identifier": "prod4.localdev.mywordpress.io", "challenge_type": "http-01", "ca": "https://acme-v02.api.letsencrypt.org/directory"}
May 19 08:32:48  2023/05/19 15:32:48.382        INFO        layer4.handlers.proxy.health_checker.active        host is down        {"address": "172.16.46.54:25566", "timeout": 5, "error": "dial tcp 172.16.46.54:25566: connect: connection refused"}
May 19 08:32:48  2023/05/19 15:32:48.787        ERROR        http.acme_client        challenge failed        {"identifier": "prod4.localdev.mywordpress.io", "challenge_type": "http-01", "problem": {"type": "urn:ietf:params:acme:error:unauthorized", "title": "", "detail": "216.137.45.107: Invalid response from https://prod4.localdev.mywordpress.io/wp-admin/install.php: \"<!DOCTYPE html>\\n<html lang=\\\"en-US\\\" xml:lang=\\\"en-US\\\">\\n<head>\\n\\t<meta name=\\\"viewport\\\" content=\\\"width=device-width\\\" />\\n\\t<meta http-e\"", "instance": "", "subproblems": []}}
May 19 08:32:48  2023/05/19 15:32:48.787        ERROR        http.acme_client        validating authorization        {"identifier": "prod4.localdev.mywordpress.io", "problem": {"type": "urn:ietf:params:acme:error:unauthorized", "title": "", "detail": "216.137.45.107: Invalid response from https://prod4.localdev.mywordpress.io/wp-admin/install.php: \"<!DOCTYPE html>\\n<html lang=\\\"en-US\\\" xml:lang=\\\"en-US\\\">\\n<head>\\n\\t<meta name=\\\"viewport\\\" content=\\\"width=device-width\\\" />\\n\\t<meta http-e\"", "instance": "", "subproblems": []}, "order": "https://acme-v02.api.letsencrypt.org/acme/order/1117020427/183199142017", "attempt": 1, "max_attempts": 3}
May 19 08:32:50  2023/05/19 15:32:50.066        INFO        http.acme_client        trying to solve challenge        {"identifier": "prod4.localdev.mywordpress.io", "challenge_type": "tls-alpn-01", "ca": "https://acme-v02.api.letsencrypt.org/directory"}
May 19 08:32:50  2023/05/19 15:32:50.635        ERROR        http.acme_client        challenge failed        {"identifier": "prod4.localdev.mywordpress.io", "challenge_type": "tls-alpn-01", "problem": {"type": "urn:ietf:params:acme:error:unauthorized", "title": "", "detail": "Cannot negotiate ALPN protocol \"acme-tls/1\" for tls-alpn-01 challenge", "instance": "", "subproblems": []}}
May 19 08:32:50  2023/05/19 15:32:50.636        ERROR        http.acme_client        validating authorization        {"identifier": "prod4.localdev.mywordpress.io", "problem": {"type": "urn:ietf:params:acme:error:unauthorized", "title": "", "detail": "Cannot negotiate ALPN protocol \"acme-tls/1\" for tls-alpn-01 challenge", "instance": "", "subproblems": []}, "order": "https://acme-v02.api.letsencrypt.org/acme/order/1117020427/183199151007", "attempt": 2, "max_attempts": 3}
May 19 08:32:50  2023/05/19 15:32:50.636        ERROR        tls.obtain        could not get certificate from issuer        {"identifier": "prod4.localdev.mywordpress.io", "issuer": "acme-v02.api.letsencrypt.org-directory", "error": "HTTP 403 urn:ietf:params:acme:error:unauthorized - Cannot negotiate ALPN protocol \"acme-tls/1\" for tls-alpn-01 challenge"}
May 19 08:32:50  2023/05/19 15:32:50.636        ERROR        tls.obtain        will retry        {"error": "[prod4.localdev.mywordpress.io] Obtain: [prod4.localdev.mywordpress.io] solving challenge: prod4.localdev.mywordpress.io: [prod4.localdev.mywordpress.io] authorization failed: HTTP 403 urn:ietf:params:acme:error:unauthorized - Cannot negotiate ALPN protocol \"acme-tls/1\" for tls-alpn-01 challenge (ca=https://acme-v02.api.letsencrypt.org/directory)", "attempt": 1, "retrying_in": 60, "elapsed": 5.277105841, "max_duration": 2592000}

Caddy Config

{
  "admin": {
    "config": {
      "persist": false
    },
    "disabled": false,
    "enforce_origin": false,
    "listen": "127.0.0.1:8443",
    "origins": [
      "http://127.0.0.1:8443"
    ]
  },
  "logging": {
    "logs": {
      "default": {
        "encoder": {
          "format": "console"
        },
        "level": "info",
        "writer": {
          "output": "stdout"
        }
      }
    },
    "sink": {
      "writer": {
        "output": "stderr"
      }
    }
  },
  "storage": {
    "address": "https://vault:8200",
    "approle_login_path": "auth/approle/login",
    "approle_logout_path": "auth/token/revoke-self",
    "approle_role_id": "dead-beef",
    "approle_secret_id": "ea7-beef",
    "insecure_skip_verify": true,
    "module": "vault",
    "path_prefix": "production/caddy",
    "secrets_path": "klm/secrets"
  },
  "apps": {
    "tls": {
      "automation": {
        "on_demand": {
          "ask": "https://a.b.c.d/caddy/onDemand",
          "rate_limit": {
            "burst": 3,
            "interval": "1m"
          }
        },
        "policies": [
          {
            "issuers": [
              {
                "ca": "https://acme-v02.api.letsencrypt.org/directory",
                "challenges": {
                  "http": {
                    "disabled": false
                  },
                  "tls-alpn": {
                    "disabled": false
                  }
                },
                "email": "info@mywordpress.io",
                "module": "acme"
              }
            ],
            "on_demand": true
          }
        ]
      }
    },
    "http": {
      "grace_period": "30s",
      "http_port": 80,
      "https_port": 443,
      "servers": {
        "http": {
          "listen": [
            "0.0.0.0:80"
          ],
          "routes": [
            {
              "group": "default",
              "handle": [
                {
                  "body": "DEFAULT ROUTE",
                  "close": true,
                  "handler": "static_response",
                  "status_code": "200"
                }
              ]
            }
          ]
        },
        "https": {
          "listen": [
            "0.0.0.0:443"
          ],
          "metrics": {},
          "routes": [
            {
              "group": "default",
              "handle": [
                {
                  "dynamic_upstreams": {
                    "dial_timeout": "750ms",
                    "name": "service.tld",
                    "port": "443",
                    "refresh": "30s",
                    "source": "a"
                  },
                  "handler": "reverse_proxy",
                  "load_balancing": {
                    "selection_policy": {
                      "policy": "least_conn"
                    }
                  },
                  "transport": {
                    "protocol": "http",
                    "tls": {
                      "insecure_skip_verify": true
                    }
                  }
                }
              ]
            }
          ],
          "tls_connection_policies": [
            {}
          ]
        }
      }
    }
  }
}

I am not quite able to understand why the HTTP-01 challenge is not being served by Caddy when it’s requested through port 80. It seems like the HTTP-01 challenge is getting all the way to the backend service (which then says “hey, go here for installation”), even though I would expect Caddy to directly serve the HTTP-01 challenge response first, before it gets that far? Maybe I’m not understanding how that part of HTTP-01 works.

2

Could log at the “DEBUG” level (change “level” to “DEBUG” instead of info) and see if that yields any more info. Usually Caddy will log – and maybe it’s even at info-level, but definitely debug – when it receives HTTP challenge requests.

It’s quite possible that CF isn’t forwarding those or is changing them somehow. Not sure. I would definitely scrutinize the CloudFront setup to make sure ACME challenge requests are going to Caddy unaltered (or even at all). Check DNS as well.

3

After fumbling around with CloudFront for entirely too long, I came up with a solution that seems to work reliably.

AWS CloudFront

Verify you have the following enabled:

    - ViewerProtocolPolicy  = "allow-all"
    - OriginProtocolPolicy = "match-viewer"
    - DistributionHTTPProtocolVersions = "http2-and-http3"
    - DistributionSSLSupportMethod = "sni-only"
    - CachePolicyExclusionPath = "/.well-known/*"
    - OriginRequestPolicy
        - Include Headers
            - Origin
            - Accept-Charset
            - Accept
            - Access-Control-Request-Method
            - Access-Control-Request-Headers
            - Referer
            - Host
            - Accept-Language
            - Accept-Datetime
        - Include Query Strings: all
        - Include Cookies: all

Caddy

The config for Caddy is straightforward, this is my tls.automation.policy block (to support HTTP/3, make sure tcp+udp/443 are open to your Caddy instance):

          {
            "issuers": [
              {
                "ca": "https://acme-v02.api.letsencrypt.org/directory",
                "challenges": {
                  "http": {
                    "disabled": false
                  },
                  "tls-alpn": {
                    "disabled": false
                  }
                },
                "email": "somebody@example.org",
                "module": "acme"
              }
            ],
            "on_demand": true
          }

And, on my http listener, I added a route/match like this to redirect to https, if the URL does not contain /.well-known/...:

      "servers": {
        "http": {
          "listen": [
            "0.0.0.0:80"
          ],
          "routes": [
            {
              "group": "default",
              "handle": [
                {
                  "handler": "subroute",
                  "routes": [
                    {
                      "handle": [
                        {
                          "handler": "static_response",
                          "headers": {
                            "Location": [
                              "https://{http.request.host}{http.request.uri}"
                            ]
                          },
                          "status_code": 301
                        }
                      ],
                      "match": [
                        {
                          "not": [
                            {
                              "path": [
                                "/.well-known/*"
                              ]
                            }
                          ]
                        }
                      ]
                    }
                  ]
                },
                {
                  "body": "DEFAULT ROUTE",
                  "close": true,
                  "handler": "static_response",
                  "status_code": "200"
                }
              ]
            }
          ]
        },

Notes

  • In this setup, you will NOT have AWS CloudFront doing any http → https redirection because you need at least some requests from ACME to get through to the origin via http.
  • Caveat: You may be able to add a CachePolicy that would allow http or https for /.well-known/* only, but I decided not to do that since my I need my Caddy instance to generally redirect http → https for other use-cases anyways (but I have an exception in for /.well-known/*). If you did that in AWS CloudFront, that would allow you to only enable http + https access to the origin for a single URL /.well-known/*–YMMV, and I did not test that case since I did not need it.

Thank you for the help and pointers!!

1 Like
4

Would you be interested in making a wiki post out of this? Looks useful for others!

5

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.