Requests to http .well-known/acme-challenge respond with 308 redirect to https (instead of the challenge response)

1. Output of caddy version:

$ caddy version
v2.5.2 h1:eCJdLyEyAGzuQTa5Mh3gETnYWDClo1LjtQm2q9RNZrs=

2. How I run Caddy:

Dockerized inside of AWS ECS Fargate containers

RUN go install github.com/caddyserver/caddy/v2/cmd/caddy@v2.5.2

WORKDIR /root/project/

COPY Caddyfile /root/project/Caddyfile
CMD caddy run
TaskDefinition:
  Type: AWS::ECS::TaskDefinition
  Properties:
    RequiresCompatibilities:
      - FARGATE
    ContainerDefinitions:
      - Name: caddy-proxy
        PortMappings:
          - ContainerPort: 8080
          - ContainerPort: 8081

CaddyProxyTargetGroupHTTP:
  Type: AWS::ElasticLoadBalancingV2::TargetGroup
  Properties:
    TargetType: ip
    Port: 8081

Listener80:
  Type: AWS::ElasticLoadBalancingV2::Listener
  Properties:
    Port: 80
    Protocol: TCP
    DefaultActions:
      - Type: forward
        Order: 1
        TargetGroupArn: !Ref CaddyProxyTargetGroupHTTP
    LoadBalancerArn: !Ref CaddyProxyLoadBalancer

Service:
  Type: AWS::ECS::Service
  Properties:
    LoadBalancers:
      - TargetGroupArn: !Ref CaddyProxyTargetGroup
        ContainerName: caddy-proxy
        ContainerPort: 8080
      - TargetGroupArn: !Ref CaddyProxyTargetGroupHTTP
        ContainerName: caddy-proxy
        ContainerPort: 8081

d. My complete Caddy config:

{
	debug
	log
	acme_ca https://acme-staging-v02.api.letsencrypt.org/directory

	http_port 8081
	https_port 8080
}

tcp/https://localhost:8080,
tcp/https://localhost:8081 {
	log
	tls internal

	handle {
		respond 200 {
			body `200_CADDYPROXY_OK`
		}
	}
}

tcp/https://textio.tech:8080 {
	log
	tls {
		issuer acme {
			dir https://acme-staging-v02.api.letsencrypt.org/directory
		}
	}

	handle {
		respond 200 {
			body `200_CADDYPROXY_OK`
		}
	}
}

3. The problem I’m having:

$ curl http://textio.tech/.well-known/acme-challenge/somerandomstringhere -i

HTTP/1.1 308 Permanent Redirect
Connection: close
Location: https://textio.tech/.well-known/acme-challenge/somerandomstringhere
Server: Caddy
Date: Wed, 31 Aug 2022 19:38:00 GMT
Content-Length: 0

4. caddy log output:

From oldest (top) to newest (bottom)

{
  "logger":"tls.issuance.acme.acme_client",
  "msg":"trying to solve challenge",
  "identifier":"textio.tech",
  "challenge_type":"http-01",
  "ca":"https://acme-staging-v02.api.letsencrypt.org/directory"
}
{
  "logger":"tls.issuance.acme.acme_client",
  "msg":"challenge accepted",
  "identifier":"textio.tech",
  "challenge_type":"http-01"
}
{
  "logger":"tls.issuance.acme.acme_client",
  "msg":"validating authorization",
  "identifier":"textio.tech",
  "problem":{
  "type":"urn:ietf:params:acme:error:connection",
  "title":"",
  "detail":"34.223.128.227: Fetching http://textio.tech/.well-known/acme-challenge/somerandomstringhere: Timeout during connect (likely firewall problem)",
  "instance":"",
  "subproblems":[]},"order":"https://acme-staging-v02.api.letsencrypt.org/acme/order/66614213/3865304433",
  "attempt":1,"max_attempts":3}
{
  "logger":"tls.issuance.acme.acme_client",
  "msg":"challenge failed",
  "identifier":"textio.tech",
  "challenge_type":"http-01",
  "problem":{
  "type":"urn:ietf:params:acme:error:connection",
  "title":"",
  "detail":"34.223.128.227: Fetching http://textio.tech/.well-known/acme-challenge/somerandomstringhere: Timeout during connect (likely firewall problem)",
  "instance":"",
  "subproblems":[]}}
{
  "logger":"tls.obtain",
  "msg":"could not get certificate from issuer",
  "identifier":"textio.tech",
  "issuer":"acme-staging-v02.api.letsencrypt.org-directory",
  "error":"HTTP 400 urn:ietf:params:acme:error:tls - 34.223.128.227: Fetching https://textio.tech/.well-known/acme-challenge/somerandomstringhere: remote error: tls: internal error"}

5. What I think is happening

The docs say

[The http challenge] challenge is enabled by default and does not require explicit configuration.

But I get the impression that there is some configuration required here. How do I get caddy to stop 308’ing me and actually respond to the http challenge? Does anyone have a fully working example of a Caddyfile that does a http challenge?

6. Links to relevant resources:

My goal is to implement the “http challenge” as described here: Automatic HTTPS — Caddy Documentation

Update! It looks like the server did manage to provision a cert eventually, which this config? Perhaps my config was correct and I was just stuck behind a rate limit window?

I’m going to investigate more

That means there’s a firewall or network configuration problem – make sure ports 80 and 443 go to your Caddy instance at ports 8081 and 8080.

(Caddy won’t try to solve challenges it doesn’t have challenge info for. So “somerandomstringhere” will of course not be handled like you expect. Otherwise, please don’t redact info from your logs, according to our forum rules. As you can see we can’t effectively help you if you do.)

I believe what’s happening is a combination of 1. wait time when requesting a new cert and 2. wait time between rate limit resets on the lets encrypt staging server

Here’s a set of logs for a newly deployed server that hasn’t yet managed to get a cert

{
    "level": "debug",
    "ts": 1661991668.0266001,
    "logger": "tls.handshake",
    "msg": "no certificate matching TLS ClientHello",
    "server_name": "textio.tech",
    "remote": "10.20.12.199:28007",
    "identifier": "textio.tech",
    "cipher_suites": [
        49195,
        49199,
        49196,
        49200,
        52393,
        52392,
        49161,
        49171,
        49162,
        49172,
        156,
        157,
        47,
        53,
        49170,
        10,
        4865,
        4866,
        4867
    ],
    "cert_cache_fill": 0.0001,
    "load_if_necessary": true,
    "obtain_if_necessary": true,
    "on_demand": false
}
{
    "level": "debug",
    "ts": 1661991668.0266817,
    "logger": "http.stdlib",
    "msg": "http: TLS handshake error from 10.20.12.199:28007: no certificate available for 'textio.tech'"
}
{
    "level": "debug",
    "ts": 1661991668.0285478,
    "logger": "tls.issuance.acme.acme_client",
    "msg": "http request",
    "method": "POST",
    "url": "https://acme-staging-v02.api.letsencrypt.org/acme/chall-v3/3497799373/EeSzoQ",
    "headers": {
        "Content-Type": [
            "application/jose+json"
        ],
        "User-Agent": [
            "Caddy/2.5.2 CertMagic acmez (linux; amd64)"
        ]
    },
    "response_headers": {
        "Boulder-Requester": [
            "66725913"
        ],
        "Cache-Control": [
            "public, max-age=0, no-cache"
        ],
        "Content-Length": [
            "193"
        ],
        "Content-Type": [
            "application/json"
        ],
        "Date": [
            "Thu, 01 Sep 2022 00:21:08 GMT"
        ],
        "Link": [
            "<https://acme-staging-v02.api.letsencrypt.org/directory>;rel=\"index\"",
            "<https://acme-staging-v02.api.letsencrypt.org/acme/authz-v3/3497799373>;rel=\"up\""
        ],
        "Location": [
            "https://acme-staging-v02.api.letsencrypt.org/acme/chall-v3/3497799373/EeSzoQ"
        ],
        "Replay-Nonce": [
            "0002rk_Z9yR_2eEaW4nXhROGsWfVRk3mm4YxNWt7fmuqlao"
        ],
        "Server": [
            "nginx"
        ],
        "Strict-Transport-Security": [
            "max-age=604800"
        ],
        "X-Frame-Options": [
            "DENY"
        ]
    },
    "status_code": 200
}
{
    "level": "debug",
    "ts": 1661991668.0286267,
    "logger": "tls.issuance.acme.acme_client",
    "msg": "challenge accepted",
    "identifier": "textio.tech",
    "challenge_type": "http-01"
}
{
    "level": "debug",
    "ts": 1661991668.32594,
    "logger": "tls.issuance.acme.acme_client",
    "msg": "http request",
    "method": "POST",
    "url": "https://acme-staging-v02.api.letsencrypt.org/acme/authz-v3/3497799373",
    "headers": {
        "Content-Type": [
            "application/jose+json"
        ],
        "User-Agent": [
            "Caddy/2.5.2 CertMagic acmez (linux; amd64)"
        ]
    },
    "response_headers": {
        "Boulder-Requester": [
            "66725913"
        ],
        "Cache-Control": [
            "public, max-age=0, no-cache"
        ],
        "Content-Length": [
            "1387"
        ],
        "Content-Type": [
            "application/json"
        ],
        "Date": [
            "Thu, 01 Sep 2022 00:21:08 GMT"
        ],
        "Link": [
            "<https://acme-staging-v02.api.letsencrypt.org/directory>;rel=\"index\""
        ],
        "Replay-Nonce": [
            "0002FwU8Vi5-iOtK515QoFJOKd3-vqCRok543F0dVeQv4-4"
        ],
        "Server": [
            "nginx"
        ],
        "Strict-Transport-Security": [
            "max-age=604800"
        ],
        "X-Frame-Options": [
            "DENY"
        ]
    },
    "status_code": 200
}
{
    "level": "error",
    "ts": 1661991668.3261902,
    "logger": "tls.issuance.acme.acme_client",
    "msg": "challenge failed",
    "identifier": "textio.tech",
    "challenge_type": "http-01",
    "problem": {
        "type": "urn:ietf:params:acme:error:tls",
        "title": "",
        "detail": "52.11.190.80: Fetching https://textio.tech/.well-known/acme-challenge/fDm2npLVx1iOkz9nLw0A8M1u0VbMr1WFAVNf7kTuXJY: remote error: tls: internal error",
        "instance": "",
        "subproblems": []
    }
}
{
    "level": "error",
    "ts": 1661991668.3262205,
    "logger": "tls.issuance.acme.acme_client",
    "msg": "validating authorization",
    "identifier": "textio.tech",
    "problem": {
        "type": "urn:ietf:params:acme:error:tls",
        "title": "",
        "detail": "52.11.190.80: Fetching https://textio.tech/.well-known/acme-challenge/fDm2npLVx1iOkz9nLw0A8M1u0VbMr1WFAVNf7kTuXJY: remote error: tls: internal error",
        "instance": "",
        "subproblems": []
    },
    "order": "https://acme-staging-v02.api.letsencrypt.org/acme/order/66725913/3874150223",
    "attempt": 2,
    "max_attempts": 3
}
{
    "level": "error",
    "ts": 1661991668.32625,
    "logger": "tls.obtain",
    "msg": "could not get certificate from issuer",
    "identifier": "textio.tech",
    "issuer": "acme-staging-v02.api.letsencrypt.org-directory",
    "error": "HTTP 400 urn:ietf:params:acme:error:tls - 52.11.190.80: Fetching https://textio.tech/.well-known/acme-challenge/fDm2npLVx1iOkz9nLw0A8M1u0VbMr1WFAVNf7kTuXJY: remote error: tls: internal error"
}
{
    "level": "error",
    "ts": 1661991668.3262691,
    "logger": "tls.obtain",
    "msg": "will retry",
    "error": "[textio.tech] Obtain: [textio.tech] solving challenge: textio.tech: [textio.tech] authorization failed: HTTP 400 urn:ietf:params:acme:error:tls - 52.11.190.80: Fetching https://textio.tech/.well-known/acme-challenge/fDm2npLVx1iOkz9nLw0A8M1u0VbMr1WFAVNf7kTuXJY: remote error: tls: internal error (ca=https://acme-staging-v02.api.letsencrypt.org/directory)",
    "attempt": 4,
    "retrying_in": 300,
    "elapsed": 309.112320974,
    "max_duration": 2592000
}

Here’s the logs from the moment where the cert becomes valid

{
    "level": "debug",
    "ts": 1661992571.1044464,
    "logger": "tls.issuance.acme.acme_client",
    "msg": "http request",
    "method": "POST",
    "url": "https://acme-staging-v02.api.letsencrypt.org/acme/chall-v3/3497930103/b17TZA",
    "headers": {
        "Content-Type": [
            "application/jose+json"
        ],
        "User-Agent": [
            "Caddy/2.5.2 CertMagic acmez (linux; amd64)"
        ]
    },
    "response_headers": {
        "Boulder-Requester": [
            "66725913"
        ],
        "Cache-Control": [
            "public, max-age=0, no-cache"
        ],
        "Content-Length": [
            "197"
        ],
        "Content-Type": [
            "application/json"
        ],
        "Date": [
            "Thu, 01 Sep 2022 00:36:11 GMT"
        ],
        "Link": [
            "<https://acme-staging-v02.api.letsencrypt.org/directory>;rel=\"index\"",
            "<https://acme-staging-v02.api.letsencrypt.org/acme/authz-v3/3497930103>;rel=\"up\""
        ],
        "Location": [
            "https://acme-staging-v02.api.letsencrypt.org/acme/chall-v3/3497930103/b17TZA"
        ],
        "Replay-Nonce": [
            "00014bbAYQW7k5wx8tL9xMk_il_Wjf0SK1qE8wKClA-zuvQ"
        ],
        "Server": [
            "nginx"
        ],
        "Strict-Transport-Security": [
            "max-age=604800"
        ],
        "X-Frame-Options": [
            "DENY"
        ]
    },
    "status_code": 200
}
{
    "level": "debug",
    "ts": 1661992571.1045372,
    "logger": "tls.issuance.acme.acme_client",
    "msg": "challenge accepted",
    "identifier": "textio.tech",
    "challenge_type": "tls-alpn-01"
}
{
    "level": "info",
    "ts": 1661992571.2511332,
    "logger": "tls",
    "msg": "served key authentication certificate",
    "server_name": "textio.tech",
    "challenge": "tls-alpn-01",
    "remote": "10.20.12.199:61534",
    "distributed": false
}
{
    "level": "info",
    "ts": 1661992571.2586117,
    "logger": "tls",
    "msg": "served key authentication certificate",
    "server_name": "textio.tech",
    "challenge": "tls-alpn-01",
    "remote": "10.20.12.199:41239",
    "distributed": false
}
{
    "level": "info",
    "ts": 1661992571.384922,
    "logger": "tls",
    "msg": "served key authentication certificate",
    "server_name": "textio.tech",
    "challenge": "tls-alpn-01",
    "remote": "10.20.12.199:47006",
    "distributed": false
}
{
    "level": "debug",
    "ts": 1661992571.3988109,
    "logger": "tls.issuance.acme.acme_client",
    "msg": "http request",
    "method": "POST",
    "url": "https://acme-staging-v02.api.letsencrypt.org/acme/authz-v3/3497930103",
    "headers": {
        "Content-Type": [
            "application/jose+json"
        ],
        "User-Agent": [
            "Caddy/2.5.2 CertMagic acmez (linux; amd64)"
        ]
    },
    "response_headers": {
        "Boulder-Requester": [
            "66725913"
        ],
        "Cache-Control": [
            "public, max-age=0, no-cache"
        ],
        "Content-Length": [
            "813"
        ],
        "Content-Type": [
            "application/json"
        ],
        "Date": [
            "Thu, 01 Sep 2022 00:36:11 GMT"
        ],
        "Link": [
            "<https://acme-staging-v02.api.letsencrypt.org/directory>;rel=\"index\""
        ],
        "Replay-Nonce": [
            "0001N0LP1eNwnIEDps0fAIXHisffVZ7msjN53b1rKAavxGc"
        ],
        "Server": [
            "nginx"
        ],
        "Strict-Transport-Security": [
            "max-age=604800"
        ],
        "X-Frame-Options": [
            "DENY"
        ]
    },
    "status_code": 200
}
{
    "level": "debug",
    "ts": 1661992571.695166,
    "logger": "tls.issuance.acme.acme_client",
    "msg": "http request",
    "method": "POST",
    "url": "https://acme-staging-v02.api.letsencrypt.org/acme/authz-v3/3497930103",
    "headers": {
        "Content-Type": [
            "application/jose+json"
        ],
        "User-Agent": [
            "Caddy/2.5.2 CertMagic acmez (linux; amd64)"
        ]
    },
    "response_headers": {
        "Boulder-Requester": [
            "66725913"
        ],
        "Cache-Control": [
            "public, max-age=0, no-cache"
        ],
        "Content-Length": [
            "675"
        ],
        "Content-Type": [
            "application/json"
        ],
        "Date": [
            "Thu, 01 Sep 2022 00:36:11 GMT"
        ],
        "Link": [
            "<https://acme-staging-v02.api.letsencrypt.org/directory>;rel=\"index\""
        ],
        "Replay-Nonce": [
            "0002WqvkjEq0dKoSUHtsOzCnhfyeF2FZJLLwKgNflNDCgGE"
        ],
        "Server": [
            "nginx"
        ],
        "Strict-Transport-Security": [
            "max-age=604800"
        ],
        "X-Frame-Options": [
            "DENY"
        ]
    },
    "status_code": 200
}
{
    "level": "info",
    "ts": 1661992571.6953752,
    "logger": "tls.issuance.acme.acme_client",
    "msg": "validations succeeded; finalizing order",
    "order": "https://acme-staging-v02.api.letsencrypt.org/acme/order/66725913/3874307333"
}
{
    "level": "debug",
    "ts": 1661992572.0417929,
    "logger": "tls.issuance.acme.acme_client",
    "msg": "http request",
    "method": "POST",
    "url": "https://acme-staging-v02.api.letsencrypt.org/acme/finalize/66725913/3874307333",
    "headers": {
        "Content-Type": [
            "application/jose+json"
        ],
        "User-Agent": [
            "Caddy/2.5.2 CertMagic acmez (linux; amd64)"
        ]
    },
    "response_headers": {
        "Boulder-Requester": [
            "66725913"
        ],
        "Cache-Control": [
            "public, max-age=0, no-cache"
        ],
        "Content-Length": [
            "457"
        ],
        "Content-Type": [
            "application/json"
        ],
        "Date": [
            "Thu, 01 Sep 2022 00:36:12 GMT"
        ],
        "Link": [
            "<https://acme-staging-v02.api.letsencrypt.org/directory>;rel=\"index\""
        ],
        "Location": [
            "https://acme-staging-v02.api.letsencrypt.org/acme/order/66725913/3874307333"
        ],
        "Replay-Nonce": [
            "0001O09aKhUugOaJBLZJ6pdtuxQV9ic_ucfVjcjoqYPJAK4"
        ],
        "Server": [
            "nginx"
        ],
        "Strict-Transport-Security": [
            "max-age=604800"
        ],
        "X-Frame-Options": [
            "DENY"
        ]
    },
    "status_code": 200
}
{
    "level": "debug",
    "ts": 1661992572.0891848,
    "logger": "tls.issuance.acme.acme_client",
    "msg": "http request",
    "method": "POST",
    "url": "https://acme-staging-v02.api.letsencrypt.org/acme/cert/fa3a9c1c2ce0ff2796088639c625fc403957",
    "headers": {
        "Content-Type": [
            "application/jose+json"
        ],
        "User-Agent": [
            "Caddy/2.5.2 CertMagic acmez (linux; amd64)"
        ]
    },
    "response_headers": {
        "Cache-Control": [
            "public, max-age=0, no-cache"
        ],
        "Content-Length": [
            "4171"
        ],
        "Content-Type": [
            "application/pem-certificate-chain"
        ],
        "Date": [
            "Thu, 01 Sep 2022 00:36:12 GMT"
        ],
        "Link": [
            "<https://acme-staging-v02.api.letsencrypt.org/directory>;rel=\"index\""
        ],
        "Replay-Nonce": [
            "0002sJPOyLlxOyz7qSnOPoGmsZx3GJx2DCn2bxG2Kn7vj2I"
        ],
        "Server": [
            "nginx"
        ],
        "Strict-Transport-Security": [
            "max-age=604800"
        ],
        "X-Frame-Options": [
            "DENY"
        ]
    },
    "status_code": 200
}
{
    "level": "info",
    "ts": 1661992572.0892928,
    "logger": "tls.issuance.acme.acme_client",
    "msg": "successfully downloaded available certificate chains",
    "count": 1,
    "first_url": "https://acme-staging-v02.api.letsencrypt.org/acme/cert/fa3a9c1c2ce0ff2796088639c625fc403957"
}
{
    "level": "info",
    "ts": 1661992572.08982,
    "logger": "tls.obtain",
    "msg": "certificate obtained successfully",
    "identifier": "textio.tech"
}
{
    "level": "info",
    "ts": 1661992572.089838,
    "logger": "tls.obtain",
    "msg": "releasing lock",
    "identifier": "textio.tech"
}

So the root cause of my problem was that my certs were still “pending” and the docs didn’t provide me any guidance about what I should look at to check the status of the certs. I was making requests to random URLs in the logs trying to infer what was going on, but ultimately what I would have liked to see was a log line that shows the cert’s provisioning status in a simple way, like so (example)

{
    "level": "error",
    "msg": "cert is pending, will retry later. 20 retries over 2 hours left in retry window",
    "error": "[textio.tech] Obtain: [textio.tech] solving challenge: textio.tech: [textio.tech] authorization failed: HTTP 400 urn:ietf:params:acme:error:tls - 52.11.190.80: Fetching https://textio.tech/.well-known/acme-challenge/fDm2npLVx1iOkz9nLw0A8M1u0VbMr1WFAVNf7kTuXJY: remote error: tls: internal error (ca=https://acme-staging-v02.api.letsencrypt.org/directory)",
    "attempt": 4,
    "retrying_in": 300,
    "elapsed": 309.112320974,
    "max_duration": 2592000
}

I assume the underlying 400 error from api.letsencrypt.org above does actually contain some of this detail.

On a simpler note, it would also be helpful if the HTTP Challenge docs included

  1. a mention that SSL requests will generally fail for the first X minutes (~30 minutes, IME) while a cert is pending
  2. a mention of what the log output should look like when the cert is pending
  3. some guidance on how to inspect the status of the provisioning process to confirm that the cert is pending
  4. a full example of a Caddyfile using lets encrypt staging

Hey Kai, glad you figured it out, but I’m still not really sure what’s going on here. You seem to understand, which is great, but I still have questions.

52.11.190.80: Fetching https://textio.tech/.well-known/acme-challenge/fDm2npLVx1iOkz9nLw0A8M1u0VbMr1WFAVNf7kTuXJY: remote error: tls: internal error

This error indicates that the CA couldn’t connect to Caddy. Again, this is evidence of a network, DNS, firewall, or other system misconfiguration.

I don’t think that’s the root cause… why were they still “pending”? Are you talking about the order status?

and the docs didn’t provide me any guidance about what I should look at to check the status of the certs.

(To clarify, this has nothing to do with Caddy, so it’s not in our docs. But you can learn about how the ACME protocol works if you wish to investigate. I also recommend https://letsdebug.net/ along the way of troubleshooting!)

I was making requests to random URLs in the logs trying to infer what was going on

Sometimes that’ll work, but most ACME endpoints require GET-as-POST requests where you have to submit some sort of credential to see the information. Just FYI.

Yes, you’re right! And in fact we do already log all the details that the CA gives us, notice:

    "problem": {
        "type": "urn:ietf:params:acme:error:tls",
        "title": "",
        "detail": "52.11.190.80: Fetching https://textio.tech/.well-known/acme-challenge/fDm2npLVx1iOkz9nLw0A8M1u0VbMr1WFAVNf7kTuXJY: remote error: tls: internal error",
        "instance": "",
        "subproblems": []
    },

I think our current logs are pretty similar already; you are looking for the order status to be printed though?

This topic was automatically closed after 30 days. New replies are no longer allowed.