Caddy not responding to .well-known/acme-challenge

delfick · February 21, 2021, 12:09pm

1. Caddy version (`caddy version`):

I’ve tried with

v2.3.0 h1:fnrqJLa3G5vfxcxmOH/+kJOcunPLhSBnjgIvjXV/QTA=

and

v2.4.0-beta.1 h1:Ed/tIaN3p6z8M3pEiXWJL/T8JmCqV62FrSJCHKquW/I=

2. How I run Caddy:

Using brew services with this plist

<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
<plist version="1.0">
  <dict>
    <key>KeepAlive</key>
    <true/>
    <key>Label</key>
    <string>homebrew.mxcl.caddy</string>
    <key>ProgramArguments</key>
    <array>
      <string>/usr/local/opt/caddy/bin/caddy</string>
      <string>run</string>
    </array>
    <key>RunAtLoad</key>
    <true/>
    <key>StandardOutPath</key>
    <string>/usr/local/var/log/caddy.log</string>
    <key>StandardErrorPath</key>
    <string>/usr/local/var/log/caddy.log</string>
  </dict>
</plist>

brew services start caddy

a. System environment:

Mac OSX 11.2.1, 2019 Macbook Pro

d. My complete Caddyfile or JSON config:

curl http://localhost:2019/config/

{
  "apps": {
    "http": {
      "servers": {
        "cloud": {
          "listen": [
            ":443",
            ":80"
          ],
          "routes": [
            {
              "handle": [
                {
                  "handler": "acme_server"
                },
                {
                  "handler": "reverse_proxy",
                  "transport": {
                    "protocol": "http"
                  },
                  "upstreams": [
                    {
                      "dial": ":1337"
                    }
                  ]
                }
              ],
              "match": [
                {
                  "host": [
                    "thing.local.lifx.co"
                  ]
                }
              ]
            }
          ]
        }
      }
    },
    "tls": {
      "automation": {
        "policies": [
          {
            "issuers": [
              {
                "ca": "https://step-ca.local.lifx.co:666/acme/acme/directory",
                "module": "acme"
              }
            ],
            "subjects": [
              "thing.local.lifx.co"
            ]
          }
        ]
      }
    }
  }
}

For the purposes of this test, I’m running python -m http.server 1337

3. The problem I’m having:

When I try something like

step ca certificate "thing.local.lifx.co" server.crt server.key --provisioner acme

It tries to do a GET on http://thing.local.lifx.co/.well-known/acme-challenge/QX2K0JYcvPC22mimNh0MqWkDScoXoSFq and Caddy passes it onto the reverse_proxy

As far as I can tell, Caddy should be catching this before my reverse_proxy and returning information that is then given to step-ca.

4. Error messages and/or full log output:

Caddy will repeat this line

{
  "level": "error",
  "ts": 1613909043.799864,
  "logger": "tls.issuance.acme",
  "msg": "looking up info for HTTP challenge",
  "host": "thing.local.lifx.co",
  "error": "no information found to solve challenge for identifier: thing.local.lifx.co"
}

And step will say

time="2021-02-21T23:04:03+11:00"
level=info
duration=4.640762ms
duration-ns=4640762
fields.time="2021-02-21T23:04:03+11:00"
method=POST
name=ca
nonce=ZEFGQnNnMU9DUUJYczZCU29jWTZqdzRMN1ZWSlRGNUY
path=/acme/acme/challenge/hHdanv3Ag6hOgIUPVKwmx9WptnYphdc6
protocol=HTTP/2.0
referer=
remote-address=127.0.0.1
request-id=c0p4ocucie6ggdsqtq1g
response="{\"type\":\"http-01\",\"status\":\"pending\",\"token\":\"QX2K0JYcvPC22mimNh0MqWkDScoXoSFq\",\"url\":\"https://step-ca.local.lifx.co:666/acme/acme/challenge/hHdanv3Ag6hOgIUPVKwmx9WptnYphdc6\",\"error\":{\"type\":\"urn:ietf:params:acme:error:connection\",\"detail\":\"error doing http GET for url http://thing.local.lifx.co/.well-known/acme-challenge/QX2K0JYcvPC22mimNh0MqWkDScoXoSFq with status code 404\"}}"
size=381
status=200 
user-agent=Go-http-client/2.0
user-id=

And the server will say

::ffff:127.0.0.1 - - [21/Feb/2021 23:04:03] code 404, message File not found
::ffff:127.0.0.1 - - [21/Feb/2021 23:04:03] "GET /.well-known/acme-challenge/QX2K0JYcvPC22mimNh0MqWkDScoXoSFq HTTP/1.1" 404 -

5. What I already tried:

I spent a couple hours desperately googling .well-known and acme-challenge and related things and afaict it should “just work” and yet caddy doesn’t seem to recognise this path.

If I tell the reverse_proxy to ignore this path, then caddy will just return an empty reply and I get similar problems.

francislavoie · February 21, 2021, 4:07pm

Hmm. I think if you remove:

Then it might work. Caddy will automatically set up an HTTP->HTTPS redirect for you as part of the Automatic HTTPS procedures, but I think having both ports being listened to on the same server is messing with that.

delfick · February 21, 2021, 8:59pm

Nup, after removing that, Caddy still doesn’t catch those requests before the reverse_proxy and so step gets a 308 instead of a 404.

Also, my understanding of this stuff is low, but I believe it happens over port 80 because this is the process it uses to be able to get the certificates necessary to do https.

matt · February 21, 2021, 9:20pm

Hmm, I imagine the acme_server handler should catch the ACME HTTP challenge? Maybe the Smallstep team would know. I’ll point them here and see if we can get their thoughts.

FWIW, Caddy passes the challenge request thru if it doesn’t have an outstanding challenge for that name which it initiated.

delfick · February 21, 2021, 9:42pm

Thanks

I think the acme_server is a different thing, cause that handles everything after /acme

I feel like maybe there’s something I’m missing here? I’m pretty sure it’s caddy that should have that challenge, rather than my http server.

francislavoie · February 21, 2021, 10:36pm

I’m confused as to what you’re trying to do in the first place. You’re running a step CA which you’re pointing Caddy to, and also running an ACME server in Caddy (which is also step under the hood if you weren’t aware). What are you trying to solve here?

matt · February 21, 2021, 11:03pm

I realize now my previous comment doesn’t make much sense, sorry about that. That’s what I get for helping with troubleshooting at church.

Now that I’m home…

You are right, my bad.

github.com

caddyserver/caddy/blob/5376e5113e25d85725e1f64d06a288700a50bbaa/modules/caddypki/acmeserver/acmeserver.go#L154


      
          	r := chi.NewRouter()
          	r.Route(ash.PathPrefix, func(r chi.Router) {
          		acmeRouterHandler.Route(r)
          	})
          	ash.acmeEndpoints = r
          
          	return nil
          }
          
          func (ash Handler) ServeHTTP(w http.ResponseWriter, r *http.Request, next caddyhttp.Handler) error {
          	if strings.HasPrefix(r.URL.Path, ash.PathPrefix) {
          		ash.acmeEndpoints.ServeHTTP(w, r)
          		return nil
          	}
          	return next.ServeHTTP(w, r)
          }
          
          func (ash Handler) getDatabaseKey() string {
          	key := ash.CA
          	key = strings.ToLower(key)
          	key = strings.TrimSpace(key)

However… Caddy shouldn’t be getting any ACME challenge requests for other ACME clients’ orders. You’re running step ca certificate which should be initiating the ACME transaction, right? If so, it should be solving the challenge, not Caddy. Is Caddy supposed to be the ACME server or the ACME client here?

I’m not really sure what you’re trying to do, but I think something about your setup is wrong. What are you trying to do, exactly?

delfick · February 21, 2021, 11:14pm

I get this feeling what I’m trying to do is indeed wrong and probably should have started with that. My apologies.

I want to start my server as doing the https itself (the project I’m working on isn’t always behind Caddy and needs to do https itself). But on my computer where I have Caddy, I wanted caddy to reverse_proxy https requests to my https server.

So I used caddy to generate those certs like it does for everything. So curl https://thing.local.lifx.co and then I stole "~/Library/Application Support/Caddy/certificates/step-ca.local.lifx.co-666-acme-acme-directory/thing.local.lifx.co/thing.local.lifx.co.{crt,key}" and started my server with that.

but that’s kinda horrible so I wanted to use step-ca to generate those certs and then I can have this to reverse_proxy to my https server.

            {
              "handle": [
                {
                  "handler": "acme_server"
                },
                {
                  "handler": "reverse_proxy",
                  "transport": {
                    "protocol": "http",
                    "tls": {
                      "client_certificate_automate": "thing.local.lifx.co",
                      "server_name": "thing.local.lifx.co"
                    }
                  },
                  "upstreams": [
                    {
                      "dial": ":1337"
                    }
                  ]
                }
              ],
              "match": [
                {
                  "host": [
                    "thing.local.lifx.co"
                  ]
                }
              ]
            },

delfick · February 27, 2021, 8:22am

I had another go and realised I can run this instead to generate those certs

        step certificate create $name $name/$name.crt $name/$name.key \
            --ca $HOME/.step/certs/intermediate_ca.crt \
            --ca-key $HOME/.step/secrets/intermediate_ca_key \
            --no-password \
            --insecure \
            --san $name \
            --bundle

with an added --not-after to change how long it takes to expire

system · March 23, 2021, 12:10pm

This topic was automatically closed after 30 days. New replies are no longer allowed.