Caddy seems to to try HTTP-01 using 127.0.0.1

Help
1

1. Caddy version (caddy version):

v2.1.1

2. How I run Caddy:

a. System environment:

OS: Ubuntu 18.04.4 LTS

Running with SystemD.

b. Command:

systemctl start caddy

(see below for the service file)

c. Service/unit/compose file:

# caddy.service
#
# For using Caddy with a config file.
#
# Make sure the ExecStart and ExecReload commands are correct
# for your installation.
#
# See https://caddyserver.com/docs/install for instructions.
#
# WARNING: This service does not use the --resume flag, so if you
# use the API to make changes, they will be overwritten by the
# Caddyfile next time the service is restarted. If you intend to
# use Caddy's API to configure it, add the --resume flag to the
# `caddy run` command or use the caddy-api.service file instead.

[Unit]
Description=Caddy
Documentation=https://caddyserver.com/docs/
After=network.target

[Service]
User=www-data
Group=www-data
ExecStart=/usr/local/bin/caddy run --environ --config /etc/caddy/Caddyfile
ExecReload=/usr/local/bin/caddy reload --config /etc/caddy/Caddyfile
TimeoutStopSec=5s
LimitNOFILE=1048576
LimitNPROC=512
PrivateTmp=true
ProtectSystem=full
AmbientCapabilities=CAP_NET_BIND_SERVICE

[Install]
WantedBy=multi-user.target

d. My complete Caddyfile or JSON config:

{
        email   [redacted]
        acme_ca https://acme-staging-v02.api.letsencrypt.org/directory
}

matrix.club-tech.fr, matrix.club-tech.fr:8448 {
        reverse_proxy /_matrix/* 127.0.0.1:8008

        header Access-Control-Allow-Origin *

        root * /srv/http/riot-v1.6.0
        file_server
}

3. The problem I’m having:

I’ve just updated to Caddy 2, and the HTTP-01 challenge seems to fail. Looking at the logs, it looks like it’s trying to use 127.0.0.1 as the address for it, which sounds bogus to me. It also seems to be trying and failing the ALPN-01 challenge but I don’t know if that’s related.

4. Error messages and/or full log output:

Jul 07 11:51:41 matrix-club-tech systemd[1]: Started Caddy.
Jul 07 11:51:41 matrix-club-tech caddy[30374]: caddy.HomeDir=/var/www
Jul 07 11:51:41 matrix-club-tech caddy[30374]: caddy.AppDataDir=/var/www/.local/share/caddy
Jul 07 11:51:41 matrix-club-tech caddy[30374]: caddy.AppConfigDir=/var/www/.config/caddy
Jul 07 11:51:41 matrix-club-tech caddy[30374]: caddy.ConfigAutosavePath=/var/www/.config/caddy/autosave.json
Jul 07 11:51:41 matrix-club-tech caddy[30374]: runtime.GOOS=linux
Jul 07 11:51:41 matrix-club-tech caddy[30374]: runtime.GOARCH=amd64
Jul 07 11:51:41 matrix-club-tech caddy[30374]: runtime.Compiler=gc
Jul 07 11:51:41 matrix-club-tech caddy[30374]: runtime.NumCPU=1
Jul 07 11:51:41 matrix-club-tech caddy[30374]: runtime.GOMAXPROCS=1
Jul 07 11:51:41 matrix-club-tech caddy[30374]: runtime.Version=go1.14.4
Jul 07 11:51:41 matrix-club-tech caddy[30374]: os.Getwd=/
Jul 07 11:51:41 matrix-club-tech caddy[30374]: LANG=C.UTF-8
Jul 07 11:51:41 matrix-club-tech caddy[30374]: PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/snap/bin
Jul 07 11:51:41 matrix-club-tech caddy[30374]: HOME=/var/www
Jul 07 11:51:41 matrix-club-tech caddy[30374]: LOGNAME=www-data
Jul 07 11:51:41 matrix-club-tech caddy[30374]: USER=www-data
Jul 07 11:51:41 matrix-club-tech caddy[30374]: INVOCATION_ID=cda4586cdb774ab18aeaef16ba14597f
Jul 07 11:51:41 matrix-club-tech caddy[30374]: JOURNAL_STREAM=9:1236885
Jul 07 11:51:41 matrix-club-tech caddy[30374]: {"level":"info","ts":1594122701.6306107,"msg":"using provided configuration","config_file":"/etc/caddy/Caddyfile","config_adapter":""}
Jul 07 11:51:41 matrix-club-tech caddy[30374]: {"level":"info","ts":1594122701.637606,"logger":"admin","msg":"admin endpoint started","address":"tcp/localhost:2019","enforce_origin":false,"origins":["localhost:2019","[::1]:2019","127.0.0.1:2019"]}
Jul 07 11:51:41 matrix-club-tech caddy[30374]: {"level":"info","ts":1594122701.638253,"logger":"http","msg":"enabling automatic HTTP->HTTPS redirects","server_name":"srv1"}
Jul 07 11:51:41 matrix-club-tech caddy[30374]: {"level":"info","ts":1594122701.638481,"logger":"http","msg":"server is listening only on the HTTPS port but has no TLS connection policies; adding one to enable TLS","server_name":"srv0","https_port":443}
Jul 07 11:51:41 matrix-club-tech caddy[30374]: {"level":"info","ts":1594122701.6384995,"logger":"http","msg":"enabling automatic HTTP->HTTPS redirects","server_name":"srv0"}
Jul 07 11:51:41 matrix-club-tech caddy[30374]: {"level":"info","ts":1594122701.6390934,"logger":"tls","msg":"cleaned up storage units"}
Jul 07 11:51:41 matrix-club-tech caddy[30374]: {"level":"info","ts":1594122701.6393309,"logger":"http","msg":"enabling automatic TLS certificate management","domains":["matrix.club-tech.fr"]}
Jul 07 11:51:41 matrix-club-tech caddy[30374]: {"level":"info","ts":1594122701.6395383,"msg":"autosaved config","file":"/var/www/.config/caddy/autosave.json"}
Jul 07 11:51:41 matrix-club-tech caddy[30374]: {"level":"info","ts":1594122701.6395538,"msg":"serving initial configuration"}
Jul 07 11:51:41 matrix-club-tech caddy[30374]: 2020/07/07 11:51:41 [INFO][cache:0xc000247b00] Started certificate maintenance routine
Jul 07 11:51:41 matrix-club-tech caddy[30374]: 2020/07/07 11:51:41 [INFO][matrix.club-tech.fr] Obtain certificate; acquiring lock...
Jul 07 11:51:41 matrix-club-tech caddy[30374]: 2020/07/07 11:51:41 [INFO][matrix.club-tech.fr] Obtain: Lock acquired; proceeding...
Jul 07 11:51:42 matrix-club-tech caddy[30374]: 2020/07/07 11:51:42 [INFO][matrix.club-tech.fr] Waiting on rate limiter...
Jul 07 11:51:42 matrix-club-tech caddy[30374]: 2020/07/07 11:51:42 [INFO][matrix.club-tech.fr] Done waiting
Jul 07 11:51:42 matrix-club-tech caddy[30374]: 2020/07/07 11:51:42 [INFO] [matrix.club-tech.fr] acme: Obtaining bundled SAN certificate given a CSR
Jul 07 11:51:43 matrix-club-tech caddy[30374]: 2020/07/07 11:51:43 [INFO] [matrix.club-tech.fr] AuthURL: https://acme-staging-v02.api.letsencrypt.org/acme/authz-v3/73664028
Jul 07 11:51:43 matrix-club-tech caddy[30374]: 2020/07/07 11:51:43 [INFO] [matrix.club-tech.fr] acme: Could not find solver for: tls-alpn-01
Jul 07 11:51:43 matrix-club-tech caddy[30374]: 2020/07/07 11:51:43 [INFO] [matrix.club-tech.fr] acme: use http-01 solver
Jul 07 11:51:43 matrix-club-tech caddy[30374]: 2020/07/07 11:51:43 [INFO] [matrix.club-tech.fr] acme: Trying to solve HTTP-01
Jul 07 11:51:47 matrix-club-tech caddy[30374]: 2020/07/07 11:51:47 http: TLS handshake error from 54.37.23.75:51050: no certificate available for 'matrix.club-tech.fr'
Jul 07 11:51:47 matrix-club-tech caddy[30374]: 2020/07/07 11:51:47 http: TLS handshake error from 54.37.23.75:51052: no certificate available for 'matrix.club-tech.fr'
Jul 07 11:51:49 matrix-club-tech caddy[30374]: 2020/07/07 11:51:49 http: TLS handshake error from 54.37.23.75:51054: no certificate available for 'matrix.club-tech.fr'
Jul 07 11:51:49 matrix-club-tech caddy[30374]: 2020/07/07 11:51:49 [INFO] Deactivating auth: https://acme-staging-v02.api.letsencrypt.org/acme/authz-v3/73664028
Jul 07 11:51:49 matrix-club-tech caddy[30374]: 2020/07/07 11:51:49 [INFO] Unable to deactivate the authorization: https://acme-staging-v02.api.letsencrypt.org/acme/authz-v3/73664028
Jul 07 11:51:49 matrix-club-tech caddy[30374]: 2020/07/07 11:51:49 [ERROR] error: one or more domains had a problem:
Jul 07 11:51:49 matrix-club-tech caddy[30374]: [matrix.club-tech.fr] acme: error: 400 :: urn:ietf:params:acme:error:connection :: Fetching https://127.0.0.1/.well-known/acme-challenge/coH63QyLNqoL6-lmczedWKucOntmUrf6dZPMIMTgnRc: Invalid host in redirect target "127.0.0.1". Only domain names are supported, not IP addresses, url:
Jul 07 11:51:49 matrix-club-tech caddy[30374]:  (challenge=http-01 remaining=[tls-alpn-01])
Jul 07 11:51:50 matrix-club-tech caddy[30374]: 2020/07/07 11:51:50 http: TLS handshake error from 54.37.23.75:51056: no certificate available for 'matrix.club-tech.fr'
Jul 07 11:51:51 matrix-club-tech caddy[30374]: 2020/07/07 11:51:51 [INFO] [matrix.club-tech.fr] acme: Obtaining bundled SAN certificate given a CSR
Jul 07 11:51:52 matrix-club-tech caddy[30374]: 2020/07/07 11:51:52 [INFO] [matrix.club-tech.fr] AuthURL: https://acme-staging-v02.api.letsencrypt.org/acme/authz-v3/73664083
Jul 07 11:51:52 matrix-club-tech caddy[30374]: 2020/07/07 11:51:52 [INFO] [matrix.club-tech.fr] acme: use tls-alpn-01 solver
Jul 07 11:51:52 matrix-club-tech caddy[30374]: 2020/07/07 11:51:52 [INFO] [matrix.club-tech.fr] acme: Trying to solve TLS-ALPN-01
Jul 07 11:51:52 matrix-club-tech caddy[30374]: 2020/07/07 11:51:52 http: TLS handshake error from 127.0.0.1:42492: EOF
Jul 07 11:51:55 matrix-club-tech caddy[30374]: 2020/07/07 11:51:55 http: TLS handshake error from 54.37.23.75:51070: no certificate available for 'matrix.club-tech.fr'
Jul 07 11:51:57 matrix-club-tech caddy[30374]: 2020/07/07 11:51:57 [INFO] Deactivating auth: https://acme-staging-v02.api.letsencrypt.org/acme/authz-v3/73664083
Jul 07 11:51:58 matrix-club-tech caddy[30374]: 2020/07/07 11:51:58 [INFO] Unable to deactivate the authorization: https://acme-staging-v02.api.letsencrypt.org/acme/authz-v3/73664083
Jul 07 11:51:58 matrix-club-tech caddy[30374]: 2020/07/07 11:51:58 [ERROR] error: one or more domains had a problem:
Jul 07 11:51:58 matrix-club-tech caddy[30374]: [matrix.club-tech.fr] acme: error: 403 :: urn:ietf:params:acme:error:unauthorized :: Cannot negotiate ALPN protocol "acme-tls/1" for tls-alpn-01 challenge, url:
Jul 07 11:51:58 matrix-club-tech caddy[30374]:  (challenge=tls-alpn-01 remaining=[])
Jul 07 11:52:00 matrix-club-tech caddy[30374]: 2020/07/07 11:52:00 [ERROR] attempt 1: [matrix.club-tech.fr] Obtain: [matrix.club-tech.fr] error: one or more domains had a problem:
Jul 07 11:52:00 matrix-club-tech caddy[30374]: [matrix.club-tech.fr] acme: error: 403 :: urn:ietf:params:acme:error:unauthorized :: Cannot negotiate ALPN protocol "acme-tls/1" for tls-alpn-01 challenge, url:
Jul 07 11:52:00 matrix-club-tech caddy[30374]:  - retrying in 1m0s (18.480022652s/720h0m0s elapsed)...
Jul 07 11:52:03 matrix-club-tech caddy[30374]: {"level":"info","ts":1594122723.7889454,"msg":"shutting down apps then terminating","signal":"SIGTERM"}
Jul 07 11:52:03 matrix-club-tech caddy[30374]: 2020/07/07 11:52:03 [INFO][cache:0xc000247b00] Stopped certificate maintenance routine
Jul 07 11:52:03 matrix-club-tech caddy[30374]: {"level":"info","ts":1594122723.7907484,"logger":"admin","msg":"stopped previous server"}
Jul 07 11:52:03 matrix-club-tech caddy[30374]: {"level":"info","ts":1594122723.7910244,"msg":"shutdown done","signal":"SIGTERM"}

5. What I already tried:

Read up on docs, try to make my config file as simple as possible, ask a friend who’s been admining his own instance of Caddy 2, but no luck.

6. Links to relevant resources:

Not sure there’s anything to link in here?

2

Howdy @babolivier, welcome to the Caddy community!

:grimacing:

Yeah, that’s pretty bogus alright.

Wait, hold on.

Invalid host in redirect target

Redirect target?

Odd… Lets check:

~/Projects/test
➜ curl -IL https://matrix.club-tech.fr
HTTP/2 308
date: Fri, 10 Jul 2020 04:29:30 GMT
location: https://127.0.0.1/
server: Caddy
server: Caddy

curl: (7) Failed to connect to 127.0.0.1 port 443: Connection refused

Oh. The server listening there is redirecting to the IPv4 loopback address… Doesn’t seem like ideal behaviour, and this will be what’s tripping up LetsEncrypt.

But why is it doing that at all? The server is ostensibly Caddy. This challenge-response should essentially Just Work™️.

On top of this, the Caddyfile you’ve given above doesn’t have any proxy for the web root (i.e. URI /). And yet, the server header profile we got just now:

server: Caddy
server: Caddy

Is typical of a Caddy server reverse proxying to another Caddy instance.

So the question must be asked: Is the server listening on port 80/443 actually the Caddy instance that’s requesting the certificate? Having a different instance listening publicly with different config would explain all the oddities above, even the TLS-ALPN challenge failure.

2 Likes
3

Hey, and thanks for your response!

Yes I’m afraid it looks like there’s only one version of caddy running:

~$ ps -ef | grep caddy
www-data 15013     1  4 13:35 ?        00:00:00 /usr/local/bin/caddy run --environ --config /etc/caddy/Caddyfile
brendan  15045 13667  0 13:35 pts/0    00:00:00 grep --color=auto caddy
~$

And running systemctl stop caddy effectively kills that process.

Also, I hadn’t noticed that redirect, and yes it looks very weird…

Edit: Now that I think about it, maybe the host machine my VPS lives on does some reverse proxying through Caddy, I’ll have a look.

4

Right, it appears the hypervisor was also doing some reverse proxying with Caddy to the VM, hence this mess. It’s now fixed, thanks for looking into this @Whitestrake! :slight_smile:

1 Like
5

This topic was automatically closed after 30 days. New replies are no longer allowed.