.well-known/acme-challenge keeps hitting symfony

Help
1

1. The problem I’m having:

i have a basic symfony project hosted on docker using frankenphp,
i want to enable auto SSL for the public domain im using, but it seems like the validation process is looking for http://hireants.oshanrube.com/.well-known/acme-challenge/ but it keeps hitting the symfony code and returning 404,
should i read the code from symfony? it looks like it should be handled by the server. sorry if this is something obvious, i tried playing with various config, this is 1st time using caddy

2. Error messages and/or full log output:

{"level":"info","ts":1732149967.0644767,"msg":"using config from file","file":"/etc/caddy/Caddyfile"}
{"level":"info","ts":1732149967.104434,"msg":"adapted config to JSON","adapter":"caddyfile"}
{"level":"warn","ts":1732149967.1046617,"msg":"Caddyfile input is not formatted; run 'caddy fmt --overwrite' to fix inconsistencies","adapter":"caddyfile","file":"/etc/caddy/Caddyfile","line":10}
{"level":"info","ts":1732149967.117441,"logger":"admin","msg":"admin endpoint started","address":"localhost:2019","enforce_origin":false,"origins":["//localhost:2019","//[::1]:2019","//127.0.0.1:2019"]}
{"level":"info","ts":1732149967.1208313,"logger":"tls.cache.maintenance","msg":"started background certificate maintenance","cache":"0x400078c500"}
{"level":"info","ts":1732149967.1427524,"logger":"http.auto_https","msg":"server is listening only on the HTTPS port but has no TLS connection policies; adding one to enable TLS","server_name":"srv0","https_port":443}
{"level":"info","ts":1732149967.1429598,"logger":"http.auto_https","msg":"automatic HTTP->HTTPS redirects are disabled","server_name":"srv0"}
{"level":"warn","ts":1732149967.1431665,"logger":"http.auto_https","msg":"server is listening only on the HTTP port, so no automatic HTTPS will be applied to this server","server_name":"srv1","http_port":80}
{"level":"warn","ts":1732149967.1469126,"logger":"http.handlers.mercure","msg":"Setting the transport_url or the MERCURE_TRANSPORT_URL environment variable is deprecated, use the \"transport\" directive instead"}
{"level":"warn","ts":1732149967.1532807,"logger":"http.handlers.mercure","msg":"Setting the transport_url or the MERCURE_TRANSPORT_URL environment variable is deprecated, use the \"transport\" directive instead"}
{"level":"warn","ts":1732149967.1618142,"logger":"pki.ca.local","msg":"installing root certificate (you might be prompted for password)","path":"storage:pki/authorities/local/root.crt"}
{"level":"info","ts":1732149967.1653664,"msg":"warning: \"certutil\" is not available, install \"certutil\" with \"apt install libnss3-tools\" or \"yum install nss-tools\" and try again"}
{"level":"info","ts":1732149967.165857,"msg":"define JAVA_HOME environment variable to use the Java trust"}
{"level":"info","ts":1732149972.5864866,"msg":"certificate installed properly in linux trusts"}
{"level":"info","ts":1732149972.6177154,"logger":"tls","msg":"storage cleaning happened too recently; skipping for now","storage":"FileStorage:/data/caddy","instance":"96307f7c-5338-4e40-84f6-494da2735aa6","try_again":1732236372.6177065,"try_again_in":86399.999997666}
{"level":"info","ts":1732149972.6197915,"logger":"tls","msg":"finished cleaning storage units"}
{"level":"info","ts":1732149975.8097885,"msg":"FrankenPHP started 🐘","php_version":"8.3.13","num_threads":9}
{"level":"info","ts":1732149975.8108835,"logger":"http","msg":"enabling HTTP/3 listener","addr":":443"}
{"level":"info","ts":1732149975.8145328,"msg":"failed to sufficiently increase receive buffer size (was: 208 kiB, wanted: 7168 kiB, got: 416 kiB). See https://github.com/quic-go/quic-go/wiki/UDP-Buffer-Sizes for details."}
{"level":"info","ts":1732149975.816135,"logger":"http.log","msg":"server running","name":"srv0","protocols":["h1","h2","h3"]}
{"level":"info","ts":1732149975.8174279,"logger":"http.log","msg":"server running","name":"srv1","protocols":["h1","h2","h3"]}
{"level":"info","ts":1732149975.8179202,"logger":"http","msg":"enabling automatic TLS certificate management","domains":["localhost","hireants.oshanrube.com"]}
{"level":"warn","ts":1732149975.8254898,"logger":"tls","msg":"stapling OCSP","error":"no OCSP stapling for [localhost]: no OCSP server specified in certificate","identifiers":["localhost"]}
{"level":"info","ts":1732149975.8301926,"msg":"autosaved config (load with --resume flag)","file":"/config/caddy/autosave.json"}
{"level":"info","ts":1732149975.8317518,"msg":"serving initial configuration"}
{"level":"info","ts":1732149975.8356278,"logger":"tls","msg":"certificate is in configured renewal window based on expiration date","subjects":["localhost"],"expiration":1732145765,"ari_cert_id":"","next_ari_update":null,"renew_check_interval":600,"window_start":-6795364578.8713455,"window_end":-6795364578.8713455,"remaining":-4210.835622554}
{"level":"info","ts":1732149975.8468702,"logger":"tls.renew","msg":"acquiring lock","identifier":"localhost"}
{"level":"info","ts":1732149975.8491871,"logger":"tls.obtain","msg":"acquiring lock","identifier":"hireants.oshanrube.com"}
{"level":"info","ts":1732149975.853402,"logger":"tls.renew","msg":"lock acquired","identifier":"localhost"}
{"level":"info","ts":1732149975.8585157,"logger":"tls.renew","msg":"renewing certificate","identifier":"localhost","remaining":-4210.857368179}
{"level":"info","ts":1732149975.8583121,"logger":"tls.obtain","msg":"lock acquired","identifier":"hireants.oshanrube.com"}
{"level":"info","ts":1732149975.8651845,"logger":"tls.obtain","msg":"obtaining certificate","identifier":"hireants.oshanrube.com"}
{"level":"info","ts":1732149975.8851473,"logger":"tls","msg":"waiting on internal rate limiter","identifiers":["hireants.oshanrube.com"],"ca":"https://acme-v02.api.letsencrypt.org/directory","account":""}
{"level":"info","ts":1732149975.8858788,"logger":"tls","msg":"done waiting on internal rate limiter","identifiers":["hireants.oshanrube.com"],"ca":"https://acme-v02.api.letsencrypt.org/directory","account":""}
{"level":"info","ts":1732149975.8872085,"logger":"tls","msg":"using ACME account","account_id":"https://acme-v02.api.letsencrypt.org/acme/acct/2067394677","account_contact":[]}
{"level":"info","ts":1732149975.9258716,"logger":"tls.renew","msg":"certificate renewed successfully","identifier":"localhost","issuer":"local"}
{"level":"info","ts":1732149975.9269865,"logger":"tls.renew","msg":"releasing lock","identifier":"localhost"}
{"level":"info","ts":1732149975.927908,"logger":"tls","msg":"reloading managed certificate","identifiers":["localhost"]}
{"level":"warn","ts":1732149975.930193,"logger":"tls","msg":"stapling OCSP","error":"no OCSP stapling for [localhost]: no OCSP server specified in certificate","identifiers":["localhost"]}
{"level":"info","ts":1732149975.9308767,"logger":"tls.cache","msg":"replaced certificate in cache","subjects":["localhost"],"new_expiration":1732193176}
{"level":"info","ts":1732149977.6563618,"logger":"tls.acme_client","msg":"trying to solve challenge","identifier":"hireants.oshanrube.com","challenge_type":"tls-alpn-01","ca":"https://acme-v02.api.letsencrypt.org/directory"}
{"level":"error","ts":1732149988.3286004,"logger":"tls.acme_client","msg":"challenge failed","identifier":"hireants.oshanrube.com","challenge_type":"tls-alpn-01","problem":{"type":"urn:ietf:params:acme:error:connection","title":"","detail":"124.43.66.105: Timeout during connect (likely firewall problem)","instance":"","subproblems":[]}}
{"level":"error","ts":1732149988.3287733,"logger":"tls.acme_client","msg":"validating authorization","identifier":"hireants.oshanrube.com","problem":{"type":"urn:ietf:params:acme:error:connection","title":"","detail":"124.43.66.105: Timeout during connect (likely firewall problem)","instance":"","subproblems":[]},"order":"https://acme-v02.api.letsencrypt.org/acme/order/2067394677/325169341727","attempt":1,"max_attempts":3}
{"level":"info","ts":1732149990.0208874,"logger":"tls.acme_client","msg":"trying to solve challenge","identifier":"hireants.oshanrube.com","challenge_type":"http-01","ca":"https://acme-v02.api.letsencrypt.org/directory"}
{"level":"error","ts":1732150000.6701221,"logger":"tls.acme_client","msg":"challenge failed","identifier":"hireants.oshanrube.com","challenge_type":"http-01","problem":{"type":"urn:ietf:params:acme:error:connection","title":"","detail":"124.43.66.105: Fetching http://hireants.oshanrube.com/.well-known/acme-challenge/WagjXz2Yxt12EDkOSMKpJKhUhSVpKtyCocpBc9gnfDs: Timeout during connect (likely firewall problem)","instance":"","subproblems":[]}}
{"level":"error","ts":1732150000.6703324,"logger":"tls.acme_client","msg":"validating authorization","identifier":"hireants.oshanrube.com","problem":{"type":"urn:ietf:params:acme:error:connection","title":"","detail":"124.43.66.105: Fetching http://hireants.oshanrube.com/.well-known/acme-challenge/WagjXz2Yxt12EDkOSMKpJKhUhSVpKtyCocpBc9gnfDs: Timeout during connect (likely firewall problem)","instance":"","subproblems":[]},"order":"https://acme-v02.api.letsencrypt.org/acme/order/2067394677/325169390167","attempt":2,"max_attempts":3}
{"level":"error","ts":1732150000.6704526,"logger":"tls.obtain","msg":"could not get certificate from issuer","identifier":"hireants.oshanrube.com","issuer":"acme-v02.api.letsencrypt.org-directory","error":"HTTP 400 urn:ietf:params:acme:error:connection - 124.43.66.105: Fetching http://hireants.oshanrube.com/.well-known/acme-challenge/WagjXz2Yxt12EDkOSMKpJKhUhSVpKtyCocpBc9gnfDs: Timeout during connect (likely firewall problem)"}
{"level":"error","ts":1732150000.6707504,"logger":"tls.obtain","msg":"will retry","error":"[hireants.oshanrube.com] Obtain: [hireants.oshanrube.com] solving challenge: hireants.oshanrube.com: [hireants.oshanrube.com] authorization failed: HTTP 400 urn:ietf:params:acme:error:connection - 124.43.66.105: Fetching http://hireants.oshanrube.com/.well-known/acme-challenge/WagjXz2Yxt12EDkOSMKpJKhUhSVpKtyCocpBc9gnfDs: Timeout during connect (likely firewall problem) (ca=https://acme-v02.api.letsencrypt.org/directory)","attempt":1,"retrying_in":60,"elapsed":24.807367945,"max_duration":2592000}

3. Caddy version:

im using dockerfile dunglas/frankenphp:1-php8.3

4. How I installed and ran Caddy:

on docker dunglas/frankenphp:1-php8.3

a. System environment:

Docker

b. Command:

CMD [ "frankenphp", "run", "--config", "/etc/caddy/Caddyfile" ]

d. My complete Caddy config:

{
	{$CADDY_GLOBAL_OPTIONS}

	frankenphp {
		{$FRANKENPHP_CONFIG}
	}
	auto_https disable_redirects
}

{$CADDY_EXTRA_CONFIG}

localhost, http://localhost, http://127.0.0.1, http://10.0.68.53, http://192.168.10.200, hireants.oshanrube.com, http://hireants.oshanrube.com {
	{$CADDY_TLS}

	log {
        {$CADDY_SERVER_LOG_OPTIONS}
        # Redact the authorization query parameter that can be set by Mercure
        format filter {
            request>uri query {
                replace authorization REDACTED
            }
        }
    }

    root /app/public
    encode zstd br gzip

    mercure {
        # Transport to use (default to Bolt)
        transport_url {$MERCURE_TRANSPORT_URL:bolt:///data/mercure.db}
        # Publisher JWT key
        publisher_jwt {env.MERCURE_PUBLISHER_JWT_KEY} {env.MERCURE_PUBLISHER_JWT_ALG}
        # Subscriber JWT key
        subscriber_jwt {env.MERCURE_SUBSCRIBER_JWT_KEY} {env.MERCURE_SUBSCRIBER_JWT_ALG}
        # Allow anonymous subscribers (double-check that it's what you want)
        anonymous
        # Enable the subscription API (double-check that it's what you want)
        subscriptions
        # Extra directives
        {$MERCURE_EXTRA_DIRECTIVES}
    }

    vulcain

    {$CADDY_SERVER_EXTRA_DIRECTIVES}

    # Disable Topics tracking if not enabled explicitly: https://github.com/jkarlin/topics
    header ?Permissions-Policy "browsing-topics=()"

    @phpRoute {
        not path /.well-known/mercure*
        not file {path}
    }
    rewrite @phpRoute index.php

    @frontController path index.php
    php @frontController

    file_server
}

5. Links to relevant resources:

2

Howdy @Oshan_Rubesinghe, welcome to the Caddy community!

When you say it keeps hitting symfony code and returning 404, what exactly do you see that’s indicating that result?

Looking at your logs, I can’t see any 404s. This is what we can see coming back from LetsEncrypt:

urn:ietf:params:acme:error:connection: 124.43.66.105: Fetching http://hireants.oshanrube.com/.well-known/acme-challenge/WagjXz2Yxt12EDkOSMKpJKhUhSVpKtyCocpBc9gnfDs: Timeout during connect (likely firewall problem)

That particular error means LetsEncrypt never actually successfully connected to your server at all. A firewall is given as a possible reason, although having DNS resolve to the wrong IP address is another possible cause.

I can’t say I’ve ever seen Caddy fail to intercept an ACME challenge it initiated. As far as I know, it isn’t possible, as CertMagic consumes the .well-known request before any user-configured routes ever execute, so if Caddy kicked it off, Caddy will handle it - as long as the request actually reaches Caddy at all.

3

Hi @Whitestrake thanks for looking into this
on the logs theres the following line which says HTTP 400

{"level":"error","ts":1732150000.6704526,"logger":"tls.obtain","msg":"could not get certificate from issuer","identifier":"hireants.oshanrube.com","issuer":"acme-v02.api.letsencrypt.org-directory","error":"HTTP 400 urn:ietf:params:acme:error:connection - 124.43.66.105: Fetching http://hireants.oshanrube.com/.well-known/acme-challenge/WagjXz2Yxt12EDkOSMKpJKhUhSVpKtyCocpBc9gnfDs: Timeout during connect (likely firewall problem)"}

when i browse to the url directly its showing

image
image695×223 11.3 KB

which seems to be the symfony 404 page,
if it is a firewall error it should be a unreachable error right? also the port 80 is working
image

on otherhand, does it mean that something is blocked on the outgoing port? should i open any specific outgoing port? though i dont think its blocked

4

That line is a response from LetsEncrypt to Caddy. Under the hood, Caddy is actually making a request to the ACME server to tell it to try make a request back to Caddy via the domain name Caddy wants a cert for. If it works - LetsEncrypt knows that the entity requesting a cert is the entity at the domain in question, so it’s OK to issue a cert. In that case, LetsEncrypt is returning an error to Caddy to say “your request for a certificate didn’t work, and here’s why”. It’s a 400 error (Bad Request), not a 404 (Not Found).

You’re seeing a different result, which is incidentally how Caddy can allow a server behind it to request certificates with HTTP validation. CertMagic doesn’t blanket intercept all .well-known requests, only the ones it is trying to solve. .well-known requests it didn’t initiate/isn’t trying to solve get passed to the user-configured routes just like any other request would.

So It doesn’t matter what symfony would be returning, because Caddy would respond differently to LetsEncrypt than it does to you.

I’d guess you’re probably accessing Caddy from within the same firewalled network, which is allowing your traffic but not traffic from across the internet?

LetsEncrypt can’t reach you, and neither can I:

➜ curl -kIL hireants.oshanrube.com
curl: (28) Failed to connect to hireants.oshanrube.com port 80 after 75004 ms: Could not connect to server
5

Oh ok thanks, there seems to be a config error on the router, which allowed only the internal traffic, now its fixed!!

1 Like