Accessing subdomains with reverse_proxy & wildcard certificate does not work for me

T13nou · November 24, 2021, 2:54pm

1. Caddy version (`caddy version`):

v2.4.6

2. How I run Caddy:

caddy run

a. System environment:

Linux Debian 11

b. Command:

caddy run

c. Service/unit/compose file:

Paste full file contents here.
Make sure backticks stay on their own lines,
and the post looks nice in the preview pane.

d. My complete Caddyfile or JSON config:

# Global config
{
        # Let's Encrypt
        email etienne.blanchet@soverin.net

        # No admin
        admin off
}

*.etienneblanchet.fr {
        tls {
                dns gandi SECRET
        }

        #@password host password.etienneblanchet.fr
        #handle @password {
        #respond "Password"
        #       reverse_proxy 192.168.2.60:443 {
        #               header_up Host {upstream_hostport}
        #               header_up X-Forwarded-Host {host}
        #       }
        #}

        @drive host drive.etienneblanchet.fr
        handle @drive {
                #respond "Drive"
                reverse_proxy 192.168.2.100:10003
        }

        @diskstation host diskstation.etienneblanchet.fr
        handle @diskstation {
                #respond "Diskstation"
                reverse_proxy 192.168.2.100:5001
                header / Strict-Transport-Security "max-age=63072000"
        }

        #Fallback for otherwise unhandled domains
        handle {
                abort
                respond "Ce domaine n'existe pas!"
        }
}

3. The problem I’m having:

Hi, I’m trying to use Caddy for reverse proxy purposes.

I need a wildcard certificate for my domain etienneblanchet.fr and redirect each subdomain to the specific server.

So far the wildcard certificate runs perfectly but I’m having an issue with redirections.

With the Caddyfile above, I’m not able to have access to my HTTPS services. I’m always having an answer on the site.

Use case

In my Pi-hole, I’m having directed all my subdomains to Caddy IP
I run caddy with “caddy run”
Accessing from my browser (Edge or Firefox) the URL : https://diskstation.etienneblanchet.fr gives me the following error

"400 Bad Request
The plain HTTP request was sent to HTTPS port"

Am I missing some instructions in my Caddyfile ?

Thanks for helping

4. Error messages and/or full log output:

Preformatted text

nov. 24 13:28:48 Caddy systemd[1]: Starting Caddy...
nov. 24 13:28:48 Caddy caddy[8681]: caddy.HomeDir=/var/lib/caddy
nov. 24 13:28:48 Caddy caddy[8681]: caddy.AppDataDir=/var/lib/caddy/.local/share/caddy
nov. 24 13:28:48 Caddy caddy[8681]: caddy.AppConfigDir=/var/lib/caddy/.config/caddy
nov. 24 13:28:48 Caddy caddy[8681]: caddy.ConfigAutosavePath=/var/lib/caddy/.config/caddy/autosave.json
nov. 24 13:28:48 Caddy caddy[8681]: caddy.Version=v2.4.6 h1:HGkGICFGvyrodcqOOclHKfvJC0qTU7vny/7FhYp9hNw=
nov. 24 13:28:48 Caddy caddy[8681]: runtime.GOOS=linux
nov. 24 13:28:48 Caddy caddy[8681]: runtime.GOARCH=amd64
nov. 24 13:28:48 Caddy caddy[8681]: runtime.Compiler=gc
nov. 24 13:28:48 Caddy caddy[8681]: runtime.NumCPU=8
nov. 24 13:28:48 Caddy caddy[8681]: runtime.GOMAXPROCS=8
nov. 24 13:28:48 Caddy caddy[8681]: runtime.Version=go1.17
nov. 24 13:28:48 Caddy caddy[8681]: os.Getwd=/
nov. 24 13:28:48 Caddy caddy[8681]: LANG=fr_FR.UTF-8
nov. 24 13:28:48 Caddy caddy[8681]: PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
nov. 24 13:28:48 Caddy caddy[8681]: NOTIFY_SOCKET=/run/systemd/notify
nov. 24 13:28:48 Caddy caddy[8681]: HOME=/var/lib/caddy
nov. 24 13:28:48 Caddy caddy[8681]: LOGNAME=caddy
nov. 24 13:28:48 Caddy caddy[8681]: USER=caddy
nov. 24 13:28:48 Caddy caddy[8681]: INVOCATION_ID=30af1ba617d04ba7a133183dceed7f18
nov. 24 13:28:48 Caddy caddy[8681]: JOURNAL_STREAM=8:29456
nov. 24 13:28:48 Caddy caddy[8681]: {"level":"info","ts":1637756928.3894627,"msg":"using provided configuration","config_file":"/etc/caddy/Caddyfile","config_adapter":""}
nov. 24 13:28:48 Caddy caddy[8681]: {"level":"warn","ts":1637756928.3905225,"logger":"admin","msg":"admin endpoint disabled"}
nov. 24 13:28:48 Caddy caddy[8681]: {"level":"info","ts":1637756928.3906913,"logger":"http","msg":"server is listening only on the HTTPS port but has no TLS connection policies; adding one to enable TLS","server_name":"srv0","https_port":443}
nov. 24 13:28:48 Caddy caddy[8681]: {"level":"info","ts":1637756928.3907025,"logger":"http","msg":"enabling automatic HTTP->HTTPS redirects","server_name":"srv0"}
nov. 24 13:28:48 Caddy caddy[8681]: {"level":"info","ts":1637756928.3907456,"logger":"tls.cache.maintenance","msg":"started background certificate maintenance","cache":"0xc00038d7a0"}
nov. 24 13:28:48 Caddy caddy[8681]: {"level":"info","ts":1637756928.3910933,"logger":"tls","msg":"cleaning storage unit","description":"FileStorage:/etc/caddy"}
nov. 24 13:28:48 Caddy caddy[8681]: {"level":"info","ts":1637756928.3912022,"logger":"tls.cache.maintenance","msg":"stopped background certificate maintenance","cache":"0xc00038d7a0"}
nov. 24 13:28:48 Caddy caddy[8681]: run: loading initial config: loading new config: http app module: start: tcp: listening on :443: listen tcp :443: bind: address already in use
nov. 24 13:28:48 Caddy systemd[1]: Started Caddy.
nov. 24 13:28:48 Caddy systemd[1]: caddy.service: Main process exited, code=exited, status=1/FAILURE
nov. 24 13:28:48 Caddy systemd[1]: caddy.service: Failed with result 'exit-code'.

5. What I already tried:

I tried added header_up Host {upstream_hostport} & header_up X-Forwarded-Host {host} to the reverse proxy section but does not work

6. Links to relevant resources:

francislavoie · November 24, 2021, 4:08pm

When posting configs and logs on the forums, please use triple backticks ``` on the lines immediately before and after the content. This way, formatting will be preserved, and it’ll be easier to read. You may also use single backticks for inline code chunks, like this.

I don’t believe that you have no logs. You didn’t say how you’re actually running Caddy. Are you running it as a systemd service? If so, see here in the docs for how to get your logs.

T13nou · November 24, 2021, 4:19pm

You’re right @francislavoie I’ve added the logs from journalctl

Thanks,

Etienne

francislavoie · November 24, 2021, 4:35pm

Again – please use code formatting when posting logs and configs!!

Caddy wasn’t able to start because you have something else already running that’s binding to port 443. Do you have another server running? Maybe an old Apache server? You’ll need to turn that off.

T13nou · November 24, 2021, 4:40pm

Ok I used the right formatting now

I’ve restarted the service and that works correctly

● caddy.service - Caddy
     Loaded: loaded (/lib/systemd/system/caddy.service; enabled; vendor preset: enabled)
     Active: active (running) since Wed 2021-11-24 17:38:54 CET; 7s ago
       Docs: https://caddyserver.com/docs/
   Main PID: 9236 (caddy)
      Tasks: 9 (limit: 2297)
     Memory: 10.3M
        CPU: 29ms
     CGroup: /system.slice/caddy.service
             └─9236 /usr/bin/caddy run --environ --config /etc/caddy/Caddyfile

nov. 24 17:38:54 Caddy caddy[9236]: {"level":"info","ts":1637771934.3179755,"logger":"tls.cache.maintenance","msg":"started background certificate maintenance","cache":"0xc0003deee0"}
nov. 24 17:38:54 Caddy caddy[9236]: {"level":"info","ts":1637771934.3180137,"logger":"http","msg":"server is listening only on the HTTPS port but has no TLS connection policies; adding one to enable TLS","server_name":"srv0","https_port":443}
nov. 24 17:38:54 Caddy caddy[9236]: {"level":"info","ts":1637771934.318024,"logger":"http","msg":"enabling automatic HTTP->HTTPS redirects","server_name":"srv0"}
nov. 24 17:38:54 Caddy caddy[9236]: {"level":"info","ts":1637771934.3184154,"logger":"tls","msg":"cleaning storage unit","description":"FileStorage:/etc/caddy"}
nov. 24 17:38:54 Caddy caddy[9236]: {"level":"info","ts":1637771934.3184445,"logger":"tls","msg":"finished cleaning storage units"}
nov. 24 17:38:54 Caddy caddy[9236]: {"level":"info","ts":1637771934.3187895,"logger":"http","msg":"enabling automatic TLS certificate management","domains":["*.etienneblanchet.fr"]}
nov. 24 17:38:54 Caddy caddy[9236]: {"level":"info","ts":1637771934.3191054,"msg":"autosaved config (load with --resume flag)","file":"/var/lib/caddy/.config/caddy/autosave.json"}
nov. 24 17:38:54 Caddy caddy[9236]: {"level":"info","ts":1637771934.3191686,"msg":"serving initial configuration"}
nov. 24 17:38:54 Caddy caddy[9236]: {"level":"error","ts":1637771934.3191836,"logger":"tls","msg":"job failed","error":"*.etienneblanchet.fr: caching certificate after obtaining it: decoding certificate metadata: unexpected end of JSON input"}
nov. 24 17:38:54 Caddy systemd[1]: Started Caddy.

But unfortunately I have no chance to access my backend servers with the URL

Gives me “SSL_ERROR_INTERNAL_ERROR_ALERT”

francislavoie · November 24, 2021, 4:54pm

Hmm. I’m gonna guess the certificate storage got corrupted somehow.

Please wipe /var/lib/caddy/.local/share/caddy (Caddy’s storage location when running as a systemd service), then restart Caddy to try again.

T13nou · November 24, 2021, 5:21pm

In my Caddyfile I’ve put this. Would it be the reason of failure ?

T13nou · November 24, 2021, 5:22pm

Ok just wiped it and restarted the service

Here’s what I have

● caddy.service - Caddy
     Loaded: loaded (/lib/systemd/system/caddy.service; enabled; vendor preset: enabled)
     Active: active (running) since Wed 2021-11-24 18:22:15 CET; 4s ago
       Docs: https://caddyserver.com/docs/
   Main PID: 9320 (caddy)
      Tasks: 13 (limit: 2297)
     Memory: 12.9M
        CPU: 26ms
     CGroup: /system.slice/caddy.service
             └─9320 /usr/bin/caddy run --environ --config /etc/caddy/Caddyfile

nov. 24 18:22:15 Caddy caddy[9320]: {"level":"info","ts":1637774535.7153714,"logger":"tls.cache.maintenance","msg":"started background certificate maintenance","cache":"0xc00045ccb0"}
nov. 24 18:22:15 Caddy caddy[9320]: {"level":"info","ts":1637774535.715488,"logger":"http","msg":"server is listening only on the HTTPS port but has no TLS connection policies; adding one to enable TLS","server_name":"srv0","https_port":443}
nov. 24 18:22:15 Caddy caddy[9320]: {"level":"info","ts":1637774535.71557,"logger":"http","msg":"enabling automatic HTTP->HTTPS redirects","server_name":"srv0"}
nov. 24 18:22:15 Caddy caddy[9320]: {"level":"info","ts":1637774535.7160413,"logger":"tls","msg":"cleaning storage unit","description":"FileStorage:/etc/caddy"}
nov. 24 18:22:15 Caddy caddy[9320]: {"level":"info","ts":1637774535.7160718,"logger":"tls","msg":"finished cleaning storage units"}
nov. 24 18:22:15 Caddy caddy[9320]: {"level":"info","ts":1637774535.7162852,"logger":"http","msg":"enabling automatic TLS certificate management","domains":["*.etienneblanchet.fr"]}
nov. 24 18:22:15 Caddy caddy[9320]: {"level":"info","ts":1637774535.7168794,"msg":"autosaved config (load with --resume flag)","file":"/var/lib/caddy/.config/caddy/autosave.json"}
nov. 24 18:22:15 Caddy caddy[9320]: {"level":"info","ts":1637774535.717072,"msg":"serving initial configuration"}
nov. 24 18:22:15 Caddy systemd[1]: Started Caddy.
nov. 24 18:22:15 Caddy caddy[9320]: {"level":"error","ts":1637774535.7172358,"logger":"tls","msg":"job failed","error":"*.etienneblanchet.fr: caching certificate after obtaining it: decoding certificate metadata: unexpected end of JSON input"}

francislavoie · November 24, 2021, 5:25pm

Yes. Remove that.

Again – your config in your original post isn’t using code formatting, so it’s messy and hard to follow.

T13nou · November 24, 2021, 5:30pm

Sorry … just put it in the right form (1st time I use it :/)

Much better after removing the storage thing

● caddy.service - Caddy
     Loaded: loaded (/lib/systemd/system/caddy.service; enabled; vendor preset: enabled)
     Active: active (running) since Wed 2021-11-24 18:26:15 CET; 2s ago
       Docs: https://caddyserver.com/docs/
   Main PID: 9352 (caddy)
      Tasks: 9 (limit: 2297)
     Memory: 12.9M
        CPU: 74ms
     CGroup: /system.slice/caddy.service
             └─9352 /usr/bin/caddy run --environ --config /etc/caddy/Caddyfile

nov. 24 18:26:15 Caddy caddy[9352]: {"level":"info","ts":1637774775.91527,"logger":"tls","msg":"cleaning storage unit","description":"FileStorage:/var/lib/caddy/.local/share/caddy"}
nov. 24 18:26:15 Caddy caddy[9352]: {"level":"info","ts":1637774775.9152925,"logger":"tls","msg":"finished cleaning storage units"}
nov. 24 18:26:15 Caddy caddy[9352]: {"level":"info","ts":1637774775.9154081,"msg":"autosaved config (load with --resume flag)","file":"/var/lib/caddy/.config/caddy/autosave.json"}
nov. 24 18:26:15 Caddy caddy[9352]: {"level":"info","ts":1637774775.9155195,"logger":"tls.obtain","msg":"acquiring lock","identifier":"*.etienneblanchet.fr"}
nov. 24 18:26:15 Caddy caddy[9352]: {"level":"info","ts":1637774775.916086,"msg":"serving initial configuration"}
nov. 24 18:26:15 Caddy systemd[1]: Started Caddy.
nov. 24 18:26:15 Caddy caddy[9352]: {"level":"info","ts":1637774775.918639,"logger":"tls.obtain","msg":"lock acquired","identifier":"*.etienneblanchet.fr"}
nov. 24 18:26:16 Caddy caddy[9352]: {"level":"info","ts":1637774776.6619027,"logger":"tls.issuance.acme","msg":"waiting on internal rate limiter","identifiers":["*.etienneblanchet.fr"],"ca":"https://acme-v02.api.letsencrypt.org/directory","account":"etienne.blanchet@soverin.net"}
nov. 24 18:26:16 Caddy caddy[9352]: {"level":"info","ts":1637774776.6619399,"logger":"tls.issuance.acme","msg":"done waiting on internal rate limiter","identifiers":["*.etienneblanchet.fr"],"ca":"https://acme-v02.api.letsencrypt.org/directory","account":"etienne.blanchet@soverin.net"}
nov. 24 18:26:17 Caddy caddy[9352]: {"level":"info","ts":1637774777.0689776,"logger":"tls.issuance.acme.acme_client","msg":"trying to solve challenge","identifier":"*.etienneblanchet.fr","challenge_type":"dns-01","ca":"https://acme-v02.api.letsencrypt.org/directory"}

Now better for certificate error but still have the 400 issue when trying to reverse proxy to my backend

francislavoie · November 24, 2021, 5:36pm

Oops looks like you left in your Gandi secret this time. We can’t redact it from the forums (edit history is preserved) so you’ll probably need to reset the key.

It’s pretty clear that you’re trying to proxy to the upstream’s HTTPS port. Since it’s in the same network, and Caddy is terminating TLS, you should use your nginx server’s HTTP port.

FYI, turning off the admin endpoint means you won’t be able to do graceful reloads of Caddy. It’s generally best to leave it on.

Here, you’re using a / header matcher, which will only match requests to / exactly, so that header would only get set for requests to / and nothing else. Remove the / to make it apply to all requests.

It doesn’t make sense to have both abort and respond, only one will run, and in this case it will be abort because it’s higher in the directive order.

T13nou · November 24, 2021, 5:54pm

Thanks for that, I’ve applied the fixes

What I’m trying to achieve is for example

In my local network & on internet having some services with SSL

Let’s take the example of Bitwarden
LAN
PC > https://bitwarden > Resolved by Pihole to Caddy > Redirected to Bitwarden server with HTTPS

Internet
PC > https://bitwarden > Resolved by Gandi to NATed Caddy > Redirected to Bitwarden server with HTTPS

Is it possible with Caddy ?

francislavoie · November 24, 2021, 6:23pm

Yep. That’s called split-horizon DNS:

T13nou · November 24, 2021, 7:07pm

Yes I know split dns and it works perfectly for me but I’m just wondering why I am having a code 400 when trying to access my services thru Caddy

Any idea ?

francislavoie · November 24, 2021, 7:17pm

The error message is clear. You’re proxying to nginx’s HTTPS port. You should proxy to its HTTP port instead.

T13nou · November 24, 2021, 7:28pm

and why is it a problem to proxy to HTTPS ?

I’m asking for HTTPS://URL that redirects to HTTPS://backend. It’s all https and should work no ?

francislavoie · November 24, 2021, 7:50pm

Caddy terminates TLS, so there’s no use in establishing another TLS connection with the upstream, because the request has already reached your network. It’ll only add overhead, and it only complicates the setup. You need to make sure that Caddy can trust the certificate that the upstream would serve, and make sure that the Host header is set correctly, etc.

system · December 24, 2021, 2:54pm

This topic was automatically closed after 30 days. New replies are no longer allowed.