Solved: Lets Encrypt Fails on Reverse Proxies

Help
1

1. Caddy version (caddy version):

2.4.6

2. How I run Caddy:

As a service with Caddyfile

a. System environment:

Ubuntu 21.04 x64 (ESXI VM)
CloudFlare is only used for DNS (WAF is Disabled)

b. Command:

caddy reload

c. Service/unit/compose file:

Not used

d. My complete Caddyfile or JSON config:


#Test Web App (Only One that works but originally was a file server)
plex.yourdomain.com {
reverse_proxy 192.168.86.3:80
}

#Test Server (Not Working)
www.yourdomain2.com {
reverse_proxy 192.168.86.9:443
}

#Test Admin Server (Not Working)
yourdomain3.com {
reverse_proxy 192.168.86.12:9000
}

3. The problem I’m having:

Both walzglobal.com and walztrading.com are not working. They seem to be failing on the let’s encrypt challenge. However CaddyServer IP is responding to 80/443 externally.

I am trying to setup Let’s Encrypt on both of these top level domains with certificates.

How do I get Let’s Encrypt to generate certificates on these domains if port 80/443 is not exposed on the individual reverse proxies. What am I missing? I would of thought Caddy could handle this without any additional configuration but I seem to be missing something.

4. Error messages and/or full log output:

I am not see any logs get generated. Tried adding these values but didn’t generate anything.

yourdomain.com {
reverse_proxy 192.168.86.12:9000
    log {
       level DEBUG
       output file /etc/caddy/caddy.log {
           roll_size 10MiB
           roll_keep 10
           roll_keep_for 336h
       }
}
}

5. What I already tried:

yourdomain.com is probably working because I originally setup as a test file server to verify all was good, but this also generated the certificate at that time.

6. Links to relevant resources:

2

This doesn’t have anything to do with the port numbers used for the proxies, that’s a separate layer from Caddy’s certificate automation.

The ACME issuers will hit Caddy on port 80 or 443 depending on the challenge type used, and Caddy does not proxy those requests for solving the challenges (doesn’t run the request handlers you defined).

The log directive only enables access logging, which shows logs for the request/responses served by that site. If Caddy isn’t able to serve the request because there’s an issue with TLS, then there won’t be any logs in there.

Caddy emits its runtime logs to stdout/stderr, which when running as a systemd service will end up in the journal. See this page in the docs which explains how to find your logs when running as a service:

Also, I’m not sure /etc/caddy is writable by the caddy user. A better place to write your logs to would be /var/logs/caddy which is a directory that is created by the apt package installation and will have the right permissions set.

Also, setting the log level to DEBUG won’t have any effect, because only INFO and ERROR level logs are emitted for access logs currently. If you need more detail in your runtime logs though, you can enable the debug global option, which will show more detailed logs in the journal.

By the port number you’re proxying to here, I assume this is the HTTPS port of whatever upstream app you’re running.

You didn’t tell Caddy to proxy over HTTPS though, so Caddy will attempt to proxy over HTTP (the default for reverse_proxy).

If that app has an HTTP port, I’d recommend using that instead, because it’ll be much simpler, and there’s rarely any benefit in proxying over HTTPS when the upstream app is in your local network – the only concern would be if there’s untrusted software or users with access to the network that would perform a man-in-the-middle between the machines in your network.

1 Like
3

Thank you for replying.

So when you say “enable https” do you mean by adding :443 to the domains like this? Also would you be able to provide me an example of how to proxy an https port in Caddy2?

esxi.yourdomain.com:443 {
        reverse_proxy 192.168.86.9:443
}

port.yourdomain.com:443 {
        reverse_proxy 192.168.86.12:9000
}

hello.yourdomain.com:443 {
        reverse_proxy 192.168.86.3:80
}
4

No, Caddy uses HTTPS for your sites by default, so you can remove :443.

Just specify HTTPS in the reverse proxy upstreams:

reverse_proxy https://192.168.86.9

See:

5

Still not working for the local https systems. Guessing something else needs to be configured somewhere.

I can connect to this server locally where CaddyServer is running. The server is an ESXI box and the full local url is https://192.168.86.9/ui/#/login all i get is a 502 gateway error.

esxi.yourdomain.com: {
        reverse_proxy https://192.168.86.9
}
6

Remove the colon here.

Check Caddy’s logs, like I wrote above. It will tell you why it wasn’t able to connect.

Like I said earlier though, proxying over HTTPS is probably going to be complicated for you here, and have very little benefit (and actually introduces overhead). I’m guessing Caddy probably doesn’t trust the upstream’s certificate if it’s a self-signed one. You can turn off cert verification, but if you do that, you would lose all security that HTTPS provides, so there’s no point to using HTTPS, and proxying over HTTP would be easier.

7

Solved, had to enable this directive. I also tried this before but turns out my syntax was off. Thank you to the both of you for helping pinpoint the issue.

esxi.yourdomain.com {
        reverse_proxy https://192.168.86.9 {
                transport http {
                    tls_insecure_skip_verify
                }
        }
}
1 Like
8

This topic was automatically closed after 30 days. New replies are no longer allowed.