Caddy behind Apache reverse proxy can't get SSL to work

1. The problem I’m having:

We have a running apache and caddy instance, each living in their own world on different external IPs. We want to slowly migrate everything to caddy but we have to keep them both alive for production constraints. So the plan we brainstormed is to make apache a simple reverse proxy to caddy until we are ready to shut it down:

  • convert apache virtualhost definitions to the equivalent in a caddyfile
  • replace the existing virtual host definition so that all traffic is handled to caddy: here we are having some issues.

2. Error messages and/or full log output:

Error from the browser

SSL_ERROR_RX_RECORD_TOO_LONG

Errors from the caddy logs, should have stripped sensitive data:

{"level":"error","ts":1709209050.5225616,"logger":"tls.obtain","msg":"will retry","error":"[ftp.<redacted>.it] Obtain: [ftp.<redacted>.it] solving challenges: authz https://acme.zerossl.com/v2/DV90/authz/W_7RSOVpYLokEF3t6vJ4-Q has unexpected status; order will fail: invalid (order=https://acme.zerossl.com/v2/DV90/order/VFuDDFB0Nn146Hs43bLo1A) (ca=https://acme.zerossl.com/v2/DV90)","attempt":1,"retrying_in":60,"elapsed":6.182826252,"max_duration":2592000},
{"level":"error","ts":1709209049.821224,"logger":"tls.issuance.acme.acme_client","msg":"challenge failed","identifier":"ftp.<redacted>.it","challenge_type":"tls-alpn-01","status_code":400,"problem_type":"urn:ietf:params:acme:error:malformed","error":"84.247.247.54: Server only speaks HTTP, not TLS"},{"level":"error","ts":1709208012.5871758,"logger":"tls.issuance.acme.acme_client","msg":"challenge failed","identifier":"ftp.<redacted>.it","challenge_type":"tls-alpn-01","status_code":400,"problem_type":"urn:ietf:params:acme:error:connection","error":"84.247.247.54: Error getting validation data"},
{"level":"error","ts":1709209047.3053246,"logger":"tls.issuance.acme.acme_client","msg":"challenge failed","identifier":"ftp.<redacted>.it","challenge_type":"http-01","status_code":403,"problem_type":"urn:ietf:params:acme:error:unauthorized","error":"84.247.247.54: Invalid response from http://ftp.<redacted>.it/.well-known/acme-challenge/hmD_L5m31zvL5MYE4VnZaPVFZx-0Mve__LoSY8BjdBo: 502"},
{"level":"error","ts":1709209047.3054154,"logger":"tls.issuance.acme.acme_client","msg":"validating authorization","identifier":"ftp.<redacted>.it","error":"authorization failed: HTTP 403 urn:ietf:params:acme:error:unauthorized - 84.247.247.54: Invalid response from http://ftp.<redacted>.it/.well-known/acme-challenge/hmD_L5m31zvL5MYE4VnZaPVFZx-0Mve__LoSY8BjdBo: 502","order":"https://acme-v02.api.letsencrypt.org/acme/order/1499062116/248442159267","attempt":1,"max_attempts":3},
{"level":"info","ts":1709209044.3384137,"logger":"tls.obtain","msg":"acquiring lock","identifier":"ftp.<redacted>.it"},
{"level":"info","ts":1709209044.3396912,"logger":"tls.obtain","msg":"lock acquired","identifier":"ftp.<redacted>.it"},
{"level":"info","ts":1709209044.3725355,"logger":"tls.issuance.acme","msg":"waiting on internal rate limiter","identifiers":["ftp.<redacted>.it"]},
{"level":"info","ts":1709209044.3725798,"logger":"tls.issuance.acme","msg":"done waiting on internal rate limiter","identifiers":["ftp.<redacted>.it"]},
{"level":"info","ts":1709209045.7976952,"logger":"tls.issuance.acme.acme_client","msg":"trying to solve challenge","identifier":"ftp.<redacted>.it","challenge_type":"http-01","ca":"https://acme-v02.api.letsencrypt.org/directory"},
{"level":"info","ts":1709209048.7436318,"logger":"tls.issuance.acme.acme_client","msg":"trying to solve challenge","identifier":"ftp.<redacted>.it","challenge_type":"tls-alpn-01","ca":"https://acme-v02.api.letsencrypt.org/directory"},
{"level":"error","ts":1709209049.821328,"logger":"tls.issuance.acme.acme_client","msg":"validating authorization","identifier":"ftp.<redacted>.it","error":"[ftp.<redacted>.it] authorization failed: HTTP 400 urn:ietf:params:acme:error:malformed - 84.247.247.54: Server only speaks HTTP, not TLS","order":"https://acme-v02.api.letsencrypt.org/acme/order/1499062116/248442166827","attempt":2,"max_attempts":3},
{"level":"info","ts":1709209049.8220773,"logger":"tls.issuance.acme","msg":"waiting on internal rate limiter","identifiers":["ftp.<redacted>.it"]}

Apache doesn’t seem to log any error, likely indicating that request are proxied correctly.

3. Caddy version:

/ # caddy version
v2.4.0 h1:yHnnbawH2G3ZBP2mAJF4XBLnJanqhULLP/wu01Qi9Io=

4. How I installed and ran Caddy:

a. System environment:

Docker container using the “wrapper” provided by lucaslorentz for automatic caddyfile creation based on docker labels.

lucaslorentz/caddy-docker-proxy:2.3-alpine@sha256:e3268b88f0c3e3a9f93ad527375e6e57bf8aa17c2aa9c8c4d62fc519e61342eb

d. My complete Caddy config:

compiled caddyfile extracted from the logs of the container. Hopefully redacted correctly.
Some reverse proxies uses the container name since they live in the same docker network of the caddy container.

Don’t be misleaded by the ftp prefix, we are not trying to proxy ftp traffic but just a webui exposed by the upstream ftp server.


2024/02/29 12:26:06 [INFO] New Caddyfile:

{

	email it@<redacted>.it

}

ftp.<redacted>.it {

	reverse_proxy sftpgo-v2.3.0:8080

	respond /web/admin 404

}

5. Links to relevant resources:

Apache conf used:

############################ WEBUI
## Reverse Proxy verso il webserver Caddy v2 su docker
<VirtualHost *:80> 
  ServerName ftp.<redacted>.it <--- this matches a caddyfile host name
  ProxyPass "/" "http://192.168.0.80"
  ProxyPassReverse "/" "http://192.168.0.80"
  ProxyPreserveHost On
</VirtualHost>

<VirtualHost *:443> 
  ServerName ftp.<redacted>.it
  ProxyPass "/" "https://192.168.0.80:443"
  ProxyPassReverse "/" "https://192.168.0.80:443"
  ProxyPreserveHost On
</VirtualHost>

That’s a very old version. Upgrade to the latest, v2.7.6. Don’t use outdated software!

Why are you putting Apache in front of Caddy? Why not put Caddy in front? Caddy can automate TLS for all your domains.

You won’t really be able to automate TLS if Caddy is sitting behind another proxy, unless you use the DNS challenge. The HTTP challenge can sometimes work, but it’s dependent on requests not being redirected. The TLS-ALPN challenge can never work behind a proxy.

You could just serve only HTTP from Caddy (prefix your site address with http://). There’s no reason to serve HTTPS if Caddy isn’t in front, since the traffic has already entered your network. HTTPS between servers in your LAN doesn’t have any benefit, only overhead (unless you don’t trust your LAN to not be compromised… in which case it’s already game-over anyway).

That’s a very old version. Upgrade to the latest, v2.7.6. Don’t use outdated software!

Thanks for the feedback, will do!

Why are you putting Apache in front of Caddy?

Some reasons, not necessarily blocking but they are on my mind:

  • there are a lot of apache vhosts with underlying php pages and script and I haven’t figured out how to correctly configure php with caddy while the wamp server installed on the Windows host is ready out of the box.

    I suppose I could leave the apache config as is and use caddy to reverse proxy all the apache vhosts to apache? Will php still work? Will my collegues understand how to configure caddy for future vhosts they will add to apache? Maybe I could configure a fallback so that anything that doesn’t matches whats configured in Caddy is reverse proxied to apache? If so I don’t know how to do such rule.

  • There are some oddly configured vhosts that I am still working on how to translate to caddy


Maybe I could configure a fallback so that anything that doesn’t matches whats configured in Caddy is reverse proxied to apache? If so I don’t know how to do such rule.

Putting my thoughts in order, maybe I could try this approach.

Yes, you could do that. Have Caddy terminate TLS, then proxy to Apache over HTTP. PHP will still work. Probably as simple as blocks like this for each domain:

foo.example.com {
	reverse_proxy localhost:8080
}

bar.example.com {
	reverse_proxy localhost:8080
}

Or if they’re always identical you can list out the domains for one block:

foo.example.com, bar.example.com {
	reverse_proxy localhost:8080
}

You’d have to reconfigure Apache to listen on port 8080 or whatever.

I mean, it’s two lines of config, so I would hope so. It’s very easy.

Problem is, there’s no way to do fallback for TLS, a valid certificate is necessary for the connection to succeed. Listing the domains in the config is how Caddy knows to issue and maintain TLS certs for that domain.

You could use On-Demand TLS, but that’s probably not what you want to do, that’s mainly for dealing with domains not under your control (i.e. customer’s domain pointing to your server).

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.