Caddy behind caddy proxy

1. Caddy version (caddy version):

v2.3.0

2. How I run Caddy:

I run Caddy with docker, both Caddys.

a. System environment:

Docker

b. Command:

does not apply

c. Service/unit/compose file:

  1. Caddy, proxy server
services:
  caddy:
    image: caddy/caddy:alpine
    volumes:
      - /opt/docker-common/config/caddy/Caddyfile:/etc/caddy/Caddyfile
      - /opt/docker-common/config/caddy/shared:/etc/caddy/shared
      - /opt/docker-common/data/caddy:/data/caddy
      - /opt/docker-common/logs/caddy:/var/log/caddy
    ports:
      - "XXX.XXX.XXX.XXX:80:80"
      - "XXX.XXX.XXX.XXX:443:443"
    networks:
      - frontend
      - backend
    restart: always

networks:
  frontend:
  backend:
  1. Caddy which serves as the http server
services:
  caddy:
    image: caddy/caddy:alpine
    volumes:
      - /opt/docker-hosting/config/caddy/Caddyfile:/etc/caddy/Caddyfile
      - /opt/docker-hosting/config/caddy/shared:/etc/caddy/shared
      - /opt/docker-hosting/data/caddy:/data/caddy
      - /opt/docker-hosting/www:/srv
      - /opt/docker-hosting/logs/caddy:/var/log/caddy
    expose:
      - 80
    networks:
      - backend
    restart: always

networks:
  backend:
    external:
      name: docker-common_backend

d. My complete Caddyfile or JSON config:

  1. Caddy as a proxy server
portainer.domain.com {
  reverse_proxy docker-common_portainer_1:9000
}

:80, :443 {
  reverse_proxy docker-hosting_caddy_1:80 {
    # I found the below 2 lines in the forum
    header_up Host {http.reverse_proxy.upstream.hostport}
    header_up X-Forwarded-Host {host}
  }
}

  1. Caddy behind the other Caddy
{
  auto_https off
}

anotherdomain.com {
  root * /srv/anotherdomain.com
  file_server
}

3. The problem I’m having:

I am obivously missing something here so sorry for maybe a duplicate question…

I need to use Cady behind Caddy, proxy from one to another. I am sure I found something in the forum but nothing seems to work reliably.

The idea is to have one Caddy exposed on ports 80 and 443. This first Caddy will run as a proxy server and will take care of ACME requests. I am now testing with portainer and then trying to use “catch-all” to redirect all the rest to the other Caddy instance.

It works okay with portainer, it loads fine but nothing gets forwarded to the other instance of Caddy. I get “ERR_SSL_PROTOCOL_ERROR” and then 502 error.

I assume it has something to do with passing the Host but again, I see no clue in the logs at all.

4. Error messages and/or full log output:

  1. Caddy, exposed proxy server
caddy_1      | {"level":"error","ts":1615978305.2401366,"logger":"http.log.error","msg":"dial tcp 172.19.0.5:80: connect: connection refused","request":{"remote_addr":"XXX.XXX.XXX.XXX:56062","proto":"HTTP/1.1","method":"GET","host":"domain.com","uri":"/index.html","headers":{"User-Agent":["Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/88.0.4324.186 Safari/537.36"],"Accept-Encoding":["gzip, deflate"],"Accept-Language":["cs-CZ,cs;q=0.9,en-US;q=0.8,en;q=0.7"],"If-Modified-Since":["Tue, 16 Mar 2021 23:12:14 GMT"],"Dnt":["1"],"Accept":["text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.9"],"If-None-Match":["\"qq34ge8\""],"Connection":["keep-alive"],"Upgrade-Insecure-Requests":["1"]}},"duration":0.006175518,"status":502,"err_id":"ut93ajtu1","err_trace":"reverseproxy.statusError (reverseproxy.go:783)"}
  1. Caddy behind caddy (the request does not even arrive)
caddy_1  | {"level":"info","ts":1615975795.8612862,"logger":"admin","msg":"stopped previous server"}
caddy_1  | {"level":"info","ts":1615975795.8657508,"msg":"shutdown done","signal":"SIGTERM"}
caddy_1  | {"level":"info","ts":1615975798.3833447,"msg":"using provided configuration","config_file":"/etc/caddy/Caddyfile","config_adapter":"caddyfile"}
caddy_1  | {"level":"info","ts":1615975798.3958807,"logger":"admin","msg":"admin endpoint started","address":"tcp/localhost:2019","enforce_origin":false,"origins":["localhost:2019","[::1]:2019","127.0.0.1:2019"]}
caddy_1  | {"level":"info","ts":1615975798.3964393,"logger":"tls.cache.maintenance","msg":"started background certificate maintenance","cache":"0xc00041c540"}
caddy_1  | {"level":"info","ts":1615975798.397503,"msg":"autosaved config","file":"/config/caddy/autosave.json"}
caddy_1  | {"level":"info","ts":1615975798.3975322,"msg":"serving initial configuration"}
caddy_1  | {"level":"info","ts":1615975798.398879,"logger":"tls","msg":"cleaned up storage units"}

5. What I already tried:

I searched forum, online via search engines and also used my previous experience with proxy in Caddy. I also looked in the docs.

6. Links to relevant resources:

does not apply

I tried something similar like in this case but catch-all simply does not work.

When i access portainer on port 80, although it is on top of Caddyfile, it get overriden by the the catch-all part which tells Caddy to forward port :80 to the other Caddy.

This post is also related, as I see it.

You should use caddy:2.3.0-alpine, not caddy/caddy. The latter is not the official image, but rather our CI artifacts.

You should persist /data, not /data/caddy. See the docs on Docker Hub

I think you probably want just http://anotherdomain.com, then you can skip auto_https off. Or if you only have a single thing to serve in that instance, use :80 as your site label instead.

You didn’t specify a domain for Caddy to serve here, so it won’t have issued a certificate. What are you trying to do exactly?

1 Like

You are right, did not know that though. So this is fixed now.

Okay, true. Fixed.

This is the second Caddy, behind proxy. It should accept what is forwarded to it by the first Caddy proxy server. As I was digging into it even more, I should probably have this second Caddy make the certificates request as that is the only place where the actualy domains are listed. It is not meant to be used only by a single domain though.

As I said, you were right - there is no domain, hence Caddy cannot ask for a certificate. It should be the Caddy behind proxy quering for the certificates.


So to describe the flow even more clear, let me try to put it this way:

public IP (port 80/443) → Caddy (proxy server) →
→ if there is a record in the Caddyfile to send to another container (like portainer), do so
→ if there is any other record, forward it
→ if no condition was met, use catch-all forward → hosted by the second Caddy, behind proxy

This second Caddy then needs to query for certificates and host the websites that are in its Caddyfile.

Maybe I am complicating it too much, I don’t know. Should I make a more precise drawing?

That’s not really possible with vanilla Caddy. Caddy is an HTTP server, it can’t proxy raw TCP by default. You would need to use GitHub - mholt/caddy-l4: Layer 4 (TCP/UDP) app for Caddy, a plugin, to do that. But I wouldn’t recommend it here, it would only needlessly complicate things further.

You should let your first Caddy terminate TLS, then proxy to your 2nd over HTTP.

1 Like

Oh, thanks for the information.

I tried to use Traefik just before but I could not do it either. It does TCP and UDP, but Caddy does not seem to care about the headers. I had it pass the headers over to Caddy. Obviously it would receive it, but with Host empty, while X-Forwarded-Host was properly filled. Does Caddy not parse this? I found something like that in the forum.

This line will set the Host to docker-hosting_caddy_1

This line will set X-Forwarded-Host to whatever was originally in the request.

If you omit the header_up line for Host, then the Host will be passed through transparently by default.

This matters because your 2nd Caddy instance will be looking at the Host to match, and since you’re passing docker-hosting_caddy_1, that won’t match anotherdomain.com.

Alright, thank you very much for explanation and help.

As it stands, the best way is not to complicate it, so one Caddy - for proxy and websites. All that is needed, no extra hopping, no extra header issues … I forgot about the KISS concept.

1 Like

Definitely best to keep it simple here. :+1:

This topic was automatically closed after 30 days. New replies are no longer allowed.