Caddy behind caddy proxy

[kunago (Miroslav Abrahám)]

1. Caddy version (caddy version):

v2.3.0

2. How I run Caddy:

I run Caddy with docker, both Caddys.

a. System environment:

Docker

b. Command:

does not apply

c. Service/unit/compose file:

1. Caddy, proxy server

services:
  caddy:
    image: caddy/caddy:alpine
    volumes:
      - /opt/docker-common/config/caddy/Caddyfile:/etc/caddy/Caddyfile
      - /opt/docker-common/config/caddy/shared:/etc/caddy/shared
      - /opt/docker-common/data/caddy:/data/caddy
      - /opt/docker-common/logs/caddy:/var/log/caddy
    ports:
      - "XXX.XXX.XXX.XXX:80:80"
      - "XXX.XXX.XXX.XXX:443:443"
    networks:
      - frontend
      - backend
    restart: always

networks:
  frontend:
  backend:

2. Caddy which serves as the http server

services:
  caddy:
    image: caddy/caddy:alpine
    volumes:
      - /opt/docker-hosting/config/caddy/Caddyfile:/etc/caddy/Caddyfile
      - /opt/docker-hosting/config/caddy/shared:/etc/caddy/shared
      - /opt/docker-hosting/data/caddy:/data/caddy
      - /opt/docker-hosting/www:/srv
      - /opt/docker-hosting/logs/caddy:/var/log/caddy
    expose:
      - 80
    networks:
      - backend
    restart: always

networks:
  backend:
    external:
      name: docker-common_backend

d. My complete Caddyfile or JSON config:

1. Caddy as a proxy server

portainer.domain.com {
  reverse_proxy docker-common_portainer_1:9000
}

:80, :443 {
  reverse_proxy docker-hosting_caddy_1:80 {
    # I found the below 2 lines in the forum
    header_up Host {http.reverse_proxy.upstream.hostport}
    header_up X-Forwarded-Host {host}
  }
}

2. Caddy behind the other Caddy

{
  auto_https off
}

anotherdomain.com {
  root * /srv/anotherdomain.com
  file_server
}

3. The problem I’m having:

I am obivously missing something here so sorry for maybe a duplicate question…

I need to use Cady behind Caddy, proxy from one to another. I am sure I found something in the forum but nothing seems to work reliably.

The idea is to have one Caddy exposed on ports 80 and 443. This first Caddy will run as a proxy server and will take care of ACME requests. I am now testing with portainer and then trying to use “catch-all” to redirect all the rest to the other Caddy instance.

It works okay with portainer, it loads fine but nothing gets forwarded to the other instance of Caddy. I get “ERR_SSL_PROTOCOL_ERROR” and then 502 error.

I assume it has something to do with passing the Host but again, I see no clue in the logs at all.

4. Error messages and/or full log output:

1. Caddy, exposed proxy server

caddy_1      | {"level":"error","ts":1615978305.2401366,"logger":"http.log.error","msg":"dial tcp 172.19.0.5:80: connect: connection refused","request":{"remote_addr":"XXX.XXX.XXX.XXX:56062","proto":"HTTP/1.1","method":"GET","host":"domain.com","uri":"/index.html","headers":{"User-Agent":["Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/88.0.4324.186 Safari/537.36"],"Accept-Encoding":["gzip, deflate"],"Accept-Language":["cs-CZ,cs;q=0.9,en-US;q=0.8,en;q=0.7"],"If-Modified-Since":["Tue, 16 Mar 2021 23:12:14 GMT"],"Dnt":["1"],"Accept":["text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.9"],"If-None-Match":["\"qq34ge8\""],"Connection":["keep-alive"],"Upgrade-Insecure-Requests":["1"]}},"duration":0.006175518,"status":502,"err_id":"ut93ajtu1","err_trace":"reverseproxy.statusError (reverseproxy.go:783)"}

2. Caddy behind caddy (the request does not even arrive)

caddy_1  | {"level":"info","ts":1615975795.8612862,"logger":"admin","msg":"stopped previous server"}
caddy_1  | {"level":"info","ts":1615975795.8657508,"msg":"shutdown done","signal":"SIGTERM"}
caddy_1  | {"level":"info","ts":1615975798.3833447,"msg":"using provided configuration","config_file":"/etc/caddy/Caddyfile","config_adapter":"caddyfile"}
caddy_1  | {"level":"info","ts":1615975798.3958807,"logger":"admin","msg":"admin endpoint started","address":"tcp/localhost:2019","enforce_origin":false,"origins":["localhost:2019","[::1]:2019","127.0.0.1:2019"]}
caddy_1  | {"level":"info","ts":1615975798.3964393,"logger":"tls.cache.maintenance","msg":"started background certificate maintenance","cache":"0xc00041c540"}
caddy_1  | {"level":"info","ts":1615975798.397503,"msg":"autosaved config","file":"/config/caddy/autosave.json"}
caddy_1  | {"level":"info","ts":1615975798.3975322,"msg":"serving initial configuration"}
caddy_1  | {"level":"info","ts":1615975798.398879,"logger":"tls","msg":"cleaned up storage units"}

5. What I already tried:

I searched forum, online via search engines and also used my previous experience with proxy in Caddy. I also looked in the docs.

6. Links to relevant resources:

does not apply

[kunago (Miroslav Abrahám)]

I tried something similar like in this case but catch-all simply does not work.

Catch-all configuration for unlimited custom domains?

Thanks for getting back to me. I have changed my config to this: :80,:443 { reverse_proxy https://myapp.herokuapp.com { header_up Host {http.reverse_proxy.upstream.hostport} header_up X-Real-IP {http.reverse-proxy.upstream.address} header_up X-Forwarded-Port {http.request.port} header_up X-Forwarded-Host {http.request.host} } tls me@mydomain.com { on_demand } } I have set the A record of my custom domain (let’s say mydomain.net) to point to the IP…

When i access portainer on port 80, although it is on top of Caddyfile, it get overriden by the the catch-all part which tells Caddy to forward port :80 to the other Caddy.

[kunago (Miroslav Abrahám)]

This post is also related, as I see it.

What is the order of server parsing

1. Caddy version (caddy version): 2.1 2. How I run Caddy: CaddyFile a. System environment: Ubuntu 16.04 3. The problem I’m having: I need to resolve some uncertain domain to reverse proxy(no ssl required), the CaddyFile like this: *.*.example.com:80 { encode zstd gzip templates file_server try_files {path} /index.html root * /web/{http.request.host.labels.2}/dist/{http.request.host.labels.3} } :80 { @extracthost { not header_regexp Host ^.*example.*$ } reverse_proxy …

[francislavoie]

@kunago

image: caddy/caddy:alpine

You should use caddy:2.3.0-alpine, not caddy/caddy. The latter is not the official image, but rather our CI artifacts.

@kunago

- /opt/docker-common/data/caddy:/data/caddy

You should persist /data, not /data/caddy. See the docs on Docker

@kunago

2. Caddy behind the other Caddy

{
  auto_https off
}

anotherdomain.com {
  root * /srv/anotherdomain.com
  file_server
}

I think you probably want just http://anotherdomain.com, then you can skip auto_https off. Or if you only have a single thing to serve in that instance, use :80 as your site label instead.

@kunago

:80, :443 {
  reverse_proxy docker-hosting_caddy_1:80 {
    # I found the below 2 lines in the forum
    header_up Host {http.reverse_proxy.upstream.hostport}
    header_up X-Forwarded-Host {host}
  }
}

You didn’t specify a domain for Caddy to serve here, so it won’t have issued a certificate. What are you trying to do exactly?

[kunago (Miroslav Abrahám)]

@francislavoie

You should use caddy:2.3.0-alpine, not caddy/caddy. The latter is not the official image, but rather our CI artifacts.

You are right, did not know that though. So this is fixed now.

@francislavoie

You should persist /data, not /data/caddy. See the docs on Docker Hub

Okay, true. Fixed.

@francislavoie

I think you probably want just http://anotherdomain.com , then you can skip auto_https off . Or if you only have a single thing to serve in that instance, use :80 as your site label instead.

This is the second Caddy, behind proxy. It should accept what is forwarded to it by the first Caddy proxy server. As I was digging into it even more, I should probably have this second Caddy make the certificates request as that is the only place where the actualy domains are listed. It is not meant to be used only by a single domain though.

@francislavoie

You didn’t specify a domain for Caddy to serve here, so it won’t have issued a certificate. What are you trying to do exactly?

As I said, you were right - there is no domain, hence Caddy cannot ask for a certificate. It should be the Caddy behind proxy quering for the certificates.

---

So to describe the flow even more clear, let me try to put it this way:

public IP (port 80/443) → Caddy (proxy server) →
→ if there is a record in the Caddyfile to send to another container (like portainer), do so
→ if there is any other record, forward it
→ if no condition was met, use catch-all forward → hosted by the second Caddy, behind proxy

This second Caddy then needs to query for certificates and host the websites that are in its Caddyfile.

Maybe I am complicating it too much, I don’t know. Should I make a more precise drawing?

[francislavoie]

@kunago

As I said, you were right - there is no domain, hence Caddy cannot ask for a certificate. It should be the Caddy behind proxy quering for the certificates.

That’s not really possible with vanilla Caddy. Caddy is an HTTP server, it can’t proxy raw TCP by default. You would need to use GitHub - mholt/caddy-l4: Layer 4 (TCP/UDP) app for Caddy, a plugin, to do that. But I wouldn’t recommend it here, it would only needlessly complicate things further.

You should let your first Caddy terminate TLS, then proxy to your 2nd over HTTP.

[kunago (Miroslav Abrahám)]

Oh, thanks for the information.

I tried to use Traefik just before but I could not do it either. It does TCP and UDP, but Caddy does not seem to care about the headers. I had it pass the headers over to Caddy. Obviously it would receive it, but with Host empty, while X-Forwarded-Host was properly filled. Does Caddy not parse this? I found something like that in the forum.

[francislavoie]

@kunago

header_up Host {http.reverse_proxy.upstream.hostport}

This line will set the Host to docker-hosting_caddy_1

@kunago

header_up X-Forwarded-Host {host}

This line will set X-Forwarded-Host to whatever was originally in the request.

If you omit the header_up line for Host, then the Host will be passed through transparently by default.

reverse_proxy (Caddyfile directive) - Caddy Documentation

This matters because your 2nd Caddy instance will be looking at the Host to match, and since you’re passing docker-hosting_caddy_1, that won’t match anotherdomain.com.

[kunago (Miroslav Abrahám)]

Alright, thank you very much for explanation and help.

As it stands, the best way is not to complicate it, so one Caddy - for proxy and websites. All that is needed, no extra hopping, no extra header issues … I forgot about the KISS concept.

[francislavoie]

Definitely best to keep it simple here.