Difference between CLI and Caddyfile for reverse-proxy

Help
1

Hello!

1. The problem I’m having:

I’m using Caddy as a reverse proxy in front of an application serving HTTP and GRPC APIs.

I tried using the CLI, but GRPC didn’t work. I then tried to use a Caddyfile and it worked. What’s the difference between the two configuration methods?

2. Error messages and/or full log output:

With the CLI:

caddy-1    | {"level":"debug","ts":1730823380.2811732,"logger":"events","msg":"event","name":"tls_get_certificate","id":"170e4e6d-2193-4873-94e9-22037da0f9e4","origin":"tls","data":{"client_hello":{"CipherSuites":[49195,49199,49196,49200,52393,52392,4865,4866,4867],"ServerName":"localhost","SupportedCurves":[29,23,24,25],"SupportedPoints":"AA==","SignatureSchemes":[2052,1027,2055,2053,2054,1025,1281,1537,1283,1539,513,515],"SupportedProtos":["h2"],"SupportedVersions":[772,771],"RemoteAddr":{"IP":"10.89.0.32","Port":33628,"Zone":""},"LocalAddr":{"IP":"10.89.0.32","Port":8443,"Zone":""}}}}
caddy-1    | {"level":"debug","ts":1730823380.2814481,"logger":"tls.handshake","msg":"choosing certificate","identifier":"localhost","num_choices":1}
caddy-1    | {"level":"debug","ts":1730823380.2814696,"logger":"tls.handshake","msg":"default certificate selection results","identifier":"localhost","subjects":["localhost"],"managed":true,"issuer_key":"local","hash":"863ccab3bfda2b546430f0046f0486b1a5177add34621adfcd350aae5ee0d9f5"}
caddy-1    | {"level":"debug","ts":1730823380.2814777,"logger":"tls.handshake","msg":"matched certificate in cache","remote_ip":"10.89.0.32","remote_port":"33628","subjects":["localhost"],"managed":true,"expiration":1730866558,"hash":"863ccab3bfda2b546430f0046f0486b1a5177add34621adfcd350aae5ee0d9f5"}
caddy-1    | {"level":"debug","ts":1730823380.283846,"logger":"http.handlers.reverse_proxy","msg":"selected upstream","dial":"zitadel:8080","total_upstreams":1}
zitadel-1  | 2024/11/05 16:16:20 ERROR: Failed to extract ServerMetadata from context

With Caddyfile:

caddy-1    | {"level":"debug","ts":1730823667.8643844,"logger":"events","msg":"event","name":"tls_get_certificate","id":"ae770161-1da8-400c-9746-3fa5be50264d","origin":"tls","data":{"client_hello":{"CipherSuites":[49195,49199,49196,49200,52393,52392,4865,4866,4867],"ServerName":"localhost","SupportedCurves":[29,23,24,25],"SupportedPoints":"AA==","SignatureSchemes":[2052,1027,2055,2053,2054,1025,1281,1537,1283,1539,513,515],"SupportedProtos":["h2"],"SupportedVersions":[772,771],"RemoteAddr":{"IP":"10.89.0.41","Port":39032,"Zone":""},"LocalAddr":{"IP":"10.89.0.41","Port":8443,"Zone":""}}}}
caddy-1    | {"level":"debug","ts":1730823667.8644094,"logger":"tls.handshake","msg":"choosing certificate","identifier":"localhost","num_choices":1}
caddy-1    | {"level":"debug","ts":1730823667.8644223,"logger":"tls.handshake","msg":"default certificate selection results","identifier":"localhost","subjects":["localhost"],"managed":true,"issuer_key":"local","hash":"dbf33036f79f0b3085611b859715ffc4ecd15e6ce0f9f2fb72cecf63e782758c"}
caddy-1    | {"level":"debug","ts":1730823667.8644297,"logger":"tls.handshake","msg":"matched certificate in cache","remote_ip":"10.89.0.41","remote_port":"39032","subjects":["localhost"],"managed":true,"expiration":1730866753,"hash":"dbf33036f79f0b3085611b859715ffc4ecd15e6ce0f9f2fb72cecf63e782758c"}
caddy-1    | {"level":"debug","ts":1730823667.8665938,"logger":"http.handlers.reverse_proxy","msg":"selected upstream","dial":"zitadel:8080","total_upstreams":1}
caddy-1    | {"level":"debug","ts":1730823667.8687286,"logger":"http.handlers.reverse_proxy","msg":"upstream roundtrip","upstream":"zitadel:8080","duration":0.002089246,"request":{"remote_ip":"10.89.0.41","remote_port":"39032","client_ip":"10.89.0.41","proto":"HTTP/2.0","method":"POST","host":"localhost:8443","uri":"/grpc.reflection.v1.ServerReflection/ServerReflectionInfo","headers":{"Content-Type":["application/grpc"],"User-Agent":["grpcurl/v1.9.1 grpc-go/1.61.0"],"Te":["trailers"],"Grpc-Accept-Encoding":["gzip"],"X-Forwarded-For":["10.89.0.41"],"X-Forwarded-Proto":["https"],"X-Forwarded-Host":["localhost:8443"]},"tls":{"resumed":false,"version":772,"cipher_suite":4865,"proto":"h2","server_name":"localhost"}},"headers":{"Content-Type":["application/grpc"],"X-Robots-Tag":["none"]},"status":200}

3. Caddy version:

v2.8.4 h1:q3pe0wpBj1OcHFZ3n/1nl4V4bxBrYoSoab7rL9BMYNk=

4. How I installed and ran Caddy:

Podman Compose

a. System environment:

Linux (Fedora Silverblue 41), x86_64, Podman 5.2.5

b. Command:

podman compose up

c. Service/unit/compose file:

docker-compose.yaml:

version: '3.8'

services:
  caddy:
    image: "caddy:latest"
    volumes:
      - './Caddyfile:/etc/caddy/Caddyfile:ro,z'
    #command: 'caddy reverse-proxy --debug --access-log --to h2c://zitadel:8080 --from localhost:8443'
    ports:
      - "8443:8443"
    networks:
      - 'zitadel'

  zitadel:
    restart: 'always'
    networks:
      - 'zitadel'
    image: 'ghcr.io/zitadel/zitadel:latest'
    command: 'start-from-init --masterkey "MasterkeyNeedsToHave32Characters" --tlsMode disabled'
    environment:
      - 'ZITADEL_DATABASE_POSTGRES_HOST=db'
      - 'ZITADEL_DATABASE_POSTGRES_PORT=5432'
      - 'ZITADEL_DATABASE_POSTGRES_DATABASE=zitadel'
      - 'ZITADEL_DATABASE_POSTGRES_USER_USERNAME=zitadel'
      - 'ZITADEL_DATABASE_POSTGRES_USER_PASSWORD=zitadel'
      - 'ZITADEL_DATABASE_POSTGRES_USER_SSL_MODE=disable'
      - 'ZITADEL_DATABASE_POSTGRES_ADMIN_USERNAME=postgres'
      - 'ZITADEL_DATABASE_POSTGRES_ADMIN_PASSWORD=postgres'
      - 'ZITADEL_DATABASE_POSTGRES_ADMIN_SSL_MODE=disable'
      - 'ZITADEL_EXTERNALSECURE=false'
      - 'ZITADEL_FIRSTINSTANCE_MACHINEKEYPATH=/machinekey/zitadel-admin-sa.json'
      - 'ZITADEL_FIRSTINSTANCE_PATPATH=/machinekey/pat.json'
      - 'ZITADEL_FIRSTINSTANCE_ORG_MACHINE_MACHINE_USERNAME=zitadel-admin-sa'
      - 'ZITADEL_FIRSTINSTANCE_ORG_MACHINE_MACHINE_NAME=Admin'
      - 'ZITADEL_FIRSTINSTANCE_ORG_MACHINE_MACHINEKEY_TYPE=1'
      - 'ZITADEL_FIRSTINSTANCE_ORG_MACHINE_PAT_EXPIRATIONDATE=2025-01-01T00:00:00Z'
    depends_on:
      db:
        condition: 'service_healthy'
    ports:
      - '8080:8080'
    volumes:
      - ./machinekey:/machinekey:z

  db:
    restart: 'always'
    image: postgres:16-alpine
    environment:
      - POSTGRES_USER=postgres
      - POSTGRES_PASSWORD=postgres
    networks:
      - 'zitadel'
    healthcheck:
      test: ["CMD-SHELL", "pg_isready", "-d", "db_prod"]
      interval: '10s'
      timeout: '30s'
      retries: 5
      start_period: '20s'

networks:
  zitadel:

d. My complete Caddy config:

CLI:

caddy reverse-proxy --debug --access-log --to h2c://zitadel:8080 --from localhost:8443

Caddyfile:

{
	debug
}

localhost:8443 {
	reverse_proxy h2c://zitadel:8080
}

5. Links to relevant resources:

2

I don’t think CLI supports the h2c:// shortcut. That’s exclusive to the Caddyfile. As the docs here explain, it’s short for setting the versions on the HTTP transport.

PRs welcome to add support for that, I guess. But it can get complicated to introduce.

Either way, using a Caddyfile is strongly recommended. Since you’re running Caddy in Docker anyway, you should prefer a config file. Much more power and control. CLI is only meant for quick-and-dirty local servers when developing and testing something, not for production usecases.

1 Like
3

Hi @francislavoie,

The problem is that my context makes it hard to pass a Caddyfile to the Caddy container, whereas I can easily use command line or environment variables.

Anyway, thanks for your answer.

4

Thanks a lot for explain.

5

I don’t understand. Does this make it infeasible to supply a bind mounted file?

If so, have you considered using a config in your Compose file itself? e.g.:

configs:
  Caddyfile:
    content: |
      example.com {
        # ...
      }
services:
  caddy:
    image: caddy:latest
    configs:
      - source: Caddyfile
        target: /etc/caddy/Caddyfile
1 Like
6

Hi, I was using AWS ECS, a container orchestration service which does not easily allow to mount a file in a container. There is no Compose file in this context. You can customize the command executed in your container, set environment variables, but not mount files without workarounds: [ECS] [Volumes]: Ability to create config "volume" and mount it into container as file · Issue #56 · aws/containers-roadmap · GitHub, amazon-ecs-configmaps-example/README.md at main · aws-samples/amazon-ecs-configmaps-example · GitHub. Every time I tried to use a simpler container orchestration solution, I always ended up missing Kubernetes features unfortunately.

7

Hmm!

I imagine you’ve probably crawled over a lot of possible solutions already and might’ve considered this already, but…

I wonder if you could host a very small static server in a VPC, configuring your ECS startup command to run a bash script to curl the Caddyfile from the static server into the container and then run Caddy on it?

Hacky but I imagine it would work.

Heck, you could strip all the private stuff from it and import it as env vars, posting the Caddyfile to Github and curling it raw from there on container startup.

8

Also, you can do something like echo '<your-Caddyfile-contents>' | caddy run -a caddyfile -c- (where -c- is short for --config - where - means take the config file from stdin).