Caddy2 + docker-compose + Amazon Elastic Beanstalk

1. Caddy version (caddy version):

2.4.6

2. How I run Caddy:

I run Caddy in my docker-compose.yml using a caddybuilder, so that I can use the caddy-dns/route53 and caddy-tlsredis plugins.

Here’s my docker-compose.yml caddy section:

  caddy:
    build: 
      context: ./caddy
      dockerfile: ./Dockerfile
    restart: unless-stopped
    command: caddy run --config /dockerapp/caddy/Caddyfile
    ports:
      - 80:80
      - 443:443
    volumes_from:
      - web
    volumes:
      - ./caddy/Caddyfile:/etc/caddy/Caddyfile
      - caddy_data:/data
      - caddy_config:/config
    networks:
      - web-net
    environment:
      - AWS_ACCESS_KEY_ID=${AWS_SECRET_ACCESS_KEY}
      - AWS_SECRET_ACCESS_KEY=${AWS_SECRET_ACCESS_KEY}

and here’s the Dockerfile used to build Caddy in docker-compose.yml:

FROM caddy:builder AS builder
RUN xcaddy build \
	--with github.com/caddy-dns/route53 \
	--with github.com/gamalan/caddy-tlsredis@cb6e2c46bea8a6cafdcba76770be2a6db1200396

FROM caddy:latest
COPY --from=builder /usr/bin/caddy /usr/bin/caddy

a. System environment:

docker-compose running on AWS Elastic Beanstalk.

b. Command:

caddy run --config /dockerapp/caddy/Caddyfile

c. Service/unit/compose file:

docker-compose.yml caddy section posted above.

d. My complete Caddyfile or JSON config:

{
  on_demand_tls {
    ask 0.0.0.0:3000/caddy-domain-check
    interval 2m
    burst 5
  }
  debug
}

(SecurityHeaders) {
  header_up X-Real-IP {remote_host}
  header_up X-Forwarded-Proto {scheme}
}

my-example.com, *.my-example.com {
    reverse_proxy 0.0.0.0:3000 {
      header_up X-Forwarded-For {remote_host}
      header_up X-Forwarded-Port {server_port}
      header_up X-Forwarded-Proto {scheme}
    }
    file_server
    log {
        format single_field common_log
        output file /data/logs/caddy.log
    }
    tls me@my-example.com{
        dns route53
    }
}
:443, :80 {
    tls {
        on_demand
    }
    tls internal {
        on_demand
    }
}

3. The problem I’m having:

curl -i https://my-example.com responds:

HTTP/1.1 400 Bad Request
transfer-encoding: chunked
Connection: keep-alive

Client sent an HTTP request to an HTTPS server.

Because my migration involves a switch to Caddy, docker-compose, and a new AWS Elastic Beanstalk environment, I’m having trouble tracking down what is to blame for the above issue. A couple things I’ve noticed:

  1. If I remove the listener on 443 from my Elastic Beanstalk load balancer, the https request times out instead of returning “Client sent an HTTP request to an HTTPS server.”

  2. The ssl certificate on this page is valid, but it’s issued by Amazon (via the certificate attached to my load balancer) rather than Caddy.

4. Error messages and/or full log output:

Caddy logs from startup until it finishes loading. Every line logged after this is a variation of the last line’s TLS handshake error

caddy_1    | {"level":"info","ts":1639782873.55904,"msg":"using provided configuration","config_file":"/dockerapp/caddy/Caddyfile","config_adapter":""}
caddy_1    | {"level":"warn","ts":1639782873.560051,"msg":"input is not formatted with 'caddy fmt'","adapter":"caddyfile","file":"/dockerapp/caddy/Caddyfile","line":11}
caddy_1    | {"level":"info","ts":1639782873.5652137,"logger":"admin","msg":"admin endpoint started","address":"tcp/localhost:2019","enforce_origin":false,"origins":["localhost:2019","[::1]:2019","127.0.0.1:2019"]}
caddy_1    | {"level":"info","ts":1639782873.5655,"logger":"tls.cache.maintenance","msg":"started background certificate maintenance","cache":"0xc000465f10"}
caddy_1    | {"level":"info","ts":1639782873.5658686,"logger":"http","msg":"server is listening only on the HTTPS port but has no TLS connection policies; adding one to enable TLS","server_name":"srv0","https_port":443}
caddy_1    | {"level":"info","ts":1639782873.5658894,"logger":"http","msg":"enabling automatic HTTP->HTTPS redirects","server_name":"srv0"}
caddy_1    | {"level":"info","ts":1639782873.565914,"logger":"http","msg":"server is listening only on the HTTP port, so no automatic HTTPS will be applied to this server","server_name":"srv1","http_port":80}
caddy_1    | {"level":"warn","ts":1639782873.627143,"logger":"pki.ca.local","msg":"installing root certificate (you might be prompted for password)","path":"storage:pki/authorities/local/root.crt"}
caddy_1    | 2021/12/17 23:14:33 define JAVA_HOME environment variable to use the Java trust
caddy_1    | 2021/12/17 23:14:33 Warning: "certutil" is not available, install "certutil" with "apt install libnss3-tools" or "yum install nss-tools" and try again
caddy_1    | 2021/12/17 23:14:33 certificate installed properly in linux trusts
caddy_1    | {"level":"info","ts":1639782873.7765002,"logger":"tls","msg":"cleaning storage unit","description":"FileStorage:/data/caddy"}
caddy_1    | {"level":"info","ts":1639782873.777416,"logger":"tls","msg":"finished cleaning storage units"}
caddy_1    | {"level":"debug","ts":1639782873.7806373,"logger":"http","msg":"starting server loop","address":"[::]:443","http3":false,"tls":true}
caddy_1    | {"level":"debug","ts":1639782873.7807045,"logger":"http","msg":"starting server loop","address":"[::]:80","http3":false,"tls":false}
caddy_1    | {"level":"info","ts":1639782873.780712,"logger":"http","msg":"enabling automatic TLS certificate management","domains":["my-example.com","*.my-example.com"]}
caddy_1    | {"level":"debug","ts":1639782873.7811437,"logger":"tls","msg":"loading managed certificate","domain":"my-example.com","expiration":1647561599,"issuer_key":"acme.zerossl.com-v2-DV90","storage":"FileStorage:/data/caddy"}
caddy_1    | {"level":"debug","ts":1639782873.7816758,"logger":"tls.cache","msg":"added certificate to cache","subjects":["my-example.com"],"expiration":1647561599,"managed":true,"issuer_key":"acme.zerossl.com-v2-DV90","hash":"16f3f4feea021d4cb16fe91d99c8b34fa65b6c5bb59b206838360b4335523d9b","cache_size":1,"cache_capacity":10000}
caddy_1    | {"level":"debug","ts":1639782873.7819345,"logger":"tls","msg":"loading managed certificate","domain":"*.my-example.com","expiration":1647542310,"issuer_key":"acme-v02.api.letsencrypt.org-directory","storage":"FileStorage:/data/caddy"}
caddy_1    | {"level":"debug","ts":1639782873.7821574,"logger":"tls.cache","msg":"added certificate to cache","subjects":["*.my-example.com"],"expiration":1647542310,"managed":true,"issuer_key":"acme-v02.api.letsencrypt.org-directory","hash":"54ef9bff38b059e94bbfdd3a2805f6c38fbcb015ce96ddc7fd3940f3b691b4ae","cache_size":2,"cache_capacity":10000}
caddy_1    | {"level":"info","ts":1639782873.782314,"msg":"autosaved config (load with --resume flag)","file":"/config/caddy/autosave.json"}
caddy_1    | {"level":"info","ts":1639782873.7823212,"msg":"serving initial configuration"}
caddy_1    | {"level":"debug","ts":1639782933.7712696,"logger":"http.stdlib","msg":"http: TLS handshake error from 172.31.17.51:21498: EOF"}

5. What I already tried:

In order to enable simple custom domains w/ SSL on my Rails SaaS app, I’m migrating to Caddy from nginx. I’m also having to migrate to a new docker-compose production environment on AWS Elastic Beanstalk at the same time.

I’ve tried countless iterations of the above Caddyfile and docker-compose.yml to arrive at what I have now. I suspect the problem has something to do with the docker-compose + Caddy configuration, but I feel like I’m at a dead end in terms of trying to understand this issue better by googling.

6. Links to relevant resources:

I’m confused – you’re mounting the Caddyfile at /etc/caddy/Caddyfile, but you override the command to use a Caddyfile at /dockerapp/caddy/Caddyfile instead. Are you sure you need that command line? The default CMD in the Docker image will run Caddy from the Caddyfile in /etc/caddy/Caddyfile.

You don’t need any of these lines. Remove them. Caddy already sets these headers appropriately:

It doesn’t make sense to enable a file_server when you’re using reverse_proxy. Only one of them will be used (in this case reverse_proxy because of the directive order), unless you use request matchers to tell Caddy which requests should use one or the other.

The common_log field is deprecated, and will be removed in v2.5.0. You may use the GitHub - caddyserver/format-encoder: Log encoder module for custom log formats plugin instead.

You can’t configure tls twice for one site. What are you trying to do here?

That likely means AWS EB sent an HTTP request to Caddy on port 443.

Is there a TCP mode for EB? Basically, you want to let requests go through to Caddy without TLS being terminated, otherwise you won’t be able to make use of Caddy’s TLS functionality.

Thank you!

Remove my HTTPS listener on port 443 from EB configuration and creating a new TCP listener on port 443 fixed it for me.

1 Like

This topic was automatically closed after 30 days. New replies are no longer allowed.