Proxying caddy to caddy in docker network

Help
1

1. The problem I’m having:

I am moving from nginx to caddy and I was trying to set up caddy (on host) → caddy container (docker compose network) → service (same docker compose network) setup.

I have to use this setup because some containers in docker compose network expect a valid (non-self-signed) certificate so I aliased caddy as internal docker network proxy between services. I mount certificates into docker volume /fullchain.pem and /key.pem so that works fine. It is probably irrelevant but cert is a wildcard and is obtained from letsencrypt with certbot and mounted in docker. Same certificate is used for both proxies.

I tested both:
curl → #1 proxy → dummy python service for logging headers
curl → #2 proxy → dummy python service for logging headers
individually, and I got same request headers logged, meaning same, correct Host: service.my_domain.com with command
curl -v --connect-to service.my_domain.com:443:localhost:443 https://service.my_domain.com/

2. Error messages and/or full log output:

This is debug log from first proxy (intended to reach service through second proxy):

2023/05/10 10:25:35.372 DEBUG   events  event   {"name": "tls_get_certificate", "id": "0507ca7e-db9c-40e1-bb10-efaff50c5cf5", "origin": "tls", "data": {"client_hello":{"CipherSuites":[4866,4867,4865,49200,49196,49192,49188,49172,49162,159,107,57,52393,52392,52394,65413,196,136,129,157,61,53,192,132,49199,49195,49191,49187,49171,49161,158,103,51,190,69,156,60,47,186,65,49169,49159,5,4,49170,49160,22,10,255],"ServerName":"service.my_domain.com","SupportedCurves":[29,23,24,25],"SupportedPoints":"AA==","SignatureSchemes":[2054,1537,1539,2053,1281,1283,2052,1025,1027,513,515],"SupportedProtos":["h2","http/1.1"],"SupportedVersions":[772,771,770,769],"Conn":{}}}}
2023/05/10 10:25:35.372 DEBUG   tls.handshake   no matching certificate; will choose from all certificates       {"identifier": "service.my_domain.com"}
2023/05/10 10:25:35.372 DEBUG   tls.handshake   choosing certificate    {"identifier": "service.my_domain.com", "num_choices": 1}
2023/05/10 10:25:35.372 DEBUG   tls.handshake   custom certificate selection results    {"identifier": "service.my_domain.com", "subjects": ["*.my_domain.com", "my_domain.com"], "managed": false, "issuer_key": "", "hash": "5c5ed19af54ff7cc75972bf2d869dfc9266d9c20f594cc12521327d629d173cd"}
2023/05/10 10:25:35.372 DEBUG   tls.handshake   matched certificate in cache    {"remote_ip": "127.0.0.1", "remote_port": "62269", "subjects": ["*.my_domain.com", "my_domain.com"], "managed": false, "expiration": "2023/06/25 05:49:16.000", "hash": "5c5ed19af54ff7cc75972bf2d869dfc9266d9c20f594cc12521327d629d173cd"}
2023/05/10 10:25:35.379 DEBUG   http.handlers.reverse_proxy     selected upstream       {"dial": "localhost:4443", "total_upstreams": 1}
2023/05/10 10:25:35.383 DEBUG   http.handlers.reverse_proxy     upstream roundtrip      {"upstream": "localhost:4443", "duration": 0.003564555, "request": {"remote_ip": "127.0.0.1", "remote_port": "62269", "proto": "HTTP/2.0", "method": "GET", "host": "service.my_domain.com", "uri": "/", "headers": {"User-Agent": ["curl/7.79.1"], "Accept": ["*/*"], "X-Forwarded-For": ["127.0.0.1"], "X-Forwarded-Proto": ["https"], "X-Forwarded-Host": ["service.my_domain.com"]}, "tls": {"resumed": false, "version": 772, "cipher_suite": 4865, "proto": "h2", "server_name": "service.my_domain.com"}}, "error": "remote error: tls: internal error"}
2023/05/10 10:25:35.383 ERROR   http.log.error  remote error: tls: internal error       {"request": {"remote_ip": "127.0.0.1", "remote_port": "62269", "proto": "HTTP/2.0", "method": "GET", "host": "service.my_domain.com", "uri": "/", "headers": {"User-Agent": ["curl/7.79.1"], "Accept": ["*/*"]}, "tls": {"resumed": false, "version": 772, "cipher_suite": 4865, "proto": "h2", "server_name": "service.my_domain.com"}}, "duration": 0.004004207, "status": 502, "err_id": "mxsd6t2xe", "err_trace": "reverseproxy.statusError (reverseproxy.go:1299)"}

And here is the output from second proxy’s debug log:

2023/05/10 10:26:04.800 DEBUG   events  event   {"name": "tls_get_certificate", "id": "db35f1b4-102d-464c-bf6c-0f66a0b14d4a", "origin": "tls", "data": {"client_hello":{"CipherSuites":[49195,49199,49196,49200,52393,52392,49161,49171,49162,49172,156,157,47,53,49170,10,4865,4866,4867],"ServerName":"localhost","SupportedCurves":[29,23,24,25],"SupportedPoints":"AA==","SignatureSchemes":[2052,1027,2055,2053,2054,1025,1281,1537,1283,1539,513,515],"SupportedProtos":["h2","http/1.1"],"SupportedVersions":[772,771],"Conn":{}}}}
2023/05/10 10:26:04.800 DEBUG   tls.handshake   no matching certificates and no custom selection logic  {"identifier": "localhost"}
2023/05/10 10:26:04.800 DEBUG   tls.handshake   no matching certificates and no custom selection logic  {"identifier": "*"}
2023/05/10 10:26:04.800 DEBUG   tls.handshake   all external certificate managers yielded no certificates and no errors  {"remote_ip": "::1", "remote_port": "62274", "sni": "localhost"}
2023/05/10 10:26:04.800 DEBUG   tls.handshake   no certificate matching TLS ClientHello {"remote_ip": "::1", "remote_port": "62274", "server_name": "localhost", "remote": "[::1]:62274", "identifier": "localhost", "cipher_suites": [49195, 49199, 49196, 49200, 52393, 52392, 49161, 49171, 49162, 49172, 156, 157, 47, 53, 49170, 10, 4865, 4866, 4867], "cert_cache_fill": 0.0001, "load_if_necessary": true, "obtain_if_necessary": true, "on_demand": false}
2023/05/10 10:26:04.800 DEBUG   http.stdlib     http: TLS handshake error from [::1]:62274: no certificate available for 'localhost'
2023/05/10 10:27:25.845 DEBUG   events  event   {"name": "tls_get_certificate", "id": "9231f569-93a8-4a23-afe2-fbec02cabe17", "origin": "tls", "data": {"client_hello":{"CipherSuites":[4866,4867,4865,49200,49196,49192,49188,49172,49162,159,107,57,52393,52392,52394,65413,196,136,129,157,61,53,192,132,49199,49195,49191,49187,49171,49161,158,103,51,190,69,156,60,47,186,65,49169,49159,5,4,49170,49160,22,10,255],"ServerName":"service.my_domain.com","SupportedCurves":[29,23,24,25],"SupportedPoints":"AA==","SignatureSchemes":[2054,1537,1539,2053,1281,1283,2052,1025,1027,513,515],"SupportedProtos":["h2","http/1.1"],"SupportedVersions":[772,771,770,769],"Conn":{}}}}
2023/05/10 10:27:25.845 DEBUG   tls.handshake   no matching certificate; will choose from all certificates       {"identifier": "service.my_domain.com"}
2023/05/10 10:27:25.845 DEBUG   tls.handshake   choosing certificate    {"identifier": "service.my_domain.com", "num_choices": 1}
2023/05/10 10:27:25.845 DEBUG   tls.handshake   custom certificate selection results    {"identifier": "service.my_domain.com", "subjects": ["*.my_domain.com", "my_domain.com"], "managed": false, "issuer_key": "", "hash": "5c5ed19af54ff7cc75972bf2d869dfc9266d9c20f594cc12521327d629d173cd"}
2023/05/10 10:27:25.846 DEBUG   tls.handshake   matched certificate in cache    {"remote_ip": "127.0.0.1", "remote_port": "62280", "subjects": ["*.my_domain.com", "my_domain.com"], "managed": false, "expiration": "2023/06/25 05:49:16.000", "hash": "5c5ed19af54ff7cc75972bf2d869dfc9266d9c20f594cc12521327d629d173cd"}
2023/05/10 10:27:25.853 DEBUG   http.handlers.reverse_proxy     selected upstream       {"dial": "127.0.0.1:8888", "total_upstreams": 1}
2023/05/10 10:27:25.875 DEBUG   http.handlers.reverse_proxy     upstream roundtrip      {"upstream": "127.0.0.1:8888", "duration": 0.021025566, "request": {"remote_ip": "127.0.0.1", "remote_port": "62280", "proto": "HTTP/2.0", "method": "GET", "host": "service.my_domain.com", "uri": "/", "headers": {"X-Forwarded-For": ["127.0.0.1"], "X-Forwarded-Proto": ["https"], "X-Forwarded-Host": ["service.my_domain.com"], "User-Agent": ["curl/7.79.1"], "Accept": ["*/*"]}, "tls": {"resumed": false, "version": 772, "cipher_suite": 4865, "proto": "h2", "server_name": "service.my_domain.com"}}, "headers": {"Server": ["SimpleHTTP/0.6 Python/3.11.1"], "Date": ["Wed, 10 May 2023 10:27:25 GMT"], "Content-Type": ["text/html; charset=utf-8"], "Content-Length": ["531"]}, "status": 200}

It seems to me that second proxy receives Host set to ‘localhost’ but that is not correct as I logged it with this simple python script:

#!/usr/bin/env python3

import http.server as SimpleHTTPServer
import socketserver as SocketServer
import logging

PORT = 8888

class GetHandler(SimpleHTTPServer.SimpleHTTPRequestHandler):

    def do_GET(self):
        print(self.headers)
        SimpleHTTPServer.SimpleHTTPRequestHandler.do_GET(self)

httpd = SocketServer.TCPServer(("", PORT), GetHandler)
httpd.serve_forever()

which logged:

Host: service.my_domain.com
User-Agent: curl/7.79.1
Accept: */*
X-Forwarded-For: 127.0.0.1
X-Forwarded-Host: service.my_domain.com
X-Forwarded-Proto: https
Accept-Encoding: gzip


127.0.0.1 - - [10/May/2023 12:28:22] "GET / HTTP/1.1" 200 -
Host: service.my_domain.com
User-Agent: curl/7.79.1
Accept: */*
X-Forwarded-For: 127.0.0.1
X-Forwarded-Host: service.my_domain.com
X-Forwarded-Proto: https
Accept-Encoding: gzip

in both instances (behind first and second proxy).

3. Caddy version: 2.6.4

4. How I installed and ran Caddy:

Extracted binary from github releases and run directly with ./caddy

a. System environment: MacOS, intel

b. Command:

./caddy  run --watch --config Caddyfile1
./caddy run --watch --config Caddyfile2

d. My complete Caddy config:

So on my host machine, I setup caddy like so:

{
    https_port 443
    debug
}

(tls-config) {
  tls fullchain.pem privkey.pem  # same cert and key are mounted in caddy inside docker
}

service.my_domain.com {
  import tls-config
  reverse_proxy http://127.0.0.1:4443 {
    header_up Host service.my_domain.com # also tried {request.header.host/hostport} & {hostport}
  }
}

And config for dockerized caddy is:

{
    https_port 4443
    debug
}

(tls-config) {
  tls /fullchain.pem /privkey.pem # mounted in container, ensured caddy can read them
}

*.my_domain.com {
  import tls-config

  @service host service.my_domain.com
  handle @service {
    reverse_proxy http://service {  # 'service' is container name in docker network
    }
  }
}

5. Links to relevant resources:

One thing that might still be relevant is docker compose config for caddy proxy, although I don’t think it is:

services:
  caddy:
    image: caddy
    volumes:
        - ./Caddyfile:/etc/caddy/Caddyfile
        - /etc/letsencrypt/live/my_domain.com/fullchain.pem:/cert.pem
        - /etc/letsencrypt/live/my_domain.com/privkey.pem:/key.pem
    ports:
      - "127.0.0.1:8080:80"
      - "127.0.0.1:4443:443"
    networks:
        compose_network:
            aliases: # so that every request goes through internal compose network proxy
                - service.my_domain.com
                - another_service.my_domain.com
                - ...
2

Why not use Caddy instead of certbot to do that? Caddy can issue and maintain wildcard certs for you using the DNS challenge.

You can remove this, it’s redundant. It’s already the default.

You wrote http:// here, so Caddy is making an HTTP request to an HTTPS server. Use https:// instead.

{request.*} isn’t valid, it would be {http.request.*} or any of these shorthands Caddyfile Concepts — Caddy Documentation.

You might also need to set tls_server_name as part of the transport config of the proxy reverse_proxy (Caddyfile directive) — Caddy Documentation

That’s definitely strange. It should be fine to connect with like http://container-name

3

Thank you very much sir!
tls_server_name missing was the problem!
Solved :tada:

4

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.