Reverse Proxy SSL failing

Help
1

1. The problem I’m having:

I’m trying to setup a linux server with a caddy docker as the reverse proxy, and several docker services using various subdomains. Right now, I’m just trying to do one, before I add anymore.

2. Error messages and/or full log output:

$ curl -vL https://localhost
*   Trying 127.0.0.1:443...
* Connected to localhost (127.0.0.1) port 443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
*  CAfile: /etc/ssl/certs/ca-certificates.crt
*  CApath: /etc/ssl/certs
* TLSv1.0 (OUT), TLS header, Certificate Status (22):
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* TLSv1.2 (IN), TLS header, Unknown (21):
* TLSv1.3 (IN), TLS alert, internal error (592):
* error:0A000438:SSL routines::tlsv1 alert internal error
* Closing connection 0
curl: (35) error:0A000438:SSL routines::tlsv1 alert internal error

3. Caddy version:

v2.6.4 h1:2hwYqiRwk1tf3VruhMpLcYTg+11fCdr8S3jhNAdnPy8=

4. How I installed and ran Caddy:

I installed caddy via docker-compose.

a. System environment:

VM running Ubuntu 22.04.2 LTS, with Docker:
Docker version 20.10.21, build 20.10.21-0ubuntu1~22.04.3

b. Command:

docker-compose up -d

c. Service/unit/compose file:

version: "3.7"

networks:
        web:
                external: true
        internal:
                external: false
                driver: bridge

services:
        caddy:
                image: caddy:2-alpine
                restart: unless-stopped
                ports:
                        - "80:80"
                        - "443:443"
                volumes:
                        - /data/caddy/Caddyfile:/etc/caddy/Caddyfile
                        - /data/caddy/data:/data # Optional
                        - /data/caddy/config:/config # Optional
                networks:
                        - web
                        - internal
        toddsoft:
                build:
                        context: /data/ToddSoft/.devcontainer/
                        dockerfile: dockerfile
                restart: unless-stopped
                ports:
                        - "8080:8080"
                environment:
                        - url=https://toddsoft.toddcom.com
                        - ServerName=toddsoft.toddcom.com
                volumes:
                        - /data/ToddSoft:/var/www/html
                networks:
                        - internal

d. My complete Caddy config:

{
    # Global options block. Entirely optional, https is on by default
    # Optional email key for lets encrypt
    email rob@toddcom.com
    # Optional staging lets encrypt for testing. Comment out for production.
    # acme_ca https://acme-staging-v02.api.letsencrypt.org/directory
    debug
}
toddsoft.toddcom.com {
    reverse_proxy toddsoft:8080
}
www.toddsoft.toddcom.com {
    redir https://toddsoft.toddcom.com{uri}
}

5. Links to relevant resources:

I started with this tutorial. But I haven’t done anything with ghost.

It’s probably important to note that I had a VM running a basic webserver at this location with SSL installed using certbot and had it all running just fine. The router and dns entries have all been configured for about two years working flawlessly. Now I had to rebuild the VM but I can’t seem to get it to work with caddy. Thanks for any help you guys can offer.

2

What’s in the server logs? That will tell you what the actual error is, that the client can only report as “internal error”

3

You made a request to localhost, but you didn’t configure localhost as a site address in your config.

4

Now see, that’s a good question. So I’m still new to this docker world. I’m trying to get into see the logs on the Caddy docker but frankly, I’m really not sure how. How do I do that?

5

Thank you for that @francislavoie! now I got this from the server:

$ curl -vL https://toddsoft.toddcom.com/
*   Trying 10.10.10.15:443...
* Connected to toddsoft.toddcom.com (10.10.10.15) port 443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
*  CAfile: /etc/ssl/certs/ca-certificates.crt
*  CApath: /etc/ssl/certs
* TLSv1.0 (OUT), TLS header, Certificate Status (22):
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* TLSv1.2 (IN), TLS header, Certificate Status (22):
* TLSv1.3 (IN), TLS handshake, Server hello (2):
* TLSv1.2 (IN), TLS header, Finished (20):
* TLSv1.2 (IN), TLS header, Supplemental data (23):
* TLSv1.3 (IN), TLS handshake, Encrypted Extensions (8):
* TLSv1.2 (IN), TLS header, Supplemental data (23):
* TLSv1.3 (IN), TLS handshake, Certificate (11):
* TLSv1.2 (IN), TLS header, Supplemental data (23):
* TLSv1.3 (IN), TLS handshake, CERT verify (15):
* TLSv1.2 (IN), TLS header, Supplemental data (23):
* TLSv1.3 (IN), TLS handshake, Finished (20):
* TLSv1.2 (OUT), TLS header, Finished (20):
* TLSv1.3 (OUT), TLS change cipher, Change cipher spec (1):
* TLSv1.2 (OUT), TLS header, Supplemental data (23):
* TLSv1.3 (OUT), TLS handshake, Finished (20):
* SSL connection using TLSv1.3 / TLS_AES_128_GCM_SHA256
* ALPN, server accepted to use h2
* Server certificate:
*  subject: CN=toddsoft.toddcom.com
*  start date: May  2 19:56:06 2023 GMT
*  expire date: Jul 31 19:56:05 2023 GMT
*  subjectAltName: host "toddsoft.toddcom.com" matched cert's "toddsoft.toddcom.com"
*  issuer: C=US; O=Let's Encrypt; CN=R3
*  SSL certificate verify ok.
* Using HTTP2, server supports multiplexing
* Connection state changed (HTTP/2 confirmed)
* Copying HTTP/2 data in stream buffer to connection buffer after upgrade: len=0
* TLSv1.2 (OUT), TLS header, Supplemental data (23):
* TLSv1.2 (OUT), TLS header, Supplemental data (23):
* TLSv1.2 (OUT), TLS header, Supplemental data (23):
* Using Stream ID: 1 (easy handle 0x561e7548de90)
* TLSv1.2 (OUT), TLS header, Supplemental data (23):
> GET / HTTP/2
> Host: toddsoft.toddcom.com
> user-agent: curl/7.81.0
> accept: */*
>
* TLSv1.2 (IN), TLS header, Supplemental data (23):
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
* TLSv1.2 (IN), TLS header, Supplemental data (23):
* Connection state changed (MAX_CONCURRENT_STREAMS == 250)!
* TLSv1.2 (OUT), TLS header, Supplemental data (23):
* TLSv1.2 (IN), TLS header, Supplemental data (23):
* TLSv1.2 (IN), TLS header, Supplemental data (23):
* TLSv1.2 (IN), TLS header, Supplemental data (23):
< HTTP/2 502
< alt-svc: h3=":443"; ma=2592000
< server: Caddy
< content-length: 0
< date: Wed, 03 May 2023 20:39:23 GMT
<
* Connection #0 to host toddsoft.toddcom.com left intact

So now I’m getting a “bad gateway.”

6

502 means Caddy couldn’t connect to your upstream app.

Turn on the debug global option and check Caddy’s logs. Make sure the address is correct. Make sure your app is actually running.

Use docker-compose logs caddy to look at your logs.

7

@francislavoie Thanks! Ok, here are the logs:

$ docker-compose logs caddy
Attaching to caddy_caddy_1
caddy_1     | {"level":"info","ts":1683146353.662285,"msg":"using provided configuration","config_file":"/etc/caddy/Caddyfile","config_adapter":"caddyfile"}
caddy_1     | {"level":"warn","ts":1683146353.6719215,"msg":"Caddyfile input is not formatted; run the 'caddy fmt' command to fix inconsistencies","adapter":"caddyfile","file":"/etc/caddy/Caddyfile","line":2}
caddy_1     | {"level":"info","ts":1683146353.6740532,"logger":"admin","msg":"admin endpoint started","address":"localhost:2019","enforce_origin":false,"origins":["//localhost:2019","//[::1]:2019","//127.0.0.1:2019"]}
caddy_1     | {"level":"info","ts":1683146353.6781716,"logger":"http","msg":"server is listening only on the HTTPS port but has no TLS connection policies; adding one to enable TLS","server_name":"srv0","https_port":443}
caddy_1     | {"level":"info","ts":1683146353.678196,"logger":"http","msg":"enabling automatic HTTP->HTTPS redirects","server_name":"srv0"}
caddy_1     | {"level":"info","ts":1683146353.6785388,"logger":"http","msg":"enabling HTTP/3 listener","addr":":443"}
caddy_1     | {"level":"info","ts":1683146353.6787715,"msg":"failed to sufficiently increase receive buffer size (was: 208 kiB, wanted: 2048 kiB, got: 416 kiB). See https://github.com/quic-go/quic-go/wiki/UDP-Receive-Buffer-Size for details."}
caddy_1     | {"level":"debug","ts":1683146353.6788926,"logger":"http","msg":"starting server loop","address":"[::]:443","tls":true,"http3":true}
caddy_1     | {"level":"info","ts":1683146353.6789796,"logger":"http.log","msg":"server running","name":"srv0","protocols":["h1","h2","h3"]}
caddy_1     | {"level":"debug","ts":1683146353.6790693,"logger":"http","msg":"starting server loop","address":"[::]:80","tls":false,"http3":false}
caddy_1     | {"level":"info","ts":1683146353.6790879,"logger":"http.log","msg":"server running","name":"remaining_auto_https_redirects","protocols":["h1","h2","h3"]}
caddy_1     | {"level":"info","ts":1683146353.6790938,"logger":"http","msg":"enabling automatic TLS certificate management","domains":["toddsoft.toddcom.com"]}
caddy_1     | {"level":"info","ts":1683146353.679432,"logger":"tls.cache.maintenance","msg":"started background certificate maintenance","cache":"0xc0003560e0"}
caddy_1     | {"level":"info","ts":1683146353.682875,"logger":"tls","msg":"cleaning storage unit","description":"FileStorage:/data/caddy"}
caddy_1     | {"level":"debug","ts":1683146353.6836348,"logger":"tls","msg":"loading managed certificate","domain":"toddsoft.toddcom.com","expiration":1690833366,"issuer_key":"acme-v02.api.letsencrypt.org-directory","storage":"FileStorage:/data/caddy"}
caddy_1     | {"level":"debug","ts":1683146353.6857355,"logger":"tls.cache","msg":"added certificate to cache","subjects":["toddsoft.toddcom.com"],"expiration":1690833366,"managed":true,"issuer_key":"acme-v02.api.letsencrypt.org-directory","hash":"bfb93f704e87622f40b0262aa4db1e3db0ef4bbbaff9052681f6c1a392c10d32","cache_size":1,"cache_capacity":10000}
caddy_1     | {"level":"debug","ts":1683146353.6859646,"logger":"events","msg":"event","name":"cached_managed_cert","id":"c673343f-7056-4c1a-b2f1-220fd67dbd3e","origin":"tls","data":{"sans":["toddsoft.toddcom.com"]}}
caddy_1     | {"level":"info","ts":1683146353.6864738,"msg":"autosaved config (load with --resume flag)","file":"/config/caddy/autosave.json"}
caddy_1     | {"level":"info","ts":1683146353.6864905,"msg":"serving initial configuration"}
caddy_1     | {"level":"info","ts":1683146353.686962,"logger":"tls","msg":"finished cleaning storage units"}
caddy_1     | {"level":"debug","ts":1683146363.5840914,"logger":"events","msg":"event","name":"tls_get_certificate","id":"3198827d-8a1b-4e6d-83c1-9b4ab2e0028e","origin":"tls","data":{"client_hello":{"CipherSuites":[4866,4867,4865,49196,49200,159,52393,52392,52394,49195,49199,158,49188,49192,107,49187,49191,103,49162,49172,57,49161,49171,51,157,156,61,60,53,47,255],"ServerName":"toddsoft.toddcom.com","SupportedCurves":[29,23,30,25,24,256,257,258,259,260],"SupportedPoints":"AAEC","SignatureSchemes":[1027,1283,1539,2055,2056,2057,2058,2059,2052,2053,2054,1025,1281,1537,771,769,770,1026,1282,1538],"SupportedProtos":["h2","http/1.1"],"SupportedVersions":[772,771],"Conn":{}}}}
caddy_1     | {"level":"debug","ts":1683146363.5842001,"logger":"tls.handshake","msg":"choosing certificate","identifier":"toddsoft.toddcom.com","num_choices":1}
caddy_1     | {"level":"debug","ts":1683146363.5842164,"logger":"tls.handshake","msg":"default certificate selection results","identifier":"toddsoft.toddcom.com","subjects":["toddsoft.toddcom.com"],"managed":true,"issuer_key":"acme-v02.api.letsencrypt.org-directory","hash":"bfb93f704e87622f40b0262aa4db1e3db0ef4bbbaff9052681f6c1a392c10d32"}
caddy_1     | {"level":"debug","ts":1683146363.5842264,"logger":"tls.handshake","msg":"matched certificate in cache","remote_ip":"10.10.10.15","remote_port":"36894","subjects":["toddsoft.toddcom.com"],"managed":true,"expiration":1690833366,"hash":"bfb93f704e87622f40b0262aa4db1e3db0ef4bbbaff9052681f6c1a392c10d32"}
caddy_1     | {"level":"debug","ts":1683146363.592016,"logger":"http.handlers.reverse_proxy","msg":"selected upstream","dial":"toddsoft:8080","total_upstreams":1}
caddy_1     | {"level":"debug","ts":1683146363.5929887,"logger":"http.handlers.reverse_proxy","msg":"upstream roundtrip","upstream":"toddsoft:8080","duration":0.000927707,"request":{"remote_ip":"10.10.10.15","remote_port":"36894","proto":"HTTP/2.0","method":"GET","host":"toddsoft.toddcom.com","uri":"/","headers":{"User-Agent":["curl/7.81.0"],"Accept":["*/*"],"X-Forwarded-For":["10.10.10.15"],"X-Forwarded-Proto":["https"],"X-Forwarded-Host":["toddsoft.toddcom.com"]},"tls":{"resumed":false,"version":772,"cipher_suite":4865,"proto":"h2","server_name":"toddsoft.toddcom.com"}},"error":"dial tcp 172.30.0.2:8080: connect: connection refused"}
caddy_1     | {"level":"error","ts":1683146363.5931878,"logger":"http.log.error","msg":"dial tcp 172.30.0.2:8080: connect: connection refused","request":{"remote_ip":"10.10.10.15","remote_port":"36894","proto":"HTTP/2.0","method":"GET","host":"toddsoft.toddcom.com","uri":"/","headers":{"User-Agent":["curl/7.81.0"],"Accept":["*/*"]},"tls":{"resumed":false,"version":772,"cipher_suite":4865,"proto":"h2","server_name":"toddsoft.toddcom.com"}},"duration":0.001198194,"status":502,"err_id":"hi0r31p2j","err_trace":"reverseproxy.statusError (reverseproxy.go:1299)"}
8 
$ docker ps
CONTAINER ID   IMAGE            COMMAND                  CREATED          STATUS          PORTS                                                                                         NAMES
f6074a5d7ac3   caddy:2-alpine   "caddy run --config …"   18 seconds ago   Up 17 seconds   0.0.0.0:80->80/tcp, :::80->80/tcp, 0.0.0.0:443->443/tcp, :::443->443/tcp, 443/udp, 2019/tcp   caddy_caddy_1
d46f6cb96e79   caddy_toddsoft   "/bin/sh -c 'apachec…"   18 seconds ago   Up 17 seconds   80/tcp, 0.0.0.0:8080->8080/tcp, :::8080->8080/tcp                                             caddy_toddsoft_1
9

:man_shrugging:

Make sure your app is actually listening on that port. That’s not a problem with Caddy in particular.

10

Ok, I know that toddsoft docker is listening to port 80 internally. But I thought I got docker to forward that to port 8080 on the outside. Am I doing it wrong?

$ docker ps
CONTAINER ID   IMAGE            COMMAND                  CREATED         STATUS         PORTS                                                                                         NAMES
2dc2c7ded6b3   caddy_toddsoft   "/bin/sh -c 'apachec…"   3 minutes ago   Up 3 minutes   8080/tcp, 0.0.0.0:8080->80/tcp, :::8080->80/tcp                                               caddy_toddsoft_1
4cd7d1e4e40b   caddy:2-alpine   "caddy run --config …"   3 minutes ago   Up 3 minutes   0.0.0.0:80->80/tcp, :::80->80/tcp, 0.0.0.0:443->443/tcp, :::443->443/tcp, 443/udp, 2019/tcp   caddy_caddy_1
11

AHAH!

OK, thank you so much for your help. I had to change the caddyfile to point to toddsoft:80 instead of toddsoft:8080.

I thought that I had forwarded the port using docker to be port 8080 to the port 80 internally in my toddsoft container. But I think I’m misunderstanding how docker handles ports. This works now. Thank you!

12

Docker networking happens inside the docker network. You don’t need to publish the port for another container to connect. You can remove your ports: - "8080:8080" mapping.

13

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.