Unable to serve https using dockerized Caddy

1. The problem I’m having:

My dockerized Caddy cannot serve https. There is a no error. Would you have any recommendation where i can check further?

2. Error messages and/or full log output:

windmill-caddy-1             | {"level":"info","ts":1680436850.6773994,"msg":"using provided configuration","config_file":"/etc/caddy/Caddyfile","config_adapter":"caddyfile"}
windmill-caddy-1             | {"level":"warn","ts":1680436850.6834738,"msg":"Caddyfile input is not formatted; run the 'caddy fmt' command to fix inconsistencies","adapter":"caddyfile","file":"/etc/caddy/Caddyfile","line":1}
windmill-caddy-1             | {"level":"info","ts":1680436850.6842742,"logger":"admin","msg":"admin endpoint started","address":"tcp/localhost:2019","enforce_origin":false,"origins":["//localhost:2019","//[::1]:2019","//127.0.0.1:2019"]}
windmill-caddy-1             | {"level":"warn","ts":1680436850.684465,"logger":"http","msg":"automatic HTTPS is completely disabled for server","server_name":"srv0"}
windmill-caddy-1             | {"level":"warn","ts":1680436850.6845076,"logger":"http","msg":"automatic HTTPS is completely disabled for server","server_name":"srv1"}
windmill-caddy-1             | {"level":"info","ts":1680436850.6848018,"logger":"tls.cache.maintenance","msg":"started background certificate maintenance","cache":"0x400032e000"}
windmill-caddy-1             | {"level":"info","ts":1680436850.6875093,"msg":"autosaved config (load with --resume flag)","file":"/config/caddy/autosave.json"}
windmill-caddy-1             | {"level":"info","ts":1680436850.6875677,"msg":"serving initial configuration"}
windmill-caddy-1             | {"level":"info","ts":1680436850.6876233,"logger":"tls","msg":"cleaning storage unit","description":"FileStorage:/data/caddy"}
windmill-caddy-1             | {"level":"info","ts":1680436850.6876886,"logger":"tls","msg":"finished cleaning storage units"}

3. Caddy version:

caddy:2.5.2-alpine

4. How I installed and ran Caddy:

Caddy was already part of the docker container that i use.

a. System environment:

Ubuntu. Docker. Arm64

b. Command:

I didn’t run any command except the standard “docker compose up” to run the docker. But, this is the list of containers that are working with Caddy in the same docker.

ubuntu@appwrite:~$ docker container ls
CONTAINER ID   IMAGE                                 COMMAND                  CREATED       STATUS                 PORTS                                                                                         NAMES
d0c851a79c2a   ghcr.io/windmill-labs/windmill:main   "./windmill"             2 hours ago   Up 2 hours             8000/tcp                                                                                      windmill-windmill_worker-7
3adb591fd3cd   ghcr.io/windmill-labs/windmill:main   "./windmill"             2 hours ago   Up 2 hours             8000/tcp                                                                                      windmill-windmill_worker-6
a5559c0bb9d6   ghcr.io/windmill-labs/windmill:main   "./windmill"             2 hours ago   Up 2 hours             8000/tcp                                                                                      windmill-windmill_worker-9
c58a2317d6e3   ghcr.io/windmill-labs/windmill:main   "./windmill"             2 hours ago   Up 2 hours             8000/tcp                                                                                      windmill-windmill_worker-2
a8fb83e38d07   ghcr.io/windmill-labs/windmill:main   "./windmill"             2 hours ago   Up 2 hours             8000/tcp                                                                                      windmill-windmill_worker-5
c3822289f813   ghcr.io/windmill-labs/windmill:main   "./windmill"             2 hours ago   Up 2 hours             8000/tcp                                                                                      windmill-windmill_worker-8
84a9756c6f43   ghcr.io/windmill-labs/windmill:main   "./windmill"             2 hours ago   Up 2 hours             8000/tcp                                                                                      windmill-windmill_worker-1
f2feced17d51   ghcr.io/windmill-labs/windmill:main   "./windmill"             2 hours ago   Up 2 hours             8000/tcp                                                                                      windmill-windmill_worker-4
3df1e2f4ecf2   ghcr.io/windmill-labs/windmill:main   "./windmill"             2 hours ago   Up 2 hours             8000/tcp                                                                                      windmill-windmill_worker-3
068eafd08bb2   ghcr.io/windmill-labs/windmill:main   "./windmill"             2 hours ago   Up 2 hours             0.0.0.0:8000->8000/tcp, :::8000->8000/tcp                                                     windmill-windmill_server-1
114d7e1be51c   ghcr.io/windmill-labs/windmill:main   "./windmill"             2 hours ago   Up 2 hours             8000/tcp                                                                                      windmill-windmill_worker-10
57f21b98b6b6   postgres:14                           "docker-entrypoint.s…"   2 hours ago   Up 2 hours (healthy)   0.0.0.0:5432->5432/tcp, :::5432->5432/tcp, 0.0.0.0:5433->5432/tcp, :::5433->5432/tcp          windmill-db-1
54802bfbee64   caddy:2.5.2-alpine                    "caddy run --config …"   2 hours ago   Up 2 hours             0.0.0.0:80->80/tcp, :::80->80/tcp, 0.0.0.0:443->443/tcp, :::443->443/tcp, 443/udp, 2019/tcp   windmill-caddy-1

d. My complete Caddy config:

My Caddyfile looks like this

root@appwrite:/home/ubuntu/windmill# cat Caddyfile

{
        auto_https off
}

http://workflow.saidhasyim.com {
        bind {$ADDRESS}
        reverse_proxy /ws/* http://lsp:3001
        reverse_proxy /* http://windmill_server:8000
}

https://workflow.saidhasyim.com  {
  bind {$ADDRESS}
 reverse_proxy /ws/* http://lsp:3001
}

the caddy config inside my docker-compose.yml file:

caddy:
    image: caddy:2.5.2-alpine
    restart: unless-stopped
    volumes:
      - ./Caddyfile:/etc/caddy/Caddyfile
    ports:
      - 80:80
      - 443:443
    environment:
      - BASE_URL=${WM_BASE_URL}

Please use the latest version, v2.6.4.

Make sure to at least also have a volume for /data. This is very important, to make sure you don’t lose your certs and keys when recreating the containers.

See our docs for using Caddy with Docker Compose

What do you mean by that? That’s very vague. What behaviour are you seeing, specifically? What do you see when you make a request with curl -v?

Are you sure you need this? I don’t think you do. When running in Docker, I can’t think of any reason to use bind, because Docker is setting up virtual networks for you. This might be breaking it.

Hello Francis,

Thanks a lot for your feedback. Sorry I wasn’t clear. I meant to say that I couldn’t launch https on my website.

I have made adjustment to the docker-compose.yml (add the latest version, and the /data).

  caddy:
    image: caddy:2.6.4-alpine
    restart: unless-stopped
    volumes:
      - ./Caddyfile:/etc/caddy/Caddyfile
      - caddy_data:/data
      - caddy_config:/config
    ports:
      - 80:80
      - 443:443
      - 443:443/udp
    environment:
      - BASE_URL=${WM_BASE_URL}

volumes:
  db_data: null
  worker_dependency_cache: null
  caddy_data:
  caddy_config:

I have also remove bind $address in caddyfile.

{
        auto_https off
}

http://{$BASE_URL} {
        reverse_proxy /ws/* http://lsp:3001
        reverse_proxy /* http://windmill_server:8000
}

https://{$BASE_URL}  {  
  reverse_proxy /ws/* http://localhost:3001
}

After restarted and reloading the caddy file, i still can’t launch the site with https. Here’s latest caddy log

windmill-caddy-1  | {"level":"info","ts":1680572655.2499247,"msg":"using provided configuration","config_file":"/etc/caddy/Caddyfile","config_adapter":"caddyfile"}
windmill-caddy-1  | {"level":"warn","ts":1680572655.2531986,"msg":"Caddyfile input is not formatted; run the 'caddy fmt' command to fix inconsistencies","adapter":"caddyfile","file":"/etc/caddy/Caddyfile","line":2}
windmill-caddy-1  | {"level":"info","ts":1680572655.2549703,"logger":"admin","msg":"admin endpoint started","address":"localhost:2019","enforce_origin":false,"origins":["//localhost:2019","//[::1]:2019","//127.0.0.1:2019"]}
windmill-caddy-1  | {"level":"info","ts":1680572655.257206,"logger":"tls.cache.maintenance","msg":"started background certificate maintenance","cache":"0x40001c6af0"}
windmill-caddy-1  | {"level":"warn","ts":1680572655.25755,"logger":"http","msg":"automatic HTTPS is completely disabled for server","server_name":"srv0"}
windmill-caddy-1  | {"level":"warn","ts":1680572655.257786,"logger":"http","msg":"automatic HTTPS is completely disabled for server","server_name":"srv1"}
windmill-caddy-1  | {"level":"info","ts":1680572655.2597566,"logger":"http.log","msg":"server running","name":"srv1","protocols":["h1","h2","h3"]}
windmill-caddy-1  | {"level":"info","ts":1680572655.2598574,"logger":"tls","msg":"cleaning storage unit","description":"FileStorage:/data/caddy"}
windmill-caddy-1  | {"level":"info","ts":1680572655.25998,"logger":"tls","msg":"finished cleaning storage units"}
windmill-caddy-1  | {"level":"info","ts":1680572655.260054,"logger":"http","msg":"enabling HTTP/3 listener","addr":":443"}
windmill-caddy-1  | {"level":"info","ts":1680572655.2601905,"msg":"failed to sufficiently increase receive buffer size (was: 208 kiB, wanted: 2048 kiB, got: 416 kiB). See https://github.com/quic-go/quic-go/wiki/UDP-Receive-Buffer-Size for details."}
windmill-caddy-1  | {"level":"info","ts":1680572655.2602882,"logger":"http.log","msg":"server running","name":"srv0","protocols":["h1","h2","h3"]}
windmill-caddy-1  | {"level":"info","ts":1680572655.260742,"msg":"autosaved config (load with --resume flag)","file":"/config/caddy/autosave.json"}
windmill-caddy-1  | {"level":"info","ts":1680572655.260789,"msg":"serving initial configuration"}
windmill-caddy-1  | {"level":"info","ts":1680572675.5702384,"logger":"admin.api","msg":"received request","method":"POST","host":"localhost:2019","uri":"/load","remote_ip":"127.0.0.1","remote_port":"46006","headers":{"Accept-Encoding":["gzip"],"Content-Length":["754"],"Content-Type":["application/json"],"Origin":["http://localhost:2019"],"User-Agent":["Go-http-client/1.1"]}}
windmill-caddy-1  | {"level":"info","ts":1680572675.5706,"msg":"config is unchanged"}
windmill-caddy-1  | {"level":"info","ts":1680572675.5706286,"logger":"admin.api","msg":"load complete"}

here’s the curl output:

PS C:\Windows\system32> curl -v

cmdlet Invoke-WebRequest at command pipeline position 1
Supply values for the following parameters:
Uri: http://workflow.saidhasyim.com
VERBOSE: GET http://workflow.saidhasyim.com/ with 0-byte payload
VERBOSE: received 1168-byte response of content type text/html


StatusCode        : 200
StatusDescription : OK
Content           : <!DOCTYPE html>
                    <html lang="en">
                        <head>
                                <meta charset="utf-8" />
                                <link rel="icon" href="/logo.svg" />
                                <meta name="viewport" content="width=device-width, initial-scale=1" />
                                <meta http-equiv="...
RawContent        : HTTP/1.1 200 OK
                    Content-Length: 1168
                    Cache-Control: no-cache, no-store, must-revalidate
                    Content-Type: text/html
                    Date: Tue, 04 Apr 2023 01:45:23 GMT
                    Server: Caddy

                    <!DOCTYPE html>
                    <html lang="en...
Forms             : {}
Headers           : {[Content-Length, 1168], [Cache-Control, no-cache, no-store, must-revalidate], [Content-Type,
                    text/html], [Date, Tue, 04 Apr 2023 01:45:23 GMT]...}
Images            : {}
InputFields       : {}
Links             : {}
ParsedHtml        : mshtml.HTMLDocumentClass
RawContentLength  : 1168



PS C:\Windows\system32> curl -v

cmdlet Invoke-WebRequest at command pipeline position 1
Supply values for the following parameters:
Uri: https://workflow.saidhasyim.com
VERBOSE: GET https://workflow.saidhasyim.com/ with 0-byte payload
curl : The request was aborted: Could not create SSL/TLS secure channel.
At line:1 char:1
+ curl -v
+ ~~~~~~~
    + CategoryInfo          : InvalidOperation: (System.Net.HttpWebRequest:HttpWebRequest) [Invoke-WebRequest], WebExc
   eption
    + FullyQualifiedErrorId : WebCmdletWebResponseException,Microsoft.PowerShell.Commands.InvokeWebRequestCommand

Would you have any other suggestion i can try?

Thanks again

Why are you turning off Automatic HTTPS? This is what causes Caddy to set up cert automation for your domain.

Why are you adding a site block with http://? This is obviously allowing traffic to be served insecurely.

Be careful here, localhost inside of a Docker container means “this container”. You don’t have something else running in the same container as Caddy.

Thanks a lot.

I remove those and my caddyfile looks like this now.

https://{$BASE_URL}  {  
  reverse_proxy /ws/* http://localhost:3001
}

Finally i can browse in HTTPS.

PS C:\Windows\system32> curl -v

cmdlet Invoke-WebRequest at command pipeline position 1
Supply values for the following parameters:
Uri: https://workflow.saidhasyim.com
VERBOSE: GET https://workflow.saidhasyim.com/ with 0-byte payload
VERBOSE: received 0-byte response of content type


StatusCode        : 200
StatusDescription : OK
Content           : {}
RawContent        : HTTP/1.1 200 OK
                    Alt-Svc: h3=":443"; ma=2592000
                    Content-Length: 0
                    Date: Tue, 04 Apr 2023 02:17:58 GMT
                    Server: Caddy


Headers           : {[Alt-Svc, h3=":443"; ma=2592000], [Content-Length, 0], [Date, Tue, 04 Apr 2023 02:17:58 GMT],
                    [Server, Caddy]}
RawContentLength  : 0

However, the page is loading a blank page.

Here’s the caddy log

root@appwrite:/home/ubuntu/windmill# docker compose logs caddy
windmill-caddy-1  | {"level":"info","ts":1680574634.6263647,"msg":"using provided configuration","config_file":"/etc/caddy/Caddyfile","config_adapter":"caddyfile"}
windmill-caddy-1  | {"level":"warn","ts":1680574634.6274283,"msg":"Caddyfile input is not formatted; run the 'caddy fmt' command to fix inconsistencies","adapter":"caddyfile","file":"/etc/caddy/Caddyfile","line":1}
windmill-caddy-1  | {"level":"info","ts":1680574634.6289527,"logger":"admin","msg":"admin endpoint started","address":"localhost:2019","enforce_origin":false,"origins":["//[::1]:2019","//127.0.0.1:2019","//localhost:2019"]}
windmill-caddy-1  | {"level":"info","ts":1680574634.6294825,"logger":"http","msg":"server is listening only on the HTTPS port but has no TLS connection policies; adding one to enable TLS","server_name":"srv0","https_port":443}
windmill-caddy-1  | {"level":"info","ts":1680574634.629523,"logger":"http","msg":"enabling automatic HTTP->HTTPS redirects","server_name":"srv0"}
windmill-caddy-1  | {"level":"info","ts":1680574634.629799,"logger":"http","msg":"enabling HTTP/3 listener","addr":":443"}
windmill-caddy-1  | {"level":"info","ts":1680574634.6299105,"logger":"tls.cache.maintenance","msg":"started background certificate maintenance","cache":"0x4000205ab0"}
windmill-caddy-1  | {"level":"info","ts":1680574634.6300387,"logger":"tls","msg":"cleaning storage unit","description":"FileStorage:/data/caddy"}
windmill-caddy-1  | {"level":"info","ts":1680574634.630061,"logger":"tls","msg":"finished cleaning storage units"}
windmill-caddy-1  | {"level":"info","ts":1680574634.6301477,"logger":"http.log","msg":"server running","name":"srv0","protocols":["h1","h2","h3"]}
windmill-caddy-1  | {"level":"info","ts":1680574634.6303318,"logger":"http.log","msg":"server running","name":"remaining_auto_https_redirects","protocols":["h1","h2","h3"]}
windmill-caddy-1  | {"level":"info","ts":1680574634.630364,"logger":"http","msg":"enabling automatic TLS certificate management","domains":["workflow.saidhasyim.com"]}
windmill-caddy-1  | {"level":"info","ts":1680574634.6305735,"msg":"autosaved config (load with --resume flag)","file":"/config/caddy/autosave.json"}
windmill-caddy-1  | {"level":"info","ts":1680574634.6305819,"msg":"serving initial configuration"}
windmill-caddy-1  | {"level":"info","ts":1680574634.630839,"logger":"tls.obtain","msg":"acquiring lock","identifier":"workflow.saidhasyim.com"}
windmill-caddy-1  | {"level":"info","ts":1680574634.6343815,"logger":"tls.obtain","msg":"lock acquired","identifier":"workflow.saidhasyim.com"}
windmill-caddy-1  | {"level":"info","ts":1680574634.6345534,"logger":"tls.obtain","msg":"obtaining certificate","identifier":"workflow.saidhasyim.com"}
windmill-caddy-1  | {"level":"info","ts":1680574634.9497604,"logger":"http","msg":"waiting on internal rate limiter","identifiers":["workflow.saidhasyim.com"],"ca":"https://acme-v02.api.letsencrypt.org/directory","account":""}
windmill-caddy-1  | {"level":"info","ts":1680574634.94981,"logger":"http","msg":"done waiting on internal rate limiter","identifiers":["workflow.saidhasyim.com"],"ca":"https://acme-v02.api.letsencrypt.org/directory","account":""}
windmill-caddy-1  | {"level":"info","ts":1680574635.190461,"logger":"http.acme_client","msg":"trying to solve challenge","identifier":"workflow.saidhasyim.com","challenge_type":"http-01","ca":"https://acme-v02.api.letsencrypt.org/directory"}
windmill-caddy-1  | {"level":"info","ts":1680574635.4389567,"logger":"http","msg":"served key authentication","identifier":"workflow.saidhasyim.com","challenge":"http-01","remote":"3.143.0.194:36732","distributed":false}
windmill-caddy-1  | {"level":"info","ts":1680574635.529792,"logger":"http","msg":"served key authentication","identifier":"workflow.saidhasyim.com","challenge":"http-01","remote":"23.178.112.208:30014","distributed":false}
windmill-caddy-1  | {"level":"info","ts":1680574635.656017,"logger":"http","msg":"served key authentication","identifier":"workflow.saidhasyim.com","challenge":"http-01","remote":"34.216.251.44:31126","distributed":false}
windmill-caddy-1  | {"level":"info","ts":1680574635.8512971,"logger":"http.acme_client","msg":"authorization finalized","identifier":"workflow.saidhasyim.com","authz_status":"valid"}
windmill-caddy-1  | {"level":"info","ts":1680574635.8513706,"logger":"http.acme_client","msg":"validations succeeded; finalizing order","order":"https://acme-v02.api.letsencrypt.org/acme/order/1044155017/174062539747"}
windmill-caddy-1  | {"level":"info","ts":1680574636.5738835,"logger":"http.acme_client","msg":"successfully downloaded available certificate chains","count":2,"first_url":"https://acme-v02.api.letsencrypt.org/acme/cert/0449b859130062e910376228a7cb8ef7e314"}
windmill-caddy-1  | {"level":"info","ts":1680574636.5749109,"logger":"tls.obtain","msg":"certificate obtained successfully","identifier":"workflow.saidhasyim.com"}
windmill-caddy-1  | {"level":"info","ts":1680574636.5750582,"logger":"tls.obtain","msg":"releasing lock","identifier":"workflow.saidhasyim.com"}
windmill-caddy-1  | {"level":"info","ts":1680574656.2971783,"logger":"admin.api","msg":"received request","method":"POST","host":"localhost:2019","uri":"/load","remote_ip":"127.0.0.1","remote_port":"48880","headers":{"Accept-Encoding":["gzip"],"Content-Length":["289"],"Content-Type":["application/json"],"Origin":["http://localhost:2019"],"User-Agent":["Go-http-client/1.1"]}}
windmill-caddy-1  | {"level":"info","ts":1680574656.2973208,"msg":"config is unchanged"}
windmill-caddy-1  | {"level":"info","ts":1680574656.2973328,"logger":"admin.api","msg":"load complete"}

Yeah, you’re only handling requests for /ws/* and nothing else. And you’re still proxying to localhost which won’t work like I mentioned.

sorry, i missed that out. Is it correct that i do just like this in the caddyfile?

https://{$BASE_URL}

I just tried this, and still blank page. Should I add something else? Sorry, I’m still trying to study the caddy documentation

Do you mean literally only that? Definitely not, because you need something in your config to handle requests (e.g. a reverse_proxy, or respond, or file_server, etc).

To be clear, the site address should only be a hostname (domain) and optionally a scheme. If no scheme is provided, then https:// is assumed. See the docs: Caddyfile Concepts — Caddy Documentation

So using an env var named BASE_URL is misleading, because there should be no path/query, it’s not really a full URL.

Thanks very much. I have hardcoded the BaseURL value inside. Now it looks like this. does it look about correct? I also change localhost to the public IP of the website

https://workflow.saidhasyim.com  {  
  reverse_proxy * http://<<IP ADDRESS>>:3001
}

Wasn’t this more correct? That would proxy to the Docker containers named lsp and windmill_server.

I can’t tell you what is correct, I don’t know what you’re trying to serve.

You can simplify this to simply workflow.saidhasyim.com (i.e. remove the scheme). Also, you can continue to use an env var as you did before, but I recommend changing the env var name to something else so it’s not misleading.

that works!! I thank you so much. Now i roughly understand the concept

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.