Caddy on Docker swarm received HTTP 404

1. The problem I’m having:

HTTP Error 404

未命名

2. Error messages and/or full log output:


2024-03-12 15:14:30 {"level":"info","ts":1710227670.7920194,"msg":"using provided configuration","config_file":"/etc/caddy/Caddyfile","config_adapter":"caddyfile"}
2024-03-12 15:14:30 {"level":"warn","ts":1710227670.7925587,"msg":"Caddyfile input is not formatted; run 'caddy fmt --overwrite' to fix inconsistencies","adapter":"caddyfile","file":"/etc/caddy/Caddyfile","line":1}
2024-03-12 15:14:30 {"level":"info","ts":1710227670.7933304,"logger":"admin","msg":"admin endpoint started","address":"localhost:2019","enforce_origin":false,"origins":["//localhost:2019","//[::1]:2019","//127.0.0.1:2019"]}
2024-03-12 15:14:30 {"level":"info","ts":1710227670.7934408,"logger":"http.auto_https","msg":"server is listening only on the HTTPS port but has no TLS connection policies; adding one to enable TLS","server_name":"srv0","https_port":443}
2024-03-12 15:14:30 {"level":"info","ts":1710227670.7934673,"logger":"http.auto_https","msg":"enabling automatic HTTP->HTTPS redirects","server_name":"srv0"}
2024-03-12 15:14:30 {"level":"info","ts":1710227670.7935898,"logger":"tls.cache.maintenance","msg":"started background certificate maintenance","cache":"0xc000419500"}
2024-03-12 15:14:30 {"level":"warn","ts":1710227670.798147,"logger":"tls","msg":"storage cleaning happened too recently; skipping for now","storage":"FileStorage:/data/caddy","instance":"5f458cdb-da79-4d2a-9146-38bb02bc2fe5","try_again":1710314070.7981458,"try_again_in":86399.9999997}
2024-03-12 15:14:30 {"level":"info","ts":1710227670.7982256,"logger":"tls","msg":"finished cleaning storage units"}
2024-03-12 15:14:30 {"level":"warn","ts":1710227670.8294501,"logger":"pki.ca.local","msg":"installing root certificate (you might be prompted for password)","path":"storage:pki/authorities/local/root.crt"}
2024-03-12 15:14:30 {"level":"info","ts":1710227670.83146,"msg":"warning: \"certutil\" is not available, install \"certutil\" with \"apt install libnss3-tools\" or \"yum install nss-tools\" and try again"}
2024-03-12 15:14:30 {"level":"info","ts":1710227670.831504,"msg":"define JAVA_HOME environment variable to use the Java trust"}
2024-03-12 15:14:30 {"level":"info","ts":1710227670.8676543,"msg":"certificate installed properly in linux trusts"}
2024-03-12 15:14:30 {"level":"info","ts":1710227670.8680847,"logger":"http.log","msg":"server running","name":"remaining_auto_https_redirects","protocols":["h1","h2","h3"]}
2024-03-12 15:14:30 {"level":"info","ts":1710227670.8681805,"logger":"http","msg":"enabling HTTP/3 listener","addr":":443"}
2024-03-12 15:14:30 {"level":"info","ts":1710227670.8683345,"msg":"failed to sufficiently increase receive buffer size (was: 208 kiB, wanted: 2048 kiB, got: 416 kiB). See https://github.com/quic-go/quic-go/wiki/UDP-Buffer-Sizes for details."}
2024-03-12 15:14:30 {"level":"info","ts":1710227670.868477,"logger":"http.log","msg":"server running","name":"srv0","protocols":["h1","h2","h3"]}
2024-03-12 15:14:30 {"level":"info","ts":1710227670.868504,"logger":"http","msg":"enabling automatic TLS certificate management","domains":["localhost"]}
2024-03-12 15:14:30 {"level":"info","ts":1710227670.868777,"logger":"tls.obtain","msg":"acquiring lock","identifier":"localhost"}
2024-03-12 15:14:30 {"level":"info","ts":1710227670.8690805,"msg":"autosaved config (load with --resume flag)","file":"/config/caddy/autosave.json"}
2024-03-12 15:14:30 {"level":"info","ts":1710227670.869109,"msg":"serving initial configuration"}
2024-03-12 15:14:30 {"level":"info","ts":1710227670.8717136,"logger":"tls.obtain","msg":"lock acquired","identifier":"localhost"}
2024-03-12 15:14:30 {"level":"info","ts":1710227670.8718152,"logger":"tls.obtain","msg":"obtaining certificate","identifier":"localhost"}
2024-03-12 15:14:30 {"level":"info","ts":1710227670.8732607,"logger":"tls.obtain","msg":"certificate obtained successfully","identifier":"localhost"}
2024-03-12 15:14:30 {"level":"info","ts":1710227670.8733213,"logger":"tls.obtain","msg":"releasing lock","identifier":"localhost"}
2024-03-12 15:14:30 {"level":"warn","ts":1710227670.8734894,"logger":"tls","msg":"stapling OCSP","error":"no OCSP stapling for [localhost]: no OCSP server specified in certificate","identifiers":["localhost"]}

3. Caddy version:

FROM caddy:2.7.6-alpine

4. How I installed and ran Caddy:

a. System environment:

Docker swarm, Docker v4.28.0 on Windows 11

b. Command:

=== docker build and push ===
docker build -f caddy.dockerfile -t andrewwang1tw/micro-caddy:1.0.2 .  
docker push andrewwang1tw/micro-caddy:1.0.2

=== docker swarm init and deploy ===
docker swarm init
docker stack deploy -c swarm.yml myapp

c. Service/unit/compose file:

Docker file and Docker swarm file

==- Docker file ===
FROM caddy:2.7.6-alpine

COPY Caddyfile /etc/caddy/Caddyfile

===  Docker swarm file =====
version: '3'

services:

  caddy:
    image: andrewwang1tw/micro-caddy:1.0.2
    deploy:
      mode: replicated
      replicas: 1
    ports:  
      - "80:80"
      - "443:443"
    volumes:  
      - caddy_data:/data
      - caddy_config:/config  

volumes: 
  caddy_data:
    external: true
  caddy_config:   

d. My complete Caddy config:

localhost

respond "Hello, world!"

5. Links to relevant resources:

That 404 isn’t coming from Caddy at all. It looks to me like an apache 404.

Are you sure you’re actually hitting Caddy?

Make a request with curl -v, show the output. You should see a Server header which should hint to you what’s happening.

If curl http://localhost:80, then
“HTTP Error 404. The requested resource is not found.”

If curl https://localhost:80, then
“The underlying connection is closed: An unexpected error occurred while sending.”

ERR_SSL_PROTOCOL_ERROR <= error if using chrome

SSL_ERROR_RX_RECORD_TOO_LONG <= error if using firefox

PS C:\Users\andrewwang1\source\repos\udemy-microservice-golang\project> curl https://localhost:80 -v
>>
詳細資訊: GET with 0-byte payload
curl : Not Found
HTTP Error 404. The requested resource is not found.
位於 線路:1 字元:1
+ curl http://localhost:80 -v
+ ~~~~~~~~~~~~~~~~~~~~~~~~~~~
    + CategoryInfo          : InvalidOperation: (System.Net.HttpWebRequest:HttpWebRequest) [Invoke-WebRequest],WebException
    + FullyQualifiedErrorId : WebCmdletWebResponseException,Microsoft.PowerShell.Commands.InvokeWebRequestCommand

PS C:\Users\andrewwang1\source\repos\udemy-microservice-golang\project> curl https://localhost:80 -v
>>
詳細資訊: GET with 0-byte payload
curl : 基礎連接已關閉: 傳送時發生未預期的錯誤。(The underlying connection is closed: An unexpected error occurred while sending.)
位於 線路:1 字元:1
+ curl https://localhost:80 -v
+ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~
    + CategoryInfo          : InvalidOperation: (System.Net.HttpWebRequest:HttpWebRequest) [Invoke-WebRequest],WebException
    + FullyQualifiedErrorId : WebCmdletWebResponseException,Microsoft.PowerShell.Commands.InvokeWebRequestCommand

You don’t have curl installed on your machine, that’s Windows’ terrible Invoke-WebRequest. Install the real curl and try that again.

This doesn’t make sense, port 80 serves HTTP, you can’t make an HTTPS request to it.

The reason I try HTTPS is because the caddy quick start tutorial(as below pic), it uses HTTPS. It’s strange and that’s why I provide both HTTP and HTTPS.

And the reason you think I have no curl is because the message “curl : Not Found”. But I had curl installed.

Even downloaded Caddy and runs on Windows 11, the caddy process exits.

Even I change to port 6666, caddy still exits

No, you don’t have curl. You have a stupid Windows command alias to Invoke-WebRequest. See Removing the PowerShell curl alias? | daniel.haxx.se from the developer of curl complaining about this stupidity of Windows.

See here, in your command output, it clearly says Invoke-WebRequest:

You can use HTTPS, but you can’t use port 80 with HTTPS. You have to use port 443 (or just omit the port, which makes it default to 443).

That’s what the tutorial does, use https://localhost, not https://localhost:80 which is invalid.

You probably need to run the Caddy process as an administrator. Caddy needs permission to bind to ports, it’s giving an error because it wasn’t able to bind to localhost:2019 for the admin endpoint.

This is to test Caddy runs on Windows 11

  1. I already started the Windows terminal with the “administrator”

  2. Change the port to 6666

Caddy still exits

  1. curl already installed.
PS C:\ProgramData\chocolatey\lib\curl\tools\curl-8.6.0_1-win64-mingw\bin> choco install curl
Chocolatey v1.1.0
Installing the following packages:
curl
By installing, you accept licenses for the packages.
curl v8.6.0 already installed.
 Use --force to reinstall, specify a version to install, or try upgrade.

Chocolatey installed 0/1 packages.
 See the log for details (C:\ProgramData\chocolatey\logs\chocolatey.log).

Warnings:
 - curl - curl v8.6.0 already installed.
 Use --force to reinstall, specify a version to install, or try upgrade.
  1. ./curl http://localhost:80 -v

This is to test Caddy on Docker swarm, with the Caddyfile as

localhost
respond "Hello, world!"

The result

PS C:\ProgramData\chocolatey\lib\curl\tools\curl-8.6.0_1-win64-mingw\bin> ./curl http://localhost:80 -v
* Host localhost:80 was resolved.
* IPv6: ::1
* IPv4: 127.0.0.1
*   Trying [::1]:80...
* Connected to localhost (::1) port 80
> GET / HTTP/1.1
> Host: localhost
> User-Agent: curl/8.6.0
> Accept: */*
>
< HTTP/1.1 404 Not Found
< Content-Type: text/html; charset=us-ascii
< Server: Microsoft-HTTPAPI/2.0
< Date: Tue, 12 Mar 2024 10:04:06 GMT
< Connection: close
< Content-Length: 315
<
<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01//EN""http://www.w3.org/TR/html4/strict.dtd">
<HTML><HEAD><TITLE>Not Found</TITLE>
<META HTTP-EQUIV="Content-Type" Content="text/html; charset=us-ascii"></HEAD>
<BODY><h2>Not Found</h2>
<hr><p>HTTP Error 404. The requested resource is not found.</p>
</BODY></HTML>
* Closing connection
PS C:\ProgramData\chocolatey\lib\curl\tools\curl-8.6.0_1-win64-mingw\bin>

You’re hitting a Microsoft server of somekind, not Caddy.

That’s strange.

However, I did not install IIS on the current Windows 11.

And cannot find HTTPAPI (Suggested by Gemini)

For not hitting the Microsoft-HTTPAPI/2.0 at port 80, I change the Caddyfile to port 6688 as below

localhost:6688

respond "Hello, world!"

Then

./curl http://localhost:6688 -v

This time, it did not hit MS-HTTPAPI but still not connect to Caddy

PS C:\ProgramData\chocolatey\lib\curl\tools\curl-8.6.0_1-win64-mingw\bin> ./curl http://localhost:6688 -v
* Host localhost:6688 was resolved.
* IPv6: ::1
* IPv4: 127.0.0.1
*   Trying [::1]:6688...
*   Trying 127.0.0.1:6688...
* connect to ::1 port 6688 from :: port 10883 failed: Connection refused
* connect to 127.0.0.1 port 6688 from 0.0.0.0 port 10884 failed: Connection refused
* Failed to connect to localhost port 6688 after 2258 ms: Couldn't connect to server
* Closing connection
curl: (7) Failed to connect to localhost port 6688 after 2258 ms: Couldn't connect to server
PS C:\ProgramData\chocolatey\lib\curl\tools\curl-8.6.0_1-win64-mingw\bin>

Did you update your docker compose after changing that port?

You could leave your Caddyfile the same (no custom port) and just change the Docker Compose file to map to a different port on the host.

Honestly, Windows networking stuff is a mystery to me. It usually just works for me when I use it; I run Caddy as a Windows service.

Docker on Windows is especially troublesome because you’re running inside of a virtual machine. For develop anything using Docker, I prefer Linux, since it’s actually native.

Yes, the port on docker is changed to 6688

If

You could leave your Caddyfile the same (no custom port) and just change the Docker Compose file to map to a different port on the host.

Same result

./curl http://localhost:6688 -v
* Host localhost:6688 was resolved.
* IPv6: ::1
* IPv4: 127.0.0.1
*   Trying [::1]:6688...
*   Trying 127.0.0.1:6688...
* connect to ::1 port 6688 from :: port 11386 failed: Connection refused
* connect to 127.0.0.1 port 6688 from 0.0.0.0 port 11387 failed: Connection refused
* Failed to connect to localhost port 6688 after 2272 ms: Couldn't connect to server
* Closing connection
curl: (7) Failed to connect to localhost port 6688 after 2272 ms: Couldn't connect to server

Well, your problem is with Docker networking, not with Caddy specifically. I don’t know what to tell you.

OK, Thanks.

The result, if changed to Nginx as the reverse proxy, is the same with Caddy.
Looks like the Docker networking problem on Windows 11.

  1. If port 80 hits the “Server: Microsoft-HTTPAPI/2.0”
PS C:\ProgramData\chocolatey\lib\curl\tools\curl-8.6.0_1-win64-mingw\bin> ./curl http://localhost -v     
* Host localhost:80 was resolved.
* IPv6: ::1
* IPv4: 127.0.0.1
*   Trying [::1]:80...
* Connected to localhost (::1) port 80
> GET / HTTP/1.1
> Host: localhost
> User-Agent: curl/8.6.0
> Accept: */*
>
< HTTP/1.1 404 Not Found
< Content-Type: text/html; charset=us-ascii
< Server: Microsoft-HTTPAPI/2.0
< Date: Wed, 13 Mar 2024 02:11:41 GMT
< Connection: close
< Content-Length: 315
<
<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01//EN""http://www.w3.org/TR/html4/strict.dtd">
<HTML><HEAD><TITLE>Not Found</TITLE>
<META HTTP-EQUIV="Content-Type" Content="text/html; charset=us-ascii"></HEAD>
<BODY><h2>Not Found</h2>
<hr><p>HTTP Error 404. The requested resource is not found.</p>
</BODY></HTML>
* Closing connection
  1. If custom port, refuse to connect

PS C:\ProgramData\chocolatey\lib\curl\tools\curl-8.6.0_1-win64-mingw\bin> ./curl http://localhost:7878 -v
* Host localhost:7878 was resolved.
* IPv6: ::1
* IPv4: 127.0.0.1
*   Trying [::1]:7878...
*   Trying 127.0.0.1:7878...
* connect to ::1 port 7878 from :: port 4056 failed: Connection refused
* connect to 127.0.0.1 port 7878 from 0.0.0.0 port 4057 failed: Connection refused
* Failed to connect to localhost port 7878 after 2244 ms: Couldn't connect to server
* Closing connection
curl: (7) Failed to connect to localhost port 7878 after 2244 ms: Couldn't connect to server
PS C:\ProgramData\chocolatey\lib\curl\tools\curl-8.6.0_1-win64-mingw\bin> ./curl http://localhost:7878 -v
* Host localhost:7878 was resolved.
* IPv6: ::1
* IPv4: 127.0.0.1
*   Trying [::1]:7878...
*   Trying 127.0.0.1:7878...
* connect to ::1 port 7878 from :: port 4325 failed: Connection refused
* connect to 127.0.0.1 port 7878 from 0.0.0.0 port 4326 failed: Connection refused
* Failed to connect to localhost port 7878 after 2225 ms: Couldn't connect to server
* Closing connection
curl: (7) Failed to connect to localhost port 7878 after 2225 ms: Couldn't connect to server
PS C:\ProgramData\chocolatey\lib\curl\tools\curl-8.6.0_1-win64-mingw\bin>

my post on docker forum

=============================
If change Caddyfile as below, it works

:7788

respond "Hello, world!"
PS C:\ProgramData\chocolatey\lib\curl\tools\curl-8.6.0_1-win64-mingw\bin> ./curl http://localhost:7788 -v
* Host localhost:7788 was resolved.
* IPv6: ::1
* IPv4: 127.0.0.1
*   Trying [::1]:7788...
* Connected to localhost (::1) port 7788
> GET / HTTP/1.1
> Host: localhost:7788
> User-Agent: curl/8.6.0
> Accept: */*
> 
< HTTP/1.1 200 OK
< Content-Type: text/plain; charset=utf-8
< Server: Caddy
< Date: Thu, 14 Mar 2024 09:22:51 GMT
< Content-Length: 13
< 
Hello, world!* Connection #0 to host localhost left intact
PS C:\ProgramData\chocolatey\lib\curl\tools\curl-8.6.0_1-win64-mingw\bin>

==================================
But this Caddyfile, not works

localhost:7788

respond Hello World
PS C:\ProgramData\chocolatey\lib\curl\tools\curl-8.6.0_1-win64-mingw\bin> ./curl http://localhost:7788 -v
* Host localhost:7788 was resolved.
* IPv6: ::1
* IPv4: 127.0.0.1
*   Trying [::1]:7788...
* Connected to localhost (::1) port 7788
> GET / HTTP/1.1
> Host: localhost:7788
> User-Agent: curl/8.6.0
> Accept: */*
> 
* HTTP 1.0, assume close after body
< HTTP/1.0 400 Bad Request
< 
Client sent an HTTP request to an HTTPS server.
* Closing connection

That’s expected, Caddy defaults to HTTPS when given a valid domain. See Caddyfile Concepts — Caddy Documentation