Stuggles with caddy on home server with mix continer apcatoins & non conaiter aplcations

Help
1

1: The problem I’m having:

to put it plainly, & simply I cannot access some of my services correctly through the caddy reverse proxy.

2: Error messages and/or full log output:

I don’t have any logs but when I try to access some of my services though caddy I get a 504 error

3: Caddy Version:

v2.11.2

4: Context on install:

I am running caddy inside a podman container on an Ubuntu server installation
I used Podman compose with a compsoe.yml and used $podman-compose up -d

a: The compose.yml:

version: "3"
services:
  caddy:
    image: docker.io/caddy:latest
    container_name: caddy
    restart: unless-stopped
    ports:
      - "8080:80"   # HTTP
      - "8543:443"  # HTTPS
    volumes:
      - ./Caddyfile:/etc/caddy/Caddyfile
      - ./data:/data
      - ./config:/config
networks:
  homelab:
    external: true

B: Caddy file:

{
    local_certs
}

# Health endpoints
:80 {
    respond "Caddy HTTP alive"
}

:443 {
    tls internal
    respond "Caddy HTTPS alive"
}

# Containerized services on Podman network
pinry.home {
    reverse_proxy pinry:80
}

wger.home {
    reverse_proxy wger:8000
}

immich.home {
    reverse_proxy immich:2283
}

# Host services using host LAN IP
nextcloud.home {
    reverse_proxy http://192.168.0.115:80
}

# Crafty Controller
crafty.home:8543 {
    reverse_proxy https://192.168.0.115:8443 {
        transport http {
            tls_insecure_skip_verify
            versions 3
        }

        # Headers for Crafty to work behind proxy
        header_up Host crafty.home
        header_up X-Forwarded-Proto https
        header_up X-Forwarded-For {remote_host}
        header_up X-Real-IP {remote_host}
    }
}

# Cockpit
cockpit.home {
    reverse_proxy https://192.168.0.115:9090 {
        transport http {
            tls_insecure_skip_verify
            versions 3
        }

        header_up Host cockpit.home
        header_up X-Forwarded-Proto https
        header_up X-Forwarded-For {remote_host}
        header_up X-Real-IP {remote_host}
    }
}

additional context

all my sites are on a home server and not exposed to the web and this is all on a Home-lab machine

2

First off, English is my fourth language, so I’m not in a position to judge anyone’s writing, and I wouldn’t want to anyway. It would really help if you could read your post over before submitting it. I tried my best, but it’s quite hard to follow.

I don’t want to just complain, so here are a few suggestions that might help:

  1. Try enabling debug in your Caddyfile. This may give you more detailed logs about what’s happening. If possible, please share the log output.

  2. Run curl -vk YOUR_SITE and share the results. This can help us see exactly what response you’re getting when accessing your site.

  3. For WebSocket connections, make sure your backend application is properly configured to tell the browser which port to use for the wss:// protocol. By default, browsers will try port 443 unless specified otherwise, while your service is running on port 8543.

2 Likes
3

apologies I hope the updated version is clearer. I will have to try what you have suggested although I suspect its a configuration problem. thanks again for responding.

1 Like
4

None of these headers are useful. Caddy sets proxy headers correctly automatically. See reverse_proxy (Caddyfile directive) — Caddy Documentation

Are you sure your upstream server has HTTP/3 support? Remove versions 3 unless you’re certain. That is likely to break things.

You haven’t shown any evidence of the actual problem you’re having, so all we can do is speculate which is unproductive. Share your logs. Share curl -vk output. Show evidence.

5

➜ ~ curl -vk crafty.home:5843

  • Host crafty.home:5843 was resolved.
  • IPv6: (none)
  • IPv4: 192.168.0.115
  • Trying 192.168.0.115:5843…
  • connect to 192.168.0.115 port 5843 from 192.168.0.127 port 41868 failed: Connection refused
  • Failed to connect to crafty.home port 5843 after 50 ms: Could not connect to server
  • closing connection #0
    curl: (7) Failed to connect to crafty.home port 5843 after 50 ms: Could not connect to server
    ➜ ~ curl -vk cockpit.home:5843
  • Host cockpit.home:5843 was resolved.
  • IPv6: (none)
  • IPv4: 192.168.0.115
  • Trying 192.168.0.115:5843…
  • connect to 192.168.0.115 port 5843 from 192.168.0.127 port 38550 failed: Connection refused
  • Failed to connect to cockpit.home port 5843 after 2227 ms: Could not connect to server
  • closing connection #0
    curl: (7) Failed to connect to cockpit.home port 5843 after 2227 ms: Could not connect to server

Here is the output of the command you asked for. crafty.home just returns the https is alive rather than the actual site running on local host:8443

PS: i have removed all headers a skip verfiy now jsut reverse proxy ip:prot

here is the updated caddyfile.

    local_certs
}

# Health endpoints
:80 {
    respond "Caddy HTTP alive"
}

:443 {
    tls internal
    respond "Caddy HTTPS alive"
}

# Containerized services on Podman network
pinry.home {
    reverse_proxy pinry:80
}

wger.home {
    reverse_proxy wger:8000
}

immich.home {
    reverse_proxy immich:2283
}

# Host services using host LAN IP
nextcloud.home {
    reverse_proxy http://192.168.0.115:80
}

# Crafty Controller
crafty.home:8543 {
    reverse_proxy https://192.168.0.115:8443 
}

# Cockpit
cockpit.home {
    reverse_proxy https://192.168.0.115:9090 {
}
6

Are you sure that’s the correct port? You didn’t mention 5843 anywhere else, but you did have 8543.

Keep in mind this has Caddy listening on port 8543 inside the container, but you only publish ports 80 and 443 to the host:

You probably need to remove :8543 from that site block.

You still haven’t shared your Caddy debug logs.

But either way, this is a networking config issue, not really a problem with Caddy config as far as I can tell at this point.

7

Ah, I see what I’ve done wrong Port there that’s the default Prot on the crafty service my mistake. (that’s the port caddy should direct to)

Here are the raw logs what should I be looking at specifically?

{"level":"info","ts":1774830977.4663255,"msg":"maxprocs: Leaving GOMAXPROCS=4: CPU quota undefined"}

{"level":"info","ts":1774830977.466337,"msg":"GOMEMLIMIT is updated","GOMEMLIMIT":7186905907,"previous":9223372036854775807}

{"level":"info","ts":1774830977.4663405,"msg":"using config from file","file":"/etc/caddy/Caddyfile"}

{"level":"info","ts":1774830977.4663427,"msg":"adapted config to JSON","adapter":"caddyfile"}

{"level":"warn","ts":1774830977.4663444,"msg":"Caddyfile input is not formatted; run 'caddy fmt --overwrite' to fix inconsistencies","adapter":"caddyfile","file":"/etc/caddy/Caddyfile","line":2}

{"level":"info","ts":1774830977.4668977,"logger":"admin","msg":"admin endpoint started","address":"localhost:2019","enforce_origin":false,"origins":["//127.0.0.1:2019","//localhost:2019","//[::1]:2019"]}

{"level":"info","ts":1774830977.4671361,"logger":"tls.cache.maintenance","msg":"started background certificate maintenance","cache":"0xdd77c78f580"}

{"level":"info","ts":1774830977.4673414,"logger":"http.auto_https","msg":"server is listening only on the HTTPS port but has no TLS connection policies; adding one to enable TLS","server_name":"srv0","https_port":443}

{"level":"info","ts":1774830977.4673572,"logger":"http.auto_https","msg":"enabling automatic HTTP->HTTPS redirects","server_name":"srv0"}

{"level":"warn","ts":1774830977.4673624,"logger":"http.auto_https","msg":"server is listening only on the HTTP port, so no automatic HTTPS will be applied to this server","server_name":"srv1","http_port":80}

{"level":"info","ts":1774830977.4673707,"logger":"http.auto_https","msg":"enabling automatic HTTP->HTTPS redirects","server_name":"srv2"}

{"level":"warn","ts":1774830977.4841487,"logger":"pki.ca.local","msg":"installing root certificate (you might be prompted for password)","path":"storage:pki/authorities/local/root.crt"}

{"level":"info","ts":1774830977.515397,"logger":"tls","msg":"storage cleaning happened too recently; skipping for now","storage":"FileStorage:/data/caddy","instance":"8af9baf6-45d6-4a87-8ac7-b918c9fde75f","try_again":1774917377.5153944,"try_again_in":86399.99999942}

{"level":"info","ts":1774830977.515767,"logger":"tls","msg":"finished cleaning storage units"}

{"level":"info","ts":1774830977.5327168,"msg":"warning: \"certutil\" is not available, install \"certutil\" with \"apt install libnss3-tools\" or \"yum install nss-tools\" and try again"}

{"level":"info","ts":1774830977.532729,"msg":"define JAVA_HOME environment variable to use the Java trust"}

{"level":"info","ts":1774830977.670596,"msg":"certificate installed properly in linux trusts"}

{"level":"info","ts":1774830977.670763,"logger":"http","msg":"enabling HTTP/3 listener","addr":":443"}

{"level":"info","ts":1774830977.6708708,"msg":"failed to sufficiently increase receive buffer size (was: 208 kiB, wanted: 7168 kiB, got: 416 kiB). See https://github.com/quic-go/quic-go/wiki/UDP-Buffer-Sizes for details."}

{"level":"info","ts":1774830977.6709483,"logger":"http.log","msg":"server running","name":"srv0","protocols":["h1","h2","h3"]}

{"level":"warn","ts":1774830977.671005,"logger":"http","msg":"HTTP/2 skipped because it requires TLS","network":"tcp","addr":":80"}

{"level":"warn","ts":1774830977.6710103,"logger":"http","msg":"HTTP/3 skipped because it requires TLS","network":"tcp","addr":":80"}

{"level":"info","ts":1774830977.671013,"logger":"http.log","msg":"server running","name":"srv1","protocols":["h1","h2","h3"]}

{"level":"info","ts":1774830977.6710455,"logger":"http","msg":"enabling HTTP/3 listener","addr":":8543"}

{"level":"info","ts":1774830977.6711464,"logger":"http.log","msg":"server running","name":"srv2","protocols":["h1","h2","h3"]}

{"level":"info","ts":1774830977.671153,"logger":"http","msg":"enabling automatic TLS certificate management","domains":["cockpit.home","immich.home","pinry.home","wger.home","nextcloud.home","crafty.home"]}

{"level":"warn","ts":1774830977.6714225,"logger":"tls","msg":"stapling OCSP","identifiers":["pinry.home"]}

{"level":"warn","ts":1774830977.6716352,"logger":"tls","msg":"stapling OCSP","identifiers":["wger.home"]}

{"level":"warn","ts":1774830977.6718142,"logger":"tls","msg":"stapling OCSP","identifiers":["nextcloud.home"]}

{"level":"warn","ts":1774830977.672034,"logger":"tls","msg":"stapling OCSP","identifiers":["crafty.home"]}

{"level":"warn","ts":1774830977.672241,"logger":"tls","msg":"stapling OCSP","identifiers":["cockpit.home"]}

{"level":"warn","ts":1774830977.6724296,"logger":"tls","msg":"stapling OCSP","identifiers":["immich.home"]}

{"level":"info","ts":1774830977.672553,"msg":"autosaved config (load with --resume flag)","file":"/config/caddy/autosave.json"}

{"level":"info","ts":1774830977.6725652,"msg":"serving initial configuration"}

{"level":"error","ts":1774831030.9423094,"logger":"http.log.error","msg":"tls: failed to verify certificate: x509: certificate is valid for 127.0.0.1, not 192.168.0.115","request":{"remote_ip":"10.89.0.9","remote_port":"38120","client_ip":"10.89.0.9","proto":"HTTP/2.0","method":"GET","host":"cockpit.home:8543","uri":"/","headers":{"Priority":["u=0, i"],"Accept-Language":["en-US,en;q=0.9"],"Sec-Fetch-Mode":["navigate"],"Te":["trailers"],"User-Agent":["Mozilla/5.0 (X11; Linux x86_64; rv:149.0) Gecko/20100101 Firefox/149.0"],"Accept":["text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8"],"Accept-Encoding":["gzip, deflate, br, zstd"],"Upgrade-Insecure-Requests":["1"],"Sec-Fetch-Site":["none"],"Sec-Fetch-Dest":["document"],"Sec-Fetch-User":["?1"],"Sec-Gpc":["1"]},"tls":{"resumed":false,"version":772,"cipher_suite":4865,"proto":"h2","server_name":"cockpit.home","ech":false}},"duration":0.007035712,"status":502,"err_id":"eyj6fr3he","err_trace":"reverseproxy.statusError (reverseproxy.go:1525)"}

{"level":"error","ts":1774831030.9828103,"logger":"http.log.error","msg":"tls: failed to verify certificate: x509: certificate is valid for 127.0.0.1, not 192.168.0.115","request":{"remote_ip":"10.89.0.9","remote_port":"38124","client_ip":"10.89.0.9","proto":"HTTP/2.0","method":"GET","host":"cockpit.home:8543","uri":"/favicon.ico","headers":{"Accept":["*/*"],"Accept-Language":["en-US,en;q=0.9"],"Accept-Encoding":["gzip, deflate, br, zstd"],"Priority":["u=4"],"Te":["trailers"],"User-Agent":["Mozilla/5.0 (X11; Linux x86_64; rv:149.0) Gecko/20100101 Firefox/149.0"]},"tls":{"resumed":false,"version":772,"cipher_suite":4865,"proto":"h2","server_name":"cockpit.home","ech":false}},"duration":0.006795546,"status":502,"err_id":"w4ezhvr80","err_trace":"reverseproxy.statusError (reverseproxy.go:1525)"}

{"level":"error","ts":1774831030.9940543,"logger":"http.log.error","msg":"tls: failed to verify certificate: x509: certificate is valid for 127.0.0.1, not 192.168.0.115","request":{"remote_ip":"10.89.0.9","remote_port":"38124","client_ip":"10.89.0.9","proto":"HTTP/2.0","method":"GET","host":"cockpit.home:8543","uri":"/favicon.ico","headers":{"Te":["trailers"],"User-Agent":["Mozilla/5.0 (X11; Linux x86_64; rv:149.0) Gecko/20100101 Firefox/149.0"],"Accept":["*/*"],"Accept-Language":["en-US,en;q=0.9"],"Accept-Encoding":["gzip, deflate, br, zstd"],"Priority":["u=4"]},"tls":{"resumed":false,"version":772,"cipher_suite":4865,"proto":"h2","server_name":"cockpit.home","ech":false}},"duration":0.004720465,"status":502,"err_id":"f5rg53g4t","err_trace":"reverseproxy.statusError (reverseproxy.go:1525)"}

{"level":"error","ts":1774831031.084907,"logger":"http.log.error","msg":"tls: failed to verify certificate: x509: certificate is valid for 127.0.0.1, not 192.168.0.115","request":{"remote_ip":"10.89.0.9","remote_port":"38124","client_ip":"10.89.0.9","proto":"HTTP/2.0","method":"GET","host":"cockpit.home:8543","uri":"/favicon.ico","headers":{"Accept-Language":["en-US,en;q=0.9"],"Accept-Encoding":["gzip, deflate, br, zstd"],"Priority":["u=4"],"Te":["trailers"],"User-Agent":["Mozilla/5.0 (X11; Linux x86_64; rv:149.0) Gecko/20100101 Firefox/149.0"],"Accept":["*/*"]},"tls":{"resumed":false,"version":772,"cipher_suite":4865,"proto":"h2","server_name":"cockpit.home","ech":false}},"duration":0.006576598,"status":502,"err_id":"77z54we4g","err_trace":"reverseproxy.statusError (reverseproxy.go:1525)"}

{"level":"error","ts":1774831031.1869912,"logger":"http.log.error","msg":"tls: failed to verify certificate: x509: certificate is valid for 127.0.0.1, not 192.168.0.115","request":{"remote_ip":"10.89.0.9","remote_port":"38124","client_ip":"10.89.0.9","proto":"HTTP/2.0","method":"GET","host":"cockpit.home:8543","uri":"/favicon.ico","headers":{"Accept-Language":["en-US,en;q=0.9"],"Accept-Encoding":["gzip, deflate, br, zstd"],"Priority":["u=4"],"Te":["trailers"],"User-Agent":["Mozilla/5.0 (X11; Linux x86_64; rv:149.0) Gecko/20100101 Firefox/149.0"],"Accept":["*/*"]},"tls":{"resumed":false,"version":772,"cipher_suite":4865,"proto":"h2","server_name":"cockpit.home","ech":false}},"duration":0.006272648,"status":502,"err_id":"mfux7mcnv","err_trace":"reverseproxy.statusError (reverseproxy.go:1525)"}
8

Caddy can’t connect to your upstream app because it can’t trust the TLS cert served by it since you’re connecting over HTTPS and the app uses a self-signed cert.

If the service has an HTTP endpoint, use that instead, it’s still secure because the part that needs to be secured is the connection over the open internet. The connection between Caddy and your apps are on your private network so there’s no real risk.

9

Thanks for the help. Unfortunately I don’t think it does have an HTTP endpoint for crafty nor cockpit. Are there any other options?

10

I think it’s a stupid choice when self hosted apps default to self signed certs for HTTPS, it gives a false sense of security.

Cockpit has a --no-tls mode https://cockpit-project.org/guide/latest/guide.html

It seems like Crafty entirely removed the HTTP listener (used to be 8000), I think that’s very dumb.

But you can configure Caddy’s reverse_proxy with tls_insecure_skip_verify which disables certificate verification when connecting to the upstream. This obviously throws away the entire point of HTTPS, disables all security HTTPS offers, but it’s the only way to do it unless you want to spend the effort of establishing trust between the app and Caddy (i.e. configuring the upstream app with a specific cert in your control, then configure Caddy to trust that certificate in particular).

1 Like