Using Caddy in docker compose, upstreams breaks when using internal network

1. The problem I’m having:

I’m using Caddy in docker compose with django and postgres.

When I have just Caddy and Django in my compose file, caddy can reach my django web app easily.

Caddy + Django in docker compose

Albeit, I get a HTTP 500 error because my django app is configured with a DATABASE_HOST=db and there’s no db/postgres container running

But then when I add my postgres/db back to my compose file (attached below) my site goes from HTTP 500 to completely Not Found.

https://i.imgur.com/Kox8Ysb.png (Caddy + Django + Postgres)

If I remove postgres/db from the internal network then I can reach the 500 Error again.

So how do I use postgres in an internal network without getting this not found proxy error… If anyone could please advise that would be amazing, been working on this for hours now.

https://medium.com/@fish_underlings/struggling-with-caddy-docker-proxy-2936f9256a79

I found this post and he had the EXACT same issue I’m having, so it’s 100% a Docker network issue. I tried his internal bridge network solution but nothing :confused:

I would really appreciate it if anyone could help me or point me in the right direction because I am exhausted at this point

2. Error messages and/or full log output:

*Logs for caddy + django + postgres (getting Not Found, not proxying correctly)

*Logs for just caddy + django (working/proxies correctly)

{"level":"info","ts":1706994138.850038,"logger":"docker-proxy","msg":"Running caddy proxy server"}
{"level":"info","ts":1706994138.8522117,"logger":"admin","msg":"admin endpoint started","address":"localhost:2019","enforce_origin":false,"origins":["//localhost:2019","//[::1]:2019","//127.0.0.1:2019"]}
{"level":"info","ts":1706994138.8527148,"msg":"autosaved config (load with --resume flag)","file":"/config/caddy/autosave.json"}
{"level":"info","ts":1706994138.852879,"logger":"docker-proxy","msg":"Running caddy proxy controller"}
{"level":"info","ts":1706994138.8545039,"logger":"docker-proxy","msg":"Start","CaddyfilePath":"","EnvFile":"","LabelPrefix":"caddy","PollingInterval":30,"ProxyServiceTasks":true,"ProcessCaddyfile":true,"ScanStoppedContainers":false,"IngressNetworks":"[caddy]","DockerSockets":[""],"DockerCertsPath":[""],"DockerAPIsVersion":[""]}
{"level":"info","ts":1706994138.857044,"logger":"docker-proxy","msg":"Connecting to docker events","DockerSocket":""}
{"level":"info","ts":1706994138.8577094,"logger":"docker-proxy","msg":"IngressNetworksMap","ingres":"map[a4e055452da979070a37cd470be23d91b618a05c66495fb0870b8620d5276182:true caddy:true]"}
{"level":"info","ts":1706994138.869174,"logger":"docker-proxy","msg":"Swarm is available","new":false}
{"level":"info","ts":1706994138.8731415,"logger":"docker-proxy","msg":"New Caddyfile","caddyfile":"supa.blog {\n\treverse_proxy 192.168.128.2:8000\n}\n"}
{"level":"info","ts":1706994138.8737316,"logger":"docker-proxy","msg":"New Config JSON","json":"{\"apps\":{\"http\":{\"servers\":{\"srv0\":{\"listen\":[\":443\"],\"routes\":[{\"match\":[{\"host\":[\"supa.blog\"]}],\"handle\":[{\"handler\":\"subroute\",\"routes\":[{\"handle\":[{\"handler\":\"reverse_proxy\",\"upstreams\":[{\"dial\":\"192.168.128.2:8000\"}]}]}]}],\"terminal\":true}]}}}}}"}
{"level":"info","ts":1706994138.8738334,"logger":"docker-proxy","msg":"Sending configuration to","server":"localhost"}
{"level":"info","ts":1706994138.8749814,"logger":"admin.api","msg":"received request","method":"POST","host":"localhost:2019","uri":"/load","remote_ip":"127.0.0.1","remote_port":"41212","headers":{"Accept-Encoding":["gzip"],"Content-Length":["290"],"Content-Type":["application/json"],"User-Agent":["Go-http-client/1.1"]}}
{"level":"info","ts":1706994138.875998,"logger":"admin","msg":"admin endpoint started","address":"localhost:2019","enforce_origin":false,"origins":["//localhost:2019","//[::1]:2019","//127.0.0.1:2019"]}
{"level":"info","ts":1706994138.8763082,"logger":"http.auto_https","msg":"server is listening only on the HTTPS port but has no TLS connection policies; adding one to enable TLS","server_name":"srv0","https_port":443}
{"level":"info","ts":1706994138.8765037,"logger":"http.auto_https","msg":"enabling automatic HTTP->HTTPS redirects","server_name":"srv0"}
{"level":"info","ts":1706994138.8772275,"logger":"http","msg":"enabling HTTP/3 listener","addr":":443"}
{"level":"info","ts":1706994138.8775022,"msg":"failed to sufficiently increase receive buffer size (was: 208 kiB, wanted: 2048 kiB, got: 416 kiB). See https://github.com/quic-go/quic-go/wiki/UDP-Buffer-Sizes for details."}
{"level":"info","ts":1706994138.8778794,"logger":"http.log","msg":"server running","name":"srv0","protocols":["h1","h2","h3"]}
{"level":"info","ts":1706994138.878126,"logger":"http.log","msg":"server running","name":"remaining_auto_https_redirects","protocols":["h1","h2","h3"]}
{"level":"info","ts":1706994138.878312,"logger":"http","msg":"enabling automatic TLS certificate management","domains":["supa.blog"]}
{"level":"info","ts":1706994138.879989,"msg":"autosaved config (load with --resume flag)","file":"/config/caddy/autosave.json"}
{"level":"info","ts":1706994138.8801703,"logger":"admin.api","msg":"load complete"}
{"level":"warn","ts":1706994138.8835196,"logger":"tls","msg":"storage cleaning happened too recently; skipping for now","storage":"FileStorage:/data/caddy","instance":"c6c47413-964c-4624-90a5-c72ffcebc8b5","try_again":1707080538.8835158,"try_again_in":86399.999998998}
{"level":"info","ts":1706994138.8839314,"logger":"tls","msg":"finished cleaning storage units"}
{"level":"info","ts":1706994138.8844116,"logger":"admin","msg":"stopped previous server","address":"localhost:2019"}
{"level":"info","ts":1706994138.8846436,"logger":"tls.cache.maintenance","msg":"started background certificate maintenance","cache":"0xc000039b80"}
{"level":"info","ts":1706994138.8853877,"logger":"docker-proxy","msg":"Successfully configured","server":"localhost"}
{"level":"error","ts":1706994912.45343,"logger":"http.log.error","msg":"EOF","request":{"remote_ip":"71.79.45.188","remote_port":"52582","client_ip":"71.79.45.188","proto":"HTTP/2.0","method":"GET","host":"supa.blog","uri":"/","headers":{"Accept-Encoding":["gzip, deflate, br"],"Cache-Control":["max-age=0"],"Sec-Ch-Ua":["\"Not A(Brand\";v=\"99\", \"Google Chrome\";v=\"121\", \"Chromium\";v=\"121\""],"Sec-Ch-Ua-Mobile":["?0"],"User-Agent":["Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/121.0.0.0 Safari/537.36"],"Accept":["text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.7"],"Sec-Fetch-Site":["none"],"Sec-Fetch-User":["?1"],"Accept-Language":["en-US,en;q=0.9"],"Sec-Ch-Ua-Platform":["\"Windows\""],"Upgrade-Insecure-Requests":["1"],"Sec-Fetch-Mode":["navigate"],"Sec-Fetch-Dest":["document"]},"tls":{"resumed":true,"version":772,"cipher_suite":4865,"proto":"h2","server_name":"supa.blog"}},"duration":30.073333047,"status":502,"err_id":"r3x8gczja","err_trace":"reverseproxy.statusError (reverseproxy.go:1267)"}

3. Caddy version:

v2.7.6 h1:w0NymbG2m9PcvKWsrXO6EEkY9Ru4FJK8uQbYcev1p3A=

4. How I installed and ran Caddy:

Docker compose on Ubuntu 22.04

version: "3.7"
services:

  web:
    build: .
    command: sh -c "gunicorn notobackend.wsgi:application --bind 0.0.0.0:8000"
    restart: always
    ports:
      - "8000:8000"
    expose:
      - 8000
    environment:
      - POSTGRES_DB=supadb
      - POSTGRES_USER=postgres
      - POSTGRES_PASSWORD=postgres
    networks:
      - caddy
      - internal
    container_name: web
    labels:
      caddy: supa.blog
      caddy.reverse_proxy: "{{upstreams 8000}}"
    env_file:
      - .env

  caddy:
      image: lucaslorentz/caddy-docker-proxy:ci-alpine
      ports:
        - 80:80
        - 443:443
      environment:
        - CADDY_INGRESS_NETWORKS=caddy
      networks:
        - caddy
        - internal
      volumes:
        - /var/run/docker.sock:/var/run/docker.sock
        - caddy_data:/data
      restart: unless-stopped
      

networks:
        caddy:
                external: true
        internal:
                external: false
                driver: bridge

volumes:
  caddy_data: {}

d. My complete Caddy config:

Not using Caddyfile, caddy-docker-proxy generates for you I believe

5. Links to relevant resources:

https://medium.com/@fish_underlings/struggling-with-caddy-docker-proxy-2936f9256a79

Caddy doesn’t emit 404s from reverse_proxy. That’s a problem with your Django app. Are you missing a route for that path? Is it missing data from your DB?

There’s no evidence that this is a Caddy problem here.

Thanks yeah I thought it was a Caddy error, didn’t realize it was Django lol

1 Like

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.