File server with fallback on reverse proxy not working

Help
1

1. The problem I’m having:

Saleor ecommerce wants to only handle media links if the files don’t exist, so it generates it, and then have nginx/caddy serving the files.
I’m using docker-caddy-proxy.
My docker-compose.yml:

services:
  caddy:
    image: lucaslorentz/caddy-docker-proxy:ci-alpine
    ports:
      - 80:80
      - 443:443
    environment:
      - CADDY_INGRESS_NETWORKS=kremik-sk_caddy
    networks:
      - caddy
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock
      - caddy:/data
      - media:/saleor-media

  api:
    image: ghcr.io/saleor/saleor:3.19.11
    networks:
      - caddy
    volumes:
      - media:/app/media
    labels:
      caddy: "api.kremik.dev"
      caddy.0_root: "* /saleor-media/"
      caddy.1_@file.file: "{path} {path}/"
      caddy.2_tls: "internal"
      caddy.3_handle: "@file"
      caddy.3_handle.0_uri: "strip_prefix /media"
      caddy.3_handle.2_file_server: ""
      caddy.4_handle.0_reverse_proxy: "{{upstreams 8000}}"

volumes:
  media:
networks:
  caddy:

This generates the following caddyfile:

api.kremik.dev {
	@file {
		file {path} {path}/
	}
	root * /saleor-media/
	tls internal
	handle @file {
		uri strip_prefix /media
		file_server
	}
	handle {
		reverse_proxy 172.24.0.12:8000
	}
}

When I try to get an image at, say https://api.kremik.dev/media/thumbnails/products/prod1_thumbnail_512.webp, I get 404, cause Saleor made that image already and expects caddy to serve it.

Unfortunately, it seems caddy never tries to use file handler and always reverse proxies, which leads to 404s.

I verified, and /saleor-media exists in caddys container.

2. Error messages and/or full log output:

╭─djkato@djkato in ~ took 59ms
[🔴] × curl -vLk https://api.kremik.dev/media/thumbnails/products/poplinBaelastSweetsoranzzlty_1691662575_14cbd5d6_thumbnail_512.webp
* Host api.kremik.dev:443 was resolved.
* IPv6: (none)
* IPv4: 10.100.110.44
*   Trying 10.100.110.44:443...
* Connected to api.kremik.dev (10.100.110.44) port 443
* ALPN: curl offers h2,http/1.1
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* TLSv1.3 (IN), TLS handshake, Server hello (2):
* TLSv1.3 (IN), TLS handshake, Encrypted Extensions (8):
* TLSv1.3 (IN), TLS handshake, Certificate (11):
* TLSv1.3 (IN), TLS handshake, CERT verify (15):
* TLSv1.3 (IN), TLS handshake, Finished (20):
* TLSv1.3 (OUT), TLS change cipher, Change cipher spec (1):
* TLSv1.3 (OUT), TLS handshake, Finished (20):
* SSL connection using TLSv1.3 / TLS_AES_128_GCM_SHA256 / x25519 / id-ecPublicKey
* ALPN: server accepted h2
* Server certificate:
*  subject: [NONE]
*  start date: May  2 14:54:40 2024 GMT
*  expire date: May  3 02:54:40 2024 GMT
*  issuer: CN=Caddy Local Authority - ECC Intermediate
*  SSL certificate verify result: unable to get local issuer certificate (20), continuing anyway.
*   Certificate level 0: Public key type EC/prime256v1 (256/128 Bits/secBits), signed using ecdsa-with-SHA256
*   Certificate level 1: Public key type EC/prime256v1 (256/128 Bits/secBits), signed using ecdsa-with-SHA256
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
* using HTTP/2
* [HTTP/2] [1] OPENED stream for https://api.kremik.dev/media/thumbnails/products/poplinBaelastSweetsoranzzlty_1691662575_14cbd5d6_thumbnail_512.webp
* [HTTP/2] [1] [:method: GET]
* [HTTP/2] [1] [:scheme: https]
* [HTTP/2] [1] [:authority: api.kremik.dev]
* [HTTP/2] [1] [:path: /media/thumbnails/products/poplinBaelastSweetsoranzzlty_1691662575_14cbd5d6_thumbnail_512.webp]
* [HTTP/2] [1] [user-agent: curl/8.7.1]
* [HTTP/2] [1] [accept: */*]
> GET /media/thumbnails/products/poplinBaelastSweetsoranzzlty_1691662575_14cbd5d6_thumbnail_512.webp HTTP/2
> Host: api.kremik.dev
> User-Agent: curl/8.7.1
> Accept: */*
>
* Request completely sent off
< HTTP/2 404
< access-control-allow-credentials: true
< alt-svc: h3=":443"; ma=2592000
< content-type: text/html
< date: Thu, 02 May 2024 17:41:47 GMT
< referrer-policy: same-origin
< server: Caddy
< server: uvicorn
< x-content-type-options: nosniff
< content-length: 179
<

<!doctype html>
<html lang="en">
<head>
<title>Not Found</title>
</head>
<body>
<h1>Not Found</h1><p>The requested resource was not found on this server.</p>
</body>
</html>
* Connection #0 to host api.kremik.dev left intact

Caddy/Saleor logs. Saleors api service returning 404 is meant to happen:

caddy-1                  | {"level":"debug","ts":1714671803.3269873,"logger":"events","msg":"event","name":"tls_get_certificate","id":"8b3949fe-aa96-4bf6-80a3-e913da989a45","origin":"tls","data":{"client_hello":{"CipherSuites":[4866,4867,4865,49196,49200,159,52393,52392,52394,49195,49199,158,49188,49192,107,49187,49191,103,49162,49172,57,49161,49171,51,157,156,61,60,53,47,255],"ServerName":"api.kremik.dev","SupportedCurves":[29,23,30,25,24,256,257,258,259,260],"SupportedPoints":"AAEC","SignatureSchemes":[1027,1283,1539,2055,2056,2074,2075,2076,2057,2058,2059,2052,2053,2054,1025,1281,1537,771,769,770,1026,1282,1538],"SupportedProtos":["h2","http/1.1"],"SupportedVersions":[772,771],"RemoteAddr":{"IP":"10.100.110.20","Port":56656,"Zone":""},"LocalAddr":{"IP":"172.24.0.3","Port":443,"Zone":""}}}}
caddy-1                  | {"level":"debug","ts":1714671803.3270543,"logger":"tls.handshake","msg":"choosing certificate","identifier":"api.kremik.dev","num_choices":1}
caddy-1                  | {"level":"debug","ts":1714671803.327069,"logger":"tls.handshake","msg":"default certificate selection results","identifier":"api.kremik.dev","subjects":["api.kremik.dev"],"managed":true,"issuer_key":"local","hash":"7990724b74ff17cddf68f3a4fb56fcdb32865bf94ddc81963ffffde7cbbb64da"}
caddy-1                  | {"level":"debug","ts":1714671803.3270762,"logger":"tls.handshake","msg":"matched certificate in cache","remote_ip":"10.100.110.20","remote_port":"56656","subjects":["api.kremik.dev"],"managed":true,"expiration":1714704881,"hash":"7990724b74ff17cddf68f3a4fb56fcdb32865bf94ddc81963ffffde7cbbb64da"}
caddy-1                  | {"level":"debug","ts":1714671803.3285081,"logger":"http.handlers.reverse_proxy","msg":"selected upstream","dial":"172.24.0.12:8000","total_upstreams":1}
api-1                    | {"asctime": "2024-05-02T17:43:23Z", "levelname": "WARNING", "lineno": 224, "message": "Not Found: /media/thumbnails/products/poplinBaelastSweetsoranzzlty_1691662575_14cbd5d6_thumbnail_512.webp", "name": "django.request", "pathname": "/usr/local/lib/python3.9/site-packages/django/utils/log.py", "process": 24, "threadName": "ThreadPoolExecutor-4_0", "status_code": 404, "request": "<ASGIRequest: GET '/media/thumbnails/products/poplinBaelastSweetsoranzzlty_1691662575_14cbd5d6_thumbnail_512.webp'>", "hostname": "cadb28923765"}
caddy-1                  | {"level":"debug","ts":1714671803.3323104,"logger":"http.handlers.reverse_proxy","msg":"upstream roundtrip","upstream":"172.24.0.12:8000","duration":0.003760408,"request":{"remote_ip":"10.100.110.20","remote_port":"56656","client_ip":"10.100.110.20","proto":"HTTP/2.0","method":"GET","host":"api.kremik.dev","uri":"/media/thumbnails/products/poplinBaelastSweetsoranzzlty_1691662575_14cbd5d6_thumbnail_512.webp","headers":{"User-Agent":["curl/8.7.1"],"Accept":["*/*"],"X-Forwarded-For":["10.100.110.20"],"X-Forwarded-Proto":["https"],"X-Forwarded-Host":["api.kremik.dev"]},"tls":{"resumed":false,"version":772,"cipher_suite":4865,"proto":"h2","server_name":"api.kremik.dev"}},"headers":{"Server":["uvicorn"],"Referrer-Policy":["same-origin"],"X-Content-Type-Options":["nosniff"],"Date":["Thu, 02 May 2024 17:43:23 GMT"],"Content-Length":["179"],"Content-Type":["text/html"],"Access-Control-Allow-Credentials":["true"]},"status":404}
caddy-1                  | {"level":"debug","ts":1714671805.9703145,"logger":"docker-proxy","msg":"Skipping default Caddyfile because no path is set"}
caddy-1                  | {"level":"debug","ts":1714671805.970336,"logger":"docker-proxy","msg":"Skipping swarm config caddyfiles because swarm is not available"}
caddy-1                  | {"level":"debug","ts":1714671805.9763718,"logger":"docker-proxy","msg":"Skipping swarm services because swarm is not available"}

3. Caddy version:

https://hub.docker.com/layers/lucaslorentz/caddy-docker-proxy/ci-alpine/images/sha256-8416a6918d2d9f7e31663c72691f7a799328503287ab70e9d6de4b5c5cda1708?context=explore

4. How I installed and ran Caddy:

Docker with lucaslorentz/caddy-docker-proxy:ci-alpine

a. System environment:

docker

2

Are you sure this path is correct? You’re mounting /app/media in the container as a volume.

So are you saying that requests come in with /media/thumbnails/... and are stored at /saleor-media/thumbnails/...?

That complicates things, because the file matcher is not aware of needing the rewrite before matching. So file is looking at /saleor-media/media/thumbnails/...

If you know all requests for media use /media, then don’t use the file matcher at all, simplify it to just use a /media/* matcher. Something like this:

api.kremik.dev {
	tls internal
	handle_path /media* {
		root * /saleor-media
		file_server
	}
	handle {
		reverse_proxy 172.24.0.12:8000
	}
}

(I did say on Discord you should have root outside, but if we drop the file matcher, that’s no longer necessary)

The handle_path directive has built-in prefix-stripping, so you can drop uri strip_prefix as well.

1 Like
3

I’m mounting media:/app/media to the api container, but from what I understand the caddy container actually serves those files, and with the caddy container I do mount it to media:/saleor-media. Unless somehow caddy got access to files inside other containers, I thought I need to mount the volumes like this so caddy also got access to the media folder and can serve it.

As for your suggestion, it seems to be working! Thank you!
By pure coincidence, when an image needs generating it calls the url “https://api.kremik.sk/thumbnail/UHJvZHVjdE1lZGlhOjIwMA==/512/webp/”, so all /thumbnails go to api, which redirect to /media and it works like magic!

1 Like