404 on fonts in Firefox when proxying Alexandrite

1. The problem I’m having:

When proxying Alexandrite on a base domain Firefox gets 404 errors on fonts, leading to fontawesome glyphs not displaying properly.

https://ctrlaltelite.xyz in Firefox shows:
alex-184-106-broken

https://a.ctrlaltelite.xyz in Firefox as well as https://ctrlaltelite.xyz in Brave show:
alex-184-106-working

In the Network console I see the working subdomain sending the font requests as HTTP/2:

But the broken base domain sends as HTTP/1.1:

I tried forcing this to HTTP/2 using

reverse_proxy alexandrite:3000 {
  transport http {
    versions 2
  }
}

but still got 404s:

2. Error messages and/or full log output:

I’m not seeing any errors watching the container with docker compose logs -f. My access logs also do not show hits, even after clearing cache.

3. Caddy version:

v2.6.4

4. How I installed and ran Caddy:

Running a custom built image.
Dockerfile:

FROM caddy:2.6.4-builder-alpine AS builder

RUN apk add --no-cache tzdata

RUN xcaddy build v2.6.4 \
    --with github.com/caddy-dns/cloudflare@latest \
    --with github.com/caddyserver/transform-encoder \
    --with github.com/sjtug/caddy2-filter

FROM caddy:2.6.4

COPY --from=builder /usr/bin/caddy /usr/bin/caddy

docker-compose.yml:

version: "3.7"
services:
  caddy:
    image: flcr.foreveratroll.com/caddy:v2.6.4-fl6
    hostname: caddy
    container_name: caddy
    restart: unless-stopped
    environment:
      - CF_API_TOKEN=${CF_API_TOKEN}
      - TZ=${TZ}
    ports:
      - "80:80"
      - "443:443"
      - "443:443/udp"
    networks:
      - frontend
      - backend
    volumes:
      - /opt/docker/caddy/Caddyfile:/etc/caddy/Caddyfile
      - /opt/docker/caddy/caddy.d:/etc/caddy/caddy.d
      - /opt/docker/caddy/data/site:/srv
      - /opt/docker/caddy/data/caddy-data:/data
      - /opt/docker/caddy/data/config:/config
      - /opt/log/caddy:/opt/log/caddy

networks:
  frontend:
    name: frontend
    external: true
  backend:
    name: backend
    external: true

volumes:
  caddy_data:
    external: true
  caddy_config:

Caddyfile:

{
	acme_ca https://acme-staging-v02.api.letsencrypt.org/directory
	storage file_system {
		root /etc/caddy
	}
}

import /etc/caddy/caddy.d/snippets.caddy
import /etc/caddy/caddy.d/sites/*

snippets.caddy:

(init) {
	header {
		x-proxy-version v2.6.4-fl6
		-Server
		Strict-Transport-Security "max-age=63072000;includeSubDomains; preload always"
	}
	log {
		format transform `{request>headers>X-Forwarded-For>[0]:request>remote_ip} - {request>user_id} [{ts}] "{request>method} {request>uri} {request>proto}" {status} {size} "{request>headers>Referer>[0]}" "{request>headers>User-Agent>[0]}"` {
			time_format "02/Jan/2006:15:04:05 -0700"
		}
		output file /opt/log/caddy/{args.0}
	}
	tls {
		dns cloudflare "{env.CF_API_TOKEN}"
	}
	@denied not remote_ip private_ranges
	respond @denied "HTTP/2 403 Forbidden" 403 {
		close
	}
}

(init_external) {
	header {
		x-proxy-version v2.6.4-fl6
		-Server
		Strict-Transport-Security "max-age=63072000;includeSubDomains; preload always"
	}
	log {
		format transform `{request>headers>X-Forwarded-For>[0]:request>remote_ip} - {request>user_id} [{ts}] "{request>method} {request>uri} {request>proto}" {status} {size} "{request>headers>Referer>[0]}" "{request>headers>User-Agent>[0]}"` {
			time_format "02/Jan/2006:15:04:05 -0700"
		}
		output file /opt/log/caddy/{args.0} {
		}
	}
	tls {
		dns cloudflare "{env.CF_API_TOKEN}"
	}
}

(logger) {
	log {
		format transform `{request>headers>X-Forwarded-For>[0]:request>remote_ip} - {request>user_id} [{ts}] "{request>method} {request>uri} {request>proto}" {status} {size} "{request>headers>Referer>[0]}" "{request>headers>User-Agent>[0]}"` {
			time_format "02/Jan/2006:15:04:05 -0700"
	}
	output file /opt/log/caddy/{args.0}
	}
}

(caddy-common) {
	encode gzip
	header {
		-Server
		Strict-Transport-Security "max-age=31536000; include-subdomains;"
		X-XSS-Protection "1; mode=block"
		X-Frame-Options "DENY"
		X-Content-Type-Options nosniff
		Referrer-Policy no-referrer-when-downgrade
		X-Robots-Tag "none"
	}
}

/etc/caddy/caddy.d/sites/ctrlaltelite.xyz:

*.ctrlaltelite.xyz, ctrlaltelite.xyz {
	import caddy-common
	import logger "ctrlaltelite.xyz"
	reverse_proxy alexandrite:3000

	@a host a.ctrlaltelite.xyz
	import logger "a.ctrlaltelite.xyz"
	handle @a {
		reverse_proxy alexandrite:3000
	}

	@lemmy {
		path /api/*
		path /pictrs/*
		path /feeds/*
		path /nodeinfo/*
		path /.well-known/*
	}

	@lemmy-hdr {
		header Accept application/*
	}

	@lemmy-post {
		method POST
	}

	handle @lemmy-hdr {
		reverse_proxy lemmy-app:8536
	}

	handle @lemmy {
		reverse_proxy lemmy-app:8536
	}

	handle @lemmy-post {
		reverse_proxy lemmy-app:8536
	}

	@m host m.ctrlaltelite.xyz
	import logger "m.ctrlaltelite.xyz"
	import caddy-common
	handle @m {
		reverse_proxy lemmy-ui:1234
	}
}

a. System environment:

Docker (see above)

b. Command:

Docker (see above)

c. Service/unit/compose file:

Docker (see above)

d. My complete Caddy config:

Docker (see above)

5. Links to relevant resources:

Issue with the developer.

That’s not a good idea. This means you’re not persisting Caddy’s certificates. By default in Docker, the storage would be /data, which you do have a volume for, but you don’t have a volume for /etc/caddy.

There’s no benefit to removing this header. It only makes it harder to debug and verify that requests are correctly getting handled by going through Caddy. It reveals no secret information.

You can’t have more than one log inside of a site; not with v2.6.4, anyway – we made changes to make this possible in v2.7.0+

You can import from one snippet in another, by the way; should help you avoid a lot of duplication in your snippets.

You can write these like this:

	@lemmy path /api/* /pictrs/* /feeds/* /nodeinfo/* /.well-known/*
	@lemmy-hdr header Accept application/*
	@lemmy-post method POST

But since you’re making all of these do the same proxy, you can combine them like this with a single expression matcher:

	@lemmy `path('/api/*', '/pictrs/*', '/feeds/*', '/nodeinfo/*', '/.well-known/*') || header({'Accept', 'application/*'}) || method('POST')`

But also I’d move those to be right before the associated handle, to make it easier to follow what’s going on.

Keep in mind that since you have all your handle in the top-level of your site, they can prevent eachother from working. Only the first matching handle will run, and they may get sorted in an order you don’t expect. You should probably be moving this into the handle for the specific subdomain you want it for.

Very strange, I see random 200 or 404s using curl to make requests for the font file. There’s something funky with your Cloudflare caching, I think.

Simply running curl -v 'https://ctrlaltelite.xyz/_app/immutable/assets/fa-solid-900.7152a693.woff2' sometimes returns a 200, sometimes a 404.

You’ll need to enable the debug global option in Caddy, and grep the Caddy logs (stdout) for requests to that file. See what’s happening to those requests.

Noted on all the things to clean up with this. I’ll make those changes once I get this working. Here’s the output of docker compose logs -f | grep woff2 from using various methods. The Firefox 404 and Curl 200 were to ctrlaltelite.xyz. The Brave 200 was a.ctrlaltelite.xyz. For some reason I couldn’t trigger a log with Firefox to a.ctrlaltelite.xyz, even in a private window, container, or a VM. It was serving the fonts though.

Edit: I enabled Development mode on Cloudflare to bypass the cache. This didn’t fix anything.

Firefox 404:

{"level":"debug","ts":1691562171.9338312,"logger":"http.handlers.reverse_proxy","msg":"upstream roundtrip","upstream":"lemmy-app:8536","duration":0.001085108,"request":{"remote_ip":"162.158.245.67","remote_port":"64850","proto":"HTTP/2.0","method":"GET","host":"ctrlaltelite.xyz","uri":"/_app/immutable/assets/fa-solid-900.7152a693.woff2","headers":{"Dnt":["1"],"Referer":["https://ctrlaltelite.xyz/_app/immutable/assets/0.e7f0c801.css"],"X-Forwarded-Host":["ctrlaltelite.xyz"],"Sec-Fetch-Dest":["font"],"User-Agent":["Mozilla/5.0 (X11; Linux x86_64; rv:109.0) Gecko/20100101 Firefox/116.0"],"Cf-Visitor":["{\"scheme\":\"https\"}"],"Accept-Language":["en-US,en;q=0.5"],"X-Forwarded-For":["162.158.245.67"],"Accept-Encoding":["gzip"],"Sec-Fetch-Site":["same-origin"],"Cf-Ipcountry":["US"],"X-Forwarded-Proto":["https"],"Cf-Ray":["7f3de8361b870a01-LAS"],"Sec-Fetch-Mode":["cors"],"Accept":["application/font-woff2;q=1.0,application/font-woff;q=0.9,*/*;q=0.8"],"Cdn-Loop":["cloudflare"],"Cf-Connecting-Ip":["198.54.133.170"]},"tls":{"resumed":false,"version":772,"cipher_suite":4865,"proto":"h2","server_name":"ctrlaltelite.xyz"}},"headers":{"Content-Encoding":["gzip"],"Vary":["accept-encoding, Origin, Access-Control-Request-Method, Access-Control-Request-Headers"],"Date":["Wed, 09 Aug 2023 06:22:51 GMT"],"Access-Control-Expose-Headers":["content-encoding, vary"]},"status":404}
{"level":"debug","ts":1691562171.9509833,"logger":"http.handlers.reverse_proxy","msg":"upstream roundtrip","upstream":"lemmy-app:8536","duration":0.000610191,"request":{"remote_ip":"162.158.245.35","remote_port":"64266","proto":"HTTP/2.0","method":"GET","host":"ctrlaltelite.xyz","uri":"/_app/immutable/assets/fa-regular-400.8e7e5ea1.woff2","headers":{"Cf-Ray":["7f3de8369bde0a01-LAS"],"Cf-Visitor":["{\"scheme\":\"https\"}"],"Cf-Ipcountry":["US"],"Cf-Connecting-Ip":["198.54.133.170"],"X-Forwarded-For":["162.158.245.35"],"Accept-Language":["en-US,en;q=0.5"],"X-Forwarded-Host":["ctrlaltelite.xyz"],"Sec-Fetch-Site":["same-origin"],"Dnt":["1"],"Cdn-Loop":["cloudflare"],"X-Forwarded-Proto":["https"],"Referer":["https://ctrlaltelite.xyz/_app/immutable/assets/0.e7f0c801.css"],"User-Agent":["Mozilla/5.0 (X11; Linux x86_64; rv:109.0) Gecko/20100101 Firefox/116.0"],"Accept":["application/font-woff2;q=1.0,application/font-woff;q=0.9,*/*;q=0.8"],"Sec-Fetch-Dest":["font"],"Sec-Fetch-Mode":["cors"],"Accept-Encoding":["gzip"]},"tls":{"resumed":false,"version":772,"cipher_suite":4865,"proto":"h2","server_name":"ctrlaltelite.xyz"}},"headers":{"Access-Control-Expose-Headers":["content-encoding, vary"],"Content-Encoding":["gzip"],"Vary":["accept-encoding, Origin, Access-Control-Request-Method, Access-Control-Request-Headers"],"Date":["Wed, 09 Aug 2023 06:22:51 GMT"]},"status":404}

Brave 200:

{"level":"debug","ts":1691563478.9533854,"logger":"http.handlers.reverse_proxy","msg":"upstream roundtrip","upstream":"alexandrite:3000","duration":0.001222748,"request":{"remote_ip":"162.158.186.164","remote_port":"45922","proto":"HTTP/2.0","method":"GET","host":"a.ctrlaltelite.xyz","uri":"/_app/immutable/assets/fa-solid-900.7152a693.woff2","headers":{"Sec-Fetch-Mode":["cors"],"X-Forwarded-Proto":["https"],"Accept":["*/*"],"Sec-Ch-Ua-Platform":["\"Linux\""],"Cdn-Loop":["cloudflare"],"Sec-Ch-Ua":["\"Not/A)Brand\";v=\"99\", \"Brave\";v=\"115\", \"Chromium\";v=\"115\""],"Accept-Language":["en-US,en;q=0.6"],"X-Forwarded-For":["162.158.186.164"],"Sec-Gpc":["1"],"Origin":["https://a.ctrlaltelite.xyz"],"Sec-Fetch-Dest":["font"],"Cf-Connecting-Ip":["70.176.207.154"],"Referer":["https://a.ctrlaltelite.xyz/_app/immutable/assets/0.e7f0c801.css"],"X-Forwarded-Host":["a.ctrlaltelite.xyz"],"Priority":["u=0"],"Cf-Ray":["7f3e081f5bc52a9d-LAX"],"Accept-Encoding":["gzip, br"],"Cf-Ipcountry":["US"],"Cf-Visitor":["{\"scheme\":\"https\"}"],"Dnt":["1"],"Sec-Ch-Ua-Mobile":["?0"],"User-Agent":["Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/115.0.0.0 Safari/537.36"],"Sec-Fetch-Site":["same-origin"]},"tls":{"resumed":false,"version":772,"cipher_suite":4865,"proto":"h2","server_name":"a.ctrlaltelite.xyz"}},"headers":{"Content-Type":["font/woff2"],"Last-Modified":["Fri, 04 Aug 2023 02:11:37 GMT"],"Date":["Wed, 09 Aug 2023 06:44:38 GMT"],"Keep-Alive":["timeout=5"],"Content-Length":["150124"],"Cache-Control":["public,max-age=31536000,immutable"],"Etag":["W/\"150124-1691115097000\""],"Connection":["keep-alive"],"Vary":["Accept-Encoding"]},"status":200}
{"level":"debug","ts":1691563478.997423,"logger":"http.handlers.reverse_proxy","msg":"upstream roundtrip","upstream":"alexandrite:3000","duration":0.001052557,"request":{"remote_ip":"162.158.186.60","remote_port":"18438","proto":"HTTP/2.0","method":"GET","host":"a.ctrlaltelite.xyz","uri":"/_app/immutable/assets/fa-regular-400.8e7e5ea1.woff2","headers":{"Sec-Ch-Ua-Platform":["\"Linux\""],"Sec-Fetch-Mode":["cors"],"X-Forwarded-For":["162.158.186.60"],"Cf-Connecting-Ip":["70.176.207.154"],"Priority":["u=0"],"Sec-Fetch-Site":["same-origin"],"Sec-Fetch-Dest":["font"],"Cf-Visitor":["{\"scheme\":\"https\"}"],"Accept-Language":["en-US,en;q=0.6"],"Sec-Ch-Ua-Mobile":["?0"],"Dnt":["1"],"Cf-Ray":["7f3e081f9c3f2a9d-LAX"],"Referer":["https://a.ctrlaltelite.xyz/_app/immutable/assets/0.e7f0c801.css"],"Origin":["https://a.ctrlaltelite.xyz"],"User-Agent":["Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/115.0.0.0 Safari/537.36"],"Accept":["*/*"],"Sec-Ch-Ua":["\"Not/A)Brand\";v=\"99\", \"Brave\";v=\"115\", \"Chromium\";v=\"115\""],"Cf-Ipcountry":["US"],"Cdn-Loop":["cloudflare"],"X-Forwarded-Host":["a.ctrlaltelite.xyz"],"Sec-Gpc":["1"],"Accept-Encoding":["gzip, br"],"X-Forwarded-Proto":["https"]},"tls":{"resumed":false,"version":772,"cipher_suite":4865,"proto":"h2","server_name":"a.ctrlaltelite.xyz"}},"headers":{"Vary":["Accept-Encoding"],"Content-Type":["font/woff2"],"Etag":["W/\"24948-1691115097000\""],"Keep-Alive":["timeout=5"],"Connection":["keep-alive"],"Cache-Control":["public,max-age=31536000,immutable"],"Content-Length":["24948"],"Last-Modified":["Fri, 04 Aug 2023 02:11:37 GMT"],"Date":["Wed, 09 Aug 2023 06:44:38 GMT"]},"status":200}

Curl 200:

{"level":"debug","ts":1691563663.133957,"logger":"http.handlers.reverse_proxy","msg":"upstream roundtrip","upstream":"alexandrite:3000","duration":0.001738502,"request":{"remote_ip":"162.158.245.67","remote_port":"47906","proto":"HTTP/2.0","method":"GET","host":"ctrlaltelite.xyz","uri":"/_app/immutable/assets/fa-solid-900.7152a693.woff2","headers":{"Accept-Encoding":["gzip"],"X-Forwarded-Proto":["https"],"Cf-Connecting-Ip":["198.54.133.170"],"Cf-Visitor":["{\"scheme\":\"https\"}"],"Accept":["*/*"],"X-Forwarded-Host":["ctrlaltelite.xyz"],"X-Forwarded-For":["162.158.245.67"],"Cf-Ray":["7f3e0c9e7ae50add-LAS"],"Cdn-Loop":["cloudflare"],"Cf-Ipcountry":["US"],"User-Agent":["curl/8.2.1"]},"tls":{"resumed":false,"version":772,"cipher_suite":4865,"proto":"h2","server_name":"ctrlaltelite.xyz"}},"headers":{"Last-Modified":["Fri, 04 Aug 2023 02:11:37 GMT"],"Connection":["keep-alive"],"Vary":["Accept-Encoding"],"Cache-Control":["public,max-age=31536000,immutable"],"Content-Length":["150124"],"Content-Type":["font/woff2"],"Etag":["W/\"150124-1691115097000\""],"Date":["Wed, 09 Aug 2023 06:47:43 GMT"],"Keep-Alive":["timeout=5"]},"status":200}

Notice it’s using "upstream":"lemmy-app:8536" for that request. It’s not hitting "upstream":"alexandrite:3000".

This is because Firefox sends: Accept: application/font-woff2;q=1.0,application/font-woff;q=0.9,*/*;q=0.8 and your header Accept application/* is matching that.

You need to move your lemmy matchers inside of your handle @m { block, because otherwise those lemmy handers are intercepting requests for your other site.

Basically, the handlers are messy right now, they overlap eachother in ways they should not, as I explained earlier.

This put me on the path to figuring it out. I checked the documented nginx config and noticed they don’t have a matcher for application/* so I just removed it and everything is functioning as expected.

I had originally pulled the config for this from their documentation. I went ahead and combined the @lemmy and @lemmy-post matchers into your suggested oneliner as well and we’re still good.

Thanks so much for your help. I’m gonna clean up my snippets next (they were on the fly tests that never got the polish they should have once things were working :sweat_smile:)

1 Like

I somehow lost the ability to edit my last response. After these changes my Lemmy stopped federating. After digging into the logs this was due to application/activity+json not getting routed to my lemmy-app container. I also saw some 404s for application/x-www-form-urlencoded so I updated my config to handle these. I suspect this is why the Lemmy devs had application/* going to the app container… even though it’s not explicitly listed in their nginx config.

This is what I added to my config:

	@lemmy-hdr {
		header Accept application/activity+json
		header Accept application/x-www-form-urlencoded
	}
	
	handle @lemmy-hdr {
		reverse_proxy lemmy-app:8536
	}

and I’m back to federating.

I had tried the fancy one liner but it was throwing a Caddy config error with a single header section.

You can do this:

@lemmy-hdr `header({"Accept": ["application/activity+json", "application/x-www-form-urlencoded"]})`

But either way, that doesn’t seem right. Shouldn’t all Lemmy traffic go to the same subdomain? Why do you need that matcher at all? Without seeing your full config at this point I have to make assumptions.

So I’m running this in a non-standard way that’s 100% the cause of my issues. The way Lemmy works by default is they have five containers:

  1. lemmy-app - The back end of the application
  2. lemmy-ui - The front end of the application
  3. nginx - Their chosen proxy
  4. pictrs - Not relevant here, but how they chose to implement image uploads
  5. postgres - Not relevant here

I yanked out Nginx and am using Caddy instead. But I’m also using an alternate front end - Alexandrite.

The entire reason I started this thread was because I wanted to run Alexandrite on my base domain. Everything functioned normally when on a subdomain (a.ctrlaltelite.xyz.) But when it was on the base domain the matcher for application/* was matching the fonts and sending them to lemmy-app instead of alexandrite and returning 404s to the browser.

Removing the application/* matcher fixed the 404s on the fonts, but then it broke federation (application/activity+json.) So you’re right, all of it should go to Lemmy, but I don’t want to use their UI on the base domain.

I went ahead and added the most recent matcher you suggested which does work now, The slight change in syntax for the header from the first one seemed to do the trick.

Here’s the full config for the domain:

*.ctrlaltelite.xyz, ctrlaltelite.xyz {
	import init_external "ctrlaltelite.xyz"
	encode gzip zstd
	import caddy-common
	import logger "ctrlaltelite.xyz"
	reverse_proxy alexandrite:3000

	@a host a.ctrlaltelite.xyz
	import logger "a.ctrlaltelite.xyz"
	handle @a {
		reverse_proxy alexandrite:3000
	}

	@lemmy `path('/api/*', '/pictrs/*', '/feeds/*', '/nodeinfo/*', '/.well-known/*') || method('POST') || header({"Accept": ["application/activity+json", "application/x-www-form-urlencoded"]})`
	handle @lemmy {
		reverse_proxy lemmy-app:8536
	}

	@m host m.ctrlaltelite.xyz
	handle @m {
		reverse_proxy lemmy-ui:1234
	}

	@status host status.ctrlaltelite.xyz
	handle @status {
		reverse_proxy iamfoo:2001
	}
}

ctrlalteilte.xyz: The base domain that I want to use Lemmy+Alexandrite through
a.ctrlaltelite.xyz: Alexandrite subdomain, mostly for testing
m.ctrlaltelite.xyz: The default lemmy-ui, intended for use on mobile and just in case Alexandrite gets buggy (Alexandrite was a little iffy on mobile when it started.)
status.ctrlaltelite.xyz: Intended to be a status check page of some sort.

I haven’t addressed the spaghetti snippets you pointed out earlier in the thread, yet.

If there’s a better way to handle my use case I’m all for making it more elegant, but for now I’m not opposed to keeping an eye on logs for any other application/ issues. I also noticed some info messages in the logs that I want to address too. I’m gonna take a crack at it before potentially starting a new thread.

I’m planning on upgrading from 2.6.4 to 2.7.3 within the next few weeks, is there anything with what I have here that’s going to need to change? I already saw the switch from {args.0} to {args[0]} for my logger,.

I suggest all your base domain stuff should either go in its own separate site block, or in a handle matching the base domain (remember that you can nest handle). A separate site block is certainly the simplest conceptually.

The problem with your current config is that it matches things that would probably break other sites. Routing all POST to Lemmy will break any subdomains that ever want to use POST for example. You need a layer of specificity to avoid breaking the subdomains.

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.