Just to give everyone using HTTP/3 aka QUIC with caddy a heads up, there appears to be major compatibility problems with all new Apple software from yesterday. Apparently Apple turned on HTTP/3 by default, and this does cause all kinds of strange errors about connection reset errors. Disabling HTTP/3 worked for me for the time being. I never had any problems before trying HTTP/3 with Chrome, but with Safari it has many problems. I used the following in my global config:
Websites hosted on Caddy are now completely unusable with iOS18/Safari. They just randomly timeout or only load half of the page content. I solved this differently so that other browsers can still use HTTP/3.
This seems to work, but only after I had cleared the cache, forced closed Safari, and switched the iPhone off and on again etc.
I assume Apple are using some kind of Quic implementation that’s not compatible with the one Caddy is using. Do we have any further clues on this?
It would be interesting to see if an alternative browser engine solves the problem, but Apple being Apple only lets you do that in the EU so I can’t test that.
This may be a long shot, but what’s the MTU configured on the interface? And can you change it? Try 1500, 1300, 1280, or 1200. Perhaps 1300 and lower will work.
That must be frustrating. Can you be more specific about the errors, more than “all kinds of strange errors about connection reset” and “They just randomly timeout or only load half of the page content.”? Are you able to use the inspector tools in Safari to reveal more?
I do not have the firewall enabled. It definitely is related to quic. I can use chrome on the same machine without problems, just now with safari also using quic everything goes awry. Disabling quic in caddy is just a stop gap measure as too much was failing. For example any oauth2 flow would fail with timeouts. I might be able to do some tracing, but not today.
Could you share more details what’s going wrong? I’m struggling to reproduce it, because Safari on macOS and on iOS refuse to use QUIC in the first place.
That is the difference between iOS 18 and versions before that, with an iPhone and iOS 18 it will use QUIC. It may be the same with MacOS and Sequoia, but I did not yet test it further.
That is the difference between iOS 18 and versions before that, with an iPhone and iOS 18 it will use QUIC
Well, as I said before, I’m unable to reproduce this on my device. How do you expect me to help you here?
there appears to be major compatibility problems with all new Apple software from yesterday.
Again, what is the problem you’re seeing?
Apparently Apple turned on HTTP/3 by default, and this does cause all kinds of strange errors about connection reset errors.
Where did you get this information from?
@jum Frankly, a bug report like this is pretty unhelpful and unproductive. Plenty of speculation and finger-pointing. But not even a description of the problem (not even after I explicitly asked for it!). No hints on how to reproduce it (not even after I explicitly asked for it!). Nor any attempt made to provide any information (qlog packet traces, for example) that would help us figure out what’s actually going on.
It is difficult for me to reproduce it as my resources are limited. I had to disable it on my web servers to be able to let normal work for other users to continue. I now have to get a spare VM, configure it and put on the services I experienced the problems with. It just takes time and I am working on it.
But what did the issue look like? For example, did it show any error message? Did the page just fail to load?
You mention “connection reset” errors, which doesn’t make any sense in terms of QUIC. Connections can be closed, but not reset. Streams can be reset, on the other hand. What made you conclude that the connection was reset?
I am just rebuilding a set of services for a new VM. The simplest timeout I had was miniflux, it would deliver pages but the marking of articles as read timed out (or was stuck, probably due to missing error handling on the js side).
Connection reset might be misleading, this was a package where I am not trusting the error messages currently.
Sorry, I’ve been out all day. Just got home. It’s interesting that you mention miniflux. I’m also using miniflux and that’s certainly the worst application for this problem. You click on any of the links in miniflux to “mark as read” etc. and it simply sits there saying “in progress”. I see 200 OK in the Caddy log files, but it doesn’t look like Safari receives the response properly. If I log out of miniflux and try logging back in again it gets an error in Safari saying something along the lines of “connection was reset by peer/page not loaded because of network error”.
However I also see it on my own website. If I click a link it sits there for a while and then times out, but interestingly if I hit refresh it loads the page properly on the second attempt. Click another link again, it fails, hit refresh, it loads.
Disabling h3/Alt-Svc in Caddy restores the previous behavior and it works perfectly, no issues. This all went wrong the second I upgraded my iPhone to iOS18. It was fine in 17 and earlier and I’m assuming this is because Apple just enabled HTTP/3 in 18 where it was experimental in 17.
I have tried the new caddy head (the one with the ability to set the env variable QLOGDIR), to log QUIC and everything works, the problem with iOS 18 is gone. I verified from the caddy logs that http/3 is being used. I also tried another run, with QLOGDIR unset, again the problem can not be reproduced. So there must be something else that has changed in that version, for the better that is clear. Can anyone else reproduce this?
You could try to bisect to find where it might’ve been fixed. Take a commit about halfway in between 2.8.4 and current head and build with that, test it. If it works, then try an older commit than that. If it doesn’t work, then try a commit newer than that. Since & repeat until you find the offending commit. Effectively you’re doing a binary search, cutting the problem-space in half each time until you find where it broke.
Thank you for doing this! Looking at the fixes in quic-go:
" http3: correctly set the http.Response.ContentLength on responses that don’t set the Content-Length header:" might be related.
Both miniflux and my website are written in Go, and Go only sets a Content-Length header if the response body is small. Larger responses don’t set the header so maybe that confused Safari.
