Caddy reverse proxy + Safari strange behaviour

1. Caddy version (caddy version):

2.4.6

2. How I run Caddy:

a. System environment:

amd64, Debian 11, Docker

b. Command:

c. Service/unit/compose file:

  caddy:
    image: caddy:latest
    container_name: caddy
    restart: unless-stopped
    ports:
      - "80:80"
      - "443:443"
    volumes:
      - ${DATADIR}/caddy/Caddyfile:/etc/caddy/Caddyfile
      - ${DATADIR}/caddy/data:/data # Optional
      - ${DATADIR}/caddy/config:/config # Optional
    environment:
      - TZ=${TZ}
    labels:
      - "com.centurylinklabs.watchtower.enable=true"
      - "com.centurylinklabs.watchtower.monitor-only=true"

d. My complete Caddyfile or JSON config:

{
    # Global options block. Entirely optional, https is on by default
    # Optional email key for lets encrypt
    # email youremail@.........
    # Optional staging lets encrypt for testing. Comment out for production.
    # acme_ca https://acme-staging-v02.api.letsencrypt.org/directory
}
ha.mini2x14.local {
    encode zstd gzip
    reverse_proxy http://192.168.88.14:8123
    header {
        # enable HSTS
        Strict-Transport-Security max-age=31536000;
        # disable clients from sniffing the media type
        X-Content-Type-Options nosniff
        # clickjacking protection
        #X-Frame-Options DENY
        # keep referrer data off of HTTP connections
        Referrer-Policy no-referrer-when-downgrade
        # Content-Security-Policy: default-src 'self'
    }
}
portainer.mini2x14.local {
    encode zstd gzip
    reverse_proxy http://portainer:9000
    header {
        # enable HSTS
        Strict-Transport-Security max-age=31536000;
        # disable clients from sniffing the media type
        X-Content-Type-Options nosniff
        # clickjacking protection
        #X-Frame-Options DENY
        # keep referrer data off of HTTP connections
        Referrer-Policy no-referrer-when-downgrade
        # Content-Security-Policy: default-src 'self'
    }
}

3. The problem I’m having:

In general, everything works. However, in Safari (macOS) I noticed such a problem - in the address bar of the site behind the Caddy, there are no buttons (refresh, share, bookmark, and so on). It looks like the connection is in timeout state, page loading does not complete. In addition, in developer mode, capturing network activity does not stop automatically (although it should after the page has finished loading). With a direct visiting to the proxied server, everything is fine. Haven't noticed this with any other site. Everything is fine in Chrome.

I understand that the problem may not be directly related to Caddy, but I can not find anything on this topic. I ask for help in diagnosing the problem (only reproducible when using Caddy as a reverse proxy). Please, advise what to try with Caddy settings to localize the cause (this is my first experience with Caddy).

4. Error messages and/or full log output:

There are no errors in the logs. Example curl output:

aleksey@mPro-2x14 ~ % curl -v https://portainer.mini2x14.local
*   Trying 192.168.88.14:443...
* Connected to portainer.mini2x14.local (192.168.88.14) port 443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* successfully set certificate verify locations:
*  CAfile: /etc/ssl/cert.pem
*  CApath: none
* TLSv1.2 (OUT), TLS handshake, Client hello (1):
* TLSv1.2 (IN), TLS handshake, Server hello (2):
* TLSv1.2 (IN), TLS handshake, Certificate (11):
* TLSv1.2 (IN), TLS handshake, Server key exchange (12):
* TLSv1.2 (IN), TLS handshake, Server finished (14):
* TLSv1.2 (OUT), TLS handshake, Client key exchange (16):
* TLSv1.2 (OUT), TLS change cipher, Change cipher spec (1):
* TLSv1.2 (OUT), TLS handshake, Finished (20):
* TLSv1.2 (IN), TLS change cipher, Change cipher spec (1):
* TLSv1.2 (IN), TLS handshake, Finished (20):
* SSL connection using TLSv1.2 / ECDHE-ECDSA-AES128-GCM-SHA256
* ALPN, server accepted to use h2
* Server certificate:
*  subject: [NONE]
*  start date: Mar 13 22:08:12 2022 GMT
*  expire date: Mar 14 10:08:12 2022 GMT
*  subjectAltName: host "portainer.mini2x14.local" matched cert's "portainer.mini2x14.local"
*  issuer: CN=Caddy Local Authority - ECC Intermediate
*  SSL certificate verify ok.
* Using HTTP2, server supports multi-use
* Connection state changed (HTTP/2 confirmed)
* Copying HTTP/2 data in stream buffer to connection buffer after upgrade: len=0
* Using Stream ID: 1 (easy handle 0x7fdcae011e00)
> GET / HTTP/2
> Host: portainer.mini2x14.local
> user-agent: curl/7.77.0
> accept: */*
>
* Connection state changed (MAX_CONCURRENT_STREAMS == 250)!
< HTTP/2 200
< accept-ranges: bytes
< cache-control: max-age=31536000
< content-type: text/html; charset=utf-8
< date: Mon, 14 Mar 2022 02:07:19 GMT
< last-modified: Wed, 09 Feb 2022 01:01:46 GMT
< referrer-policy: no-referrer-when-downgrade
< server: Caddy
< strict-transport-security: max-age=31536000;
< vary: Accept-Encoding
< x-content-type-options: nosniff
< x-content-type-options: nosniff
< x-xss-protection: 1; mode=block
<
<!doctype html>
...
</html>%

5. What I already tried:

Clearing caches with reboots, using Caddy v2.5.0-beta1

6. Links to relevant resources:

These are not optional if you have any certificates, otherwise you risk losing your certs and keys on every time you recreate the containers.

Since you’re using a .local domain, Caddy will be issuing a certificate from its internal CA (not a publicly trusted CA).

So it’s possible that Safari doesn’t trust Caddy’s root CA cert and isn’t completing the TLS handshake because of that.

You can turn on the debug global option and take a look at your logs, Caddy should log something if the handshake is failing.

If that is the problem, you’ll need to grab the root CA cert from Caddy’s storage (i.e. /data) and install it on your machine with Safari.

Or you could use a domain you actually own and get a publicly trusted certificate.

2 Likes

Meanwhile, I found the source of the problem. This morning, after I didn’t find anything suspicious in the debug log, I came up with the idea to change the domain used to .org (I made the appropriate changes to the configuration of the local DNS server and Caddyfile) - and everything was OK. A quick internet search brought me here: proxy - iOS Safari development ".local" domain issue - Stack Overflow

Bottom line: Apple seems to be following RFC 6762, which reserves the .local domain for multicast DNS, and Safari does not recognize addresses in this domain as valid addresses.

Domain .lan looks like a good alternative but doesn’t work well in Safari either…

Thank you, @francislavoie , for your attention.

These are not optional if you have any certificates

Yes, I know this, this is my starting configuration, and there is a lot of copy-paste “garbage” from the tutorials I read :slight_smile:

P.S. I mark this post as a solution, although I have not found a good solution yet …

Either .localhost if you only need to access things from the same machine, or .home.arpa are good options.

1 Like

Addition:

It looks like this issue with Safari is related to the HTTP Strict Transport Security feature. After resetting it with Strict-Transport-Security max-age=0;, addresses in the .lan zone started to work fine with Safari.

This topic was automatically closed after 30 days. New replies are no longer allowed.