I switched from nginx to caddy and the website is loading slowly

jany · August 23, 2023, 7:00pm

1. The problem I’m having:

Today I switched from nginx reverse proxy to caddy reverse proxy.
I have approx. 10 domains on (cloudflare, noip.com, changeip.com …). Domains run mostly websites, wordpress, nextcloud, etc. Everything works correctly, but some websites start very slowly (especially WordPress 2 out of 3). If I write the web address in the url and press enter, I have to wait 5-6 seconds until I see the website.

2. Error messages and/or full log output:

3. Caddy version:

v2.7.4 h1:J8nisjdOxnYHXlorUKXY75Gr6iBfudfoGhrJ8t7/flI=

4. How I installed and ran Caddy:

xcaddy build --with github.com/caddy-dns/cloudflare

systemd service

a. System environment:

Debian 12 LXC

d. My complete Caddy config:

#---------------------------http-headers-------------------------------#
(headers) {
        header /* {
    Strict-Transport-Security "max-age=31536000; includeSubDomains; preload"
    Permissions-Policy interest-cohort=()
    X-Xss-Protection "1; mode=block"
    X-Content-Type-Options "nosniff"
    X-Frame-Options "DENY"
    Content-Security-Policy "upgrade-insecure-requests"
    Referrer-Policy "strict-origin-when-cross-origin"
    Cache-Control "public, max-age=15, must-revalidate"
    Feature-Policy "accelerometer 'none'; ambient-light-sensor 'none'; autoplay 'self'; camera 'none'; encrypted-media 'none'; fullscreen 'self'; geo>
    Server "No."
        }
}
 
#---------------------collabora.example.com------------------------------#
 
collabora.example.com {
  encode gzip
 
  reverse_proxy http://192.168.20.15:9980
}
 
#---------------------example.com--------------------------------#
 
example.com {
    redir https://www.example.com{uri}
     
import headers
}
 
www.example.com {
    reverse_proxy 192.168.20.11:80
     
import headers
}
 
#----------------------examplex.com--------------------------------#

examplex.com {
    redir https://www.examplex.com{uri}
 
import headers
}
 
www.examplex.com {
    tls {
        dns cloudflare *****************************************
    }
    reverse_proxy 192.168.20.11:80
 
import headers
}
 
*.examplex.com {
    tls {
        dns cloudflare *****************************************
    }
    reverse_proxy 192.168.20.11:80
 
import headers
}
 
#-----------------------exampley.com--------------------------------#
  
etc

matt · August 24, 2023, 2:25am

The problem appears to be that your Caddyfile is for example.com and examplex.com instead of your actual domain names

jany · August 24, 2023, 4:37am

Oh no . I actually have 13 real domains. All domains are working, but some are loading very slowly.
I used example.com and examplex.com in the caddyfile because I don’t want to publish my domains.

matt · August 24, 2023, 3:58pm

Domains are already published and public information… as the help template and forum rule says, we won’t be able to help you unless we can investigate it ourselves. By redacting domains and not helping us experience the behavior you’re also seeing, you’ve tied our hands so we can’t help you

jany · August 25, 2023, 5:23am

Yes, I can also put a real caddyfile here (some domains). The first and last domains in the caddyfile are wordpress sites. If I enter the url address of the site, it starts up very slowly (I have to wait 5-7 seconds). The non-Wordpress sites are working fine. If I used reverse proxy nginx, wordpress pages worked fast too.

#---------------------------http-headers-------------------------------#
(headers) {
        header /* {
    Strict-Transport-Security "max-age=31536000; includeSubDomains; preload"
    Permissions-Policy interest-cohort=()
    X-Xss-Protection "1; mode=block"
    X-Content-Type-Options "nosniff"
    X-Frame-Options "DENY"
    Content-Security-Policy "upgrade-insecure-requests"
    Referrer-Policy "strict-origin-when-cross-origin"
    Cache-Control "public, max-age=15, must-revalidate"
    Feature-Policy "accelerometer 'none'; ambient-light-sensor 'none'; autoplay 'self'; camera 'none'; encrypted-media 'none'; fullscreen 'self'; geo>
    Server "No."
        }
}


#---------------------deb-sky.ddns.info--------------------------------#

deb-sky.ddns.info {
    redir https://www.deb-sky.ddns.info{uri}
    
import headers
}

www.deb-sky.ddns.info {
    reverse_proxy 192.168.20.11:80
    
import headers
}

#----------------------debian.ddns.info--------------------------------#

debian.ddns.info {
    redir https://www.debian.ddns.info{uri}
    
import headers
}

www.debian.ddns.info {
    reverse_proxy 192.168.20.11:80

import headers
}

#---------------------local.ddns.us------------------------------------#

local.ddns.us {
    redir https://www.local.ddns.us{uri}
    
import headers
}

www.local.ddns.us {
    reverse_proxy 192.168.20.11:80

import headers
}

#--------------------test-web.ddns.info--------------------------------#

test-web.ddns.info {
    redir https://www.test-web.ddns.info{uri}
    
import headers
}

www.test-web.ddns.info {
    reverse_proxy 192.168.20.11:80

import headers
}

#---------------------urbarstola.sk------------------------------------#

urbarstola.sk {
    redir https://www.urbarstola.sk{uri}
}

www.urbarstola.sk {
    reverse_proxy 192.168.20.11:80
    
import headers
}

matt · August 25, 2023, 3:59pm

Thanks.

Hmm, I’m not experiencing any latency like that from my end on those two domains. Have you tried accessing your sites from different networks?

jany · August 25, 2023, 4:38pm

Yes, I tried accessing from different WAN networks. And I also try LAN (respectively, the caddy is in the DMZ) and it is very slow.
hmm
I’m not an expert on these things, but if I use the F12 key in Mozilla, the developer tools will open and there you can see where the problem could be.
However, I don’t know how to solve it.

jany · August 26, 2023, 11:30am

I do not claim that the problem is in the caddy. The problem manifests itself on websites with wordpress. Where there is no wordpress, websites are loaded in 200-300 ms.
When I used nginx as a reverse proxy, wordpress loaded very quickly.
I use apache2 as backend servers and I didn’t make any changes there.
I have gigabit within the local network. WAN network upload is 200 Mbps and ping 10ms.

Edit: I temporarily switched to nginx and the results are now 747 ms

Mohammed90 · August 26, 2023, 3:03pm

What’s your NGINX config?

jany · August 26, 2023, 3:12pm

server {
    server_name deb-sky.ddns.info;
    return 301 http://www.deb-sky.ddns.info$request_uri;

    listen 443 ssl; # managed by Certbot
    ssl_certificate /etc/letsencrypt/live/deb-sky.ddns.info/fullchain.pem; # managed by Certbot
    ssl_certificate_key /etc/letsencrypt/live/deb-sky.ddns.info/privkey.pem; # managed by Certbot
    include /etc/letsencrypt/options-ssl-nginx.conf; # managed by Certbot
    ssl_dhparam /etc/letsencrypt/ssl-dhparams.pem; # managed by Certbot
    add_header Strict-Transport-Security "max-age=31536000; includeSubDomains" always;
    add_header Referrer-Policy                      "no-referrer"   always;
    add_header X-Content-Type-Options               "nosniff"       always;
    add_header X-Download-Options                   "noopen"        always;
    add_header X-Frame-Options                      "SAMEORIGIN"    always;
    add_header X-Permitted-Cross-Domain-Policies    "none"          always;
    add_header X-Robots-Tag                         "none"          always;
    add_header X-XSS-Protection                     "1; mode=block" always;
    add_header Content-Security-Policy "default-src 'self' https: data: 'unsafe-inline' 'unsafe-eval';" always;
    add_header Permissions-Policy "geolocation=(),midi=(),sync-xhr=(),microphone=(),camera=(),magnetometer=(),gyroscope=(),fullscreen=(self),payment=()";


}
 
server {
    server_name www.deb-sky.ddns.info;
 
    location / {
        proxy_pass http://192.168.20.11;
        proxy_set_header Host $host;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header X-Forwarded-Proto $scheme;
    }

    listen 443 ssl; # managed by Certbot
    ssl_certificate /etc/letsencrypt/live/deb-sky.ddns.info/fullchain.pem; # managed by Certbot
    ssl_certificate_key /etc/letsencrypt/live/deb-sky.ddns.info/privkey.pem; # managed by Certbot
    include /etc/letsencrypt/options-ssl-nginx.conf; # managed by Certbot
    ssl_dhparam /etc/letsencrypt/ssl-dhparams.pem; # managed by Certbot
    add_header Strict-Transport-Security "max-age=31536000; includeSubDomains" always;
    add_header Referrer-Policy                      "no-referrer"   always;
    add_header X-Content-Type-Options               "nosniff"       always;
    add_header X-Download-Options                   "noopen"        always;
    add_header X-Frame-Options                      "SAMEORIGIN"    always;
    add_header X-Permitted-Cross-Domain-Policies    "none"          always;
    add_header X-Robots-Tag                         "none"          always;
    add_header X-XSS-Protection                     "1; mode=block" always;
    add_header Content-Security-Policy "default-src 'self' https: data: 'unsafe-inline' 'unsafe-eval';" always;
    add_header Permissions-Policy "geolocation=(),midi=(),sync-xhr=(),microphone=(),camera=(),magnetometer=(),gyroscope=(),fullscreen=(self),payment=()";


}

server {
    if ($host = deb-sky.ddns.info) {
        return 301 https://$host$request_uri;
    } # managed by Certbot


    listen 80;
    server_name deb-sky.ddns.info;
    return 404; # managed by Certbot


}
 
server {
    if ($host = www.deb-sky.ddns.info) {
        return 301 https://$host$request_uri;
    } # managed by Certbot


    listen 80;
    server_name www.deb-sky.ddns.info;
    return 404; # managed by Certbot


}

jany · August 28, 2023, 2:09pm

I don’t understand it at all. If I use caddy as a reverse proxy, wordpress websites load very slowly (4-6 seconds or more). If I switch to the old nginx reverse proxy, everything is very fast (up to 700-800 ms).
I wanted to try ver. 2.7.3 but the Cloudflare plugin requires ver 2.7.4 caddy when compiling.
Unfortunately I was forced to go back to nginx.

Mohammed90 · August 28, 2023, 2:47pm

I asked for the NGINX config thinking maybe the slowness is due to TCP socket vs UNIX socket, but you’re using TCP socket in both. I wonder if it has to do with some sysctl parameter or AppArmor.

What’s blocking you from trying 2.7.4?

jany · August 28, 2023, 3:25pm

I’m actually using 2.7.4 (respectively I’m already using nginx), but I wanted to downgrade to 2.7.3.

If someone could take a look at it.
JavaScript or loading relatively small images cause a big delay.
But it’s currently running on nginx.
Try to look at it through firefox F12 and later I will replace LXC nginx with LXC caddy.

matt · August 28, 2023, 4:01pm

I think we’re trying to figure out why you want to use 2.7.3 specifically

When I try loading those sites in my browser, the network timings don’t look like that, but some of them do look slow (see the turtle icons):

But these are served by nginx So I don’t know that there’s really much of a difference. I’m guessing something to do with the network where your site is or even between it and where you are, since I can’t reproduce the results you’re seeing.

jany · August 28, 2023, 5:49pm

Thank you very much for trying it. 1.43 seconds is a good time. Now I just replace the LXC Debian 11 nginx container, with Dedien 12 caddy. Please clear the web browser cache and reload the website.

matt · August 29, 2023, 4:24am

Aha, thank you. I see the delay now.

So it looks like Caddy vs. nginx is a red herring. The real difference is HTTP/2 vs HTTP/3. When I use curl or disable HTTP/3 in Firefox, I see the pages load quickly even when served by Caddy.

I bet you’d see similar performance issues with nginx serving HTTP/3. (I could be wrong, but I figure a bug like this would have been caught in our HTTP/3 lib already.)

It could be that some HTTP/3 streams / QUIC connections are being impeded near your site.

I can’t yet explain why only some HTTP/3 requests are slow, but I definitely think there’s something about the network configuration of the website’s host (or near the host) that is causing UDP packets to be delayed or dropped, adding significant latency. (This is not uncommon with HTTP/3, since most network gear focuses on TCP.)

I would check with your host, your host’s ISP, and the actual OS image configuration to ensure that UDP performance is optimal.

jany · August 29, 2023, 6:35am

My ISP will surely be fine, because these problems are already in the LAN (DMZ). Now I closed the UDP port and left only TCP 80/443 open. It is understandable that HTTP/3 still works within the LAN network. Now I don’t have the opportunity to try the WAN.
This is my topology

The servers run Linux containers (LXC Debian 12/11). I download the LXC templates from here.

Please try it now that I have disabled UDP ports 80/443.
BTW, I tried disabling HTTP/3 in firefox and loading is still very long.

jany · September 2, 2023, 2:22pm

I didn’t find out why the wordpress website takes a long time to load, so I have to switch to nginx.

system · October 2, 2023, 2:23pm

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.