Is Caddy reverse proxy really slow? If not, how do I locate the cause?

[vbakke (Vegard Bakke)]

I’m planning on using Caddy as a reverse proxy in front of the Business Intelligence (BI) tool Qlik Sense. But when I’m testing it, opening a page is about 10x as slow as opening it directly.

For a difference this great, there must be an error somewhere. Can someone here help we pin point where this cause might be hiding?

The Qlik Sense Demo site is open to the public, so feel free to test if you get similar results.

1. Caddy version (caddy version):

V1: v1.0.4
V2: v2.0.0 h1:pQSaIJGFluFvu8KDGDODV8u4/QRED/OPyIR+MWYYse8=

2. How I run Caddy:

a. System environment:

localhost: Windows 10 Pro, v 1903

b. Command:

V1: caddy.exe --conf Caddyfile-QlikDemo
V2: caddy.exe run --config Caddyfile-QlikDemo

c. Service/unit/compose file:

N/A

d. My complete Caddyfile or JSON config:

V1

# Caddy v1
https://localhost:4200 {
	log ./logs/access.log
    #log stdout # "{proto} Request: {method} {path} {request}"
	tls self_signed

	proxy / https://sense-demo.qlik.com:443 { 
		#transparent
		websocket
		#insecure_skip_verify 
	}
    
}

V2

# Caddy v2
https://localhost:4200 {
		
	#tls internal

	reverse_proxy * https://sense-demo.qlik.com:443 { 
		header_up Host {http.reverse_proxy.upstream.hostport}  # Include only if you need to override browsers Host header
	}
    
}

3. The problem I’m having:

I cannot use Caddy, it it actually appears to be 10x as slow as going directly. ;-/

4. Error messages and/or full log output:

I tried a quick test, using Chrome DevTools - Network - Finnish: timestamp to clock the download time.
Tried 10 times via v1, 10 times via v2, and 10 times directly.

On averge:

• v1: 19,8 sec
• v2: 23,9 sec
• direct: 2,6 sec

Chrome DevTools is reporting TTFB as main cause of the delay. But Chrome has (of cource) no way of knowing if the wait is between Chrome and Caddy, inside Caddy, og between Caddy and Qlik Sense.

5. What I already tried:

Not really sure how to go about and debug this.

Are there any useful ways of getting timings and insight out from the inner workings of Caddy, to find out what side the problem is on?

I have not yet tried installing Caddy on the same server er Qlik Sense is running on. Since I have to change ports on a production server, I didn’t want to do that before I had som more knowledge abut the problem.

6. Links to relevant resources:

Qlik Sense is a real world case in the wild. A competitor with PowerBI and Tableau, among others.

So called “Quick guides” for using Apache or IIS as reverse proxies.
Apache: https://support.qlik.com/articles/000049958
IIS: https://support.qlik.com/articles/000043332

(Why do you think I’d like to use Caddy. )

Any help is greatly appreciated.

[vbakke (Vegard Bakke)]

Ah, forgot to mention the URL I was using when timing!

I tested: https://localhost:4200/dev-hub/ for V1 and V2.
And Qlik Sense when going directly.

It’s a fairly simple page without too much bells and whistles. Thought that was a good candidate. But behaves slow on other pages as well.

The https://localhost:4200/hub/ is the normal starting point.
(If you use https://localhost:4200/ you get redirected to the actual site.)

[francislavoie]

Looping in @hairyhenderson who works at Qlik and is the maintainer of the Caddy Docker image

[hairyhenderson (Dave Henderson)]

Hi @vbakke! Good to see people using Caddy with Qlik Sense

I presume you had browser caching disabled for all those tests? Also are you able to post any logs?

I’ll try to find some time to poke at this a bit to see if there’s anything weird going on…

[vbakke (Vegard Bakke)]

Good to see people using Caddy with Qlik Sense

Sure. Well, at least I hope I will.

I presume you had browser caching disabled for all those tests

Actually, I testet both with and without browser cache (in DevTools). And for some reason, it did not make much of a difference.

Also are you able to post any logs?

Sure. But which logs do you want? Any specific caddy parameters?

Regards,
Vegard

[hairyhenderson (Dave Henderson)]

@vbakke

Actually, I testet both with and without browser cache (in DevTools). And for some reason, it did not make much of a difference.

Strange! But perhaps there’s something in the response that means it can’t be cached. No big deal.

@vbakke

Also are you able to post any logs?

Sure. But which logs do you want? Any specific caddy parameters?

Yup - add the global debug option. Just put this at the top of your Caddyfile:

{
    debug
}

Hopefully that’ll give a bit more insight into what’s going on!

[vbakke (Vegard Bakke)]

Hmm. Looked like I could only upload images to this forum, so I placed the logfile in a Dropbox:

CaddyQlik.log

Cheers,
Ve

[vbakke (Vegard Bakke)]

Does anyone else get the same sluggish response, or is it just om my local machine?

[calisro (Rob)]

Definitely fast for me. I’ve tested running an rclone sync against the http service through a reverse proxy with many many transfers and was able to saturate my gigabit link.

I’ve also setup that same sync with a large number of ‘checkers’ (rather than just streaming data) and it completed very reasonably.

[hairyhenderson (Dave Henderson)]

Sorry for the slow response…

I’ve just now been able to try to reproduce and I’m seeing basically the same timings for both proxied and unproxied. I used the Brave DevTools (same as Chrome) performance tab to profile a dozen page loads and in both cases it varied from ~3s to ~7s.

I’m testing on macOS Catalina 10.15.4, with Brave v1.8.95, and the release binary of Caddy 2.0.0.

I wonder if there’s something special/different about Windows, but that’s the only significant difference I can think of…

[hairyhenderson (Dave Henderson)]

@hairyhenderson

I’ve just now been able to try to reproduce and I’m seeing basically the same timings for both proxied and unproxied.

Just to be clear (since upon a second reading I realize this maybe isn’t that clear):

I haven’t been able to reproduce the slow performance. I’m seeing similar performance both with and without Caddy in the way

[matt (Matt Holt)]

Is it DNS?

It’s always DNS.

What happens if you proxy to the site’s IP address instead of domain name?

[vbakke (Vegard Bakke)]

Nope, sorry. Tried now with:

# Caddy v2
{
    #debug
}

https://localhost:4200 {
	

	reverse_proxy * https://52.48.5.86:443 { 
		header_up Host sense-demo.qlik.com

		transport http {
			tls_insecure_skip_verify 
		}

	}
    
}

And same delay. But it could be something on the remote server for all I know. Or a way of pooling requests together (or not do so). I don’t know how if the remote site has any reverse proxies in addition.

@hairyhenderson did you have any notable TTFB (time-to-first-byte) in the requests in Brave DevTools?