Very slow access to self-hosted applications through Docker

OursBlanc · February 6, 2021, 12:35pm

1. Caddy version (`caddy version`):

/srv # caddy version
v2.3.0 h1:fnrqJLa3G5vfxcxmOH/+kJOcunPLhSBnjgIvjXV/QTA=

2. How I run Caddy:

I run caddy in Docker on my server (self hosted)

a. System environment:

Armbian 21.02.1 Buster with Linux 5.10.12-rockchip64

Docker 20.10.3 (API: 1.41)

b. Command:

c. Service/unit/compose file:

 	caddy run --config /etc/caddy/Caddyfile --adapter caddyfile

d. My complete Caddyfile or JSON config:

mydomain.xy {
        root * /srv/mydomain/
        file_server
}

page.mydomain.xy {
        root * /srv/page/
        file_server
}

rss.mydomain.xy {

        log {
                format single_field common_log
                output file /var/log/caddy/rss.log
        }

        reverse_proxy 192.168.1.5:48080

}


torrent.mydomain.xy {

        log {
                format single_field common_log
                output file /var/log/caddy/torrent.log
        }

        reverse_proxy 192.168.1.5:8080

}

3. The problem I’m having:

If i try to connect to FreshRSS or Qbittorent (two container in docker) with local ip : no problem.
If I try to connect by typing the address, it works but it’s extremely slow. Several minutes just to display the login page

4. Error messages and/or full log output:

No error message (unless i’m mistaken)

5. What I already tried:

I tried to run Caddy directly on my server (not through docker in a container) : same thing.
I tried to just reverse_proxy (without log) : same thing

thank you for taking the time to help me. i am a neophyte, please excuse my ignorance.

6. Links to relevant resources:

francislavoie · February 6, 2021, 4:06pm

Hmm. If you make a request with curl -v instead of via the browser, what do you see?

If you enable debug mode (by adding the below, at the top of your Caddyfile), what’s in your logs?

{
	debug
}

OursBlanc · February 7, 2021, 12:06am

Hi and thank you for your answer.

This is the result with curl -v from the server

root@helios64:/etc/caddy# curl -v rss.mywebsite.me
* Expire in 0 ms for 6 (transfer 0xaaaad6054f10)
* Expire in 1 ms for 1 (transfer 0xaaaad6054f10)
* Expire in 0 ms for 1 (transfer 0xaaaad6054f10)
* Expire in 1 ms for 1 (transfer 0xaaaad6054f10)
* Expire in 0 ms for 1 (transfer 0xaaaad6054f10)
* Expire in 0 ms for 1 (transfer 0xaaaad6054f10)
* Expire in 1 ms for 1 (transfer 0xaaaad6054f10)
* Expire in 0 ms for 1 (transfer 0xaaaad6054f10)
* Expire in 0 ms for 1 (transfer 0xaaaad6054f10)
* Expire in 1 ms for 1 (transfer 0xaaaad6054f10)
* Expire in 0 ms for 1 (transfer 0xaaaad6054f10)
* Expire in 0 ms for 1 (transfer 0xaaaad6054f10)
* Expire in 2 ms for 1 (transfer 0xaaaad6054f10)
* Expire in 0 ms for 1 (transfer 0xaaaad6054f10)
* Expire in 0 ms for 1 (transfer 0xaaaad6054f10)
* Expire in 2 ms for 1 (transfer 0xaaaad6054f10)
* Expire in 0 ms for 1 (transfer 0xaaaad6054f10)
* Expire in 0 ms for 1 (transfer 0xaaaad6054f10)
* Expire in 2 ms for 1 (transfer 0xaaaad6054f10)
* Expire in 0 ms for 1 (transfer 0xaaaad6054f10)
* Expire in 0 ms for 1 (transfer 0xaaaad6054f10)
* Expire in 2 ms for 1 (transfer 0xaaaad6054f10)
* Expire in 0 ms for 1 (transfer 0xaaaad6054f10)
* Expire in 0 ms for 1 (transfer 0xaaaad6054f10)
* Expire in 2 ms for 1 (transfer 0xaaaad6054f10)
* Expire in 0 ms for 1 (transfer 0xaaaad6054f10)
* Expire in 0 ms for 1 (transfer 0xaaaad6054f10)
* Expire in 2 ms for 1 (transfer 0xaaaad6054f10)
* Expire in 0 ms for 1 (transfer 0xaaaad6054f10)
* Expire in 0 ms for 1 (transfer 0xaaaad6054f10)
* Expire in 2 ms for 1 (transfer 0xaaaad6054f10)
* Expire in 0 ms for 1 (transfer 0xaaaad6054f10)
* Expire in 0 ms for 1 (transfer 0xaaaad6054f10)
* Expire in 2 ms for 1 (transfer 0xaaaad6054f10)
* Expire in 0 ms for 1 (transfer 0xaaaad6054f10)
* Expire in 0 ms for 1 (transfer 0xaaaad6054f10)
* Expire in 2 ms for 1 (transfer 0xaaaad6054f10)
* Expire in 0 ms for 1 (transfer 0xaaaad6054f10)
* Expire in 0 ms for 1 (transfer 0xaaaad6054f10)
* Expire in 2 ms for 1 (transfer 0xaaaad6054f10)
* Expire in 0 ms for 1 (transfer 0xaaaad6054f10)
* Expire in 0 ms for 1 (transfer 0xaaaad6054f10)
* Expire in 2 ms for 1 (transfer 0xaaaad6054f10)
* Expire in 0 ms for 1 (transfer 0xaaaad6054f10)
* Expire in 1 ms for 1 (transfer 0xaaaad6054f10)
* Expire in 2 ms for 1 (transfer 0xaaaad6054f10)
* Expire in 1 ms for 1 (transfer 0xaaaad6054f10)
* Expire in 1 ms for 1 (transfer 0xaaaad6054f10)
* Expire in 2 ms for 1 (transfer 0xaaaad6054f10)
* Expire in 1 ms for 1 (transfer 0xaaaad6054f10)
* Expire in 1 ms for 1 (transfer 0xaaaad6054f10)
* Expire in 2 ms for 1 (transfer 0xaaaad6054f10)
* Expire in 1 ms for 1 (transfer 0xaaaad6054f10)
* Expire in 1 ms for 1 (transfer 0xaaaad6054f10)
* Expire in 2 ms for 1 (transfer 0xaaaad6054f10)
* Expire in 2 ms for 1 (transfer 0xaaaad6054f10)
* Expire in 2 ms for 1 (transfer 0xaaaad6054f10)
* Expire in 4 ms for 1 (transfer 0xaaaad6054f10)
* Expire in 2 ms for 1 (transfer 0xaaaad6054f10)
* Expire in 2 ms for 1 (transfer 0xaaaad6054f10)
* Expire in 4 ms for 1 (transfer 0xaaaad6054f10)
* Expire in 3 ms for 1 (transfer 0xaaaad6054f10)
* Expire in 3 ms for 1 (transfer 0xaaaad6054f10)
* Expire in 4 ms for 1 (transfer 0xaaaad6054f10)
* Expire in 3 ms for 1 (transfer 0xaaaad6054f10)
* Expire in 3 ms for 1 (transfer 0xaaaad6054f10)
* Expire in 4 ms for 1 (transfer 0xaaaad6054f10)
* Expire in 4 ms for 1 (transfer 0xaaaad6054f10)
* Expire in 4 ms for 1 (transfer 0xaaaad6054f10)
* Expire in 4 ms for 1 (transfer 0xaaaad6054f10)
* Expire in 5 ms for 1 (transfer 0xaaaad6054f10)
* Expire in 5 ms for 1 (transfer 0xaaaad6054f10)
* Expire in 4 ms for 1 (transfer 0xaaaad6054f10)
* Expire in 5 ms for 1 (transfer 0xaaaad6054f10)
* Expire in 5 ms for 1 (transfer 0xaaaad6054f10)
* Expire in 8 ms for 1 (transfer 0xaaaad6054f10)
* Expire in 7 ms for 1 (transfer 0xaaaad6054f10)
* Expire in 7 ms for 1 (transfer 0xaaaad6054f10)
* Expire in 8 ms for 1 (transfer 0xaaaad6054f10)
* Expire in 7 ms for 1 (transfer 0xaaaad6054f10)
* Expire in 7 ms for 1 (transfer 0xaaaad6054f10)
* Expire in 8 ms for 1 (transfer 0xaaaad6054f10)
* Expire in 9 ms for 1 (transfer 0xaaaad6054f10)
* Expire in 9 ms for 1 (transfer 0xaaaad6054f10)
* Expire in 8 ms for 1 (transfer 0xaaaad6054f10)
* Expire in 10 ms for 1 (transfer 0xaaaad6054f10)
* Expire in 10 ms for 1 (transfer 0xaaaad6054f10)
* Expire in 16 ms for 1 (transfer 0xaaaad6054f10)
* Expire in 13 ms for 1 (transfer 0xaaaad6054f10)
* Expire in 13 ms for 1 (transfer 0xaaaad6054f10)
* Expire in 16 ms for 1 (transfer 0xaaaad6054f10)
* Expire in 13 ms for 1 (transfer 0xaaaad6054f10)
* Expire in 13 ms for 1 (transfer 0xaaaad6054f10)
* Expire in 16 ms for 1 (transfer 0xaaaad6054f10)
* Expire in 50 ms for 1 (transfer 0xaaaad6054f10)
* Expire in 50 ms for 1 (transfer 0xaaaad6054f10)
* Expire in 16 ms for 1 (transfer 0xaaaad6054f10)
* Expire in 50 ms for 1 (transfer 0xaaaad6054f10)
* Expire in 50 ms for 1 (transfer 0xaaaad6054f10)
* Expire in 50 ms for 1 (transfer 0xaaaad6054f10)
*   Trying 2001:xxx:yyy:zzz:xxx:yyy:zzz:3e55...
* TCP_NODELAY set
* Expire in 149948 ms for 3 (transfer 0xaaaad6054f10)
* Expire in 200 ms for 4 (transfer 0xaaaad6054f10)
* connect to 2001:xxx:yyy:zzz:xxx:yyy:zzz:3e55... port 80 failed: Connexion refusée
*   Trying 80.xx.yy.zz...
* TCP_NODELAY set
* Expire in 149948 ms for 3 (transfer 0xaaaad6054f10)
* Connected to rss.mywebsite.me (80.xx.yy.zz) port 80 (#0)
> GET / HTTP/1.1
> Host: rss.mywebsite.me
> User-Agent: curl/7.64.0
> Accept: */*
> 
< HTTP/1.1 308 Permanent Redirect
< Connection: close
< Location: https://rss.mywebsite.me/
< Server: Caddy
< Date: Sat, 06 Feb 2021 23:51:16 GMT
< Content-Length: 0
< 
* Closing connection 0

and this is the resultat from my local network

    *   Trying 2001:xxx:yyy:zzz:xxx:yyy:zzz:3e55...
* TCP_NODELAY set
* connect to 2001:xxx:yyy:zzz:xxx:yyy:zzz:3e55 port 80 failed: Connexion refusée
*   Trying 80.xx.yy.zz...
* TCP_NODELAY set
* Connected to rss.mywebsite.me (80.xx.yy.zz) port 80 (#0)
> GET / HTTP/1.1
> Host: rss.mywebsite.me
> User-Agent: curl/7.68.0
> Accept: */*
> 
* Mark bundle as not supporting multiuse
< HTTP/1.1 308 Permanent Redirect
< Connection: close
< Location: https://rss.mywebsite.me/
< Server: Caddy
< Date: Sat, 06 Feb 2021 23:54:01 GMT
< Content-Length: 0
< 
* Closing connection 0

Then this is my log (again for the RSS subdomain) .

192.168.1.1 - - [07/Feb/2021:00:05:12 +0000] “GET /themes/icons/favicon-256.png HTTP/2.0” 0 0
192.168.1.1 - - [07/Feb/2021:00:05:12 +0000] “GET /favicon.ico HTTP/2.0” 0 0
192.168.1.1 - - [07/Feb/2021:00:05:12 +0000] “GET /i/ HTTP/2.0” 200 19504
192.168.1.1 - - [07/Feb/2021:00:05:12 +0000] “GET /themes/icons/add-white.svg HTTP/2.0” 200 217
192.168.1.1 - - [07/Feb/2021:00:05:12 +0000] “GET /themes/icons/down.svg HTTP/2.0” 200 605
192.168.1.1 - - [07/Feb/2021:00:05:12 +0000] “GET /themes/icons/unread.svg HTTP/2.0” 200 778
192.168.1.1 - - [07/Feb/2021:00:05:12 +0000] “GET /themes/icons/bookmark-add.svg HTTP/2.0” 200 261
192.168.1.1 - - [07/Feb/2021:00:05:12 +0000] “GET /themes/icons/all.svg HTTP/2.0” 200 362
192.168.1.1 - - [07/Feb/2021:00:05:12 +0000] “GET /themes/icons/view-global.svg HTTP/2.0” 200 705
192.168.1.1 - - [07/Feb/2021:00:05:12 +0000] “GET /themes/icons/up.svg HTTP/2.0” 200 604
192.168.1.1 - - [07/Feb/2021:00:05:12 +0000] “GET /themes/icons/rss.svg HTTP/2.0” 200 672
192.168.1.1 - - [07/Feb/2021:00:05:12 +0000] “GET /themes/icons/category.svg HTTP/2.0” 200 567
192.168.1.1 - - [07/Feb/2021:00:05:12 +0000] “GET /themes/icons/logout.svg HTTP/2.0” 200 569
192.168.1.1 - - [07/Feb/2021:00:05:12 +0000] “GET /themes/icons/link.svg HTTP/2.0” 200 4018
192.168.1.1 - - [07/Feb/2021:00:05:12 +0000] “GET /themes/icons/next.svg HTTP/2.0” 200 589
192.168.1.1 - - [07/Feb/2021:00:05:12 +0000] “GET /themes/icons/view-reader.svg HTTP/2.0” 200 1090
192.168.1.1 - - [07/Feb/2021:00:05:12 +0000] “GET /themes/icons/read.svg HTTP/2.0” 200 1295
192.168.1.1 - - [07/Feb/2021:00:05:12 +0000] “GET /themes/icons/bookmark.svg HTTP/2.0” 200 750
192.168.1.1 - - [07/Feb/2021:00:05:12 +0000] “GET /themes/icons/prev.svg HTTP/2.0” 200 592
192.168.1.1 - - [07/Feb/2021:00:05:12 +0000] “GET /themes/icons/non-starred.svg HTTP/2.0” 200 4321
192.168.1.1 - - [07/Feb/2021:00:05:12 +0000] “GET /themes/icons/view-normal.svg HTTP/2.0” 200 216
192.168.1.1 - - [07/Feb/2021:00:05:12 +0000] “GET /themes/icons/starred.svg HTTP/2.0” 200 750
192.168.1.1 - - [07/Feb/2021:00:05:12 +0000] “GET /themes/icons/refresh.svg HTTP/2.0” 200 1132
192.168.1.1 - - [07/Feb/2021:00:05:12 +0000] “GET /themes/icons/search.svg HTTP/2.0” 200 886

francislavoie · February 7, 2021, 12:15am

Ahhh I think you’re running into NAT hairpinning issues:

Basically, DNS for your domain is resolved to the WAN IP, then when a request goes out, it reaches your router, then because it likely doesn’t support NAT hairpinning, the connection gets dropped because it doesn’t know to route the request back internally.

Please read through this article to get an idea of how it works:

The solution is typically to also run a DNS server in your home network, to implement split DNS. Basically this means that your local DNS server will resolve your domain to the LAN IP address of your server, instead of the WAN IP address, but when outside of your home network, the domain will still resolve to the WAN IP.

Those are your access logs, not the Caddy logs. You can see those by running docker logs <name-of-the-caddy-container>.

OursBlanc · February 7, 2021, 12:30pm

Hey, thank you for your answer. Unfortunately, i dont understand two points
first, if i dont use docker, i dont have this kind of problem. Indeed if i intsall my app and caddy directly on my server : everything work.

then, i watched in my router and it support NAT hairpinning, and this option is activated.

(my routeur is a ASUS DSL N 16 and i followed the official documentation https://dlcdnets.asus.com/pub/ASUS/wireless/DSL-N16/E10544_DSL_N16_Manual.pdf?_ga=2.254814592.926661751.1568901346-1551512870.1564642544 )

Then, last thing very strange. When i try to access at my service from my laptop (and not my main computer) , everything work fine, nothing is slow.
(but still slow from my main computer)

I tried to launch my browser without any pluggin (on main computer) but nothing change.

End also, see my log : (i hope is the good one)
(very long so i used a pastebin to share you the file)

thank you

matt · February 7, 2021, 6:36pm

Maybe you have an old version of curl?

Mohammed90 · February 7, 2021, 7:02pm

Are they both on the same network? What does the network look like? Also since you’re running it on Docker, what’s the Docker command you’re using?

OursBlanc · February 7, 2021, 8:34pm

Hi @Mohammed90 , all my device are on the same network.

My router (192.168.1.1)
My main computer (wired to router) (192.168.1.2)
My laptop (wifi to router) (192.168.1.4)
and the server (wired to router) (192.168.1.5)

I used portainer to manage the container and there is the command for caddy :
‘caddy’ ‘run’ ‘–config’ ‘/etc/caddy/Caddyfile’ ‘–adapter’ ‘caddyfile’

About the CURL error…
I updated CURL on my server; i followed this guide : https://blog.usejournal.com/how-to-manually-update-curl-on-ubuntu-server-899476062ad6

Now its curl 7.70.0 but nothing change on my main computer. But on my smartphone (not at all on my local network) or laptop, its fine…

OursBlanc · February 10, 2021, 5:45pm

Hi there.
I’m sorry for this mistake. But indeed my router cant manage this problem of hairpining.
I use pihole so i created with own dnsmasq to resolve this problem

thank you for your help