Trying to transition my reverse proxy set up from Caddy V1 in Freenas Jail to Caddy v2 on Debian

itsshash · June 11, 2020, 2:20pm

1. Caddy version (`caddy version` ): `v2.0.0 h1:pQSaIJGFluFvu8KDGDODV8u4/QRED/OPyIR+MWYYse8=`

2. How I run Caddy:

a. System environment:

Debian buster (Proxmox specifically. I want to move it to an Ubuntu LXC in the future)

b. Command:

caddy run -config CaddyfileV2

c. Service/unit/compose file:

n/a still running from command line before using service unit.

d. My complete Caddyfile or JSON config:

search.fakeadddress.com { 
        reverse_proxy 192.168.1.237:5000
}

rss.fakeadddress.com {
        reverse_proxy 192.168.1.47:8080 
}

walla.fakeadddress.com {
        reverse_proxy 192.168.1.47:80
}

komga.fakeadddress.com {
        reverse_proxy 192.168.1.47:81
}

radarr.fakeadddress.com {
        reverse_proxy 192.168.1.228:7878
}

sonarr.fakeadddress.com {
        reverse_proxy 192.168.1.228:8989
}

nas.fakeadddress.com {
        reverse_proxy 192.168.1.152:80
}

mine.fakeadddress.com {
        reverse_proxy 192.168.1.195:443
}

home.fakeadddress.com {
       reverse_proxy 192.168.1.228:8899
}

3. The problem I’m having:

The only 2 instances that urls that work are the search.fakeaddress.com and nas.fakeaddress.com, which are a search engine and FreeNAS web UI respectively.

4. Error messages and/or full log output:

2020/06/11 13:46:49.434	INFO	using provided configuration	{"config_file": "CaddyfileV2", "config_adapter": ""}
run: loading initial config: loading new config: starting caddy administration endpoint: listen tcp 127.0.0.1:2019: bind: address already in use
root@pve:~# caddy stop
root@pve:~# caddy run -config CaddyfileV2 
2020/06/11 13:46:56.966	INFO	using provided configuration	{"config_file": "CaddyfileV2", "config_adapter": ""}
2020/06/11 13:46:56.968	INFO	admin	admin endpoint started	{"address": "tcp/localhost:2019", "enforce_origin": false, "origins": ["127.0.0.1:2019", "localhost:2019", "[::1]:2019"]}
2020/06/11 13:46:56.968	INFO	http	server is listening only on the HTTPS port but has no TLS connection policies; adding one to enable TLS	{"server_name": "srv0", "https_port": 443}
2020/06/11 13:46:56.968	INFO	http	enabling automatic HTTP->HTTPS redirects	{"server_name": "srv0"}
2020/06/11 14:46:56 [INFO][cache:0xc0000f46e0] Started certificate maintenance routine
2020/06/11 13:46:56.970	INFO	tls	cleaned up storage units
2020/06/11 13:46:56.970	INFO	http	enabling automatic TLS certificate management	{"domains": ["radarr.fakeaddress.com", "walla.fakeaddress.com", "home.fakeaddress.com", "search.fakeaddress.com", "komga.fakeaddress.com", "mine.fakeaddress.com", "sonarr.fakeaddress.com", "rss.fakeaddress.com", "nas.fakeaddress.com"]}
2020/06/11 13:46:56.981	INFO	autosaved config	{"file": "/root/.config/caddy/autosave.json"}
2020/06/11 13:46:56.981	INFO	serving initial configuration
^C2020/06/11 13:47:02.878	INFO	shutting down	{"signal": "SIGINT"}
2020/06/11 14:47:02 [INFO][cache:0xc0000f46e0] Stopped certificate maintenance routine
2020/06/11 13:47:02.879	INFO	admin	stopped previous server
2020/06/11 13:47:02.879	INFO	shutdown done	{"signal": "SIGINT"}
root@pve:~# nano CaddyfileV2 
root@pve:~# caddy run -config CaddyfileV2 
2020/06/11 13:47:56.455	INFO	using provided configuration	{"config_file": "CaddyfileV2", "config_adapter": ""}
2020/06/11 13:47:56.456	INFO	admin	admin endpoint started	{"address": "tcp/localhost:2019", "enforce_origin": false, "origins": ["localhost:2019", "[::1]:2019", "127.0.0.1:2019"]}
2020/06/11 13:47:56.456	INFO	http	server is listening only on the HTTPS port but has no TLS connection policies; adding one to enable TLS	{"server_name": "srv0", "https_port": 443}
2020/06/11 13:47:56.457	INFO	http	enabling automatic HTTP->HTTPS redirects	{"server_name": "srv0"}
2020/06/11 14:47:56 [INFO][cache:0xc0005a26e0] Started certificate maintenance routine
2020/06/11 13:47:56.459	INFO	tls	cleaned up storage units
2020/06/11 13:47:56.459	INFO	http	enabling automatic TLS certificate management	{"domains": ["search.fakeaddress.com", "radarr.fakeaddress.com", "walla.fakeaddress.com", "komga.fakeaddress.com", "mine.fakeaddress.com", "nas.fakeaddress.com", "sonarr.fakeaddress.com", "home.fakeaddress.com", "rss.fakeaddress.com"]}
2020/06/11 13:47:56.470	INFO	autosaved config	{"file": "/root/.config/caddy/autosave.json"}
2020/06/11 13:47:56.470	INFO	serving initial configuration
2020/06/11 13:49:01.766	ERROR	http.log.error	dial tcp 192.168.1.47:8080: i/o timeout	{"request": {"method": "GET", "uri": "/", "proto": "HTTP/2.0", "remote_addr": "37.120.198.77:53400", "host": "rss.fakeaddress.com", "headers": {"User-Agent": ["Mozilla/5.0 (Windows NT 10.0; rv:68.0) Gecko/20100101 Firefox/68.0"], "Accept": ["text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8"], "Accept-Language": ["en-US,en;q=0.5"], "Accept-Encoding": ["gzip, deflate, br"], "Dnt": ["1"], "Upgrade-Insecure-Requests": ["1"], "Te": ["trailers"]}, "tls": {"resumed": false, "version": 772, "ciphersuite": 4865, "proto": "h2", "proto_mutual": true, "server_name": "rss.fakeaddress.com"}}, "duration": 10.000246137, "status": 502, "err_id": "9s6sfzyxx", "err_trace": "reverseproxy.(*Handler).ServeHTTP (reverseproxy.go:380)"}
2020/06/11 13:49:50.523	ERROR	http.log.error	dial tcp 192.168.1.47:8080: i/o timeout	{"request": {"method": "GET", "uri": "/api/greader.php/reader/api/0/unread-count?output=json", "proto": "HTTP/2.0", "remote_addr": "37.120.198.77:53400", "host": "rss.fakeaddress.com", "headers": {"Te": ["trailers"], "User-Agent": ["Mozilla/5.0 (Windows NT 10.0; rv:68.0) Gecko/20100101 Firefox/68.0"], "Accept": ["*/*"], "Accept-Language": ["en-US,en;q=0.5"], "Accept-Encoding": ["gzip, deflate, br"], "Authorization": ["GoogleLogin auth=shash/e9323bd68e2a5c25168f4d8e6de40ec140cd04c3"], "Dnt": ["1"]}, "tls": {"resumed": false, "version": 772, "ciphersuite": 4865, "proto": "h2", "proto_mutual": true, "server_name": "rss.fakeaddress.com"}}, "duration": 10.000245481, "status": 502, "err_id": "wcsghh536", "err_trace": "reverseproxy.(*Handler).ServeHTTP (reverseproxy.go:380)"}
2020/06/11 14:50:09 http: TLS handshake error from 107.178.200.195:57573: no certificate available for 'qbitt.fakeaddress.com'
2020/06/11 13:50:25.822	ERROR	http.log.error	dial tcp 192.168.1.47:80: i/o timeout	{"request": {"method": "GET", "uri": "/", "proto": "HTTP/2.0", "remote_addr": "37.120.198.77:53462", "host": "walla.fakeaddress.com", "headers": {"User-Agent": ["Mozilla/5.0 (Windows NT 10.0; rv:68.0) Gecko/20100101 Firefox/68.0"], "Accept": ["text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8"], "Accept-Language": ["en-US,en;q=0.5"], "Accept-Encoding": ["gzip, deflate, br"], "Dnt": ["1"], "Upgrade-Insecure-Requests": ["1"], "Te": ["trailers"]}, "tls": {"resumed": false, "version": 772, "ciphersuite": 4865, "proto": "h2", "proto_mutual": true, "server_name": "walla.fakeaddress.com"}}, "duration": 10.00027823, "status": 502, "err_id": "qikt3pimn", "err_trace": "reverseproxy.(*Handler).ServeHTTP (reverseproxy.go:380)"}
2020/06/11 13:50:35.924	ERROR	http.log.error	dial tcp 192.168.1.47:80: i/o timeout	{"request": {"method": "GET", "uri": "/favicon.ico", "proto": "HTTP/2.0", "remote_addr": "37.120.198.77:53462", "host": "walla.fakeaddress.com", "headers": {"Te": ["trailers"], "User-Agent": ["Mozilla/5.0 (Windows NT 10.0; rv:68.0) Gecko/20100101 Firefox/68.0"], "Accept": ["image/webp,*/*"], "Accept-Language": ["en-US,en;q=0.5"], "Accept-Encoding": ["gzip, deflate, br"], "Dnt": ["1"]}, "tls": {"resumed": false, "version": 772, "ciphersuite": 4865, "proto": "h2", "proto_mutual": true, "server_name": "walla.fakeaddress.com"}}, "duration": 10.000224067, "status": 502, "err_id": "wgfxse37j", "err_trace": "reverseproxy.(*Handler).ServeHTTP (reverseproxy.go:380)"}
2020/06/11 13:50:56.117	ERROR	http.log.error	remote error: tls: internal error	{"request": {"method": "GET", "uri": "/index.php/204", "proto": "HTTP/1.1", "remote_addr": "192.168.1.1:48225", "host": "mine.fakeaddress.com", "headers": {"User-Agent": ["Mozilla/5.0 (Android) Nextcloud-android/3.11.1"]}, "tls": {"resumed": false, "version": 772, "ciphersuite": 4865, "proto": "", "proto_mutual": true, "server_name": "mine.fakeaddress.com"}}, "duration": 0.001173779, "status": 502, "err_id": "2kx17n7zy", "err_trace": "reverseproxy.(*Handler).ServeHTTP (reverseproxy.go:380)"}
2020/06/11 13:52:09.158	ERROR	http.log.error	remote error: tls: internal error	{"request": {"method": "GET", "uri": "/", "proto": "HTTP/2.0", "remote_addr": "37.120.198.77:53524", "host": "mine.fakeaddress.com", "headers": {"User-Agent": ["Mozilla/5.0 (Windows NT 10.0; rv:68.0) Gecko/20100101 Firefox/68.0"], "Accept": ["text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8"], "Accept-Language": ["en-US,en;q=0.5"], "Accept-Encoding": ["gzip, deflate, br"], "Dnt": ["1"], "Cookie": ["oc43um0qgdpv=dqbc92mjet06m410thesh0k76e; oc_sessionPassphrase=UYCajRZqkmY5gPvhz2UXCODgXh76x5kxKQhQ8Taf%2F8%2BQJXH%2Fxtj4%2FMfhd%2BG%2Bcc%2BCPuS9cmz9PlLIi7tzrtzeZa47ZhDjz2WgdKAAe1wZILjjsj5vEQQPwLX8JjhPeV7T; __Host-nc_sameSiteCookielax=true; __Host-nc_sameSiteCookiestrict=true"], "Upgrade-Insecure-Requests": ["1"], "Te": ["trailers"]}, "tls": {"resumed": false, "version": 772, "ciphersuite": 4865, "proto": "h2", "proto_mutual": true, "server_name": "mine.fakeaddress.com"}}, "duration": 0.000849436, "status": 502, "err_id": "svq9b483f", "err_trace": "reverseproxy.(*Handler).ServeHTTP (reverseproxy.go:380)"}
2020/06/11 13:53:09.700	ERROR	http.log.error	dial tcp 192.168.1.47:8080: i/o timeout	{"request": {"method": "POST", "uri": "/api/pshb.php?k=3fccd4f193c9f88491307c9f89f32f203d02cb58", "proto": "HTTP/1.1", "remote_addr": "66.249.84.235:44477", "host": "rss.fakeaddress.com", "headers": {"Content-Length": ["4488"], "Connection": ["keep-alive"], "From": ["googlebot(at)googlebot.com"], "Accept-Encoding": ["gzip,deflate,br"], "Link": ["<http://feeds.arstechnica.com/arstechnica/index>; rel=self, <http://pubsubhubbub.appspot.com/>; rel=hub"], "Content-Type": ["application/rss+xml"], "User-Agent": ["FeedFetcher-Google; (+http://www.google.com/feedfetcher.html)"], "Cache-Control": ["no-cache,max-age=0"], "Pragma": ["no-cache"], "Accept": ["*/*"]}, "tls": {"resumed": false, "version": 772, "ciphersuite": 4865, "proto": "", "proto_mutual": true, "server_name": "rss.fakeaddress.com"}}, "duration": 10.000331364, "status": 502, "err_id": "yw3sgm54j", "err_trace": "reverseproxy.(*Handler).ServeHTTP (reverseproxy.go:380)"}
2020/06/11 13:53:45.947	ERROR	http.log.error	dial tcp 192.168.1.47:8080: i/o timeout	{"request": {"method": "GET", "uri": "/", "proto": "HTTP/2.0", "remote_addr": "37.120.198.77:53594", "host": "rss.fakeaddress.com", "headers": {"Upgrade-Insecure-Requests": ["1"], "Te": ["trailers"], "User-Agent": ["Mozilla/5.0 (Windows NT 10.0; rv:68.0) Gecko/20100101 Firefox/68.0"], "Accept": ["text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8"], "Accept-Language": ["en-US,en;q=0.5"], "Accept-Encoding": ["gzip, deflate, br"], "Dnt": ["1"]}, "tls": {"resumed": false, "version": 772, "ciphersuite": 4865, "proto": "h2", "proto_mutual": true, "server_name": "rss.fakeaddress.com"}}, "duration": 10.00025098, "status": 502, "err_id": "4882mf36t", "err_trace": "reverseproxy.(*Handler).ServeHTTP (reverseproxy.go:380)"}
2020/06/11 13:54:36.407	ERROR	http.log.error	dial tcp 192.168.1.228:8899: i/o timeout	{"request": {"method": "GET", "uri": "/", "proto": "HTTP/2.0", "remote_addr": "37.120.198.77:53646", "host": "home.fakeaddress.com", "headers": {"User-Agent": ["Mozilla/5.0 (Windows NT 10.0; rv:68.0) Gecko/20100101 Firefox/68.0"], "Accept": ["text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8"], "Accept-Language": ["en-US,en;q=0.5"], "Accept-Encoding": ["gzip, deflate, br"], "Dnt": ["1"], "Cookie": ["organizrLanguage=en"], "Upgrade-Insecure-Requests": ["1"], "Te": ["trailers"]}, "tls": {"resumed": false, "version": 772, "ciphersuite": 4865, "proto": "h2", "proto_mutual": true, "server_name": "home.fakeaddress.com"}}, "duration": 10.000227638, "status": 502, "err_id": "4kz53qn9k", "err_trace": "reverseproxy.(*Handler).ServeHTTP (reverseproxy.go:380)"}
2020/06/11 13:55:17.852	ERROR	http.log.error	remote error: tls: internal error	{"request": {"method": "GET", "uri": "/index.php/204", "proto": "HTTP/1.1", "remote_addr": "192.168.1.1:38617", "host": "mine.fakeaddress.com", "headers": {"User-Agent": ["Mozilla/5.0 (Android) Nextcloud-android/3.11.1"]}, "tls": {"resumed": false, "version": 772, "ciphersuite": 4865, "proto": "", "proto_mutual": true, "server_name": "mine.fakeaddress.com"}}, "duration": 0.000908067, "status": 502, "err_id": "fd8rqggd1", "err_trace": "reverseproxy.(*Handler).ServeHTTP (reverseproxy.go:380)"}
2020/06/11 13:55:25.513	ERROR	http.log.error	dial tcp 192.168.1.228:8899: i/o timeout	{"request": {"method": "GET", "uri": "/", "proto": "HTTP/2.0", "remote_addr": "37.120.198.77:53646", "host": "home.fakeaddress.com", "headers": {"Cache-Control": ["max-age=0"], "Accept-Encoding": ["gzip, deflate, br"], "Dnt": ["1"], "Upgrade-Insecure-Requests": ["1"], "Cookie": ["organizrLanguage=en"], "Te": ["trailers"], "User-Agent": ["Mozilla/5.0 (Windows NT 10.0; rv:68.0) Gecko/20100101 Firefox/68.0"], "Accept": ["text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8"], "Accept-Language": ["en-US,en;q=0.5"]}, "tls": {"resumed": false, "version": 772, "ciphersuite": 4865, "proto": "h2", "proto_mutual": true, "server_name": "home.fakeaddress.com"}}, "duration": 10.000229595, "status": 502, "err_id": "pzk3aby34", "err_trace": "reverseproxy.(*Handler).ServeHTTP (reverseproxy.go:380)"}
2020/06/11 13:57:00.853	ERROR	http.log.error	dial tcp 192.168.1.228:8899: i/o timeout	{"request": {"method": "GET", "uri": "/", "proto": "HTTP/2.0", "remote_addr": "37.120.198.77:53646", "host": "home.fakeaddress.com", "headers": {"Accept-Language": ["en-US,en;q=0.5"], "Accept-Encoding": ["gzip, deflate, br"], "Dnt": ["1"], "Cookie": ["organizrLanguage=en"], "Upgrade-Insecure-Requests": ["1"], "Te": ["trailers"], "User-Agent": ["Mozilla/5.0 (Windows NT 10.0; rv:68.0) Gecko/20100101 Firefox/68.0"], "Accept": ["text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8"]}, "tls": {"resumed": false, "version": 772, "ciphersuite": 4865, "proto": "h2", "proto_mutual": true, "server_name": "home.fakeaddress.com"}}, "duration": 10.000239089, "status": 502, "err_id": "q76qavs3r", "err_trace": "reverseproxy.(*Handler).ServeHTTP (reverseproxy.go:380)"}

5. What I already tried:

Tried rebooting the systems that didnt work, like wallabag container etc.
I use Caddy V1 in a FreeNAS jail currently, tried just using that config file with Caddy V1 in Debian, didn’t work, either sames issues.
So i have a feeling my config is fine for the most part. But I cant figure out why I can’t actually get to the website.

6. Links to relevant resources:

Whitestrake · June 12, 2020, 5:38am

What do the other addresses do, exactly? Can you be specific as to what you’re seeing when you try one of the sites that doesn’t work?

2020/06/11 13:49:01.766 ERROR http.log.error dial tcp 192.168.1.47:8080: i/o timeout

We’re seeing these in your logs for 192.168.1.47:8080, 192.168.1.47:80, and 192.168.1.228:8899. Either those addresses are not in use, or whatever is there is dropping packets coming from Caddy (hence timeout).

I also see no certificate available for 'qbitt.fakeaddress.com', which looks good as you don’t have that address in your Caddyfile.

We also see some like http.log.error remote error: tls: internal error - not sure about this one. They all occur for mine.fakeaddress.com, which is proxied to 192.168.1.195:443 - might be hitting this because the certificate being presented is untrusted? Except I’d expect a different error code than this (usually it’s more specific than that).

itsshash · June 12, 2020, 11:59am

So these are up, and completely locally accessible and completely accessible with the Caddy V1. The other address range from things like Radarr, Sonarr, Organizr, FreshRSS. How would I fix this timeout error?

So this is a slightly more complex one. On the freenas jail, it’s runs Nextcloud instance which uses CaddyV1 to host. My hope was that I could set up a reverse proxy to that.
So
mine.fakeaddress.com --> CaddyV2 (without HTTPS)* --> CaddyV1 ---> Nextcloud.
*the docs for CaddyV2 do not have instructions to explicitly disable HTTPS, I know how to do it on CaddyV1 (tls off)

Whitestrake · June 13, 2020, 7:50am

Could you run curl -kIL -H "Host:rss.fakeadddress.com" 192.168.1.47:8080 from the command line of the server running Caddy v2 and post the output?

So you want Caddy to accept only HTTP (not HTTPS) requests for this domain?

Have you tried specifying the scheme in the site label? (e.g. http://example.com instead of example.com.)

itsshash · June 13, 2020, 1:28pm

❯ curl -kIL -H "Host:rss.fakeaddress.com" 192.168.1.47:8080
HTTP/1.1 200 OK
Date: Sat, 13 Jun 2020 13:24:31 GMT
Server: Apache/2.4.41 (Ubuntu)
Last-Modified: Fri, 12 Jun 2020 23:09:10 GMT
Accept-Ranges: bytes
Content-Length: 774
Cache-Control: max-age=2592000, public
Expires: Mon, 13 Jul 2020 13:24:31 GMT
Vary: Accept-Encoding
Content-Type: text/html; charset=utf-8

❯ curl -kIL rss.fakeaddress.com
HTTP/1.1 308 Permanent Redirect
Connection: close
Location: https://rss.fakeaddress.com/
Server: Caddy
Date: Sat, 13 Jun 2020 13:29:35 GMT

HTTP/2 502 
server: Caddy
date: Sat, 13 Jun 2020 13:29:45 GMT

So the request will be HTTPS, but I essentially, want it forwarded from Caddy v2 to Caddy v1 to manage (So CaddyV1 deals with the certs)

itsshash · June 13, 2020, 1:43pm

I have maybe more useful info.

So when I use curl -kIL on the domains on CaddyV1 compared to CaddyV2

rss.fakeaddress.com:

CaddyV1

❯ curl -kIL rss.fakeaddress.com
HTTP/1.1 301 Moved Permanently
Connection: close
Content-Type: text/html; charset=utf-8
Location: https://rss.fakeaddress.com/
Server: Caddy
Date: Sat, 13 Jun 2020 13:32:58 GMT

HTTP/2 200 
accept-ranges: bytes
cache-control: max-age=2592000, public
content-type: text/html; charset=utf-8
date: Sat, 13 Jun 2020 13:32:58 GMT
expires: Mon, 13 Jul 2020 13:32:58 GMT
last-modified: Fri, 12 Jun 2020 23:09:10 GMT
server: Caddy
server: Apache/2.4.41 (Ubuntu)
vary: Accept-Encoding
content-length: 774

CaddyV2

❯ curl -kIL rss.fakeaddress.com
HTTP/1.1 308 Permanent Redirect
Connection: close
Location: https://rss.fakeaddress.com/
Server: Caddy
Date: Sat, 13 Jun 2020 13:32:07 GMT

HTTP/2 502 
server: Caddy
date: Sat, 13 Jun 2020 13:32:17 GMT

nas.fakeaddress.com:

CaddyV1

❯ curl -kIL nas.fakeaddress.com
HTTP/1.1 301 Moved Permanently
Connection: close
Content-Type: text/html; charset=utf-8
Location: https://nas.fakeaddress.com/
Server: Caddy
Date: Sat, 13 Jun 2020 13:33:11 GMT

HTTP/2 302 
content-type: text/html
date: Sat, 13 Jun 2020 13:33:11 GMT
location: http://nas.fakeadress.com/ui/
server: Caddy
server: nginx
strict-transport-security: max-age=0
x-content-type-options: nosniff
x-xss-protection: 1
content-length: 138

HTTP/1.1 301 Moved Permanently
Connection: close
Content-Type: text/html; charset=utf-8
Location: https://nas.fakeaddress.com/ui/
Server: Caddy
Date: Sat, 13 Jun 2020 13:33:11 GMT

HTTP/2 200 
accept-ranges: bytes
cache-control: must-revalidate
content-type: text/html
date: Sat, 13 Jun 2020 13:33:11 GMT
etag: FreeNAS-11.3-U3.2
last-modified: Tue, 09 Jun 2020 14:18:52 GMT
server: Caddy
server: nginx
content-length: 3469

CaddyV2

❯ curl -kIL nas.fakeaddress.com
HTTP/1.1 308 Permanent Redirect
Connection: close
Location: https://nas.fakeaddress.com/
Server: Caddy
Date: Sat, 13 Jun 2020 13:32:23 GMT

HTTP/2 302 
content-type: text/html
date: Sat, 13 Jun 2020 13:32:23 GMT
location: http://nas.fakeadresss.com/ui/
server: Caddy
server: nginx
strict-transport-security: max-age=0
x-content-type-options: nosniff
x-xss-protection: 1
content-length: 138

HTTP/1.1 308 Permanent Redirect
Connection: close
Location: https://nas.fakeaddress.com/ui/
Server: Caddy
Date: Sat, 13 Jun 2020 13:32:23 GMT

HTTP/2 200 
accept-ranges: bytes
cache-control: must-revalidate
content-type: text/html
date: Sat, 13 Jun 2020 13:32:23 GMT
etag: FreeNAS-11.3-U3.2
last-modified: Tue, 09 Jun 2020 14:18:52 GMT
server: Caddy
server: nginx
content-length: 3469

Whitestrake · June 13, 2020, 2:26pm

Hmm! Looks like the upstream is accepting requests just fine.

Can we get debug information from your reverse proxy?

Add this to the top of your Caddyfile:

{
  debug
}

Then try to connect to rss.fakeaddress.com again (via Caddy v2). Then take the output from Caddy and post it here. It should contain round trip information and exactly what Caddy’s seeing from the upstream server when it tries to connect.

Having a single server handle both SSL termination and SSL passthrough is actually a bit complicated. HAProxy can do this (determine whether to terminate SSL or pass it through dynamically) as a TCP server.

Caddy v2’s HTTP server cannot handle SSL passthrough though, because it is a HTTP server (layer 7), not a TCP server (layer 4). Notably, the operation of a layer 4 reverse proxy is quite limited, as many details of the HTTP request (a layer 7 concept) are not available at all, and never will be unless the connection is accepted and the request is decrypted. Basically SNI is the only thing you can pivot on dynamically.

So, Caddy v2’s layer 7 HTTP server must have its own certs if you want it to handle an incoming HTTPS request, and it must terminate TLS.

Matt Holt is working on a project called Conncept, which is a Caddy v2 app that will be able to dynamically passthrough or terminate TLS by buffering the first few bytes of a connection. Check it out here: GitHub - mholt/conncept: Project Conncept: A layer 4 app for Caddy that multiplexes raw TCP/UDP streams

itsshash · June 19, 2020, 12:48pm

After debug added,

root@pve:~# caddy run -config CaddyfileV2 
2020/06/19 12:43:25.402	INFO	using provided configuration	{"config_file": "CaddyfileV2", "config_adapter": ""}
2020/06/19 12:43:25.405	INFO	admin	admin endpoint started	{"address": "tcp/localhost:2019", "enforce_origin": false, "origins": ["localhost:2019", "[::1]:2019", "127.0.0.1:2019"]}
2020/06/19 12:43:25.405	INFO	http	server is listening only on the HTTPS port but has no TLS connection policies; adding one to enable TLS	{"server_name": "srv0", "https_port": 443}
2020/06/19 12:43:25.405	INFO	http	enabling automatic HTTP->HTTPS redirects	{"server_name": "srv0"}
2020/06/19 13:43:25 [INFO][cache:0xc0006a4500] Started certificate maintenance routine
2020/06/19 12:43:25.407	INFO	tls	cleaned up storage units
2020/06/19 12:43:25.408	DEBUG	http	starting server loop	{"address": "[::]:80", "http3": false, "tls": false}
2020/06/19 12:43:25.408	DEBUG	http	starting server loop	{"address": "[::]:443", "http3": false, "tls": true}
2020/06/19 12:43:25.408	INFO	http	enabling automatic TLS certificate management	{"domains": ["rss.fakeaddress.com", "nas.fakeaddress.com", "sonarr.fakeaddress.com", "radarr.fakeaddress.com", "walla.fakeaddress.com", "mine.fakeaddress.com", "komga.fakeaddress.com", "home.fakeaddress.com", "search.fakeaddress.com"]}
2020/06/19 12:43:25.418	INFO	autosaved config	{"file": "/root/.config/caddy/autosave.json"}
2020/06/19 12:43:25.418	INFO	serving initial configuration
2020/06/19 12:43:39.907	ERROR	http.log.error	dial tcp 192.168.1.47:8080: i/o timeout	{"request": {"method": "GET", "uri": "/", "proto": "HTTP/2.0", "remote_addr": "192.168.1.1:56856", "host": "rss.fakeaddress.com", "headers": {"Cache-Control": ["max-age=0"], "Te": ["trailers"], "User-Agent": ["Mozilla/5.0 (Windows NT 10.0; rv:68.0) Gecko/20100101 Firefox/68.0"], "Accept": ["text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8"], "Accept-Language": ["en-US,en;q=0.5"], "Accept-Encoding": ["gzip, deflate, br"], "Dnt": ["1"], "Upgrade-Insecure-Requests": ["1"]}, "tls": {"resumed": false, "version": 772, "ciphersuite": 4865, "proto": "h2", "proto_mutual": true, "server_name": "rss.fakeaddress.com"}}, "duration": 10.000307506, "status": 502, "err_id": "za8ec3xaf", "err_trace": "reverseproxy.(*Handler).ServeHTTP (reverseproxy.go:380)"}
2020/06/19 12:43:49.949	ERROR	http.log.error	dial tcp 192.168.1.47:8080: i/o timeout	{"request": {"method": "GET", "uri": "/favicon.ico", "proto": "HTTP/2.0", "remote_addr": "192.168.1.1:56856", "host": "rss.fakeaddress.com", "headers": {"User-Agent": ["Mozilla/5.0 (Windows NT 10.0; rv:68.0) Gecko/20100101 Firefox/68.0"], "Accept": ["image/webp,*/*"], "Accept-Language": ["en-US,en;q=0.5"], "Accept-Encoding": ["gzip, deflate, br"], "Dnt": ["1"], "Te": ["trailers"]}, "tls": {"resumed": false, "version": 772, "ciphersuite": 4865, "proto": "h2", "proto_mutual": true, "server_name": "rss.fakeaddress.com"}}, "duration": 10.000238997, "status": 502, "err_id": "39c0zkipg", "err_trace": "reverseproxy.(*Handler).ServeHTTP (reverseproxy.go:380)"}
2020/06/19 12:43:54.190	DEBUG	http.handlers.reverse_proxy	upstream roundtrip	{"upstream": "192.168.1.152:80", "request": {"method": "GET", "uri": "/", "proto": "HTTP/2.0", "remote_addr": "192.168.1.1:56872", "host": "nas.fakeaddress.com", "headers": {"Accept": ["text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8"], "Accept-Language": ["en-US,en;q=0.5"], "Dnt": ["1"], "Te": ["trailers"], "X-Forwarded-For": ["192.168.1.1"], "User-Agent": ["Mozilla/5.0 (Windows NT 10.0; rv:68.0) Gecko/20100101 Firefox/68.0"], "Accept-Encoding": ["gzip, deflate, br"], "Upgrade-Insecure-Requests": ["1"], "X-Forwarded-Proto": ["https"]}, "tls": {"resumed": false, "version": 772, "ciphersuite": 4865, "proto": "h2", "proto_mutual": true, "server_name": "nas.fakeaddress.com"}}, "headers": {"Connection": ["keep-alive"], "Location": ["http://nas.fakeaddress.com/ui/"], "X-Content-Type-Options": ["nosniff"], "Server": ["nginx"], "Date": ["Fri, 19 Jun 2020 12:43:54 GMT"], "Content-Type": ["text/html"], "Content-Length": ["138"], "Strict-Transport-Security": ["max-age=0"], "X-Xss-Protection": ["1"]}, "duration": 0.000602549, "status": 302}

Whitestrake · June 20, 2020, 7:01am

We’re not even getting a debug roundtrip message because it never even gets to that point. It just dials the upstream and gets a timeout, no response, for the rss subdomain.

I’m trying to wrap my head around this, because it doesn’t make much sense to me.

Are Caddy v1 and Caddy v2 being run on the same server, in the same environment?

francislavoie · June 20, 2020, 2:39pm

Could you please try again with Caddy v2.1 beta 1? There were some relevant fixes for proxy debugging since v2.0

itsshash · June 20, 2020, 4:03pm

With Caddy V2.1 beta 1

root@pve:~# caddy run -config CaddyfileV2 
2020/06/20 16:04:40.464	INFO	using provided configuration	{"config_file": "CaddyfileV2", "config_adapter": ""}
2020/06/20 16:04:40.465	INFO	admin	admin endpoint started	{"address": "tcp/localhost:2019", "enforce_origin": false, "origins": ["localhost:2019", "[::1]:2019", "127.0.0.1:2019"]}
2020/06/20 16:04:40.466	INFO	http	server is listening only on the HTTPS port but has no TLS connection policies; adding one to enable TLS	{"server_name": "srv0", "https_port": 443}
2020/06/20 16:04:40.466	INFO	http	enabling automatic HTTP->HTTPS redirects	{"server_name": "srv0"}
2020/06/20 17:04:40 [INFO][cache:0xc000a18d20] Started certificate maintenance routine
2020/06/20 16:04:40.469	INFO	tls	cleaned up storage units
2020/06/20 16:04:40.469	DEBUG	http	starting server loop	{"address": "[::]:80", "http3": false, "tls": false}
2020/06/20 16:04:40.469	DEBUG	http	starting server loop	{"address": "[::]:443", "http3": false, "tls": true}
2020/06/20 16:04:40.469	INFO	http	enabling automatic TLS certificate management	{"domains": ["radarr.fakeaddress.com", "walla.fakeaddress.com", "search.fakeaddress.com", "komga.fakeaddress.com", "rss.fakeaddress.com", "nas.fakeaddress.com", "sonarr.fakeaddress.com", "mine.fakeaddress.com", "home.fakeaddress.com"]}
2020/06/20 16:04:40.479	INFO	autosaved config	{"file": "/root/.config/caddy/autosave.json"}
2020/06/20 16:04:40.479	INFO	serving initial configuration
2020/06/20 16:04:55.970	DEBUG	http.handlers.reverse_proxy	upstream roundtrip	{"upstream": "192.168.1.47:8080", "request": {"method": "GET", "uri": "/", "proto": "HTTP/2.0", "remote_addr": "192.168.1.1:52094", "host": "rss.fakeaddress.com", "headers": {"User-Agent": ["Mozilla/5.0 (Windows NT 10.0; rv:68.0) Gecko/20100101 Firefox/68.0"], "Accept-Encoding": ["gzip, deflate, br"], "Upgrade-Insecure-Requests": ["1"], "Cache-Control": ["max-age=0"], "X-Forwarded-For": ["192.168.1.1"], "Accept": ["text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8"], "Accept-Language": ["en-US,en;q=0.5"], "Dnt": ["1"], "Te": ["trailers"], "X-Forwarded-Proto": ["https"]}, "tls": {"resumed": false, "version": 772, "ciphersuite": 4865, "proto": "h2", "proto_mutual": true, "server_name": "rss.fakeaddress.com"}}, "duration": 10.000163485, "error": "dial tcp 192.168.1.47:8080: i/o timeout"}
2020/06/20 16:04:55.970	ERROR	http.log.error	dial tcp 192.168.1.47:8080: i/o timeout	{"request": {"method": "GET", "uri": "/", "proto": "HTTP/2.0", "remote_addr": "192.168.1.1:52094", "host": "rss.fakeaddress.com", "headers": {"Cache-Control": ["max-age=0"], "Te": ["trailers"], "User-Agent": ["Mozilla/5.0 (Windows NT 10.0; rv:68.0) Gecko/20100101 Firefox/68.0"], "Accept": ["text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8"], "Accept-Language": ["en-US,en;q=0.5"], "Accept-Encoding": ["gzip, deflate, br"], "Dnt": ["1"], "Upgrade-Insecure-Requests": ["1"]}, "tls": {"resumed": false, "version": 772, "ciphersuite": 4865, "proto": "h2", "proto_mutual": true, "server_name": "rss.fakeaddress.com"}}, "duration": 10.000310624, "status": 502, "err_id": "4n8vgrr37", "err_trace": "reverseproxy.(*Handler).ServeHTTP (reverseproxy.go:411)"}

itsshash · June 20, 2020, 6:06pm

Yep same external IP, CaddyV1 runs in a Freenas Jail. CaddyV2 runs on baremetal Debian (Proxmox)

Whitestrake · June 20, 2020, 6:10pm

That definitely sounds like different severs to me, not the same server (internally). The external IP doesn’t really matter for this aspect of troubleshooting, we’re trying to figure out what’s going on within the LAN.

I’d be inclined to assume this is a networking issue if the v1 host (the FreeNAS jail) can communicate with the upstream server and the v2 host (the Proxmox server) cannot.

system · July 11, 2020, 2:20pm

This topic was automatically closed after 30 days. New replies are no longer allowed.