Local network reverse proxy does not work

1. The problem I’m having:

Hi, I have caddy set up to use my public facing domain. I have that pointing to various docker containers. That works fine. I want to now set it up so if I want to go to jellyfin, for example, I don’t have to type in 192.168.1.150:8096, I can just type watch.home. However, when I set up the reverse proxy how I thought to do it, all requests over http redirect me to just the IP, without a port (takes me to my NAS homepage). I posted my caddyfile, but have tried a ton of different variations like using :443, specifying no port, tls internal, etc.

I am using Pihole as a local DNS with this setup:

2. Error messages and/or full log output:

text  error  warn  system  array  login  

{"level":"info","ts":1702872036.7006671,"logger":"http","msg":"done waiting on internal rate limiter","identifiers":["request.home"],"ca":"https://acme-v02.api.letsencrypt.org/directory","account":""}
{"level":"info","ts":1702872036.7022467,"msg":"autosaved config (load with --resume flag)","file":"/config/caddy/autosave.json"}
{"level":"info","ts":1702872036.7022526,"msg":"serving initial configuration"}
{"level":"error","ts":1702872036.9418812,"logger":"tls.obtain","msg":"could not get certificate from issuer","identifier":"request.home","issuer":"acme-v02.api.letsencrypt.org-directory","error":"HTTP 400 urn:ietf:params:acme:error:rejectedIdentifier - Error creating new order :: Cannot issue for \"request.home\": Domain name does not end with a valid public suffix (TLD)"}
{"level":"info","ts":1702872036.9432356,"logger":"http","msg":"waiting on internal rate limiter","identifiers":["request.home"],"ca":"https://acme.zerossl.com/v2/DV90","account":"caddy@zerossl.com"}
{"level":"info","ts":1702872036.9432423,"logger":"http","msg":"done waiting on internal rate limiter","identifiers":["request.home"],"ca":"https://acme.zerossl.com/v2/DV90","account":"caddy@zerossl.com"}
{"level":"error","ts":1702872037.0875914,"logger":"tls.obtain","msg":"could not get certificate from issuer","identifier":"request.home","issuer":"acme.zerossl.com-v2-DV90","error":"[request.home] creating new order: provisioning client: HTTP 503: <html>\r\n<head><title>503 Service Temporarily Unavailable</title></head>\r\n<body>\r\n<center><h1>503 Service Temporarily Unavailable</h1></center>\r\n<hr><center>nginx</center>\r\n</body>\r\n</html>\r\n (ca=https://acme.zerossl.com/v2/DV90)"}
{"level":"error","ts":1702872037.0876317,"logger":"tls.obtain","msg":"will retry","error":"[request.home] Obtain: [request.home] creating new order: provisioning client: HTTP 503: <html>\r\n<head><title>503 Service Temporarily Unavailable</title></head>\r\n<body>\r\n<center><h1>503 Service Temporarily Unavailable</h1></center>\r\n<hr><center>nginx</center>\r\n</body>\r\n</html>\r\n (ca=https://acme.zerossl.com/v2/DV90)","attempt":1,"retrying_in":60,"elapsed":0.389009814,"max_duration":2592000}
{"level":"info","ts":1702872097.0892332,"logger":"tls.obtain","msg":"obtaining certificate","identifier":"request.home"}
{"level":"error","ts":1702872097.353169,"logger":"tls.obtain","msg":"could not get certificate from issuer","identifier":"request.home","issuer":"acme-v02.api.letsencrypt.org-directory","error":"HTTP 400 urn:ietf:params:acme:error:rejectedIdentifier - Error creating new order :: Cannot issue for \"request.home\": Domain name does not end with a valid public suffix (TLD)"}
{"level":"error","ts":1702872097.386799,"logger":"tls.obtain","msg":"could not get certificate from issuer","identifier":"request.home","issuer":"acme.zerossl.com-v2-DV90","error":"[request.home] creating new order: provisioning client: HTTP 503: <html>\r\n<head><title>503 Service Temporarily Unavailable</title></head>\r\n<body>\r\n<center><h1>503 Service Temporarily Unavailable</h1></center>\r\n<hr><center>nginx</center>\r\n</body>\r\n</html>\r\n (ca=https://acme.zerossl.com/v2/DV90)"}
{"level":"error","ts":1702872097.3868227,"logger":"tls.obtain","msg":"will retry","error":"[request.home] Obtain: [request.home] creating new order: provisioning client: HTTP 503: <html>\r\n<head><title>503 Service Temporarily Unavailable</title></head>\r\n<body>\r\n<center><h1>503 Service Temporarily Unavailable</h1></center>\r\n<hr><center>nginx</center>\r\n</body>\r\n</html>\r\n (ca=https://acme.zerossl.com/v2/DV90)","attempt":2,"retrying_in":120,"elapsed":60.688200714,"max_duration":2592000}
{"level":"info","ts":1702872217.3878748,"logger":"tls.obtain","msg":"obtaining certificate","identifier":"request.home"}
{"level":"error","ts":1702872217.4938257,"logger":"tls.obtain","msg":"could not get certificate from issuer","identifier":"request.home","issuer":"acme-v02.api.letsencrypt.org-directory","error":"HTTP 400 urn:ietf:params:acme:error:rejectedIdentifier - Error creating new order :: Cannot issue for \"request.home\": Domain name does not end with a valid public suffix (TLD)"}
{"level":"warn","ts":1702872247.6399386,"logger":"http.acme_client","msg":"HTTP request failed; retrying","url":"https://acme.zerossl.com/v2/DV90/newNonce","error":"performing request: Head \"https://acme.zerossl.com/v2/DV90/newNonce\": http2: timeout awaiting response headers (Client.Timeout exceeded while awaiting headers)"}
{"level":"warn","ts":1702872277.8910935,"logger":"http.acme_client","msg":"HTTP request failed; retrying","url":"https://acme.zerossl.com/v2/DV90/newNonce","error":"performing request: Head \"https://acme.zerossl.com/v2/DV90/newNonce\": http2: timeout awaiting response headers (Client.Timeout exceeded while awaiting headers)"}
{"level":"warn","ts":1702872308.142219,"logger":"http.acme_client","msg":"HTTP request failed; retrying","url":"https://acme.zerossl.com/v2/DV90/newNonce","error":"performing request: Head \"https://acme.zerossl.com/v2/DV90/newNonce\": http2: timeout awaiting response headers (Client.Timeout exceeded while awaiting headers)"}
{"level":"error","ts":1702872308.1422727,"logger":"tls.obtain","msg":"could not get certificate from issuer","identifier":"request.home","issuer":"acme.zerossl.com-v2-DV90","error":"[request.home] creating new order: fetching new nonce from server: performing request: Head \"https://acme.zerossl.com/v2/DV90/newNonce\": http2: timeout awaiting response headers (Client.Timeout exceeded while awaiting headers) (ca=https://acme.zerossl.com/v2/DV90)"}
{"level":"error","ts":1702872308.1422887,"logger":"tls.obtain","msg":"will retry","error":"[request.home] Obtain: [request.home] creating new order: fetching new nonce from server: performing request: Head \"https://acme.zerossl.com/v2/DV90/newNonce\": http2: timeout awaiting response headers (Client.Timeout exceeded while awaiting headers) (ca=https://acme.zerossl.com/v2/DV90)","attempt":3,"retrying_in":120,"elapsed":271.443666411,"max_duration":2592000}
s6-rc: info: service legacy-services: stopping
s6-rc: info: service legacy-services successfully stopped
s6-rc: info: service service-caddy: stopping
{"level":"info","ts":1702872382.440447,"msg":"shutting down apps, then terminating","signal":"SIGTERM"}
{"level":"warn","ts":1702872382.440462,"msg":"exiting; byeee!! 👋","signal":"SIGTERM"}
{"level":"info","ts":1702872382.4404821,"logger":"http","msg":"servers shutting down with eternal grace period"}
{"level":"info","ts":1702872382.4406106,"logger":"tls.obtain","msg":"releasing lock","identifier":"request.home"}
{"level":"info","ts":1702872382.4408197,"logger":"admin","msg":"stopped previous server","address":"localhost:2019"}
{"level":"info","ts":1702872382.4408293,"msg":"shutdown complete","signal":"SIGTERM","exit_code":0}
s6-rc: info: service service-caddy successfully stopped
s6-rc: info: service init-setup: stopping
s6-rc: info: service init-setup successfully stopped
s6-rc: info: service init-secrets: stopping
s6-rc: info: service init-secrets successfully stopped
s6-rc: info: service init-perms: stopping
s6-rc: info: service init-perms successfully stopped
s6-rc: info: service legacy-cont-init: stopping
s6-rc: info: service legacy-cont-init successfully stopped
s6-rc: info: service fix-attrs: stopping
s6-rc: info: service fix-attrs successfully stopped
s6-rc: info: service s6rc-oneshot-runner: stopping
s6-rc: info: service s6rc-oneshot-runner successfully stopped
s6-rc: info: service s6rc-oneshot-runner: starting
s6-rc: info: service s6rc-oneshot-runner successfully started
s6-rc: info: service fix-attrs: starting
s6-rc: info: service fix-attrs successfully started
s6-rc: info: service legacy-cont-init: starting
s6-rc: info: service legacy-cont-init successfully started
s6-rc: info: service init-perms: starting
s6-rc: info: service init-perms successfully started
s6-rc: info: service init-secrets: starting
s6-rc: info: service init-secrets successfully started
s6-rc: info: service init-setup: starting
s6-rc: info: service init-setup successfully started
s6-rc: info: service service-caddy: starting
s6-rc: info: service service-caddy successfully started
s6-rc: info: service legacy-services: starting
s6-rc: info: service legacy-services successfully started
{"level":"info","ts":1702872386.9368484,"msg":"using provided configuration","config_file":"/config/Caddyfile","config_adapter":""}
{"level":"info","ts":1702872386.9389095,"logger":"admin","msg":"admin endpoint started","address":"localhost:2019","enforce_origin":false,"origins":["//localhost:2019","//[::1]:2019","//127.0.0.1:2019"]}
{"level":"info","ts":1702872386.9390419,"logger":"tls.cache.maintenance","msg":"started background certificate maintenance","cache":"0xc000613980"}
{"level":"info","ts":1702872386.9398887,"logger":"http.auto_https","msg":"server is listening only on the HTTPS port but has no TLS connection policies; adding one to enable TLS","server_name":"srv0","https_port":443}
{"level":"info","ts":1702872386.9399025,"logger":"http.auto_https","msg":"enabling automatic HTTP->HTTPS redirects","server_name":"srv0"}
{"level":"warn","ts":1702872386.9399254,"logger":"http.auto_https","msg":"server is listening only on the HTTP port, so no automatic HTTPS will be applied to this server","server_name":"srv1","http_port":80}
{"level":"warn","ts":1702872386.9462602,"logger":"pki.ca.local","msg":"installing root certificate (you might be prompted for password)","path":"storage:pki/authorities/local/root.crt"}
{"level":"info","ts":1702872386.9463372,"msg":"warning: \"certutil\" is not available, install \"certutil\" with \"apt install libnss3-tools\" or \"yum install nss-tools\" and try again"}
{"level":"info","ts":1702872386.9463413,"msg":"define JAVA_HOME environment variable to use the Java trust"}
{"level":"error","ts":1702872386.9469514,"logger":"pki.ca.local","msg":"failed to install root certificate","error":"failed to execute tee: exit status 1","certificate_file":"storage:pki/authorities/local/root.crt"}
{"level":"info","ts":1702872386.9470177,"logger":"http","msg":"enabling HTTP/3 listener","addr":":443"}
{"level":"info","ts":1702872386.9473712,"logger":"http.log","msg":"server running","name":"srv0","protocols":["h1","h2","h3"]}
{"level":"info","ts":1702872386.9474971,"logger":"http.log","msg":"server running","name":"srv1","protocols":["h1","h2","h3"]}
{"level":"info","ts":1702872386.9475155,"logger":"http","msg":"enabling automatic TLS certificate management","domains":["request.home","library-server.exampledomain.com","jackett.exampledomain..com","request.exampledomain..com","movies.exampledomain..com","watch.exampledomain..com","www.exampledomain..com","library.exampledomain..com","tv.exampledomain..com","download.exampledomain..com","books.exampledomain..com"]}
{"level":"warn","ts":1702872386.9484022,"logger":"tls","msg":"stapling OCSP","error":"no OCSP stapling for [request.home]: no OCSP server specified in certificate","identifiers":["request.home"]}
{"level":"warn","ts":1702872386.9492655,"logger":"tls","msg":"storage cleaning happened too recently; skipping for now","storage":"FileStorage:/config/caddy","instance":"fc9797af-3498-46ff-883d-cae1927c7c7e","try_again":1702958786.9492645,"try_again_in":86399.999999814}
{"level":"info","ts":1702872386.94939,"logger":"tls","msg":"finished cleaning storage units"}
{"level":"info","ts":1702872386.9549901,"msg":"autosaved config (load with --resume flag)","file":"/config/caddy/autosave.json"}
{"level":"info","ts":1702872386.9549968,"msg":"serving initial configuration"}

----------------------------------------------------------------------
ENVIRONMENT
----------------------------------------------------------------------
PUID=99
PGID=100
UMASK=002
TZ=America/Los_Angeles
CUSTOM_BUILD=
----------------------------------------------------------------------

Executing usermod...
usermod: no changes
Applying permissions to /config

3. Caddy version:

v2.7.6 h1:w0NymbG2m9PcvKWsrXO6EEkY9Ru4FJK8uQbYcev1p3A=

4. How I installed and ran Caddy:

Running via a docker container on a NAS.

a. System environment:

Unraid and docker.

b. Command:

Should run automatically with the container, but if not I run this:

caddy run -c /config/Caddyfile

c. Service/unit/compose file:

Using Unraid so no direct ocmpose file.

PASTE OVER THIS, BETWEEN THE ``` LINES.
Please use the preview pane to ensure it looks nice.

d. My complete Caddy config:

www.exampledomain {
	reverse_proxy 192.168.1.50:3000
	basicauth {
		myname hashedpw
	}
}
watch.exampledomain {
	reverse_proxy 192.168.1.50:8096
	#basicauth {
	#myname hashedpw
	#}
}

download.exampledomain{
	reverse_proxy 192.168.1.50:8091
	basicauth {
		myname hashedpw
	}
}

jackett.exampledomain {
	reverse_proxy 192.168.1.50:9117
	basicauth {
		myname hashedpw
	}
}

tv.exampledomain {
	reverse_proxy 192.168.1.50:8989
	basicauth {
		myname hashedpw
		friend3 hashedpw
	}
}

movies.exampledomain {
	reverse_proxy 192.168.1.50:7878
	basicauth {
		myname hashedpw
		friend3 hashedpw
	}
}

books.exampledomain {
	reverse_proxy 192.168.1.50:8787
	basicauth {
		myname hashedpw
		friend3 hashedpw
	}
}

library.exampledomain {
	reverse_proxy 192.168.1.50:8083 {
		header_up X-Scheme https
	}
}

library-server.exampledomain {
	reverse_proxy 192.168.1.50:8084
	basicauth {
		myname hashedpw
		#friend1 hashedpw
		#friend2 hashedpw
	}
}

request.exampledomain {
	reverse_proxy 192.168.1.50:5055
	#basicauth {
	#myname hashedpw
	#}
}

home:80 {
	reverse_proxy 192.168.1.50:3000
}

watch.home:80 {
	reverse_proxy 192.168.1.50:8096
}

download.home:80 {
	reverse_proxy 192.168.1.50:8091
}

jackett.home:80 {
	reverse_proxy 192.168.1.50:9117
}

tv.home:80 {
	reverse_proxy 192.168.1.50:8989
}

movies.home:80 {
	reverse_proxy 192.168.1.50:7878
}

books.home:80 {
	reverse_proxy 192.168.1.50:8787
}

manage.home:80 {
	reverse_proxy 192.168.1.50/Main
}

library.home:80 {
	reverse_proxy 192.168.1.50:8083 {
		header_up X-Scheme https
	}
}

library-server.home:80 {
	reverse_proxy 192.168.1.50:8084
}

request.home:80 {
	reverse_proxy 192.168.1.50:5055
}

Howdy @kissmekiwi, welcome to the Caddy community.

I can see from your logs:

{"level":"error","ts":1702872217.4938257,"logger":"tls.obtain","msg":"could not get certificate from issuer","identifier":"request.home","issuer":"acme-v02.api.letsencrypt.org-directory","error":"HTTP 400 urn:ietf:params:acme:error:rejectedIdentifier - Error creating new order :: Cannot issue for \"request.home\": Domain name does not end with a valid public suffix (TLD)"}

Caddy is attempting to get a domain-validated certificate for the nonexistent .home TLD. Caddy doesn’t really have any logic to keep track of all the TLDs that exist; it just tries to handle almost anything that looks like a domain (other than IP addresses, localhost, .ts.net, and soon to be .home.arpa TLDs) by getting a cert. In this case, you can’t do that for .home.

However, this behaviour does not align with the configuration in the Caddyfile you supplied. You’ve specified port 80 - the HTTP port - and I expect Caddy not to activate Automatic HTTPS at all for this site address:

Any of the following will prevent automatic HTTPS from being activated, either in whole or in part:

  • […]
  • Listening exclusively on the HTTP port
  • […]

https://caddyserver.com/docs/automatic-https#activation

I went and tested very quickly on a local installation as well just to be sure - request.home:80 should not be activating any HTTPS features at all by default.

This implies that it’s possible the configuration you think you’re running is not actually the configuration Caddy is running. You may have to double check to ensure that you have successfully reloaded Caddy with the correct Caddyfile, or that you’ve successfully stopped the server/container and started it again with the correct Caddyfile.

There’s also the question of exactly how you’re getting these redirects, which might also be explained by a misconfiguration, or it might be something else.

I suggest that you start with a new configuration:

{
  debug
}

http://request.home {
  reverse_proxy 192.168.1.50:5055
}

With only the one single site address in it and the debug global option enabled. Then start your Caddy server and observe the logs. Then, run curl -IL http://request.home from another computer in the network and observe the results as well as the debug logs from Caddy. If successful, I expect to see full round-trip reverse proxy logs from Caddy itself telling us how Caddy and the upstream server responded.

Hi, thanks for the help. Got me a little closer but feel like I am just missing a config piece on the container.

My container is set up as the following:

When trying your config, it did not work. What did work was changing it to

{
  debug
}

http://request.home:8080 {
  reverse_proxy 192.168.1.50:5055
}

And when entering http://request.home:8080 it took me to the correct site. I thought to try to get a port mapping from 80:80 on the container would help but that port is already in use somehow, I think due to unraid using port 80. I tries messing around and giving the caddy container port 80, but was not able to get that to go. With my current set up, the request is not making its way to Caddy. I really am a noob at ports and especially in docker. Can I just use any port for http for unraid and caddy, or do they have to be 80?

1 Like

You can move unraid to a different port then. Quick google says it’s in Settings → Management Access, I think.

It’s easier/nicer if you let Caddy take ports 80/443, and you can even have Caddy proxy unraid.home to the port you use for unraid.

1 Like

Okay yeah I tried that and it didn’t work. But did it again by changing unraid off 80, and moving caddy to 80.

Then for each link I have http and :80 added like so:

http://request.home:80 {
        reverse_proxy 192.168.1.50:5055
}

and it works! Thanks to the both of you!

2 Likes

You don’t need both http:// and :80 on your site address, you can just use one of them. I suggest http://.

1 Like

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.