Using Caddy in Ubunutu to reverse proxy to local devices

Help
1

1. The problem I’m having:

I have added 2 devices to the caddy file so that I may use a reverse proxy to access them. The issue is that it works just fine when I try to access through northcisco.localhost; I get the login screen for my device. However, whenever I try to browse firewall.localhost, I only get a blank page.

2. Error messages and/or full log output:

My browser displays a blank page whenever I attempt to access firewall.localhost. I don’t notice any other errors on the terminal except from this.

3. Caddy version:

v2.6.4 h1:2hwYqiRwk1tf3VruhMpLcYTg+11fCdr8S3jhNAdnPy8=

4. How I installed and ran Caddy:

a. System environment:

Ubuntu 22.04.2 LTS

b. Command:

caddy run

d. My complete Caddy config:

northcisco.localhost {
reverse_proxy 10.10.10.10
}

firewall.localhost {
reverse_proxy 10.10.10.1:17322
}

2

Please enable the debug global option and show your logs. Make a request with curl -v and show the output. Please review the help topic template, it asks for this information. We can’t help otherwise.

3

I tried executing the curl -v command and got the following output, and then I ran systemctl -l status caddy and got the following output.

1. Curl -v output:

idfyinfra@LA-255:~$ curl -v firewall.localhost

  • Trying ::1:80…
  • Connected to firewall.localhost (::1) port 80 (#0)

GET / HTTP/1.1
Host: firewall.localhost
User-Agent: curl/7.81.0
Accept: /

  • Mark bundle as not supporting multiuse
    < HTTP/1.1 308 Permanent Redirect
    < Connection: close
    < Location: https://firewall.localhost/
    < Server: Caddy
    < Date: Fri, 14 Jul 2023 05:55:32 GMT
    < Content-Length: 0
    <
  • Closing connection 0

2. systemctl -l status caddy output:

idfyinfra@LA-255:~$ systemctl -l status caddy
● caddy.service - Caddy
Loaded: loaded (/lib/systemd/system/caddy.service; enabled; vendor preset: enabled)
Active: active (running) since Fri 2023-07-14 11:25:41 IST; 14s ago
Docs: Welcome — Caddy Documentation
Main PID: 4549 (caddy)
Tasks: 13 (limit: 18811)
Memory: 11.1M
CPU: 66ms
CGroup: /system.slice/caddy.service
└─4549 /usr/bin/caddy run --environ --config /etc/caddy/Caddyfile

Jul 14 11:25:41 LA-255 caddy[4549]: {“level”:“warn”,“ts”:1689314141.6611602,“logger”:“tls”,“msg”:“stapling OCSP”,“error”:"no OCSP stapling for [firewall.localhost]: no OCSP server specified in certifica>
Jul 14 11:25:41 LA-255 caddy[4549]: {“level”:“debug”,“ts”:1689314141.6611662,“logger”:“tls.cache”,“msg”:“added certificate to cache”,“subjects”:[“firewall.localhost”],“expiration”:1689353095,“managed”:t>
Jul 14 11:25:41 LA-255 caddy[4549]: {“level”:“debug”,“ts”:1689314141.661173,“logger”:“events”,“msg”:“event”,“name”:“cached_managed_cert”,“id”:“5842c42b-808f-4985-bea9-37f004f9971d”,“origin”:“tls”,“data”>
Jul 14 11:25:41 LA-255 caddy[4549]: {“level”:“warn”,“ts”:1689314141.6698303,“logger”:“pki.ca.local”,“msg”:“installing root certificate (you might be prompted for password)”,“path”:"storage:pki/authoriti>
Jul 14 11:25:41 LA-255 caddy[4549]: {“level”:“info”,“ts”:1689314141.6699955,“msg”:“warning: "certutil" is not available, install "certutil" with "apt install libnss3-tools" or "yum install nss-to>
Jul 14 11:25:41 LA-255 caddy[4549]: {“level”:“info”,“ts”:1689314141.6700025,“msg”:“define JAVA_HOME environment variable to use the Java trust”}
Jul 14 11:25:41 LA-255 caddy[4549]: {“level”:“error”,“ts”:1689314141.6711965,“logger”:“pki.ca.local”,“msg”:“failed to install root certificate”,“error”:“failed to execute sudo: exit status 1”,“certifica>
Jul 14 11:25:41 LA-255 caddy[4549]: {“level”:“info”,“ts”:1689314141.671299,“msg”:“autosaved config (load with --resume flag)”,“file”:”/var/lib/caddy/.config/caddy/autosave.json”}
Jul 14 11:25:41 LA-255 caddy[4549]: {“level”:“info”,“ts”:1689314141.671344,“msg”:“serving initial configuration”}
Jul 14 11:25:41 LA-255 systemd[1]: Started Caddy.

3. Caddy Config:

{
debug
}

northcisco.localhost {
reverse_proxy 10.10.10.10
tls internal
}

firewall.localhost {
reverse_proxy 10.10.10.1:17322
tls internal
}

southcisco.localhost {
reverse_proxy 10.10.20.10:443
tls internal
}

4

You made an HTTP request, and Caddy responded with an HTTP->HTTPS redirect (see the Location header). Make a request with https:// to actually make a useful request.

Also, please format your logs and config with code blocks. Click the </> button to insert a code block. Your post is very difficult to read otherwise. It’s a forum rule to correctly format your posts. See FAQ - Caddy Community

Your posts must be properly formatted. Use the formatting buttons if you don’t know Markdown. Improperly formatted posts are hard to read and will discourage helpers to the point where you might not get an answer at all.

This command truncates your logs. Please use the command mentioned here: Keep Caddy Running — Caddy Documentation

5

I hope this is in the correct format and will help you understand the problem. Actually, I am new to linux and caddy, thus it may take some time to learn each thing.

1. Curl -v output:

idfyinfra@LA-255:~$ curl -v https://firewall.localhost
*   Trying ::1:443...
* Connected to firewall.localhost (::1) port 443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
*  CAfile: /etc/ssl/certs/ca-certificates.crt
*  CApath: /etc/ssl/certs
* TLSv1.0 (OUT), TLS header, Certificate Status (22):
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* TLSv1.2 (IN), TLS header, Certificate Status (22):
* TLSv1.3 (IN), TLS handshake, Server hello (2):
* TLSv1.2 (IN), TLS header, Finished (20):
* TLSv1.2 (IN), TLS header, Supplemental data (23):
* TLSv1.3 (IN), TLS handshake, Encrypted Extensions (8):
* TLSv1.2 (IN), TLS header, Supplemental data (23):
* TLSv1.3 (IN), TLS handshake, Certificate (11):
* TLSv1.2 (OUT), TLS header, Unknown (21):
* TLSv1.3 (OUT), TLS alert, unknown CA (560):
* SSL certificate problem: unable to get local issuer certificate
* Closing connection 0
curl: (60) SSL certificate problem: unable to get local issuer certificate
More details here: https://curl.se/docs/sslcerts.html

curl failed to verify the legitimacy of the server and therefore could not
establish a secure connection to it. To learn more about this situation and
how to fix it, please visit the web page mentioned above.

2. systemctl status caddy output:

idfyinfra@LA-255:~$ systemctl status caddy
● caddy.service - Caddy
     Loaded: loaded (/lib/systemd/system/caddy.service; enabled; vendor preset: enabled)
     Active: active (running) since Fri 2023-07-14 11:25:41 IST; 29min ago
       Docs: https://caddyserver.com/docs/
   Main PID: 4549 (caddy)
      Tasks: 13 (limit: 18811)
     Memory: 10.0M
        CPU: 199ms
     CGroup: /system.slice/caddy.service
             └─4549 /usr/bin/caddy run --environ --config /etc/caddy/Caddyfile

Jul 14 11:25:41 LA-255 caddy[4549]: {"level":"info","ts":1689314141.6700025,"msg":"define JAVA_HOME environment variable to use the Java trust"}
Jul 14 11:25:41 LA-255 caddy[4549]: {"level":"error","ts":1689314141.6711965,"logger":"pki.ca.local","msg":"failed to install root certificate","error":"failed to execute sudo: exit status 1","certifica>
Jul 14 11:25:41 LA-255 caddy[4549]: {"level":"info","ts":1689314141.671299,"msg":"autosaved config (load with --resume flag)","file":"/var/lib/caddy/.config/caddy/autosave.json"}
Jul 14 11:25:41 LA-255 caddy[4549]: {"level":"info","ts":1689314141.671344,"msg":"serving initial configuration"}
Jul 14 11:25:41 LA-255 systemd[1]: Started Caddy.
Jul 14 11:54:37 LA-255 caddy[4549]: {"level":"debug","ts":1689315877.8533323,"logger":"events","msg":"event","name":"tls_get_certificate","id":"46dbd400-d461-4954-8c20-aaa12adb4081","origin":"tls","data>
Jul 14 11:54:37 LA-255 caddy[4549]: {"level":"debug","ts":1689315877.8534453,"logger":"tls.handshake","msg":"choosing certificate","identifier":"firewall.localhost","num_choices":1}
Jul 14 11:54:37 LA-255 caddy[4549]: {"level":"debug","ts":1689315877.8534567,"logger":"tls.handshake","msg":"default certificate selection results","identifier":"firewall.localhost","subjects":["firewal>
Jul 14 11:54:37 LA-255 caddy[4549]: {"level":"debug","ts":1689315877.8534625,"logger":"tls.handshake","msg":"matched certificate in cache","remote_ip":"::1","remote_port":"35052","subjects":["firewall.l>
Jul 14 11:54:37 LA-255 caddy[4549]: {"level":"debug","ts":1689315877.8550754,"logger":"http.stdlib","msg":"http: TLS handshake error from [::1]:35052: local error: tls: bad record MAC"}
6

Thanks, that curl is better, and the post formatting is better.

But your logs are still truncated. Notice the lines that end in >. See the journalctl command in the docs I linked above.

Okay, so Caddy wasn’t able to install its root CA cert in your system’s trust store, because it’s not running as a privileged user. To fix this, run this:

sudo caddy trust

Then try the curl -v command again.

7

Thanks francis. There appears to be an error in the logs. Could you please assist me?

1. Curl -v output:

idfyinfra@LA-255:~$ curl -v https://firewall.localhost
*   Trying ::1:443...
* Connected to firewall.localhost (::1) port 443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
*  CAfile: /etc/ssl/certs/ca-certificates.crt
*  CApath: /etc/ssl/certs
* TLSv1.0 (OUT), TLS header, Certificate Status (22):
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* TLSv1.2 (IN), TLS header, Certificate Status (22):
* TLSv1.3 (IN), TLS handshake, Server hello (2):
* TLSv1.2 (IN), TLS header, Finished (20):
* TLSv1.2 (IN), TLS header, Supplemental data (23):
* TLSv1.3 (IN), TLS handshake, Encrypted Extensions (8):
* TLSv1.2 (IN), TLS header, Supplemental data (23):
* TLSv1.3 (IN), TLS handshake, Certificate (11):
* TLSv1.2 (IN), TLS header, Supplemental data (23):
* TLSv1.3 (IN), TLS handshake, CERT verify (15):
* TLSv1.2 (IN), TLS header, Supplemental data (23):
* TLSv1.3 (IN), TLS handshake, Finished (20):
* TLSv1.2 (OUT), TLS header, Finished (20):
* TLSv1.3 (OUT), TLS change cipher, Change cipher spec (1):
* TLSv1.2 (OUT), TLS header, Supplemental data (23):
* TLSv1.3 (OUT), TLS handshake, Finished (20):
* SSL connection using TLSv1.3 / TLS_AES_128_GCM_SHA256
* ALPN, server accepted to use h2
* Server certificate:
*  subject: [NONE]
*  start date: Jul 14 04:44:54 2023 GMT
*  expire date: Jul 14 16:44:54 2023 GMT
*  subjectAltName: host "firewall.localhost" matched cert's "firewall.localhost"
*  issuer: CN=Caddy Local Authority - ECC Intermediate
*  SSL certificate verify ok.
* Using HTTP2, server supports multiplexing
* Connection state changed (HTTP/2 confirmed)
* Copying HTTP/2 data in stream buffer to connection buffer after upgrade: len=0
* TLSv1.2 (OUT), TLS header, Supplemental data (23):
* TLSv1.2 (OUT), TLS header, Supplemental data (23):
* TLSv1.2 (OUT), TLS header, Supplemental data (23):
* Using Stream ID: 1 (easy handle 0x55b72e19de90)
* TLSv1.2 (OUT), TLS header, Supplemental data (23):
> GET / HTTP/2
> Host: firewall.localhost
> user-agent: curl/7.81.0
> accept: */*
> 
* TLSv1.2 (IN), TLS header, Supplemental data (23):
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
* TLSv1.2 (IN), TLS header, Supplemental data (23):
* Connection state changed (MAX_CONCURRENT_STREAMS == 250)!
* TLSv1.2 (OUT), TLS header, Supplemental data (23):
* TLSv1.2 (IN), TLS header, Supplemental data (23):
* TLSv1.2 (IN), TLS header, Supplemental data (23):
* TLSv1.2 (IN), TLS header, Supplemental data (23):
< HTTP/2 502 
< alt-svc: h3=":443"; ma=2592000
< server: Caddy
< content-length: 0
< date: Fri, 14 Jul 2023 07:58:52 GMT
< 
* Connection #0 to host firewall.localhost left intact

2. journalctl output:

Jul 14 13:32:52 LA-255 caddy[4549]: {"level":"debug","ts":1689321772.7606726,"logger":"http.handlers.reverse_proxy","msg":"selected upstream","dial":"10.10.20.10:443","total_upstreams":1}
Jul 14 13:32:52 LA-255 caddy[4549]: {"level":"debug","ts":1689321772.798875,"logger":"http.handlers.reverse_proxy","msg":"upstream roundtrip","upstream":"10.10.20.10:443","duration":0.038129803,"request":{"remote_ip":"127.0.0.1","remote_port":"48512","proto":"HTTP/2.0","method":"GET","host":"southcisco.localhost","uri":"/","headers":{"X-Forwarded-Host":["southcisco.localhost"],"User-Agent":["Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:109.0) Gecko/20100101 Firefox/110.0"],"Accept":["text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,*/*;q=0.8"],"Sec-Fetch-Dest":["document"],"Sec-Fetch-Site":["none"],"X-Forwarded-Proto":["https"],"Upgrade-Insecure-Requests":["1"],"X-Forwarded-For":["127.0.0.1"],"Accept-Language":["en-US,en;q=0.5"],"Accept-Encoding":["gzip, deflate, br"],"Sec-Fetch-Mode":["navigate"],"Sec-Fetch-User":["?1"],"Te":["trailers"]},"tls":{"resumed":true,"version":772,"cipher_suite":4865,"proto":"h2","server_name":"southcisco.localhost"}},"error":"EOF"}
Jul 14 13:32:52 LA-255 caddy[4549]: {"level":"error","ts":1689321772.7989633,"logger":"http.log.error","msg":"EOF","request":{"remote_ip":"127.0.0.1","remote_port":"48512","proto":"HTTP/2.0","method":"GET","host":"southcisco.localhost","uri":"/","headers":{"Sec-Fetch-Mode":["navigate"],"Accept-Language":["en-US,en;q=0.5"],"Accept-Encoding":["gzip, deflate, br"],"Upgrade-Insecure-Requests":["1"],"Sec-Fetch-Dest":["document"],"Sec-Fetch-Site":["none"],"Sec-Fetch-User":["?1"],"Te":["trailers"],"User-Agent":["Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:109.0) Gecko/20100101 Firefox/110.0"],"Accept":["text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,*/*;q=0.8"]},"tls":{"resumed":true,"version":772,"cipher_suite":4865,"proto":"h2","server_name":"southcisco.localhost"}},"duration":0.038306246,"status":502,"err_id":"31v31v18p","err_trace":"reverseproxy.statusError (reverseproxy.go:1299)"}
Jul 14 13:32:58 LA-255 caddy[4549]: {"level":"debug","ts":1689321778.605507,"logger":"http.handlers.reverse_proxy","msg":"selected upstream","dial":"10.10.10.1:17322","total_upstreams":1}
Jul 14 13:32:58 LA-255 caddy[4549]: {"level":"debug","ts":1689321778.6197236,"logger":"http.handlers.reverse_proxy","msg":"upstream roundtrip","upstream":"10.10.10.1:17322","duration":0.014158445,"request":{"remote_ip":"127.0.0.1","remote_port":"48522","proto":"HTTP/2.0","method":"GET","host":"firewall.localhost","uri":"/","headers":{"Sec-Fetch-Site":["none"],"Accept":["text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,*/*;q=0.8"],"Cookie":[],"Sec-Fetch-Dest":["document"],"X-Forwarded-Proto":["https"],"X-Forwarded-Host":["firewall.localhost"],"User-Agent":["Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:109.0) Gecko/20100101 Firefox/110.0"],"Accept-Language":["en-US,en;q=0.5"],"Accept-Encoding":["gzip, deflate, br"],"Te":["trailers"],"Sec-Fetch-Mode":["navigate"],"X-Forwarded-For":["127.0.0.1"],"Upgrade-Insecure-Requests":["1"],"Sec-Fetch-User":["?1"]},"tls":{"resumed":true,"version":772,"cipher_suite":4865,"proto":"h2","server_name":"firewall.localhost"}},"error":"EOF"}
Jul 14 13:32:58 LA-255 caddy[4549]: {"level":"error","ts":1689321778.6198046,"logger":"http.log.error","msg":"EOF","request":{"remote_ip":"127.0.0.1","remote_port":"48522","proto":"HTTP/2.0","method":"GET","host":"firewall.localhost","uri":"/","headers":{"Accept":["text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,*/*;q=0.8"],"Accept-Encoding":["gzip, deflate, br"],"Cookie":[],"Sec-Fetch-Dest":["document"],"Sec-Fetch-Mode":["navigate"],"User-Agent":["Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:109.0) Gecko/20100101 Firefox/110.0"],"Accept-Language":["en-US,en;q=0.5"],"Upgrade-Insecure-Requests":["1"],"Sec-Fetch-Site":["none"],"Sec-Fetch-User":["?1"],"Te":["trailers"]},"tls":{"resumed":true,"version":772,"cipher_suite":4865,"proto":"h2","server_name":"firewall.localhost"}},"duration":0.014314546,"status":502,"err_id":"rv47y2bve","err_trace":"reverseproxy.statusError (reverseproxy.go:1299)"}
8

I hope you can able to see all the logs.

9

Caddy isn’t able to connect to your upstream:

Are you sure this address is correct? Are you sure your networking allows that route? Make sure there isn’t some firewall rules preventing it.

At this point it’s not a problem with Caddy, it’s a networking problem between Caddy and your upstream app.

10

Yes, the address is correct, and I have disabled the Ubuntu system firewall. When I try to reach the device via the address, it works properly and I can even ping it.

idfyinfra@LA-255:~$ ping 10.10.10.1
PING 10.10.10.1 (10.10.10.1) 56(84) bytes of data.
64 bytes from 10.10.10.1: icmp_seq=1 ttl=255 time=5.76 ms
64 bytes from 10.10.10.1: icmp_seq=2 ttl=255 time=5.28 ms
64 bytes from 10.10.10.1: icmp_seq=3 ttl=255 time=6.70 ms
64 bytes from 10.10.10.1: icmp_seq=4 ttl=255 time=21.3 ms
64 bytes from 10.10.10.1: icmp_seq=5 ttl=255 time=4.36 ms
11

Could you please review the updated logs? It states that the certificate cannot be validated and that the error code is dypg0ehdp.

Jul 14 15:52:14 LA-255 caddy[12989]: {"level":"error","ts":1689330134.7739594,"logger":"http.log.error","msg":"tls: failed to verify certificate: x509: cannot validate certificate for 10.10.10.1 because it doesn't contain any IP SANs","request":{"remote_ip":"127.0.0.1","remote_port":"53812","proto":"HTTP/2.0","method":"GET","host":"firewall.localhost","uri":"/","headers":{"Accept-Encoding":["gzip, deflate, br"],"Sec-Fetch-Dest":["document"],"Sec-Fetch-Mode":["navigate"],"User-Agent":["Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:109.0) Gecko/20100101 Firefox/110.0"],"Accept":["text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,*/*;q=0.8"],"Accept-Language":["en-US,en;q=0.5"],"Upgrade-Insecure-Requests":["1"],"Sec-Fetch-Site":["none"],"Sec-Fetch-User":["?1"],"Te":["trailers"]},"tls":{"resumed":false,"version":772,"cipher_suite":4865,"proto":"h2","server_name":"firewall.localhost"}},"duration":0.017779996,"status":502,"err_id":"dypg0ehdp","err_trace":"reverseproxy.statusError (reverseproxy.go:1299)"}
12

This implies you proxied over HTTPS, and the upstream doesn’t have a certificate Caddy can trust.

Try proxying to your app’s HTTP port instead. It’s simpler, and if this is in your private network, there’s no need to re-encrypt traffic between Caddy and the upstream app.

13

Thank you, Francis. When I tried it using the HTTP port, it worked. Now, one quick question: when I access the device from my client system, it displays HTTPS but not secure in red. Is it possible to fix this?

14

Caddy passes the X-Forwarded-Proto: https header, which properly written apps should consult to know whether the original request was HTTPS or not.

Essentially, that’s a problem with your upstream app and not Caddy.

15

Is it possible to export the caddy certificates? I believe that if I manually load caddy certificates on my Windows machine, this not secure notification will not appear. This was just an idea that came to mind, and I wanted to try once.

16

You can grab Caddy’s root CA cert from /var/lib/caddy/.local/share/caddy/pki/authorities/local/root.crt

17

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.