mTLS under FreeBSD

Uh, I wouldn’t really word it that way, but sure :stuck_out_tongue:

Anyways if what I said was unclear, from your current config I’d write it like this:

(authorise) {
  basicauth {args.0} {
    admin [REDACTED]
  }
}

*.udance.com.au {

  map {labels.3} {backend} {online} {mtls} {phpmyadmin} {

#   HOSTNAME     BACKEND         ONLINE mTLS PHPMYADMIN #COMMENT
#---------------------------------------------------------------

    test         test.lan:443    yes    yes  yes        # test.udance.com.au
    basil        10.1.1.56:80    yes    no   yes        # basil.udance.com.au
    sachika      10.1.1.57:80    yes    no   yes        # sachika.udance.com.au
    default      unknown         yes    no   no         # subdomain does not exist
  }

# Error handling
  @unknown expression `{backend} == "unknown"`
  handle @unknown {
    respond "Denied" 403
  }

# Site offline
  @offline expression `{online} == "no"`
  redir @offline https://udance.statuspage.io temporary

  @split {
    expression `{online} == "split"`
    not remote_ip 10.1.1.0/24 10.1.2.0/24
  }
  redir @split https://udance.statuspage.io temporary

# Authenticate phpMyAdmin on production WordPress sites
  @phpmyadmin expression `{phpmyadmin} == "yes"`
  route @phpmyadmin {
    import authorise /phpmyadmin*
  }

# Fix when using the Nextcloud+Apache Docker image with Caddy.
  @nc-apache host nc-apache.udance.com.au
  route @nc-apache {
    redir /.well-known/carddav /remote.php/carddav permanent
    redir /.well-known/caldav /remote.php/caldav permanent
  }

# Enable HSTS for Nextcloud
  @hsts host cloud.udance.com.au
  header @hsts "Strict-Transport-Security max-age=31536000;"

# Secure backend communication
  @mtls expression `{mtls} == "yes"`
  handle @mtls {
    reverse_proxy {backend} {
      header_up Host {http.reverse_proxy.upstream.hostport}
      header_up X-Forwarded-Host {host}
      transport http {
        tls
      }
    }
  }

# Unsecured backend communication
  @nomtls expression `{mtls} == "no"`
  handle @nomtls {
    reverse_proxy {backend}
  }
}

No, you were very clear, but several related questions have sprung up…

As reverse_proxy was used here…

…could I have changed this to…

# Unsecured backend communication
  @nomtls expression `{mtls} == "no"`
  reverse_proxy @nomtls {backend}

…and as respond was used here…

…could this have been reduced as well to…

# Error handling
  @unknown expression `{backend} == "unknown"`
  respond @unknown "Denied" 403

Also, if I move the handle block below further up above other handle blocks, or, happen to add a new handle block after this one, would I need to change the handle directive to route so that if there’s a match, it runs through to the next handle block?

…or, because I’m using reverse_proxy, which is terminal, to avoid any ambiguity, would this work?

# Secure backend communication
  @mtls expression `{mtls} == "yes"`
  reverse_proxy @mtls {backend} {
    header_up Host {http.reverse_proxy.upstream.hostport}
    header_up X-Forwarded-Host {host}
    transport http {
      tls
    }
  }

If you unwrap those, then you lose the benefits of handle's mutual exclusivity and directive sorting order (which is guaranteed to be in the order they are written in the Caddyfile in relation to eachother) and the directives like respond and reverse_proxy would get sorted according to their directive order. The effect of this would be that respond @unknown "Denied" 403 would happen after the redir and header and basicauth which is probably not correct (doesn’t make sense to trigger auth when you’re going to respond with a 403 for example).

That said, I realized I made a mistake, because route is ordered after handle, so all the route blocks will get put at the end :man_facepalming: (you can see this in the adapt output).

The dumb/easy fix it to just wrap the whole thing in a single top-level route to tell the Caddyfile adapter “no, do it in this order” but that’s not so nice :grimacing: but in that case you don’t need some of the handle anymore because you can safely place directives in your Caddyfile in the right order by hand.

Ultimately, you’re writing a super complex config here, so it’s pretty tricky to get right (cause you have multiple paths through which a single request could go because you have multiple variables at play).

So this should work I guess:

(authorise) {
  basicauth {args.0} {
    admin [REDACTED]
  }
}

*.udance.com.au {

  map {labels.3} {backend} {online} {mtls} {phpmyadmin} {

#   HOSTNAME     BACKEND         ONLINE mTLS PHPMYADMIN #COMMENT
#---------------------------------------------------------------

    test         test.lan:443    yes    yes  yes        # test.udance.com.au
    basil        10.1.1.56:80    yes    no   yes        # basil.udance.com.au
    sachika      10.1.1.57:80    yes    no   yes        # sachika.udance.com.au
    default      unknown         yes    no   no         # subdomain does not exist
  }

  route {
# Error handling
    @unknown expression `{backend} == "unknown"`
    respond @unknown "Denied" 403

# Site offline
    @offline expression `{online} == "no"`
    redir @offline https://udance.statuspage.io temporary

    @split {
      expression `{online} == "split"`
      not remote_ip 10.1.1.0/24 10.1.2.0/24
    }
    redir @split https://udance.statuspage.io temporary

# Authenticate phpMyAdmin on production WordPress sites
    @phpmyadmin expression `{phpmyadmin} == "yes"`
    route @phpmyadmin {
      import authorise /phpmyadmin*
    }

# Fix when using the Nextcloud+Apache Docker image with Caddy.
    @nc-apache host nc-apache.udance.com.au
    route @nc-apache {
      redir /.well-known/carddav /remote.php/carddav permanent
      redir /.well-known/caldav /remote.php/caldav permanent
    }

# Enable HSTS for Nextcloud
    @hsts host cloud.udance.com.au
    header @hsts "Strict-Transport-Security max-age=31536000;"

# Secure backend communication
    @mtls expression `{mtls} == "yes"`
    reverse_proxy @mtls {backend} {
      header_up Host {http.reverse_proxy.upstream.hostport}
      header_up X-Forwarded-Host {host}
      transport http {
        tls
      }
    }

# Unsecured backend communication
    @nomtls expression `{mtls} == "no"`
    reverse_proxy @nomtls {backend}

  }
}
2 Likes

Ah hah…this is the routing first approach @matt refers to in said wiki article! @francislavoie this is all good stuff as it’s helping me cement my understanding.

… but logical and straightforward enough that mere mortals like myself can follow.

I know I’m pushing the envelope with Caddyfile, and I haven’t (yet, but could be coming soon!) hit any limitations using it, but I just love its simplicity with all the complexity handled under the hood.

I’ll test this out and report back later this evening (local time). Thank you for your help getting me to this point. :wine_glass: :canada:

2 Likes

There are those that are less than in love with the Caddyfile, but the truth is there are very few things it can’t do.

The place where it can become onerous is when the complexity of a given configuration (such as yours) starts to expose nuances in the abstraction. Specifically, we begin to require a high level of understanding of how and why the Caddyfile maps out into Caddy’s “real” config (JSON) in order to write it properly, at which point some see it as more pragmatic to simply write directly in JSON.

But, in truth, the Caddyfile can handle this, which is pretty cool.

3 Likes

The top-level route wrapper worked a treat. Adopting a routing-first approach was the way to go in this case. Interestingly, all the handle blocks/wrapping have disappeared from my Caddyfile making it even more streamlined than it was before, while improving its overall readability as well. :trophy:

Time to tinker with the backend.

EDIT: I had to remove the quotes from the header line…

# Enable HSTS for Nextcloud
    @hsts host cloud.udance.com.au
    header @hsts "Strict-Transport-Security max-age=31536000;"

Changed to…

# Enable HSTS for Nextcloud
    @hsts host cloud.udance.com.au
    header @hsts Strict-Transport-Security max-age=31536000;

Leaving them in resulted in Nextcloud complaining about Strict Transport Security and, curiously, also broke the connection between Nextcloud and OnlyOffice.

1 Like

Derp, I put the quotes around the wrong thing, it was meant to be:

header @hsts Strict-Transport-Security "max-age=31536000;"

I just prefer to include the quotes around header values in situation where it could eventually have spaces in the value, because if you have a 3rd parameter (i.e. third Caddyfile token, separated by spaces), it becomes a header replacement operation which is rarely the intended effect.

1 Like

Cool, that makes sense. I’ll adopt that practice.

My working backend Caddyfile under Caddy 2.3.0 before adding in the mTLS constructs:

:80 {
  root * /usr/local/www/wordpress
  encode gzip
  php_fastcgi 127.0.0.1:9000 {
    env SERVER_PORT 80
  }
  file_server

  log {
    format json 
    output file /var/log/access.log {
      roll_keep 7
    }
  }

  # External access denied to these files.
  @forbidden {
    path /wp-content/uploads/*.php
    path /wp-includes/*.php
    path /wp-config.php
    path /.user.ini
    path /wp-content/debug.log
  }
  respond @forbidden 404
}

I then followed the instructions under the section Backend in the wiki article Use Caddy for local HTTPS (TLS) between front-end reverse proxy and LAN hosts . My backend Caddyfile under Caddy 2.4.0-RC.1 including mTLS constructs.

{
  log {
    format json {
      time_format iso8601
    }
  }
#  debug

# TLS Options
  acme_ca https://acme.lan/acme/local/directory # point to ACME server
  acme_ca_root /etc/ssl/certs/root.crt  # define root certificate
}

test.lan {
  root * /usr/local/www/wordpress
  encode gzip
  php_fastcgi 127.0.0.1:9000 {
    env SERVER_PORT 80
  }
  file_server

  log {
    format json {
      time_format iso8601
    }
    output file /var/log/access.log {
      roll_keep 7
    }
  }

  # External access denied to these files.
  @forbidden {
    path /wp-content/uploads/*.php
    path /wp-includes/*.php
    path /wp-config.php
    path /.user.ini
    path /wp-content/debug.log
  }
  respond @forbidden 404
}

After restarting both frontend and backend Caddy services, I attempt to access the test site. Unfortunately, I get a 502 error.


Where to from here?
Here’s an extract of the frontend Caddy log post-restart.

{"level":"info","ts":"2021-05-09T22:55:21.393+0800","msg":"shutting down apps, then terminating","signal":"SIGTERM"}
{"level":"warn","ts":"2021-05-09T22:55:21.393+0800","msg":"exiting; byeee!! 👋","signal":"SIGTERM"}
{"level":"info","ts":"2021-05-09T22:55:21.950+0800","logger":"tls.cache.maintenance","msg":"stopped background certificate maintenance","cache":"0xc000344930"}
{"level":"debug","ts":"2021-05-09T22:55:21.950+0800","logger":"http.handlers.acme_server","msg":"unloading unused CA database","db_key":"local"}
{"level":"info","ts":"2021-05-09T22:55:21.952+0800","logger":"admin","msg":"stopped previous server","address":"tcp/localhost:2019"}
{"level":"info","ts":"2021-05-09T22:55:21.952+0800","msg":"shutdown complete","signal":"SIGTERM","exit_code":0}
{"level":"info","ts":1620572122.0956097,"msg":"using provided configuration","config_file":"/usr/local/www/Caddyfile","config_adapter":"caddyfile"}
{"level":"warn","ts":1620572122.1175373,"msg":"input is not formatted with 'caddy fmt'","adapter":"caddyfile","file":"/usr/local/www/Caddyfile","line":2}
{"level":"info","ts":"2021-05-09T22:55:22.125+0800","logger":"admin","msg":"admin endpoint started","address":"tcp/localhost:2019","enforce_origin":false,"origins":["localhost:2019","[::1]:2019","127.0.0.1:2019"]}
{"level":"info","ts":"2021-05-09T22:55:22.126+0800","logger":"tls.cache.maintenance","msg":"started background certificate maintenance","cache":"0xc00033e850"}
{"level":"info","ts":"2021-05-09T22:55:22.155+0800","logger":"http","msg":"server is listening only on the HTTPS port but has no TLS connection policies; adding one to enable TLS","server_name":"srv0","https_port":443}
{"level":"info","ts":"2021-05-09T22:55:22.155+0800","logger":"http","msg":"enabling automatic HTTP->HTTPS redirects","server_name":"srv0"}
{"level":"warn","ts":"2021-05-09T22:55:35.176+0800","logger":"pki.ca.local","msg":"installing root certificate (you might be prompted for password)","path":"storage:pki/authorities/local/root.crt"}
2021/05/09 22:55:35 Note: NSS support is not available on your platform
2021/05/09 22:55:35 define JAVA_HOME environment variable to use the Java trust
{"level":"error","ts":"2021-05-09T22:55:35.176+0800","logger":"pki.ca.local","msg":"failed to install root
certificate","error":"trust not supported","certificate_file":"storage:pki/authorities/local/root.crt"}
{"level":"info","ts":"2021-05-09T22:55:35.176+0800","logger":"tls","msg":"cleaning storage unit","description":"FileStorage:/.local/share/caddy"}
{"level":"info","ts":"2021-05-09T22:55:35.176+0800","logger":"http","msg":"enabling automatic TLS certificate management","domains":["www.xenografix.com.au","*.udance.com.au","acme.lan","caffigoalkeeping.com.au","www.caffigoalkeeping.com","caffigoalkeeping.com","www.caffigoalkeeping.com.au","readymcgetty.com.au","www.readymcgetty.com.au","xenografix.com.au","udance.com.au","www.udance.com.au"]}
{"level":"warn","ts":"2021-05-09T22:55:35.191+0800","logger":"tls","msg":"stapling OCSP","error":"no OCSP stapling for [acme.lan]: no OCSP server specified in certificate"}
{"level":"info","ts":"2021-05-09T22:55:35.191+0800","logger":"tls","msg":"finished cleaning storage units"}
{"level":"info","ts":"2021-05-09T22:55:35.191+0800","msg":"autosaved config (load with --resume flag)","file":"/.config/caddy/autosave.json"}
{"level":"info","ts":"2021-05-09T22:55:35.191+0800","msg":"serving initial configuration"}
Successfully started Caddy (pid=77501) - Caddy is running in the background

…and an extract from the backend Caddy log post-restart:

{"level":"info","ts":"2021-05-09T22:55:57.526+0800","msg":"shutting down apps, then terminating","signal":"SIGTERM"}
{"level":"warn","ts":"2021-05-09T22:55:57.526+0800","msg":"exiting; byeee!! 👋","signal":"SIGTERM"}
{"level":"info","ts":"2021-05-09T22:55:57.529+0800","logger":"tls.cache.maintenance","msg":"stopped background certificate maintenance","cache":"0xc0002c6d90"}
{"level":"info","ts":"2021-05-09T22:55:57.531+0800","logger":"admin","msg":"stopped previous server","address":"tcp/localhost:2019"}
{"level":"info","ts":"2021-05-09T22:55:57.531+0800","msg":"shutdown complete","signal":"SIGTERM","exit_code":0}
{"level":"info","ts":1620572157.6563253,"msg":"using provided configuration","config_file":"/usr/local/www/Caddyfile","config_adapter":"caddyfile"}
{"level":"warn","ts":1620572157.6602314,"msg":"input is not formatted with 'caddy fmt'","adapter":"caddyfile","file":"/usr/local/www/Caddyfile","line":2}
{"level":"info","ts":"2021-05-09T22:55:57.662+0800","logger":"admin","msg":"admin endpoint started","address":"tcp/localhost:2019","enforce_origin":false,"origins":["localhost:2019","[::1]:2019","127.0.0.1:2019"]}
{"level":"info","ts":"2021-05-09T22:55:57.663+0800","logger":"tls.cache.maintenance","msg":"started backgroundcertificate maintenance","cache":"0xc0002de380"}
{"level":"info","ts":"2021-05-09T22:55:57.663+0800","logger":"http","msg":"server is listening only on theHTTPS port but has no TLS connection policies; adding one to enable TLS","server_name":"srv0","https_port":443}
{"level":"info","ts":"2021-05-09T22:55:57.664+0800","logger":"http","msg":"enabling automatic HTTP->HTTPS redirects","server_name":"srv0"}
{"level":"info","ts":"2021-05-09T22:55:57.665+0800","logger":"http","msg":"enabling automatic TLS certificate management","domains":["test.lan"]}
{"level":"info","ts":"2021-05-09T22:55:57.665+0800","logger":"tls","msg":"cleaning storage unit","description":"FileStorage:/.local/share/caddy"}
{"level":"info","ts":"2021-05-09T22:55:57.667+0800","logger":"tls","msg":"finished cleaning storage units"}
{"level":"warn","ts":"2021-05-09T22:55:57.688+0800","logger":"tls","msg":"stapling OCSP","error":"no OCSP stapling for [test.lan]: no OCSP server specified in certificate"}
{"level":"info","ts":"2021-05-09T22:55:57.689+0800","msg":"autosaved config (load with --resume flag)","file":"/.config/caddy/autosave.json"}
{"level":"info","ts":"2021-05-09T22:55:57.689+0800","msg":"serving initial configuration"}
Successfully started Caddy (pid=80607) - Caddy is running in the background

There’s an error about OCSP stapling in both Caddy logs, but I’m not sure what this means. Help! Please!

Hi Basil,

This specific OCSP error can be ignored, I have it too in a working setup. Explaination

This is something I don’t remember seeing. Looks like something is wrong with the certificate. COuld it be something with user rights?

Edit: or the content of the certificate is wrong?

Could be, but I’m not sure.

root@caddy:/.local/share/caddy/pki/authorities/local # ls -l
total 26
-rw-------  1 root  wheel  680 May  8 04:00 intermediate.crt
-rw-------  1 root  wheel  227 May  8 04:00 intermediate.key
-rw-------  1 root  wheel  627 May  2 13:35 root.crt
-rw-------  1 root  wheel  227 May  2 13:35 root.key

I wonder if it has something to do with the acme server and the FreeBSD platform. The message first appears when I enable the acme server. See post #2 in this thread.

There must be more logs after that for both of them, i.e. doing the ACME on the backend for that domain. That looks like you only got the logs from the restart and that’s it. Need to see the logs after it started running too, because some of this stuff doesn’t happen right in the startup phase, but after it.

Note sure why there’s a newline in there, but anyways I don’t think this is a problem, this just means Caddy wasn’t able to install the root certificate into the trust store of that system which might be fine.

But you could run caddy trust manually to make sure it was done right anyways (make sure to run that command such that Caddy knows that its storage location is /var/lib/caddy, i.e. the $HOME of the caddy user when running as a service). Caddy needs sudo/root access to install that which is the trouble.

Ignore that, not a problem. It’s just a warning. OCSP is not required, but strongly recommended, but isn’t really viable for mTLS. Basically it’s supposed to be a URL that Caddy can send a request to, to see if the certificate is still valid (i.e. hasn’t been revoked for any number of reasons). It’s decently important but public trust but for private it doesn’t matter so much.

My bad. That’s me breaking up long lines to copy across and, in this instance, forgetting to restore the line.

Good to know.

The first three related messages post-restart from the frontend log. There are a few more messages similar to this that follow, and then what appears to be a cert refresh, and then more error messages much like the first three.

...
{"level":"error","ts":"2021-05-09T23:00:37.235+0800","logger":"http.log.error.log4","msg":"x509: certificate signed by unknown authority","request":{"remote_addr":"108.162.221.21:22832","proto":"HTTP/2.0","method":"HEAD","host":"test.udance.com.au","uri":"/","headers":{"Cf-Ray":["64cbcc60ddf29b97-DFW"],"X-Forwarded-Proto":["https"],"User-Agent":["jetmon/1.0 (Jetpack Site Uptime Monitor by WordPress.com)"],"Cf-Connecting-Ip":["192.0.91.177"],"Cf-Request-Id":["09f33e108b00009b979e2cf000000001"],"Cdn-Loop":["cloudflare"],"Cf-Ipcountry":["US"],"Accept-Encoding":["gzip"],"X-Forwarded-For":["192.0.91.177"],"Cf-Visitor":["{\"scheme\":\"https\"}"]},"tls":{"resumed":false,"version":772,"cipher_suite":4865,"proto":"h2","proto_mutual":true,"server_name":"test.udance.com.au"}},"duration":0.006550785,"status":502,"err_id":"kx7sdf71h","err_trace":"reverseproxy.statusError (reverseproxy.go:850)"}
...
{"level":"error","ts":"2021-05-09T23:06:06.006+0800","logger":"http.log.error.log4","msg":"x509: certificate signed by unknown authority","request":{"remote_addr":"10.1.1.54:21726","proto":"HTTP/1.1","method":"POST","host":"test.udance.com.au","uri":"/wp-admin/admin-ajax.php?action=updraft_central","headers":{"Accept-Encoding":["deflate, gzip"],"Referer":["https://test.udance.com.au/wp-admin/admin-ajax.php?action=updraft_central"],"Connection":["close"],"Content-Length":["1146"],"Content-Type":["application/x-www-form-urlencoded"],"User-Agent":["WordPress/5.7.1; class-udrpc.php/1.4.21; https://blog.udance.com.au"],"Accept":["*/*"]},"tls":{"resumed":false,"version":772,"cipher_suite":4865,"proto":"http/1.1","proto_mutual":true,"server_name":"test.udance.com.au"}},"duration":0.006877555,"status":502,"err_id":"jkzpr9x4p","err_trace":"reverseproxy.statusError (reverseproxy.go:850)"}
{"level":"error","ts":"2021-05-09T23:07:06.187+0800","logger":"http.log.error.log4","msg":"x509: certificate signed by unknown authority","request":{"remote_addr":"108.162.221.118:47290","proto":"HTTP/1.1","method":"HEAD","host":"test.udance.com.au","uri":"/","headers":{"User-Agent":["jetmon/1.0 (Jetpack Site Uptime Monitor by WordPress.com)"],"Connection":["Keep-Alive"],"Accept-Encoding":["gzip"],"X-Forwarded-Proto":["https"],"Cf-Visitor":["{\"scheme\":\"https\"}"],"Cf-Request-Id":["09f343ffe800000f3e8815d000000001"],"Cf-Ipcountry":["US"],"X-Forwarded-For":["192.0.91.177"],"Cf-Ray":["64cbd5dfde3a0f3e-DFW"],"Cf-Connecting-Ip":["192.0.91.177"],"Cdn-Loop":["cloudflare"]},"tls":{"resumed":false,"version":772,"cipher_suite":4865,"proto":"","proto_mutual":true,"server_name":"test.udance.com.au"}},"duration":0.006013134,"status":502,"err_id":"xv3yykkdr","err_trace":"reverseproxy.statusError (reverseproxy.go:850)"}
...
{"level":"info","ts":"2021-05-10T00:15:22.128+0800","logger":"tls.cache.maintenance","msg":"certificate expires soon; queuing for renewal","identifiers":["acme.lan"],"remaining":14096.871491679}
{"level":"info","ts":"2021-05-10T00:15:22.128+0800","logger":"tls.cache.maintenance","msg":"attempting certificate renewal","identifiers":["acme.lan"],"remaining":14096.871356348}
{"level":"info","ts":"2021-05-10T00:15:22.129+0800","logger":"tls.renew","msg":"acquiring lock","identifier":"acme.lan"}
{"level":"info","ts":"2021-05-10T00:15:22.134+0800","logger":"tls.renew","msg":"lock acquired","identifier":"acme.lan"}
{"level":"info","ts":"2021-05-10T00:15:22.135+0800","logger":"tls.renew","msg":"renewing certificate","identifier":"acme.lan","remaining":14096.864875394}
{"level":"info","ts":"2021-05-10T00:15:22.137+0800","logger":"tls.renew","msg":"certificate renewed successfully","identifier":"acme.lan"}
{"level":"info","ts":"2021-05-10T00:15:22.137+0800","logger":"tls.renew","msg":"releasing lock","identifier":"acme.lan"}
{"level":"info","ts":"2021-05-10T00:15:22.138+0800","logger":"tls","msg":"reloading managed certificate","identifiers":["acme.lan"]}
{"level":"warn","ts":"2021-05-10T00:15:22.139+0800","logger":"tls","msg":"stapling OCSP","error":"no OCSP stapling for [acme.lan]: no OCSP server specified in certificate"}
{"level":"info","ts":"2021-05-10T00:15:22.139+0800","logger":"tls.cache","msg":"replaced certificate in cache","identifiers":["acme.lan"],"new_expiration":"2021-05-10T04:15:22.000Z"}
{"level":"error","ts":"2021-05-10T00:18:32.293+0800","logger":"http.log.error.log4","msg":"x509: certificate signed by unknown authority","request":{"remote_addr":"108.162.221.6:43830","proto":"HTTP/1.1","method":"HEAD","host":"test.udance.com.au","uri":"/","headers":{"Connection":["Keep-Alive"],"Accept-Encoding":["gzip"],"X-Forwarded-For":["192.0.91.177"],"Cf-Ray":["64cc3e840a3a2f25-DFW"],"User-Agent":["jetmon/1.0 (Jetpack Site Uptime Monitor by WordPress.com)"],"Cf-Connecting-Ip":["192.0.91.177"],"Cf-Request-Id":["09f385668500002f25a01dc000000001"],"Cf-Ipcountry":["US"],"X-Forwarded-Proto":["https"],"Cf-Visitor":["{\"scheme\":\"https\"}"],"Cdn-Loop":["cloudflare"]},"tls":{"resumed":false,"version":772,"cipher_suite":4865,"proto":"","proto_mutual":true,"server_name":"test.udance.com.au"}},"duration":0.006488916,"status":502,"err_id":"pvzr5v6v0","err_trace":"reverseproxy.statusError (reverseproxy.go:850)"}
...

Where there’s an error in the log, I notice this at the end of the line

"err_trace":"reverseproxy.statusError (reverseproxy.go:850)"

Strange, nothing new added in the backend log.

1 Like

I think this is the relevant bit. So like I said earlier, I think you need to run caddy trust on the front-end. Basically the backend did get certificates signed by the front-end, but then when the front-end goes to proxy, it doesn’t trust the backend’s cert because the root CA cert isn’t in the front-end’s system trust store (which is used when making HTTPS requests with the proxy).

Alternatively you can configure transport http with the tls_trusted_ca_certs with the path to the root CA cert file (i.e. /var/lib/caddy/.local/share/caddy/pki/authorities/local/root.crt), but that’s a bit of a dumb workaround. Running caddy trust is the better option.

Basically, just run this on your frontend server:

sudo HOME=/var/lib/caddy caddy trust

Kinda dumb, but it should do the trick.

The caddy trust command is not very intelligent right now… the trouble is the user it runs as. If you just run it with caddy trust as your own user then it’ll try to use your user’s home. If you just run it with sudo caddy trust then it’ll use the root user’s home (i.e. /root). But what we actually want is the home of the caddy user (under which the systemd service runs Caddy) which is /var/lib/caddy. Hence hacking it by overriding HOME in that command :roll_eyes:

I’ll see if that can be smoothed out…

This made me realise I’d copied over the incorrect certificate to the backend Caddy server. I had copied across /.local/share/caddy/pki/authorities/local/root.crt instead of /var/lib/caddy/.local/share/caddy/pki/authorities/local/root.crt.

@Rob789 The above paragraph in your wiki article under the heading Backends should be updated to use the absolute path to avoid any ambiguity:

root@caddy:~ # sudo HOME=/var/lib/caddy caddy trust
2021/05/10 13:40:41.376 WARN    ca.local        installing root certificate (you might be prompted for password) {"path": "storage:pki/authorities/local/root.crt"}
2021/05/10 21:40:41 Note: NSS support is not available on your platform
2021/05/10 21:40:41 define JAVA_HOME environment variable to use the Java trust
trust: trust not supported

After copying across the correct cert to the backend server, I then restarted both frontend and backend Caddy services and attempted to access the test site. Sadly still no go :cry:

Frontend caddy log extract

{"level":"info","ts":"2021-05-10T21:41:44.437+0800","msg":"shutting down apps, then terminating","signal":"SIGTERM"}
{"level":"warn","ts":"2021-05-10T21:41:44.437+0800","msg":"exiting; byeee!! 👋","signal":"SIGTERM"}
{"level":"info","ts":"2021-05-10T21:41:44.443+0800","logger":"tls.cache.maintenance","msg":"stopped background certificate maintenance","cache":"0xc00035a690"}
{"level":"info","ts":"2021-05-10T21:41:44.444+0800","logger":"admin","msg":"stopped previous server","address":"tcp/localhost:2019"}
{"level":"info","ts":"2021-05-10T21:41:44.444+0800","msg":"shutdown complete","signal":"SIGTERM","exit_code":0}
{"level":"info","ts":1620654104.5705228,"msg":"using provided configuration","config_file":"/usr/local/www/Caddyfile","config_adapter":"caddyfile"}
{"level":"warn","ts":1620654104.6052015,"msg":"input is not formatted with 'caddy fmt'","adapter":"caddyfile","file":"/usr/local/www/Caddyfile","line":2}
{"level":"info","ts":"2021-05-10T21:41:44.613+0800","logger":"admin","msg":"admin endpoint started","address":"tcp/localhost:2019","enforce_origin":false,"origins":["127.0.0.1:2019","localhost:2019","[::1]:2019"]}
{"level":"info","ts":"2021-05-10T21:41:44.614+0800","logger":"tls.cache.maintenance","msg":"started background certificate maintenance","cache":"0xc00032ff80"}
{"level":"info","ts":"2021-05-10T21:41:44.644+0800","logger":"http","msg":"server is listening only on the HTTPS port but has no TLS connection policies; adding one to enable TLS","server_name":"srv0","https_port":443}
{"level":"info","ts":"2021-05-10T21:41:44.645+0800","logger":"http","msg":"enabling automatic HTTP->HTTPS redirects","server_name":"srv0"}
{"level":"warn","ts":"2021-05-10T21:41:57.225+0800","logger":"pki.ca.local","msg":"installing root certificate (you might be prompted for password)","path":"storage:pki/authorities/local/root.crt"}
2021/05/10 21:41:57 Note: NSS support is not available on your platform
2021/05/10 21:41:57 define JAVA_HOME environment variable to use the Java trust
{"level":"error","ts":"2021-05-10T21:41:57.226+0800","logger":"pki.ca.local","msg":"failed to install root certificate","error":"trust not supported","certificate_file":"storage:pki/authorities/local/root.crt"}
{"level":"info","ts":"2021-05-10T21:41:57.226+0800","logger":"http","msg":"enabling automatic TLS certificate management","domains":["www.xenografix.com.au","www.caffigoalkeeping.com","caffigoalkeeping.com","readymcgetty.com.au","www.udance.com.au","*.udance.com.au","caffigoalkeeping.com.au","udance.com.au","www.caffigoalkeeping.com.au","www.readymcgetty.com.au","xenografix.com.au","acme.lan"]}
{"level":"info","ts":"2021-05-10T21:41:57.226+0800","logger":"tls","msg":"cleaning storage unit","description":"FileStorage:/.local/share/caddy"}
{"level":"warn","ts":"2021-05-10T21:41:57.258+0800","logger":"tls","msg":"stapling OCSP","error":"no OCSP stapling for [acme.lan]: no OCSP server specified in certificate"}
{"level":"info","ts":"2021-05-10T21:41:57.259+0800","msg":"autosaved config (load with --resume flag)","file":"/.config/caddy/autosave.json"}
{"level":"info","ts":"2021-05-10T21:41:57.259+0800","msg":"serving initial configuration"}
Successfully started Caddy (pid=96255) - Caddy is running in the background
{"level":"info","ts":"2021-05-10T21:41:57.260+0800","logger":"tls","msg":"finished cleaning storage units"}
{"level":"error","ts":"2021-05-10T21:42:08.298+0800","logger":"http.log.error.log4","msg":"x509: certificate signed by unknown authority","request":{"remote_addr":"162.158.166.143:50592","proto":"HTTP/1.1","method":"GET","host":"test.udance.com.au","uri":"/","headers":{"Accept-Encoding":["gzip"],"Cf-Ipcountry":["AU"],"X-Forwarded-Proto":["https"],"User-Agent":["Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/90.0.4430.93 Safari/537.36"],"Cookie":["tk_or=%22https%3A%2F%2Fheimdall.udance.com.au%2F%22; wordpress_logged_in_6f3be6329744d07b768d1565b796af6d=basil%7C1645523796%7Cp7aCGGrrNSGs9CR4PRXTX8cgzxAPWRW5H3EMiHsK5v1%7Ca678f9423db6329357457c5b072e5840131c33f05e2c861be9d697954217ae95; wp-settings-1=libraryContent%3Dbrowse%26editor%3Dtinymce%26hidetb%3D1%26editor_plain_text_paste_warning%3D1; wp-settings-time-1=1613987799; __cfduid=d73e22d7aa4142037dc86a5d6da76202d1618656636; tk_lr=%22https%3A%2F%2Fheimdall.udance.com.au%2F%22"],"Cf-Request-Id":["09f81c940700001aaca3904000000001"],"Cf-Ray":["64d396ccd9631aac-SIN"],"Cf-Visitor":["{\"scheme\":\"https\"}"],"Accept-Language":["en-AU,en-GB;q=0.9,en-US;q=0.8,en;q=0.7"],"Connection":["Keep-Alive"],"X-Forwarded-For":["106.69.185.51"],"Cache-Control":["max-age=0"],"Sec-Ch-Ua-Mobile":["?0"],"Upgrade-Insecure-Requests":["1"],"Sec-Fetch-Site":["same-origin"],"Sec-Fetch-Mode":["navigate"],"Sec-Fetch-User":["?1"],"Sec-Fetch-Dest":["document"],"Sec-Gpc":["1"],"Cdn-Loop":["cloudflare"],"Sec-Ch-Ua":["\" Not A;Brand\";v=\"99\", \"Chromium\";v=\"90\", \"Google Chrome\";v=\"90\""],"Accept":["text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.9"],"Referer":["https://test.udance.com.au/"],"Cf-Connecting-Ip":["106.69.185.51"]},"tls":{"resumed":false,"version":772,"cipher_suite":4865,"proto":"","proto_mutual":true,"server_name":"test.udance.com.au"}},"duration":0.006602499,"status":502,"err_id":"nfeudwy0e","err_trace":"reverseproxy.statusError (reverseproxy.go:850)"}
{"level":"error","ts":"2021-05-10T21:42:11.888+0800","logger":"http.log.error.log4","msg":"x509: certificate signed by unknown authority","request":{"remote_addr":"162.158.165.126:18316","proto":"HTTP/1.1","method":"GET","host":"test.udance.com.au","uri":"/favicon.ico","headers":{"Accept":["image/avif,image/webp,image/apng,image/svg+xml,image/*,*/*;q=0.8"],"Sec-Fetch-Mode":["no-cors"],"Sec-Fetch-Dest":["image"],"Referer":["https://test.udance.com.au/"],"Accept-Encoding":["gzip"],"X-Forwarded-For":["106.69.185.51"],"Cf-Visitor":["{\"scheme\":\"https\"}"],"Pragma":["no-cache"],"Sec-Gpc":["1"],"Cookie":["tk_or=%22https%3A%2F%2Fheimdall.udance.com.au%2F%22; wordpress_logged_in_6f3be6329744d07b768d1565b796af6d=basil%7C1645523796%7Cp7aCGGrrNSGs9CR4PRXTX8cgzxAPWRW5H3EMiHsK5v1%7Ca678f9423db6329357457c5b072e5840131c33f05e2c861be9d697954217ae95; wp-settings-1=libraryContent%3Dbrowse%26editor%3Dtinymce%26hidetb%3D1%26editor_plain_text_paste_warning%3D1; wp-settings-time-1=1613987799; __cfduid=d73e22d7aa4142037dc86a5d6da76202d1618656636; tk_lr=%22https%3A%2F%2Fheimdall.udance.com.au%2F%22; cf_ob_info=502:64d396ccd9631aac:SIN; cf_use_ob=0"],"Cdn-Loop":["cloudflare"],"Sec-Ch-Ua-Mobile":["?0"],"User-Agent":["Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/90.0.4430.93 Safari/537.36"],"Sec-Fetch-Site":["same-origin"],"Cf-Connecting-Ip":["106.69.185.51"],"Cf-Ipcountry":["AU"],"Cf-Ray":["64d396e34aea1aac-SIN"],"X-Forwarded-Proto":["https"],"Cf-Request-Id":["09f81ca21200001aacac959000000001"],"Connection":["Keep-Alive"],"Cache-Control":["no-cache"],"Sec-Ch-Ua":["\" Not A;Brand\";v=\"99\", \"Chromium\";v=\"90\", \"Google Chrome\";v=\"90\""],"Accept-Language":["en-AU,en-GB;q=0.9,en-US;q=0.8,en;q=0.7"]},"tls":{"resumed":false,"version":772,"cipher_suite":4865,"proto":"","proto_mutual":true,"server_name":"test.udance.com.au"}},"duration":0.005877493,"status":502,"err_id":"0vkbgp8eh","err_trace":"reverseproxy.statusError (reverseproxy.go:850)"}

Backend Caddy log extract

{"level":"info","ts":"2021-05-10T21:41:50.878+0800","msg":"shutting down apps, then terminating","signal":"SIGTERM"}
{"level":"warn","ts":"2021-05-10T21:41:50.879+0800","msg":"exiting; byeee!! 👋","signal":"SIGTERM"}
{"level":"info","ts":"2021-05-10T21:41:50.887+0800","logger":"tls.cache.maintenance","msg":"stopped background certificate maintenance","cache":"0xc0003347e0"}
{"level":"info","ts":"2021-05-10T21:41:50.888+0800","logger":"admin","msg":"stopped previous server","address":"tcp/localhost:2019"}
{"level":"info","ts":"2021-05-10T21:41:50.888+0800","msg":"shutdown complete","signal":"SIGTERM","exit_code":0}
{"level":"info","ts":1620654111.0122516,"msg":"using provided configuration","config_file":"/usr/local/www/Caddyfile","config_adapter":"caddyfile"}
{"level":"warn","ts":1620654111.0166075,"msg":"input is not formatted with 'caddy fmt'","adapter":"caddyfile","file":"/usr/local/www/Caddyfile","line":2}
{"level":"info","ts":"2021-05-10T21:41:51.019+0800","logger":"admin","msg":"admin endpoint started","address":"tcp/localhost:2019","enforce_origin":false,"origins":["localhost:2019","[::1]:2019","127.0.0.1:2019"]}
{"level":"info","ts":"2021-05-10T21:41:51.019+0800","logger":"tls.cache.maintenance","msg":"started background certificate maintenance","cache":"0xc0002f2700"}
{"level":"info","ts":"2021-05-10T21:41:51.020+0800","logger":"http","msg":"server is listening only on the HTTPS port but has no TLS connection policies; adding one to enable TLS","server_name":"srv0","https_port":443}
{"level":"info","ts":"2021-05-10T21:41:51.020+0800","logger":"http","msg":"enabling automatic HTTP->HTTPS redirects","server_name":"srv0"}
{"level":"info","ts":"2021-05-10T21:41:51.021+0800","logger":"http","msg":"enabling automatic TLS certificate management","domains":["test.lan"]}
{"level":"info","ts":"2021-05-10T21:41:51.021+0800","logger":"tls","msg":"cleaning storage unit","description":"FileStorage:/.local/share/caddy"}
{"level":"info","ts":"2021-05-10T21:41:51.023+0800","logger":"tls","msg":"finished cleaning storage units"}
{"level":"warn","ts":"2021-05-10T21:41:51.045+0800","logger":"tls","msg":"stapling OCSP","error":"no OCSP stapling for [test.lan]: no OCSP server specified in certificate"}
{"level":"info","ts":"2021-05-10T21:41:51.045+0800","msg":"autosaved config (load with --resume flag)","file":"/.config/caddy/autosave.json"}
{"level":"info","ts":"2021-05-10T21:41:51.045+0800","msg":"serving initial configuration"}
Successfully started Caddy (pid=91239) - Caddy is running in the background

Wait a minute… Do you have the ca-certificates package installed? Or maybe that’s stored in a different location on your system… You’re on a BSD right?

Edit: Well I’m going to guess that since you’re on FreeBSD, the smallstep trust install command doesn’t know what to do which is why we see “trust not supported”. I’ll need to dig into that to find out when they message is written.

From a bit of Googling, looks like FreeBSD now uses Mozilla’s trust store (aka NSS) as the system trust. This seems to be shipped as a single file that apps are meant to read from. Idk what to suggest, cause I’m not a BSD user.

Aha - yeah, the trust tool doesn’t seem to support FreeBSD.

If you could comment on the issue and help move it along, that’d be cool. If you can figure out the right procedure to install a custom root cert on FreeBSD and help them test it, changes there should eventually land in Caddy. :+1:

I will make a note but the absolute path depends on a few things. For example, in my Wiki setup I run caddy with caddy start in /root which makes the absolute path /root/.local/share/caddy/pki/authorities/local but when you use systemd the path would be (for at least Debian) /var/lib/caddy/.local/share/caddy/pki/authorities/local

:sob:

Okay (after regaining some composure), I must confess to being bamboozled at this point as this is now starting to go over my head. I’m having some difficulty connecting the dots.I don’t pretend to understand any of this stuff and what follows is just a little detective work on my part to see if anything has slipped through the cracks.

The last entry for the above issue was dated 10 April 2020. Seven days later on 17 April 2020 step-certificates and step-cli were ported to FreeBSD. Does this imply anything about the trust tool and FreeBSD, or am I barking up the wrong tree? I’m hoping this ignites a spark of hope. Assuming it does, are step-certificates and step-cli package dependencies for Caddy, and, with them installed, it may be possible to skirt around the ‘trust not supported’ error?