mTLS under FreeBSD

basil · May 4, 2021, 8:00pm

Changed that. Still getting the 502 error.

Thanks for the tip

Also, good to know.

I had hoped to only highlight the minimal parts of my Caddyfile that are relevant to solving the mTLS puzzle, but for the sake of completeness, here are the relevant parts of my Caddyfile, including the global section and snippets, which relate to the udance.com.au domain.

{
  email basil.hendroff@udance.com.au
  acme_dns cloudflare [REDACTED]
#  debug
  log {
    format json {
      time_format iso8601
    }
  }
}

#----------------------------------------------------------------------
# Snippet   : Description                           : Arguments
#----------------------------------------------------------------------
# authorise : Basic authentication                  : subdirectory
# logging   : Rolling access log                    : subdomain
# online    : Domain availability                   : {yes|no|split}

(authorise) {
  basicauth {args.0} {
    admin [REDACTED]
  }
}

(logging) {
  log {
    format json {
      time_format iso8601
    }
    output file /var/log/caddy/{args.0}.log {
      roll_size 100MiB    # Default 100MiB
      roll_keep 10        # Default 10
      roll_keep_for 90d   # Default 90d
    }
  }
}

www.udance.com.au {
  redir https://udance.com.au{uri} permanent
}

udance.com.au {

  encode gzip zstd
  import logging udance.com.au
  import authorise /phpmyadmin*

  map {path} {backend} {online} {

#   PATH                BACKEND          ONLINE
#---------------------------------------------------------------
    ~^/tautulli.*       10.1.1.26:8181   yes
    ~^/transmission.*   10.1.1.28:9091   yes
    ~^/.*               10.1.1.55:80     yes
  }

# Offline handling

  @offline expression `{online} == "no"`
  handle @offline {
    redir https://udance.statuspage.io temporary
  }

  @split {
    expression `{online} == "split"`
    not remote_ip 10.1.1.0/24 10.1.2.0/24
  }
  handle @split {
    redir https://udance.statuspage.io temporary
  }

  reverse_proxy {backend}
}


*.udance.com.au {

  encode gzip zstd
  import logging udance.com.au

  map {labels.3} {backend} {online} {mtls} {phpmyadmin} {

#   HOSTNAME     BACKEND         ONLINE mTLS PHPMYADMIN #COMMENT
#---------------------------------------------------------------

    # Docker containers

    office       10.1.1.13:8880  yes    no   no         # OnlyOffice
    portainer    10.1.1.13:9000  yes    no   no         # Portainer
    truecommand  10.1.1.13:8080  yes    no   no         # TrueCommand
    tc123        10.1.1.13:8082  yes    no   no         # TrueCommand v1.2.3
    nc-fpm       10.1.1.13:8031  yes    no   no         # Nextcloud+Caddy
    wordpress    10.1.1.13:5050  yes    no   no         # WordPress
    nc-apache    10.1.1.13:8030  yes    no   no         # Nextcloud+Apache
    collabora    10.1.1.13:9980  yes    no   no         # Collabora

    # Jails

    rslsync      10.1.1.22:8888  yes    no   no         # Resilio Sync
    cloud        10.1.1.29:80    yes    no   no         # Nextcloud
    heimdall     10.1.1.23:80    yes    no   no         # Heimdall
    blog         10.1.1.54:80    yes    no   yes        # blog.udance.com.au
    test         10.1.1.50:443   yes    yes  yes        # test.udance.com.au
    basil        10.1.1.56:80    yes    no   yes        # basil.udance.com.au
    sachika      10.1.1.57:80    yes    no   yes        # sachika.udance.com.au
    default      unknown         yes    no   no         # subdomain does not exist
  }

# Error handling

  @unknown expression `{backend} == "unknown"`
  handle @unknown {
    respond "Denied" 403
  }

# Site offline

  @offline expression `{online} == "no"`
  handle @offline {
    redir https://udance.statuspage.io temporary
  }

  @split {
    expression `{online} == "split"`
    not remote_ip 10.1.1.0/24 10.1.2.0/24
  }
  handle @split {
    redir https://udance.statuspage.io temporary
  }

# Authenticate phpMyAdmin on production WordPress sites

  @phpmyadmin expression `{phpmyadmin} == "yes"`
  handle @phpmyadmin {
    import authorise /phpmyadmin*
  }

# Fix when using the Nextcloud+Apache Docker image with Caddy.

  @nc-apache host nc-apache.udance.com.au
  handle @nc-apache {
    redir /.well-known/carddav /remote.php/carddav 301
    redir /.well-known/caldav /remote.php/caldav 301
  }

# Enable HSTS for Nextcloud

  @hsts host cloud.udance.com.au
  handle @hsts {
    header {
      Strict-Transport-Security max-age=31536000;
    }
  }

# Secure backend communication

  route {
    @mtls expression `{mtls} == "yes"`
    handle @mtls {
      reverse_proxy {backend} {
        header_up Host {http.reverse_proxy.upstream.hostport}
        header_up X-Forwarded-Host {host}
        transport http {
          tls
        }
      }
    }
  }

# Unsecured backend communication

  route {
    @nomtls expression `{mtls} == "no"`
    handle @nomtls {
      reverse_proxy {backend}
    }
  }
}

francislavoie · May 4, 2021, 8:30pm

Ah, it’s cause handle is mutually exclusive, so since you’re using that all over the place beforehand, it only ever runs the first matching handle block. Most of those should be changed to route instead probably since you do want them to fall through (like the ones for headers and basicauth). Specifically I think cause you set phpmyadmin to yes for that domain, it hits that handle then never runs the one with mtls.

Also I think you probably want to use the actual domain instead of the IP address for that one (and its domain needs to resolve to that IP address with your internal DNS server). That’s probably why it doesn’t work – your backend Caddy can’t complete the TLS handshake because it’s expecting to see a domain in the handshake but you’re just giving it an IP address and it doesn’t have a certificate for that IP address. I think. But check your logs.

basil · May 4, 2021, 8:47pm

Please correct me if I’m wrong, but I’m interpreting this as 'add a route wrapper for all the surrounding handle checks`.

Oh, I see.

I assume in the map table?

I haven’t yet configured the backend. Shall I do that first and test before updating the map table with the domain name?

Time flies when you’re having fun. It’s almost 5 AM and I need to get some sleep. I’ll report back later on today.

francislavoie · May 4, 2021, 9:00pm

No I mean literally change the word handle with route for any of the blocks that you want to actually fall through to another one after being matched. The handle directive is specifically designed to not run other handle blocks after the first matching one is run.

Yeah.

Oh, well the proxy obviously won’t work then. It won’t be able to connect over TLS if the thing on the other end isn’t expecting that.

basil · May 6, 2021, 12:45am

So, before proceeding with backend configuration, I thought it useful to take stock and highlight key changes in my environment relating to mTLS configuration.

Local DNS additions

acme.lan → 10.1.1.4
test.lan → 10.1.1.50

Relevant frontend Caddyfile mTLS constructs

...
# ACME server
acme.lan {
  acme_server
  tls internal
}
...
*.udance.com.au {
...
  map {labels.3} {backend} {online} {mtls} {phpmyadmin} {

#   HOSTNAME     BACKEND         ONLINE mTLS PHPMYADMIN #COMMENT
#---------------------------------------------------------------
...
    test         test.lan:443    yes    yes  yes        # test.udance.com.au
...
# Secure backend communication

  @mtls expression `{mtls} == "yes"`
  route @mtls {
    reverse_proxy {backend} {
      header_up Host {http.reverse_proxy.upstream.hostport}
      header_up X-Forwarded-Host {host}
      transport http {
        tls
      }
    }
  }

The next stop is backend configuration, however, before proceeding, I’d like to review my understanding of the route vs handle directives. A lot has happened in the last few posts that spun me out. A separate post to follow…

basil · May 6, 2021, 1:29am

This was one of those lightbulb moments for me. I had read @matt’s excellent wiki article Composing in the Caddyfile many times over and I’ll continue to read it over and over again I’m sure. There’s so much to take in. Sometimes though, saying things differently can make a world of difference. This was one of those times for me.

I’ve gone through my Caddyfile with a fine-tooth comb reviewing places where I used the handle directive. I’d like to check off where I’ve replaced this with the route directive. This will either confirm for me that I understand the differences in these two directives, or, that I still need to reshape and deepen my understanding.

(online) {
  @offline expression `"{args.0}" == "no"`
  handle @offline {
    redir https://udance.statuspage.io temporary
  }

  @split {
    expression `"{args.0}" == "split"`
    not remote_ip 10.1.1.0/24 10.1.2.0/24
  }
  route @split {
    redir https://udance.statuspage.io temporary
  }
}

In the above snippet, I used the handle directive with the @offline matcher because I figured that if there is a match here, I won’t need to run any other handle blocks after this one. On the other hand, I used the route directive with the @split matcher. If there’s a match, then for the external network, the redir is invoked. For the internal network, however, I might still want to fall through to other handle blocks outside this snippet if this snippet happens to be invoked. Is this logic sound?

I applied similar logic to other handle blocks within the wildcard subdomain Caddy block…

*.udance.com.au {
...
  map {labels.3} {backend} {online} {mtls} {phpmyadmin} {

#   HOSTNAME     BACKEND         ONLINE mTLS PHPMYADMIN #COMMENT
#---------------------------------------------------------------
...
    test         test.lan:443    yes    yes  yes        # test.udance.com.au
    basil        10.1.1.56:80    yes    no   yes        # basil.udance.com.au
    sachika      10.1.1.57:80    yes    no   yes        # sachika.udance.com.au
    default      unknown         yes    no   no         # subdomain does not exist
  }

# Error handling

  @unknown expression `{backend} == "unknown"`
  handle @unknown {
    respond "Denied" 403
  }

# Site offline

  @offline expression `{online} == "no"`
  handle @offline {
    redir https://udance.statuspage.io temporary
  }

  @split {
    expression `{online} == "split"`
    not remote_ip 10.1.1.0/24 10.1.2.0/24
  }
  route @split {
    redir https://udance.statuspage.io temporary
  }

# Authenticate phpMyAdmin on production WordPress sites

  @phpmyadmin expression `{phpmyadmin} == "yes"`
  route @phpmyadmin {
    import authorise /phpmyadmin*
  }

# Fix when using the Nextcloud+Apache Docker image with Caddy.

  @nc-apache host nc-apache.udance.com.au
  route @nc-apache {
    redir /.well-known/carddav /remote.php/carddav 301
    redir /.well-known/caldav /remote.php/caldav 301
  }

# Enable HSTS for Nextcloud

  @hsts host cloud.udance.com.au
  route @hsts {
    header {
      Strict-Transport-Security max-age=31536000;
    }
  }

# Secure backend communication

  @mtls expression `{mtls} == "yes"`
  route @mtls {
    reverse_proxy {backend} {
      header_up Host {http.reverse_proxy.upstream.hostport}
      header_up X-Forwarded-Host {host}
      transport http {
        tls
      }
    }
  }

# Unsecured backend communication

  @nomtls expression `{mtls} == "no"`
  route @nomtls {
    reverse_proxy {backend}
  }
}
...

francislavoie · May 6, 2021, 1:57am

Redirects are always terminal (as in nothing runs after that) because they straight up write a response right away. Same with file_server, respond, reverse_proxy, etc.

In that case, since redir is ordered high in the directive order, you can remove the route/handle wrapping them and put the matcher on redir itself. So like:

(online) {
  @offline expression `"{args.0}" == "no"`
  redir @offline https://udance.statuspage.io temporary

  @split {
    expression `"{args.0}" == "split"`
    not remote_ip 10.1.1.0/24 10.1.2.0/24
  }
  redir @split https://udance.statuspage.io temporary
}

And you can also reduce the HSTS one to:

  @hsts host cloud.udance.com.au
  header @hsts Strict-Transport-Security max-age=31536000;

The only ones that you had that were problematic were basicauth (i.e. import authorise), header for HSTS (improved with the above), and the @nc-apache (because it only matches two specific paths and then should fall through for anything else, route is correct there). The rest can be handle.

basil · May 6, 2021, 3:04am

Ahh…so, in a sense, directives like redir, etc…that are terminal trump handle and route.

francislavoie · May 6, 2021, 3:33am

Uh, I wouldn’t really word it that way, but sure

Anyways if what I said was unclear, from your current config I’d write it like this:

(authorise) {
  basicauth {args.0} {
    admin [REDACTED]
  }
}

*.udance.com.au {

  map {labels.3} {backend} {online} {mtls} {phpmyadmin} {

#   HOSTNAME     BACKEND         ONLINE mTLS PHPMYADMIN #COMMENT
#---------------------------------------------------------------

    test         test.lan:443    yes    yes  yes        # test.udance.com.au
    basil        10.1.1.56:80    yes    no   yes        # basil.udance.com.au
    sachika      10.1.1.57:80    yes    no   yes        # sachika.udance.com.au
    default      unknown         yes    no   no         # subdomain does not exist
  }

# Error handling
  @unknown expression `{backend} == "unknown"`
  handle @unknown {
    respond "Denied" 403
  }

# Site offline
  @offline expression `{online} == "no"`
  redir @offline https://udance.statuspage.io temporary

  @split {
    expression `{online} == "split"`
    not remote_ip 10.1.1.0/24 10.1.2.0/24
  }
  redir @split https://udance.statuspage.io temporary

# Authenticate phpMyAdmin on production WordPress sites
  @phpmyadmin expression `{phpmyadmin} == "yes"`
  route @phpmyadmin {
    import authorise /phpmyadmin*
  }

# Fix when using the Nextcloud+Apache Docker image with Caddy.
  @nc-apache host nc-apache.udance.com.au
  route @nc-apache {
    redir /.well-known/carddav /remote.php/carddav permanent
    redir /.well-known/caldav /remote.php/caldav permanent
  }

# Enable HSTS for Nextcloud
  @hsts host cloud.udance.com.au
  header @hsts "Strict-Transport-Security max-age=31536000;"

# Secure backend communication
  @mtls expression `{mtls} == "yes"`
  handle @mtls {
    reverse_proxy {backend} {
      header_up Host {http.reverse_proxy.upstream.hostport}
      header_up X-Forwarded-Host {host}
      transport http {
        tls
      }
    }
  }

# Unsecured backend communication
  @nomtls expression `{mtls} == "no"`
  handle @nomtls {
    reverse_proxy {backend}
  }
}

basil · May 6, 2021, 4:02am

No, you were very clear, but several related questions have sprung up…

As reverse_proxy was used here…

…could I have changed this to…

# Unsecured backend communication
  @nomtls expression `{mtls} == "no"`
  reverse_proxy @nomtls {backend}

…and as respond was used here…

…could this have been reduced as well to…

# Error handling
  @unknown expression `{backend} == "unknown"`
  respond @unknown "Denied" 403

Also, if I move the handle block below further up above other handle blocks, or, happen to add a new handle block after this one, would I need to change the handle directive to route so that if there’s a match, it runs through to the next handle block?

francislavoie:

# Secure backend communication
  @mtls expression `{mtls} == "yes"`
  handle @mtls {
    reverse_proxy {backend} {
      header_up Host {http.reverse_proxy.upstream.hostport}
      header_up X-Forwarded-Host {host}
      transport http {
        tls
      }
    }
  }

…or, because I’m using reverse_proxy, which is terminal, to avoid any ambiguity, would this work?

# Secure backend communication
  @mtls expression `{mtls} == "yes"`
  reverse_proxy @mtls {backend} {
    header_up Host {http.reverse_proxy.upstream.hostport}
    header_up X-Forwarded-Host {host}
    transport http {
      tls
    }
  }

francislavoie · May 6, 2021, 4:23am

If you unwrap those, then you lose the benefits of handle's mutual exclusivity and directive sorting order (which is guaranteed to be in the order they are written in the Caddyfile in relation to eachother) and the directives like respond and reverse_proxy would get sorted according to their directive order. The effect of this would be that respond @unknown "Denied" 403 would happen after the redir and header and basicauth which is probably not correct (doesn’t make sense to trigger auth when you’re going to respond with a 403 for example).

That said, I realized I made a mistake, because route is ordered after handle, so all the route blocks will get put at the end (you can see this in the adapt output).

The dumb/easy fix it to just wrap the whole thing in a single top-level route to tell the Caddyfile adapter “no, do it in this order” but that’s not so nice but in that case you don’t need some of the handle anymore because you can safely place directives in your Caddyfile in the right order by hand.

Ultimately, you’re writing a super complex config here, so it’s pretty tricky to get right (cause you have multiple paths through which a single request could go because you have multiple variables at play).

So this should work I guess:

(authorise) {
  basicauth {args.0} {
    admin [REDACTED]
  }
}

*.udance.com.au {

  map {labels.3} {backend} {online} {mtls} {phpmyadmin} {

#   HOSTNAME     BACKEND         ONLINE mTLS PHPMYADMIN #COMMENT
#---------------------------------------------------------------

    test         test.lan:443    yes    yes  yes        # test.udance.com.au
    basil        10.1.1.56:80    yes    no   yes        # basil.udance.com.au
    sachika      10.1.1.57:80    yes    no   yes        # sachika.udance.com.au
    default      unknown         yes    no   no         # subdomain does not exist
  }

  route {
# Error handling
    @unknown expression `{backend} == "unknown"`
    respond @unknown "Denied" 403

# Site offline
    @offline expression `{online} == "no"`
    redir @offline https://udance.statuspage.io temporary

    @split {
      expression `{online} == "split"`
      not remote_ip 10.1.1.0/24 10.1.2.0/24
    }
    redir @split https://udance.statuspage.io temporary

# Authenticate phpMyAdmin on production WordPress sites
    @phpmyadmin expression `{phpmyadmin} == "yes"`
    route @phpmyadmin {
      import authorise /phpmyadmin*
    }

# Fix when using the Nextcloud+Apache Docker image with Caddy.
    @nc-apache host nc-apache.udance.com.au
    route @nc-apache {
      redir /.well-known/carddav /remote.php/carddav permanent
      redir /.well-known/caldav /remote.php/caldav permanent
    }

# Enable HSTS for Nextcloud
    @hsts host cloud.udance.com.au
    header @hsts "Strict-Transport-Security max-age=31536000;"

# Secure backend communication
    @mtls expression `{mtls} == "yes"`
    reverse_proxy @mtls {backend} {
      header_up Host {http.reverse_proxy.upstream.hostport}
      header_up X-Forwarded-Host {host}
      transport http {
        tls
      }
    }

# Unsecured backend communication
    @nomtls expression `{mtls} == "no"`
    reverse_proxy @nomtls {backend}

  }
}

basil · May 6, 2021, 4:43am

Ah hah…this is the routing first approach @matt refers to in said wiki article! @francislavoie this is all good stuff as it’s helping me cement my understanding.

… but logical and straightforward enough that mere mortals like myself can follow.

I know I’m pushing the envelope with Caddyfile, and I haven’t (yet, but could be coming soon!) hit any limitations using it, but I just love its simplicity with all the complexity handled under the hood.

I’ll test this out and report back later this evening (local time). Thank you for your help getting me to this point.

Whitestrake · May 6, 2021, 7:01am

There are those that are less than in love with the Caddyfile, but the truth is there are very few things it can’t do.

The place where it can become onerous is when the complexity of a given configuration (such as yours) starts to expose nuances in the abstraction. Specifically, we begin to require a high level of understanding of how and why the Caddyfile maps out into Caddy’s “real” config (JSON) in order to write it properly, at which point some see it as more pragmatic to simply write directly in JSON.

But, in truth, the Caddyfile can handle this, which is pretty cool.

basil · May 7, 2021, 3:44am

The top-level route wrapper worked a treat. Adopting a routing-first approach was the way to go in this case. Interestingly, all the handle blocks/wrapping have disappeared from my Caddyfile making it even more streamlined than it was before, while improving its overall readability as well.

Time to tinker with the backend.

EDIT: I had to remove the quotes from the header line…

# Enable HSTS for Nextcloud
    @hsts host cloud.udance.com.au
    header @hsts "Strict-Transport-Security max-age=31536000;"

Changed to…

# Enable HSTS for Nextcloud
    @hsts host cloud.udance.com.au
    header @hsts Strict-Transport-Security max-age=31536000;

Leaving them in resulted in Nextcloud complaining about Strict Transport Security and, curiously, also broke the connection between Nextcloud and OnlyOffice.

francislavoie · May 8, 2021, 4:57am

Derp, I put the quotes around the wrong thing, it was meant to be:

header @hsts Strict-Transport-Security "max-age=31536000;"

I just prefer to include the quotes around header values in situation where it could eventually have spaces in the value, because if you have a 3rd parameter (i.e. third Caddyfile token, separated by spaces), it becomes a header replacement operation which is rarely the intended effect.

basil · May 8, 2021, 5:03am

Cool, that makes sense. I’ll adopt that practice.

basil · May 9, 2021, 3:19pm

My working backend Caddyfile under Caddy 2.3.0 before adding in the mTLS constructs:

:80 {
  root * /usr/local/www/wordpress
  encode gzip
  php_fastcgi 127.0.0.1:9000 {
    env SERVER_PORT 80
  }
  file_server

  log {
    format json 
    output file /var/log/access.log {
      roll_keep 7
    }
  }

  # External access denied to these files.
  @forbidden {
    path /wp-content/uploads/*.php
    path /wp-includes/*.php
    path /wp-config.php
    path /.user.ini
    path /wp-content/debug.log
  }
  respond @forbidden 404
}

I then followed the instructions under the section Backend in the wiki article Use Caddy for local HTTPS (TLS) between front-end reverse proxy and LAN hosts . My backend Caddyfile under Caddy 2.4.0-RC.1 including mTLS constructs.

{
  log {
    format json {
      time_format iso8601
    }
  }
#  debug

# TLS Options
  acme_ca https://acme.lan/acme/local/directory # point to ACME server
  acme_ca_root /etc/ssl/certs/root.crt  # define root certificate
}

test.lan {
  root * /usr/local/www/wordpress
  encode gzip
  php_fastcgi 127.0.0.1:9000 {
    env SERVER_PORT 80
  }
  file_server

  log {
    format json {
      time_format iso8601
    }
    output file /var/log/access.log {
      roll_keep 7
    }
  }

  # External access denied to these files.
  @forbidden {
    path /wp-content/uploads/*.php
    path /wp-includes/*.php
    path /wp-config.php
    path /.user.ini
    path /wp-content/debug.log
  }
  respond @forbidden 404
}

After restarting both frontend and backend Caddy services, I attempt to access the test site. Unfortunately, I get a 502 error.

Where to from here?
Here’s an extract of the frontend Caddy log post-restart.

{"level":"info","ts":"2021-05-09T22:55:21.393+0800","msg":"shutting down apps, then terminating","signal":"SIGTERM"}
{"level":"warn","ts":"2021-05-09T22:55:21.393+0800","msg":"exiting; byeee!! 👋","signal":"SIGTERM"}
{"level":"info","ts":"2021-05-09T22:55:21.950+0800","logger":"tls.cache.maintenance","msg":"stopped background certificate maintenance","cache":"0xc000344930"}
{"level":"debug","ts":"2021-05-09T22:55:21.950+0800","logger":"http.handlers.acme_server","msg":"unloading unused CA database","db_key":"local"}
{"level":"info","ts":"2021-05-09T22:55:21.952+0800","logger":"admin","msg":"stopped previous server","address":"tcp/localhost:2019"}
{"level":"info","ts":"2021-05-09T22:55:21.952+0800","msg":"shutdown complete","signal":"SIGTERM","exit_code":0}
{"level":"info","ts":1620572122.0956097,"msg":"using provided configuration","config_file":"/usr/local/www/Caddyfile","config_adapter":"caddyfile"}
{"level":"warn","ts":1620572122.1175373,"msg":"input is not formatted with 'caddy fmt'","adapter":"caddyfile","file":"/usr/local/www/Caddyfile","line":2}
{"level":"info","ts":"2021-05-09T22:55:22.125+0800","logger":"admin","msg":"admin endpoint started","address":"tcp/localhost:2019","enforce_origin":false,"origins":["localhost:2019","[::1]:2019","127.0.0.1:2019"]}
{"level":"info","ts":"2021-05-09T22:55:22.126+0800","logger":"tls.cache.maintenance","msg":"started background certificate maintenance","cache":"0xc00033e850"}
{"level":"info","ts":"2021-05-09T22:55:22.155+0800","logger":"http","msg":"server is listening only on the HTTPS port but has no TLS connection policies; adding one to enable TLS","server_name":"srv0","https_port":443}
{"level":"info","ts":"2021-05-09T22:55:22.155+0800","logger":"http","msg":"enabling automatic HTTP->HTTPS redirects","server_name":"srv0"}
{"level":"warn","ts":"2021-05-09T22:55:35.176+0800","logger":"pki.ca.local","msg":"installing root certificate (you might be prompted for password)","path":"storage:pki/authorities/local/root.crt"}
2021/05/09 22:55:35 Note: NSS support is not available on your platform
2021/05/09 22:55:35 define JAVA_HOME environment variable to use the Java trust
{"level":"error","ts":"2021-05-09T22:55:35.176+0800","logger":"pki.ca.local","msg":"failed to install root
certificate","error":"trust not supported","certificate_file":"storage:pki/authorities/local/root.crt"}
{"level":"info","ts":"2021-05-09T22:55:35.176+0800","logger":"tls","msg":"cleaning storage unit","description":"FileStorage:/.local/share/caddy"}
{"level":"info","ts":"2021-05-09T22:55:35.176+0800","logger":"http","msg":"enabling automatic TLS certificate management","domains":["www.xenografix.com.au","*.udance.com.au","acme.lan","caffigoalkeeping.com.au","www.caffigoalkeeping.com","caffigoalkeeping.com","www.caffigoalkeeping.com.au","readymcgetty.com.au","www.readymcgetty.com.au","xenografix.com.au","udance.com.au","www.udance.com.au"]}
{"level":"warn","ts":"2021-05-09T22:55:35.191+0800","logger":"tls","msg":"stapling OCSP","error":"no OCSP stapling for [acme.lan]: no OCSP server specified in certificate"}
{"level":"info","ts":"2021-05-09T22:55:35.191+0800","logger":"tls","msg":"finished cleaning storage units"}
{"level":"info","ts":"2021-05-09T22:55:35.191+0800","msg":"autosaved config (load with --resume flag)","file":"/.config/caddy/autosave.json"}
{"level":"info","ts":"2021-05-09T22:55:35.191+0800","msg":"serving initial configuration"}
Successfully started Caddy (pid=77501) - Caddy is running in the background

…and an extract from the backend Caddy log post-restart:

{"level":"info","ts":"2021-05-09T22:55:57.526+0800","msg":"shutting down apps, then terminating","signal":"SIGTERM"}
{"level":"warn","ts":"2021-05-09T22:55:57.526+0800","msg":"exiting; byeee!! 👋","signal":"SIGTERM"}
{"level":"info","ts":"2021-05-09T22:55:57.529+0800","logger":"tls.cache.maintenance","msg":"stopped background certificate maintenance","cache":"0xc0002c6d90"}
{"level":"info","ts":"2021-05-09T22:55:57.531+0800","logger":"admin","msg":"stopped previous server","address":"tcp/localhost:2019"}
{"level":"info","ts":"2021-05-09T22:55:57.531+0800","msg":"shutdown complete","signal":"SIGTERM","exit_code":0}
{"level":"info","ts":1620572157.6563253,"msg":"using provided configuration","config_file":"/usr/local/www/Caddyfile","config_adapter":"caddyfile"}
{"level":"warn","ts":1620572157.6602314,"msg":"input is not formatted with 'caddy fmt'","adapter":"caddyfile","file":"/usr/local/www/Caddyfile","line":2}
{"level":"info","ts":"2021-05-09T22:55:57.662+0800","logger":"admin","msg":"admin endpoint started","address":"tcp/localhost:2019","enforce_origin":false,"origins":["localhost:2019","[::1]:2019","127.0.0.1:2019"]}
{"level":"info","ts":"2021-05-09T22:55:57.663+0800","logger":"tls.cache.maintenance","msg":"started backgroundcertificate maintenance","cache":"0xc0002de380"}
{"level":"info","ts":"2021-05-09T22:55:57.663+0800","logger":"http","msg":"server is listening only on theHTTPS port but has no TLS connection policies; adding one to enable TLS","server_name":"srv0","https_port":443}
{"level":"info","ts":"2021-05-09T22:55:57.664+0800","logger":"http","msg":"enabling automatic HTTP->HTTPS redirects","server_name":"srv0"}
{"level":"info","ts":"2021-05-09T22:55:57.665+0800","logger":"http","msg":"enabling automatic TLS certificate management","domains":["test.lan"]}
{"level":"info","ts":"2021-05-09T22:55:57.665+0800","logger":"tls","msg":"cleaning storage unit","description":"FileStorage:/.local/share/caddy"}
{"level":"info","ts":"2021-05-09T22:55:57.667+0800","logger":"tls","msg":"finished cleaning storage units"}
{"level":"warn","ts":"2021-05-09T22:55:57.688+0800","logger":"tls","msg":"stapling OCSP","error":"no OCSP stapling for [test.lan]: no OCSP server specified in certificate"}
{"level":"info","ts":"2021-05-09T22:55:57.689+0800","msg":"autosaved config (load with --resume flag)","file":"/.config/caddy/autosave.json"}
{"level":"info","ts":"2021-05-09T22:55:57.689+0800","msg":"serving initial configuration"}
Successfully started Caddy (pid=80607) - Caddy is running in the background

There’s an error about OCSP stapling in both Caddy logs, but I’m not sure what this means. Help! Please!

Rob789 · May 9, 2021, 4:49pm

Hi Basil,

This specific OCSP error can be ignored, I have it too in a working setup. Explaination

basil:

{"level":"error","ts":"2021-05-09T22:55:35.176+0800","logger":"pki.ca.local","msg":"failed to install root
certificate","error":"trust not supported","certificate_file":"storage:pki/authorities/local/root.crt"}

This is something I don’t remember seeing. Looks like something is wrong with the certificate. COuld it be something with user rights?

Edit: or the content of the certificate is wrong?

basil · May 9, 2021, 5:28pm

Could be, but I’m not sure.

root@caddy:/.local/share/caddy/pki/authorities/local # ls -l
total 26
-rw-------  1 root  wheel  680 May  8 04:00 intermediate.crt
-rw-------  1 root  wheel  227 May  8 04:00 intermediate.key
-rw-------  1 root  wheel  627 May  2 13:35 root.crt
-rw-------  1 root  wheel  227 May  2 13:35 root.key

I wonder if it has something to do with the acme server and the FreeBSD platform. The message first appears when I enable the acme server. See post #2 in this thread.

francislavoie · May 9, 2021, 5:47pm

There must be more logs after that for both of them, i.e. doing the ACME on the backend for that domain. That looks like you only got the logs from the restart and that’s it. Need to see the logs after it started running too, because some of this stuff doesn’t happen right in the startup phase, but after it.

basil:

{"level":"error","ts":"2021-05-09T22:55:35.176+0800","logger":"pki.ca.local","msg":"failed to install root
certificate","error":"trust not supported","certificate_file":"storage:pki/authorities/local/root.crt"}

Note sure why there’s a newline in there, but anyways I don’t think this is a problem, this just means Caddy wasn’t able to install the root certificate into the trust store of that system which might be fine.

But you could run caddy trust manually to make sure it was done right anyways (make sure to run that command such that Caddy knows that its storage location is /var/lib/caddy, i.e. the $HOME of the caddy user when running as a service). Caddy needs sudo/root access to install that which is the trouble.

Ignore that, not a problem. It’s just a warning. OCSP is not required, but strongly recommended, but isn’t really viable for mTLS. Basically it’s supposed to be a URL that Caddy can send a request to, to see if the certificate is still valid (i.e. hasn’t been revoked for any number of reasons). It’s decently important but public trust but for private it doesn’t matter so much.