Yeah there’s the bug. So it’s a problem with the Caddyfile adapter.
So, a newer build should allow me to move forward? Should I wait till 2.4.0 is officially released? I am working with some client systems (other domains) as well and I’m just a little concerned about using a Caddy beta with those systems, or am I unnecessarily worried?
I just tried to replicate the issue on latest master and it’s still broken unfortunately. I’ll see if I can write a quick fix, but we’ve had a lot of churn with this part of the code so we’ll see.
Don’t stress. I’ll roll back for the moment. At least the issue has been flagged.
@francislavoie Thanks for the rapid follow-up. I’ve been thinking it may be possible for me to keep the 2.3.0 version of Caddy I’m currently using and set up a new instance of the master that I can rapidly switch between for testing. This should minimise any disruption to other production systems, though, I won’t have a rig set up until at least later tonight (seven hours from now) or tomorrow.
I’m pleased to say that the patch at post #25 has fixed the issue identified at post #16. Also, all other production domains seem to be working well under the 2.4.0 beta. The udance subdomains are now using a wildcard certificate. The working subdomain Caddy block is reproduced below.
*.udance.com.au {
map {labels.3} {upstream} {
rslsync 10.1.1.22:8888 # Resilio Sync
cloud 10.1.1.29:80 # Nextcloud
heimdall 10.1.1.23:80 # Heimdall
blog 10.1.1.54:80 # blog.udance.com.au
test 10.1.1.50:80 # test.udance.com.au
basil 10.1.1.56:80 # basil.udance.com.au
sachika 10.1.1.57:80 # sachika.udance.com.au
# www 10.1.1.55:80 # www.udance.com.au
# default 10.1.1.55:80 # udance.com.au
}
encode gzip
import tlsdns
import authproxy /phpmyadmin*
import logging udance
reverse_proxy {upstream}
}
The final step of the design calls for the udance domain Caddy block and its subdomains Caddy block to be merged into a single Caddy block.
1 Like
This is the Caddyfile with the domain+www and subdomain blocks merged.
*.udance.com.au, udance.com.au {
map {labels.3} {upstream} {
rslsync 10.1.1.22:8888 # Resilio Sync
cloud 10.1.1.29:80 # Nextcloud
heimdall 10.1.1.23:80 # Heimdall
blog 10.1.1.54:80 # blog.udance.com.au
test 10.1.1.50:80 # test.udance.com.au
basil 10.1.1.56:80 # basil.udance.com.au
sachika 10.1.1.57:80 # sachika.udance.com.au
www 10.1.1.55:80 # www.udance.com.au
default 10.1.1.55:80 # udance.com.au
}
encode gzip
import tlsdns
import authproxy /phpmyadmin*
import logging udance
@udance host udance.com.au, www.udance.com.au
handle @udance {
reverse_proxy /tautulli* 10.1.1.26:8181
reverse_proxy /transmission* 10.1.1.28:9091
}
reverse_proxy {upstream}
}
The udance domain and www subdomain are working, however, the subdirectories are not. Entering udance.com.au/tautulli or udance.com.au/transmission in the address bar returns a ‘Page not found error’ from the udance WordPress site. This suggests that there’s a problem around the handle block. I’ve not used a handle block before, so it’s probably a logical error on my part somewhere. I’ve included an extract from caddy adapt, which might reveal some clues (I’m still learning to find my way around this).
{
"match": [
{
"host": [
"*.udance.com.au",
"udance.com.au"
]
}
],
"handle": [
{
"handler": "subroute",
"routes": [
{
"handle": [
{
"defaults": [
"10.1.1.55:80"
],
"destinations": [
"{upstream}"
],
"handler": "map",
"mappings": [
{
"input": "rslsync",
"outputs": [
"10.1.1.22:8888"
]
},
{
"input": "cloud",
"outputs": [
"10.1.1.29:80"
]
},
{
"input": "heimdall",
"outputs": [
"10.1.1.23:80"
]
},
{
"input": "blog",
"outputs": [
"10.1.1.54:80"
]
},
{
"input": "test",
"outputs": [
"10.1.1.50:80"
]
},
{
"input": "basil",
"outputs": [
"10.1.1.56:80"
]
},
{
"input": "sachika",
"outputs": [
"10.1.1.57:80"
]
},
{
"input": "www",
"outputs": [
"10.1.1.55:80"
]
}
],
"source": "{http.request.host.labels.3}"
}
]
},
{
"handle": [
{
"handler": "authentication",
"providers": {
"http_basic": {
"accounts": [
{
"password": [REDACTED],
"username": "admin"
}
],
"hash": {
"algorithm": "bcrypt"
},
"hash_cache": {}
}
}
}
],
"match": [
{
"path": [
"/phpmyadmin*"
]
}
]
},
{
"handle": [
{
"encodings": {
"gzip": {}
},
"handler": "encode"
}
]
},
{
"handle": [
{
"handler": "subroute",
"routes": [
{
"handle": [
{
"handler": "reverse_proxy",
"upstreams": [
{
"dial": "10.1.1.28:9091"
}
]
}
],
"match": [
{
"path": [
"/transmission*"
]
}
]
},
{
"handle": [
{
"handler": "reverse_proxy",
"upstreams": [
{
"dial": "10.1.1.26:8181"
}
]
}
],
"match": [
{
"host": [
"udance.com.au,",
"www.udance.com.au"
]
}
]
},
{
"handle": [
{
"handler": "reverse_proxy",
"upstreams": [
{
"dial": "{upstream}"
}
]
}
]
}
]
}
],
"terminal": true
}
],
Turn on the debug global option, then make a request. What do you see in your logs? Run journalctl -u caddy --no-pager | less to see.
Apologies for the delay. journalctl doesn’t appear to be a command available under FreeBSD. I’ve asked for an equivalent command in the FreeNAS forum. The lines are really long in the logs and I’m not sure how to present them at this stage. Still checking.
Necessity is the mother of invention
Still no response on the FreeNAS forum regarding journalctl, so, this is what I did. I deleted the Caddy log; restarted Caddy; issued the request udance.com.au/tautulli; took a snapshot of the Caddy log; isolated the lines with tautulli in them; split the (four) lines so they could be copied and this is the result.
{"level":"debug","ts":1614884735.433205,"logger":"http.handlers.reverse_proxy","msg":"upstream roundtrip","upstream":"{upstream}","request":{"remote_addr":"10.1.1.136:51911","proto":"HTTP/2.0","method":"GET","host":"udance.com.au","uri":"/tautulli","headers":{"Upgrade-Insecure-Requests"
:["1"],"Accept":["text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.9"],"Sec-Fetch-User":["?1"],"Accept-Encoding":["gzip, deflate, br"],"Accept-Language":["en-GB,en-US;q=0.9,en;q=0.8"],"Cookie":["tk_or=%
22%22; tk_lr=%22%22; wordpress_logged_in_01ad53d5bf4f84d52a16c29bd439eb90=basil%7C1645606391%7Cg0BlFaRlmpOtRQIufy2y3zkZMaJQ5kBwHgodcQZTCol%7C21fc4a76a8a48a1988b5664d18c3115b251ad355646ec0b0bbf00b4cd0c0aa34; wp-settings-1=libraryContent%3Dbrowse; wp-settings-time-1=1614070400; wfwaf-auth
cookie-d3ec030b92199ac6467163e799082cd3=1%7Cadministrator%7Cb1d638a92a5d5466468d1bae6bc4744a68e7577cf00bb7bd68043c3dbc3f974d; tk_ai=woo%3AzWH9NWshxhnKTLOrnIn7Nbvx; mailchimp_landing_site=https%3A%2F%2Fudance.com.au%2Fwp-admin%2Fadmin.php%3Fpage%3Dstats%26noheader%26proxy%26chart%3Dadmin
-bar-hours-scale"],"X-Forwarded-For":["10.1.1.136"],"X-Forwarded-Proto":["https"],"User-Agent":["Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/88.0.4324.190 Safari/537.36"],"Sec-Fetch-Site":["none"],"Sec-Fetch-Mode":["navigate"],"Sec-Fetch-Dest"
:["document"]},"tls":{"resumed":false,"version":772,"cipher_suite":4865,"proto":"h2","proto_mutual":true,"server_name":"udance.com.au"}},"duration":50.928469734,"headers":{"X-Powered-By":["PHP/7.4.15"],"Date":["Thu, 04 Mar 2021 19:05:35 GMT"],"Cache-Control":["no-cache, must-revalidate,
max-age=0"],"Content-Encoding":["gzip"],"Expires":["Wed, 11 Jan 1984 05:00:00 GMT"],"Link":["<https://udance.com.au/wp-json/>; rel=\"https://api.w.org/\""],"Server":["Caddy"],"Status":["404 Not Found"],"Content-Type":["text/html; charset=UTF-8"],"Vary":["Accept-Encoding, Cookie","Accep
t-Encoding"]},"status":404}
{"level":"error","ts":1614884735.4358041,"logger":"http.log.access.log9","msg":"handled request","request":{"remote_addr":"10.1.1.136:51911","proto":"HTTP/2.0","method":"GET","host":"udance.com.au","uri":"/tautulli","headers":{"Upgrade-Insecure-Requests":["1"],"Accept":["text/html,appli
cation/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.9"],"Sec-Fetch-User":["?1"],"Accept-Encoding":["gzip, deflate, br"],"User-Agent":["Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chro
me/88.0.4324.190 Safari/537.36"],"Sec-Fetch-Site":["none"],"Sec-Fetch-Mode":["navigate"],"Sec-Fetch-Dest":["document"],"Accept-Language":["en-GB,en-US;q=0.9,en;q=0.8"],"Cookie":["tk_or=%22%22; tk_lr=%22%22; wordpress_logged_in_01ad53d5bf4f84d52a16c29bd439eb90=basil%7C1645606391%7Cg0BlFa
RlmpOtRQIufy2y3zkZMaJQ5kBwHgodcQZTCol%7C21fc4a76a8a48a1988b5664d18c3115b251ad355646ec0b0bbf00b4cd0c0aa34; wp-settings-1=libraryContent%3Dbrowse; wp-settings-time-1=1614070400; wfwaf-authcookie-d3ec030b92199ac6467163e799082cd3=1%7Cadministrator%7Cb1d638a92a5d5466468d1bae6bc4744a68e7577cf
00bb7bd68043c3dbc3f974d; tk_ai=woo%3AzWH9NWshxhnKTLOrnIn7Nbvx; mailchimp_landing_site=https%3A%2F%2Fudance.com.au%2Fwp-admin%2Fadmin.php%3Fpage%3Dstats%26noheader%26proxy%26chart%3Dadmin-bar-hours-scale"]},"tls":{"resumed":false,"version":772,"cipher_suite":4865,"proto":"h2","proto_mutu
al":true,"server_name":"udance.com.au"}},"common_log":"10.1.1.136 - - [05/Mar/2021:03:05:35 +0800] \"GET /tautulli HTTP/2.0\" 404 11073","duration":50.931240318,"size":11073,"status":404,"resp_headers":{"Status":["404 Not Found"],"Date":["Thu, 04 Mar 2021 19:05:35 GMT"],"Expires":["Wed,
11 Jan 1984 05:00:00 GMT"],"Server":["Caddy","Caddy"],"X-Powered-By":["PHP/7.4.15"],"Cache-Control":["no-cache, must-revalidate, max-age=0"],"Content-Encoding":["gzip"],"Link":["<https://udance.com.au/wp-json/>; rel=\"https://api.w.org/\""],"Content-Type":["text/html; charset=UTF-8"],"
Vary":["Accept-Encoding, Cookie","Accept-Encoding"]}}
{"level":"debug","ts":1614884741.0708818,"logger":"http.handlers.reverse_proxy","msg":"upstream roundtrip","upstream":"{upstream}","request":{"remote_addr":"10.1.1.136:51911","proto":"HTTP/2.0","method":"POST","host":"udance.com.au","uri":"/?wc-ajax=get_refreshed_fragments","headers":{"
Content-Length":["18"],"User-Agent":["Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/88.0.4324.190 Safari/537.36"],"X-Forwarded-For":["10.1.1.136"],"X-Requested-With":["XMLHttpRequest"],"Sec-Fetch-Dest":["empty"],"Referer":["https://udance.com.au
/tautulli"],"Origin":["https://udance.com.au"],"Sec-Fetch-Mode":["cors"],"Accept-Encoding":["gzip, deflate, br"],"Cookie":["tk_or=%22%22; tk_lr=%22%22; wordpress_logged_in_01ad53d5bf4f84d52a16c29bd439eb90=basil%7C1645606391%7Cg0BlFaRlmpOtRQIufy2y3zkZMaJQ5kBwHgodcQZTCol%7C21fc4a76a8a48a1
988b5664d18c3115b251ad355646ec0b0bbf00b4cd0c0aa34; wp-settings-1=libraryContent%3Dbrowse; wp-settings-time-1=1614070400; wfwaf-authcookie-d3ec030b92199ac6467163e799082cd3=1%7Cadministrator%7Cb1d638a92a5d5466468d1bae6bc4744a68e7577cf00bb7bd68043c3dbc3f974d; tk_ai=woo%3AzWH9NWshxhnKTLOrnI
n7Nbvx; mailchimp_landing_site=https%3A%2F%2Fudance.com.au%2Fwp-admin%2Fadmin.php%3Fpage%3Dstats%26noheader%26proxy%26chart%3Dadmin-bar-hours-scale"],"Accept":["*/*"],"Content-Type":["application/x-www-form-urlencoded; charset=UTF-8"],"Sec-Fetch-Site":["same-origin"],"Accept-Language":[
"en-GB,en-US;q=0.9,en;q=0.8"],"X-Forwarded-Proto":["https"]},"tls":{"resumed":false,"version":772,"cipher_suite":4865,"proto":"h2","proto_mutual":true,"server_name":"udance.com.au"}},"duration":5.009920019,"error":"context canceled"}
{"level":"info","ts":1614884741.0712857,"logger":"http.log.access.log9","msg":"handled request","request":{"remote_addr":"10.1.1.136:51911","proto":"HTTP/2.0","method":"POST","host":"udance.com.au","uri":"/?wc-ajax=get_refreshed_fragments","headers":{"Sec-Fetch-Mode":["cors"],"Accept-En
coding":["gzip, deflate, br"],"Cookie":["tk_or=%22%22; tk_lr=%22%22; wordpress_logged_in_01ad53d5bf4f84d52a16c29bd439eb90=basil%7C1645606391%7Cg0BlFaRlmpOtRQIufy2y3zkZMaJQ5kBwHgodcQZTCol%7C21fc4a76a8a48a1988b5664d18c3115b251ad355646ec0b0bbf00b4cd0c0aa34; wp-settings-1=libraryContent%3Db
rowse; wp-settings-time-1=1614070400; wfwaf-authcookie-d3ec030b92199ac6467163e799082cd3=1%7Cadministrator%7Cb1d638a92a5d5466468d1bae6bc4744a68e7577cf00bb7bd68043c3dbc3f974d; tk_ai=woo%3AzWH9NWshxhnKTLOrnIn7Nbvx; mailchimp_landing_site=https%3A%2F%2Fudance.com.au%2Fwp-admin%2Fadmin.php%3
Fpage%3Dstats%26noheader%26proxy%26chart%3Dadmin-bar-hours-scale"],"Content-Length":["18"],"User-Agent":["Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/88.0.4324.190 Safari/537.36"],"Origin":["https://udance.com.au"],"Sec-Fetch-Site":["same-orig
in"],"Sec-Fetch-Dest":["empty"],"Referer":["https://udance.com.au/tautulli"],"Accept-Language":["en-GB,en-US;q=0.9,en;q=0.8"],"Accept":["*/*"],"X-Requested-With":["XMLHttpRequest"],"Content-Type":["application/x-www-form-urlencoded; charset=UTF-8"]},"tls":{"resumed":false,"version":772,
"cipher_suite":4865,"proto":"h2","proto_mutual":true,"server_name":"udance.com.au"}},"common_log":"10.1.1.136 - - [05/Mar/2021:03:05:41 +0800] \"POST /?wc-ajax=get_refreshed_fragments HTTP/2.0\" 0 0","duration":5.010550976,"size":0,"status":0,"resp_headers":{"Server":["Caddy"]}}
Aha, I see what’s going on, you put a comma in the host matcher:
You can see in the JSON that this produced a bad matcher:
So it didn’t match and fell through to the {upstream} proxy rule.
1 Like
Are you serious!? One comma broke the subdirectories. Technology can be so unforgiving at times. 
I removed the comma and reloaded the Caddyfile and subdirectories burst into life. 
1 Like
So, I started with a Caddyfile excerpt, which used individual certificates and relied on a specific snippet.
(online) {
{args.0}.udance.com.au {
encode gzip
import tlsdns
import authproxy /phpmyadmin*
import logging {args.0}
reverse_proxy http://{args.1}
}
}
www.udance.com.au, udance.com.au {
encode gzip
import tlsdns
import authproxy /phpmyadmin*
import logging udance
reverse_proxy http://10.1.1.55
reverse_proxy /tautulli* http://10.1.1.26:8181
reverse_proxy /transmission* http://10.1.1.28:9091
}
import online rslsync 10.1.1.22:8888 # Resilio Sync
import online cloud 10.1.1.29 # Nextcloud
import online heimdall 10.1.1.23 # Heimdall
import online blog 10.1.1.54 # blog.udance.com.au
import online test 10.1.1.50 # test.udance.com.au
import online basil 10.1.1.56 # basil.udance.com.au
import online sachika 10.1.1.57 # sachika.udance.com.au
The excerpt has been transformed to use a wildcard certificate for the subdomains and the snippet is now defunct.
*.udance.com.au, udance.com.au {
map {labels.3} {upstream} {
rslsync 10.1.1.22:8888 # Resilio Sync
cloud 10.1.1.29:80 # Nextcloud
heimdall 10.1.1.23:80 # Heimdall
blog 10.1.1.54:80 # blog.udance.com.au
test 10.1.1.50:80 # test.udance.com.au
basil 10.1.1.56:80 # basil.udance.com.au
sachika 10.1.1.57:80 # sachika.udance.com.au
www 10.1.1.55:80 # www.udance.com.au
default 10.1.1.55:80 # udance.com.au
}
encode gzip
import tlsdns
import authproxy /phpmyadmin*
import logging udance
@udance host udance.com.au www.udance.com.au
handle @udance {
reverse_proxy /tautulli* 10.1.1.26:8181
reverse_proxy /transmission* 10.1.1.28:9091
}
reverse_proxy {upstream}
}
So what have I learnt along the way…heaps!.. such as:
- The usefulness of wildcard certificates for subdomains;
- Use of directives and constructs I was previously not familiar with including
map,handle,{labels.[ptr]}; -
logsandtlsare at the top level of a site. -
caddy adapt --prettyis a useful debugging tool. - It’s possible to combine a domain and its subdomains in a single Caddy block. This is useful when working with a number of domains.
- Caddy 2.4.0 beta appears to be fine with existing configurations. You may unearth an issue if you push the envelope with it though.
- A comma is optional until it’s not.
Thanks @francislavoie and @matt for supporting me on this journey. It’s been a very useful learning exercise and I’m confident I’ll be able to apply this newfound knowledge to the rest of my Caddyfile.
2 Likes
The only place commas are valid, really, is in the site address, and it’s only actually there to make multi-line site address lists functional. Caddy uses spaces for tokenizing everywhere else.
2 Likes
It was a very poor assumption on my part. Forgive me @matt 
1 Like
During this exercise, I became aware that the map directive feels a little bit like a case construct in other programming languages, and a named matcher serves to act a little bit like an if-then statement. I wonder if new users to Caddy would find these types of analogies useful.
1 Like
The way I’ve used the default switch in the map directive seems to suggest even the domain udance.com.au is using a wildcard certificate. Am I correct in this assertion?
A possible issue with the approach I’ve taken is that when a request is made to an invalid subdomain e.g. notexist.udance.com.au, the browser won’t throw up an error message, but will always redirect to the domain udance.com.au. Should I be overly concerned? I have this vague feeling that this might not be the best practice.
No, it’ll be using its own certificate because *.example.com does not match example.com, and you specified example.com in the site address. There’s no overlap in matched domains here.
Yeah - I would suggest to not use map’s default for your base domain and instead handle the remaining paths in your handle @udance block instead, then that opens you up to do something like default unknown in the map, followed by a matcher like:
@unknown expression `{upstream} == "unknown"`
handle @unknown {
# do whatever
}
Which would catch any subdomains that you didn’t explicitly configure; you could do like respond "Denied" 403 or you could redirect, or whatever you like.
1 Like
In my original Caddyfile, import authproxy /phpmyadmin* was applied across all subdomains, however, it really should have only been applied to WordPress sites within FreeBSD jails. Now, armed with my newfound knowledge, I thought to myself 'I can do better!’.
This was my first attempt, which worked well:
@phpmyadmin host blog.udance.com.au test.udance.com.au basil.udance.com.au sachika.udance.com.au www.udance.com.au udance.com.au
handle @phpmyadmin {
import authproxy /phpmyadmin*
}
The only problem is, the definition of the named matcher was a bit long and would only get longer the more WordPress sites I added. So, here’s my second attempt:
@phpmyadmin {
host blog.udance.com.au test.udance.com.au basil.udance.com.au
host sachika.udance.com.au www.udance.com.au udance.com.au
}
handle @phpmyadmin {
import authproxy /phpmyadmin*
}
This seemed to work when I tested it. The only problem is that I had to remember that if I added a new WordPress site, I’d have to update the map directive and the @phpmyadmin named matcher. It would be quite easy to forget to do the latter.
The solution I settled upon, which addressed this issue relied on extending the map directive by adding a new column:
*.udance.com.au, udance.com.au {
map {labels.3} {upstream} {phpmyadmin} {
# HOSTNAME IPADDRESS PHPMYADMIN
#---------------------------------------------------------------
# Docker containers
office 10.1.1.17:80 no # OnlyOffice
portainer 10.1.1.13:9000 no # Portainer
truecommand 10.1.1.17:8080 no # TrueCommand
tc123 10.1.1.13:8080 no # TrueCommand v1.2.3
nc-fpm 10.1.1.13:8031 no # Nextcloud+Caddy
wordpress 10.1.1.13:5050 no # WordPress
nc-apache 10.1.1.13:8030 no # Nextcloud+Apache
collabora 10.1.1.17:9980 no # Collabora
# Jails
rslsync 10.1.1.22:8888 no # Resilio Sync
cloud 10.1.1.29:80 no # Nextcloud
heimdall 10.1.1.23:80 no # Heimdall
blog 10.1.1.54:80 yes # blog.udance.com.au
test 10.1.1.50:80 yes # test.udance.com.au
basil 10.1.1.56:80 yes # basil.udance.com.au
sachika 10.1.1.57:80 yes # sachika.udance.com.au
www 10.1.1.55:80 yes # www.udance.com.au
default unknown no # subdomain does not exist
}
encode gzip
import tlsdns
import logging udance
@phpmyadmin expression `{phpmyadmin} == "yes"`
handle @phpmyadmin {
import authproxy /phpmyadmin*
}
Btw, I’d forgotten I had a whole bunch of Docker containers attached to subdomains as well. So, the final Caddyfile transformation resulted in 16 individual certificates being replaced with a single wildcard certificate.
This solution worked very well except for the domain udance.com.au. I tried adjusting the named matcher as follows in the hope that an OR would apply to the matchers, but that didn’t work:
@phpmyadmin {
expression `{phpmyadmin} == "yes"`
host udance.com.au
}
handle @phpmyadmin {
import authproxy /phpmyadmin*
}
I’m open to suggestions on how to proceed from here.