custom name: dashy, jumpy - proxmox
other name is exact service
3. The problem I’m having:
Internal DNS is working but reverse proxy is not. I access the domain (https://dashy.custom_domain.tld) to access my proxmox, log doesn’t have any information from journalctl and pfsense (router). I’m not touching SSL certificate yet with the Caddyfile based on what I read in documentation and the blog ( Advanced Caddy Options :: Guru Computing Blog)…
4. Error messages and/or full log output:
Dec 03 17:09:41 connecttest systemd[1]: Stopped Caddy.
Dec 03 17:09:41 connecttest systemd[1]: Starting Caddy...
Dec 03 17:09:41 connecttest caddy[19042]: caddy.HomeDir=/var/lib/caddy
Dec 03 17:09:41 connecttest caddy[19042]: caddy.AppDataDir=/var/lib/caddy/.local/share/caddy
Dec 03 17:09:41 connecttest caddy[19042]: caddy.AppConfigDir=/var/lib/caddy/.config/caddy
Dec 03 17:09:41 connecttest caddy[19042]: caddy.ConfigAutosavePath=/var/lib/caddy/.config/caddy/autosave.json
Dec 03 17:09:41 connecttest caddy[19042]: caddy.Version=v2.6.2 h1:wKoFIxpmOJLGl3QXoo6PNbYvGW4xLEgo32GPBEjWL8o=
Dec 03 17:09:41 connecttest caddy[19042]: runtime.GOOS=linux
Dec 03 17:09:41 connecttest caddy[19042]: runtime.GOARCH=amd64
Dec 03 17:09:41 connecttest caddy[19042]: runtime.Compiler=gc
Dec 03 17:09:41 connecttest caddy[19042]: runtime.NumCPU=3
Dec 03 17:09:41 connecttest caddy[19042]: runtime.GOMAXPROCS=3
Dec 03 17:09:41 connecttest caddy[19042]: runtime.Version=go1.19.2
Dec 03 17:09:41 connecttest caddy[19042]: os.Getwd=/
Dec 03 17:09:41 connecttest caddy[19042]: LANG=en_US.UTF-8
Dec 03 17:09:41 connecttest caddy[19042]: PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
Dec 03 17:09:41 connecttest caddy[19042]: NOTIFY_SOCKET=/run/systemd/notify
Dec 03 17:09:41 connecttest caddy[19042]: HOME=/var/lib/caddy
Dec 03 17:09:41 connecttest caddy[19042]: LOGNAME=caddy
Dec 03 17:09:41 connecttest caddy[19042]: USER=caddy
Dec 03 17:09:41 connecttest caddy[19042]: INVOCATION_ID=64f2ba11250f44389aa9eaf98c6b4ccd
Dec 03 17:09:41 connecttest caddy[19042]: JOURNAL_STREAM=8:92222
Dec 03 17:09:41 connecttest caddy[19042]: {"level":"info","ts":1670058581.9877987,"msg":"using provided configuration","config_file":"/etc/caddy/Caddyfile","config_adapter":""}
Dec 03 17:09:41 connecttest caddy[19042]: {"level":"warn","ts":1670058581.988877,"msg":"Caddyfile input is not formatted; run the 'caddy fmt' command to fix inconsistencies","adapter":"caddyfile","file":"/etc/caddy/Caddyfile","line":2}
Dec 03 17:09:41 connecttest caddy[19042]: {"level":"info","ts":1670058581.989648,"logger":"admin","msg":"admin endpoint started","address":"localhost:2019","enforce_origin":false,"origins":["//localhost:2019","//[::1]:2019","//127.0.0.1:2019"]}
Dec 03 17:09:41 connecttest caddy[19042]: {"level":"info","ts":1670058581.9898903,"logger":"tls.cache.maintenance","msg":"started background certificate maintenance","cache":"0xc0000e8d90"}
Dec 03 17:09:41 connecttest caddy[19042]: {"level":"info","ts":1670058581.9900846,"logger":"http","msg":"server is listening only on the HTTPS port but has no TLS connection policies; adding one to enable TLS","server_name":"srv0","https_port":443}
Dec 03 17:09:41 connecttest caddy[19042]: {"level":"info","ts":1670058581.990101,"logger":"http","msg":"enabling automatic HTTP->HTTPS redirects","server_name":"srv0"}
Dec 03 17:09:41 connecttest caddy[19042]: {"level":"info","ts":1670058581.9904752,"logger":"pki.ca.local","msg":"root certificate trust store installation disabled; unconfigured clients may show warnings","path":"storage:pki/authorities/local/root.crt"}
Dec 03 17:09:41 connecttest caddy[19042]: {"level":"warn","ts":1670058581.9905403,"logger":"tls","msg":"YOUR SERVER MAY BE VULNERABLE TO ABUSE: on-demand TLS is enabled, but no protections are in place","docs":"https://caddyserver.com/docs/automatic-https#on-demand-tls"}
Dec 03 17:09:41 connecttest caddy[19042]: {"level":"info","ts":1670058581.9906244,"logger":"tls","msg":"cleaning storage unit","description":"FileStorage:/var/lib/caddy/.local/share/caddy"}
Dec 03 17:09:41 connecttest caddy[19042]: {"level":"info","ts":1670058581.9910107,"logger":"tls","msg":"finished cleaning storage units"}
Dec 03 17:09:41 connecttest caddy[19042]: {"level":"debug","ts":1670058581.9910572,"logger":"http","msg":"starting server loop","address":"[::]:80","tls":false,"http3":false}
Dec 03 17:09:41 connecttest caddy[19042]: {"level":"info","ts":1670058581.9910705,"logger":"http.log","msg":"server running","name":"remaining_auto_https_redirects","protocols":["h1","h2","h3"]}
Dec 03 17:09:41 connecttest caddy[19042]: {"level":"info","ts":1670058581.991088,"logger":"http","msg":"enabling HTTP/3 listener","addr":":443"}
Dec 03 17:09:41 connecttest caddy[19042]: {"level":"debug","ts":1670058581.9912093,"logger":"http","msg":"starting server loop","address":"[::]:443","tls":true,"http3":true}
Dec 03 17:09:41 connecttest caddy[19042]: {"level":"info","ts":1670058581.9912205,"logger":"http.log","msg":"server running","name":"srv0","protocols":["h1","h2","h3"]}
Dec 03 17:09:41 connecttest caddy[19042]: {"level":"info","ts":1670058581.991224,"logger":"http","msg":"enabling automatic TLS certificate management","domains":["jumpy.custom_domain.tld","pfsense.custom_domain.tld","truenas.custom_domain.tld","pihole.custom_domain.tld","dashy.custom_domain.tld"]}
Dec 03 17:09:41 connecttest caddy[19042]: {"level":"info","ts":1670058581.9914203,"msg":"autosaved config (load with --resume flag)","file":"/var/lib/caddy/.config/caddy/autosave.json"}
Dec 03 17:09:41 connecttest systemd[1]: Started Caddy.
Dec 03 17:09:41 connecttest caddy[19042]: {"level":"info","ts":1670058581.9931724,"msg":"serving initial configuration"}
5. What I already tried:
I troubleshoot for few hours and didn’t get this work. Every config is looking fine and there is no specific log tell that I’m connected to the domain url. This troubleshooting is making me trouble…
I still didn’t touch the section of request certificate using ACME or cloudflare DNS provider.
Not related…
I tried Nginx Proxy Manager to generate certificate and create a proxy host with Cloudflare A record, I also cannot access proxy host. Does my networking get restricted?
6. Links to relevant resources:
I’m following this guide but I’m using virtual machine. The guide is simple to read and step by step but I cannot achieve until reverse proxy…
Did everything before “Advanced Caddy Options”, and working fine Advanced Caddy Options :: Guru Computing Blog
The log option in the global options are for Caddy runtime logs, not access logs of your vhosts/server blocks.
For those, you need to use the log directive within the vhost/server block you want to log.
I added the log directive parameter, where can I see the log? I execute this command to run instead of systemd (suppose same outcome)
/usr/bin/caddy run --environ --config /etc/caddy/Caddyfile
Below is the log of curl -vL truenas.custom_domain.tld. They seem like not going through reverse proxy but only translating the general DNS to IP.
Just FYI, based on curl command, I observe truenas, pfsense, proxmox (truenas has nginx running, pfsense has nothing running, proxmox default listening https + 8006 only)
NETWORKING AND REVERSE PROXY IS SO HARD FOR ME D:
From caddy server to [truenas] (http://truenas.custom_domain.tld/)
From other virtual machine to truenas http://truenas.custom_domain.tld/
* Trying 192.168.1.10:80...
* Connected to truenas.custom_domain.tld (192.168.1.10) port 80 (#0)
> GET / HTTP/1.1
> Host: truenas.custom_domain.tld
> User-Agent: curl/7.74.0
> Accept: */*
>
* Mark bundle as not supporting multiuse
< HTTP/1.1 302 Moved Temporarily
< Server: nginx
< Date: Sat, 03 Dec 2022 15:25:14 GMT
< Content-Type: text/html
< Content-Length: 138
< Connection: keep-alive
< Location: http://truenas.custom_domain.tld/ui/
< Strict-Transport-Security: max-age=0; includeSubDomains; preload
< X-Content-Type-Options: nosniff
< X-XSS-Protection: 1; mode=block
< Permissions-Policy: geolocation=(),midi=(),sync-xhr=(),microphone=(),camera=(),magnetometer=(),gyroscope=(),fullscreen=(self),payment=()
< Referrer-Policy: strict-origin
< X-Frame-Options: SAMEORIGIN
<
* Ignoring the response-body
* Connection #0 to host truenas.custom_domain.tld left intact
* Issue another request to this URL: 'http://truenas.custom_domain.tld/ui/'
* Found bundle for host truenas.custom_domain.tld: 0x559f353a99d0 [serially]
* Can not multiplex, even if we wanted to!
* Re-using existing connection! (#0) with host truenas.custom_domain.tld
* Connected to truenas.custom_domain.tld (192.168.1.10) port 80 (#0)
> GET /ui/ HTTP/1.1
> Host: truenas.custom_domain.tld
> User-Agent: curl/7.74.0
> Accept: */*
>
* Mark bundle as not supporting multiuse
< HTTP/1.1 200 OK
< Server: nginx
< Date: Sat, 03 Dec 2022 15:25:14 GMT
< Content-Type: text/html
< Content-Length: 6883
< Last-Modified: Mon, 08 Aug 2022 17:27:36 GMT
< Connection: keep-alive
< Etag: TrueNAS-SCALE-22.02.3
< Cache-Control: must-revalidate
< Strict-Transport-Security: max-age=0; includeSubDomains; preload
< X-Content-Type-Options: nosniff
< X-XSS-Protection: 1; mode=block
< Permissions-Policy: geolocation=(),midi=(),sync-xhr=(),microphone=(),camera=(),magnetometer=(),gyroscope=(),fullscreen=(self),payment=()
< Referrer-Policy: strict-origin
< X-Frame-Options: SAMEORIGIN
< Accept-Ranges: bytes
<
<!DOCTYPE html><html lang="en"><head>
<meta charset="utf-8">
<meta http-equiv="Pragma" content="no-cache">
<meta http-equiv="Expires" content="-1">
<meta http-equiv="CACHE-CONTROL" content="NO-CACHE">
<script nomodule src="assets/scripts/ie-support/ie-polyfills.min.js"></script>
<title id="main-page-title"></title>
<base href="/ui/">
<meta name="viewport" content="width=device-width, initial-scale=1">
<link rel="apple-touch-icon" sizes="120x120" href="assets/favicons/apple-touch-icon.png">
<link rel="icon" type="image/png" sizes="32x32" href="assets/favicons/favicon-32x32.png">
<link rel="icon" type="image/png" sizes="16x16" href="assets/favicons/favicon-16x16.png">
<link rel="manifest" href="assets/favicons/site.webmanifest">
<link rel="mask-icon" href="assets/favicons/safari-pinned-tab.svg" color="#000000">
<link rel="shortcut icon" href="assets/favicons/favicon.ico">
<meta name="msapplication-TileColor" content="#2d89ef">
<meta name="msapplication-config" content="assets/favicons/browserconfig.xml">
<meta name="theme-color" content="#ffffff">
<link href="assets/iconfont/material-icons.css" rel="stylesheet" media="print" onload="this.media='all'"><noscript><link rel="stylesheet" href="assets/iconfont/material-icons.css"></noscript>
<link href="assets/iconfont/mdi/css/materialdesignicons.min.css" rel="stylesheet" media="print" onload="this.media='all'"><noscript><link rel="stylesheet" href="assets/iconfont/mdi/css/materialdesignicons.min.css"></noscript>
<link href="assets/iconfont/primeicons/primeicons.css" rel="stylesheet" media="print" onload="this.media='all'"><noscript><link rel="stylesheet" href="assets/iconfont/primeicons/primeicons.css"></noscript>
<style>@font-face{font-display:swap;font-family:Roboto;font-style:italic;font-weight:100;src:local("Roboto Thin Italic"),local("Roboto-ThinItalic"),url(Roboto-ThinItalic.53496aed2032f7a0bcbd.ttf) format("truetype")}@font-face{font-display:swap;font-family:Roboto;font-style:italic;font-weight:300;src:local("Roboto Light Italic"),local("Roboto-LightItalic"),url(Roboto-LightItalic.e53d1fb66605bf6cbc4b.ttf) format("truetype")}@font-face{font-display:swap;font-family:Roboto;font-style:italic;font-weight:400;src:local("Roboto Italic"),local("Roboto-Italic"),url(Roboto-Italic.7b9923bbf8996da54770.ttf) format("truetype")}@font-face{font-display:swap;font-family:Roboto;font-style:italic;font-weight:500;src:local("Roboto Medium Italic"),local("Roboto-MediumItalic"),url(Roboto-MediumItalic.ed9fbccb5bb0c1acf86c.ttf) format("truetype")}@font-face{font-display:swap;font-family:Roboto;font-style:italic;font-weight:700;src:local("Roboto Bold Italic"),local("Roboto-BoldItalic"),url(Roboto-BoldItalic.d17b86957cee6a280e7a.ttf) format("truetype")}@font-face{font-display:swap;font-family:Roboto;font-style:italic;font-weight:900;src:local("Roboto Black Italic"),local("Roboto-BlackItalic"),url(Roboto-BlackItalic.7336356dcbed62c84515.ttf) format("truetype")}@font-face{font-display:swap;font-family:Roboto;font-style:normal;font-weight:100;src:local("Roboto Thin"),local("Roboto-Thin"),url(Roboto-Thin.a76225673323123d2989.ttf) format("truetype")}@font-face{font-display:swap;font-family:Roboto;font-style:normal;font-weight:300;src:local("Roboto Light"),local("Roboto-Light"),url(Roboto-Light.73fd17b88d34c2b2f437.ttf) format("truetype")}@font-face{font-display:swap;font-family:Roboto;font-style:normal;font-weight:400;src:local("Roboto"),local("Roboto-Regular"),url(Roboto-Regular.1ba679c05036b34bf359.ttf) format("truetype")}@font-face{font-display:swap;font-family:Roboto;font-style:normal;font-weight:500;src:local("Roboto Medium"),local("Roboto-Medium"),url(Roboto-Medium.32be89b11725274cd3e8.ttf) format("truetype")}@font-face{font-display:swap;font-family:Roboto;font-style:normal;font-weight:700;src:local("Roboto Bold"),local("Roboto-Bold"),url(Roboto-Bold.6dafca5a4f1e31f2bdf1.ttf) format("truetype")}@font-face{font-display:swap;font-family:Roboto;font-style:normal;font-weight:900;src:local("Roboto Black"),local("Roboto-Black"),url(Roboto-Black.8afe3ed70f5ef2813ba8.ttf) format("truetype")}@-webkit-keyframes cdk-text-field-autofill-start{}@-webkit-keyframes cdk-text-field-autofill-end{}html{font-size:16px}body,html{height:100%;min-height:100%;position:relative;-webkit-tap-highlight-color:transparent;-webkit-text-size-adjust:100%;-moz-text-size-adjust:100%;text-size-adjust:100%;-webkit-touch-callout:none;width:100%}body:not(.safari-platform){-webkit-font-smoothing:antialiased;-moz-osx-font-smoothing:grayscale}body{color:rgba(0,0,0,.87);font-family:Roboto,Helvetica Neue,sans-serif;font-size:.875rem;height:100%;line-height:1.5;margin:0;padding:0;position:relative;width:100%}div{box-sizing:border-box}body{background:var(--bg1)}@font-face{font-family:IBM Plex Sans;src:url(IBMPlexSans-Regular.0519459d8ed646e0f90d.ttf) format("truetype")}html{font-size:87.5%}:root{--fn-dropshadow:0 1px 2px transparent;--contrast-normal:var(--bg2);--light-theme-lines:var(--contrast-darkest);--dark-theme-lines:var(--contrast-lighter);--lines:var(--dark-theme-lines);--sidenav-width:240px;--btn-default-bg:var(--alt-bg2);--hover-bg:var(--lines)!important;--font-family-body:"IBM Plex Sans"}body{font-family:IBM Plex Sans,sans-serif;font-family:var(--font-family-body),sans-serif;font-weight:400;line-height:1.65}</style><link rel="stylesheet" href="styles.30e0eaace16347472bf1.css" media="print" onload="this.media='all'"><noscript><link rel="stylesheet" href="styles.30e0eaace16347472bf1.css"></noscript></head>
<body class="ix-blue">
<app-root>
<div class="app-loadr" style="background-color:var(--bg2);height:100vh; --bg1:#171E26;--bg2:#232d35;--fg1:#aaaaaa;--fg2:#cccccc;">
<div style="position:relative;z-index:2;width:180px;height:180px;top:calc(50% - 90px);left:calc(50% - 90px);">
<div style="text-align:center; position:relative; top:calc(50% - 75px);">
<mat-icon svgicon="truenas_core_logomark" id="brandLogo" alt="" style="width: 100%; height:100%"></mat-icon>
<!-- Put a loader in here at some point -->
</div>
</div>
<script src="assets/scripts/product.js">
</script>
<script>
if (product.trim() === 'TrueNAS') {
document.getElementById('brandLogo').src = 'assets/images/TrueNAS_Logomark_White.svg';
}
</script>
<script src="runtime-es2015.6984309977a740e8d413.js" type="module"></script><script src="runtime-es5.6984309977a740e8d413.js" nomodule defer></script><script src="polyfills-es5.79dd455292550bfc27db.js" nomodule defer></script><script src="polyfills-es2015.8cb64c24c071eafaf53f.js" type="module"></script><script src="scripts.f1955eed7d9c9bef18be.js" defer></script><script src="main-es2015.e2178dd8259df19503a8.js" type="module"></script><script src="main-es5.e2178dd8259df19503a8.js" nomodule defer></script>
* Connection #0 to host truenas.custom_domain.tld left intact
From caddy server to pfsense http://pfsense.custom_domain.tld/
* Trying 192.168.1.5:80...
* connect to 192.168.1.5 port 80 failed: Connection refused
* Failed to connect to jumpy.custom_domain.tld port 80: Connection refused
* Closing connection 0
curl: (7) Failed to connect to jumpy.custom_domain.tld port 80: Connection refused
caddy server to proxmox domain
* Trying 192.168.1.5:80...
* connect to 192.168.1.5 port 80 failed: Connection refused
* Failed to connect to jumpy.custom_domain.tld port 80: Connection refused
* Closing connection 0
curl: (7) Failed to connect to jumpy.custom_domain.tld port 80: Connection refused
I added DNS to internal DNS (pihole) and global DNS (cloudflare), DNS is correct, no luck…
I have no clue why caddy server’s reverse proxy is not working… I tried another reverse proxy like nginx proxy manager as LXC container, it is not working as well. I tested many possibility…
they never hit caddy once even I execute the url from caddy… My homelab network environment is not complex:
ISP – a modem – hypervisor (proxmox) virtualized router – another hypervisor virtualzed caddy – other virtual machine