Why Caddy error with valid certificate?

Help
1

1. The problem I’m having:

I have configured Caddy to proxy a container running MeshCentral but it only works if Caddyfile has the “tls_insecure_skip_verify” option active for this transport. If I comment the “tls_insecure_skip_verify” option the site never loads and the error below appears in Caddy logs.

Interestingly, if “tls_insecure_skip_verify” command is let in/active the R3 issued SSL certificate shows “This certificate is valid” in all browsers and has an proper expiration date of 9/4/2024 so it is a valid certificate!

Also, MeshCentral seems to successfully load this valid certificate from Caddy since it’s logs show:

MeshCentral HTTP redirection server running on port 80.

MeshCentral v1.1.24, Hybrid (LAN + WAN) mode, Production mode.

MeshCentral Intel(R) AMT server running on mesh.example.com:4433.

Loaded web certificate from "https://172.22.0.5:443", host: "mesh.example.com"

  SHA384 cert hash: 9001e831ffe6a0806607c987f3429129094ac087e04e1f86778955242648d7d7059bd5a4f223857689132203fa24b67d

MeshCentral HTTPS server running on mesh.example.com:443.

Why does the error in Caddy occur if the certificate is valid and loads successfully?

2. Error messages and/or full log output:

ERR | ts=1718595207.8516228 logger=http.log.error msg=tls: failed to verify certificate: x509: certificate is valid for mesh.example.com, localhost, not meshcentral request={"remote_ip":"192.168.1.199","remote_port":"55518","client_ip":"192.168.1.199","proto":"HTTP/2.0","method":"GET","host":"mesh.example.com","uri":"/","headers":{"User-Agent":["Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/126.0.0.0 Safari/537.36"],"Upgrade-Insecure-Requests":["1"],"Sec-Fetch-User":["?1"],"Accept-Encoding":["gzip, deflate, br, zstd"],"Cookie":["REDACTED"],"Priority":["u=0, i"],"Sec-Ch-Ua":["\"Not/A)Brand\";v=\"8\", \"Chromium\";v=\"126\", \"Brave\";v=\"126\""],"Accept":["text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8"],"Sec-Fetch-Site":["none"],"Sec-Fetch-Mode":["navigate"],"Cache-Control":["max-age=0"],"Sec-Ch-Ua-Platform":["\"macOS\""],"Sec-Gpc":["1"],"Accept-Language":["en-US,en;q=0.8"],"Sec-Fetch-Dest":["document"],"Sec-Ch-Ua-Mobile":["?0"]},"tls":{"resumed":true,"version":772,"cipher_suite":4865,"proto":"h2","server_name":"mesh.example.com"}} duration=0.004046964 status=502 err_id=z8mn70msv err_trace=reverseproxy.statusError (reverseproxy.go:1269)

3. Caddy version:

caddy v2.8.4

4. How I installed and ran Caddy:

a. System environment:

Ubuntu Server
Docker
Docker compose v2
Portainer

b. Command:

n/a

c. Service/unit/compose file:

d. My complete Caddy config:

mesh.example.com {
        reverse_proxy {
                to https://meshcentral:443
                transport http {
                        tls
#                       tls_insecure_skip_verify
                }
        }
}

5. Links to relevant resources:

2

The message is pretty clear, Caddy sent meshcentral in SNI and expected the certificate to contain that domain, but it contained mesh.example.com instead, so it doesn’t validate.

If your upstream address must be meshcentral (I assume a container name or w/e) then you’ll need to override tls_server_name to be your actual domain name so that it’s used in TLS-SNI and for certificate validation.

Also, you don’t need to specify tls in your config if you have https:// in your upstream address. The :443 is also redundant (it’s the default HTTPS port).

3 Likes
3

Thanks @francislavoie, I think I got it but would appreciate your patience properly fixing.

And yes, “meshcentral” is the name of the docker container (under Portainer) that Caddy can see, as Caddy is also running as a separate container in the same Docker setup.

Could you be so kind as check the modified Caddyfile below since now I get a recurring new error in Caddy about tls: failed to verify certificate and the MeshCentral site no longer loads.

Adding tls_insecure_skip_verify of course makes the error go away and allows MeshCentral to be accessible again with it’s Valid SSL, but it defeats the purpose of checking the SSL certificate, so I hope you can show me a better/proper way to do this and stop the errors…

ERR | ts=1718650765.3287187 logger=http.log.error msg=tls: failed to verify certificate: x509: certificate signed by unknown authority request={"remote_ip":"192.168.1.199","remote_port":"56919","client_ip":"192.168.1.199","proto":"HTTP/1.1","method":"GET","host":"mesh.example.com","uri":"/agent.ashx","headers":{"Upgrade":["websocket"],"Connection":["Upgrade"],"Sec-Websocket-Key":["ZONYMCWnSk1RjcUG0NM/4H=="],"Sec-Websocket-Version":["13"]},"tls":{"resumed":false,"version":772,"cipher_suite":4865,"proto":"","server_name":"mesh.example.com"}} duration=0.003146775 status=502 err_id=tb3rd8ugy err_trace=reverseproxy.statusError (reverseproxy.go:1269) 

mesh.example.com {
        reverse_proxy {
                to https://meshcentral
                transport http {
                        tls_server_name mesh.example.com
                }
        }
}
4

This means that the system Caddy is running on is missing the root CA cert for your upstream, so it doesn’t trust the connection. You’ll need to install it on whatever you’re running Caddy on (is it in a Docker container? I don’t find your post clear about that), or configure Caddy to trust the root CA cert with the tls_trust_pool option.

1 Like
5

Thanks @francislavoie,

Caddy is indeed just running in its own Docker container, and separate from the MeshCentral container. MeshCentral simply shares the caddy: network and gets its certificate directly from Caddy.

Would the Root CA for upstream not come from / be part of Let’s Encrypt since Caddy is handling all certificates?

Also, for reference, the host OS is Ubuntu Server 22.04.4 LTS, so could it be that I need to upgrade the Let’s encrypt trusted roots?

Either way, do you have a good tutorial I could follow?

Thank you.

6

I dunno, how did you issue the cert for your upstream app? That’s outside of Caddy’s concerns.

Caddy is managing certs for incoming connections to itself, it has no responsibility to manage certs for connections it makes to upstream apps. It’s impossible for it to manage that.

2 Likes
7

Hi @francislavoie,

The upstream app is “MeshCentral” container, and it was configured to retrieve its certificate directly from Caddy. It does this successfully, as can be seen in the logs. Note that https://172.22.0.5:443 is the IP of the Caddy container.

MeshCentral log excerpt here:

MeshCentral HTTP redirection server running on port 80.
MeshCentral v1.1.24, Hybrid (LAN + WAN) mode, Production mode.
MeshCentral Intel(R) AMT server running on mesh.example.com:4433.
Loaded web certificate from "https://172.22.0.5:443", host: "mesh.example.com"
  SHA384 cert hash: 9001e831ffe6a0806607c987f3429129094ac087e04e1f86778955242648d7d7059bd5a4f223857689132203fa24b67d
MeshCentral HTTPS server running on mesh.example.com:443.

Also, for reference, the host OS is Ubuntu Server 22.04.4 LTS, so could it be that I need to upgrade the Let’s encrypt trusted roots? If so, maybe you could share a good tutorial I could follow?

Thank you.

8

How are you doing that? I don’t understand. I don’t see acme_server in your Caddy config.

1 Like
9

TL;DR: MeshCentral doesn’t get “its certificate” from Caddy. It can be configured to pull the public cert from its reverse proxy so it can configure remote agents to look for it when connecting. This DOES NOT allow MeshCentral to use this cert to provide a validated HTTPS connection since it ONLY retrieves the public cert - the correct configuration is to use TlsOffload to serve HTTP to the TLS-terminating reverse proxy.

For a little bit of very important context that’s been left out of the thread so far:

MeshCentral is RMM software that makes use of agents on client computers. It’s notable for being able to configure and monitor Intel vPro-capable machines agentlessly.

When MeshCentral agents, or Intel vPro via CIRA (Client-Initiated Remote Access), connect to MeshCentral through a TLS reverse proxy, they note the public certificate of the proxy.

MeshCentral itself then configures those agents, and as part of that process, pins them to that public certificate; to do this, MeshCentral needs to obtain the public certificate of the reverse proxy. This is done by configuring the MeshCentral TlsOffload and CertUrl configuration keys. When a CertUrl is configured, MeshCentral will simply connect to the reverse proxy to download the public cert.

CertUrl

Load the TLS certificate for this domain from this https url. For example “https://127.0.0.1:123”. This option is useful when used along with the “TlsOffload” option. When MeshCentral is not doing any TLS but has a reverse-proxy or TLS offload device doing this work in front of the server, you can use this to have MeshCentral load the certificate from the server in front of MeshCentral.

This is needed because when agents connect, they need to be told that the certificate they saw upon connecting is the correct one. Using this, MeshCentral will know what certificate the agents are expected to see.

—https://meshcentral.com/docs/MeshCentral2UserGuide.pdf, Section 7.2

See also section 10 on TLS Offloading.

image
image1000×355 44.3 KB

4 Likes
10

Thanks @Whitestrake for reference I am only using “agents” and they seem to work fine.

However I am still missing why if I have tls_insecure_skip_verify active, MeshCentral will not load and if I enable it, MeshCentral loads just fine and shows it’s using a Valid & Trusted mesh.example.com Let’s Encrypt certificate?

Here is my Caddyfile again, how should I modify it to not use 'tls_insecure_skip_verify` and still allow MeshCentral to operate?

mesh.example.com {
        reverse_proxy {
                to https://meshcentral
                transport http {
                        tls_server_name mesh.example.com
                        tls_insecure_skip_verify
                }
        }
}

Thank you.

11

You need tls_insecure_skip_verify because MeshCentral does not have a valid certificate.

Since you currently have Caddy reverse proxying to MeshCentral over HTTPS, you must either have a valid certificate, or configure Caddy to ignore the invalid certificate.

When you enable tls_insecure_skip_verify and connect to MeshCentral, the valid certificate you see in your browser is from Caddy, not from MeshCentral.

When you disable tls_insecure_skip_verify, Caddy refuses to connect to MeshCentral over HTTPS.

Reverse proxy to http://meshcentral instead.

MeshCentral should only be configured to serve TLS if it is serving the edge without a reverse proxy in front. If you put Caddy in front, offload the TLS to Caddy, as per the MeshCentral User Guide.

1 Like
12

I see. @Whitestrake, so in config.json for MeshCentral settings I keep the current:

"settings": {
     "TLSOffload": false,
     "cert": "mesh.example.com"
},

and under

"domains": {
    "certUrl": "https://172.22.0.5:443"
}

For reference, 172.22.0.5 is the IP of the Caddy container.

Then change Caddyfile config to:

mesh.example.com {
        reverse_proxy {
                to http://meshcentral
                transport http {
                        tls_server_name mesh.example.com
                }
        }
}
13

As for your MeshCentral settings, I would strongly suggest going over the MeshCentral User Guide again and reviewing Section 7.1 for the correct usage of the TlsOffload setting and Section 10 on TLS Offloading.

As for Caddy, you should remove tls_server_name as you don’t need to send SNI to a HTTP endpoint. You may also remove the entire transport http stanza and the http:// from http://meshcentral. The end result for the Caddyfile would simply be a single directive in your site block: reverse_proxy meshcentral

3 Likes
14

Got it @Whitestrake, thanks, I cleaned up my Caddyfile.

mesh.example.com {
        reverse_proxy http://meshcentral
}

As for the meshCentral config, as far as I understand from the manual, you have to set a “cert” entry to a FQDN in order to use MeshCentral in WAN mode and instruct the agents what FQDN to use.

There are still 2 entries I am unsure of referencing Caddy and they are:

      "certUrl": "https://172.22.0.5:443",
      "_TrustedCert": true

So far this is what I came up with, but unsure if 100% correct under Caddy:

{
  "$schema": "https://raw.githubusercontent.com/Ylianst/MeshCentral/master/meshcentral-config-schema.json",
  "settings": {
    "plugins":{"enabled": true},
    "_mongoDb": null,
    "cert": "mesh.example.com",
    "_WANonly": true,
    "_LANonly": true,
    "_sessionKey": "MySecretPassword",
    "port": 443,
    "_aliasPort": 443,
    "redirPort": 80,
    "_redirAliasPort": 80,
    "AgentPong": 300,
    "TLSOffload": false,
    "SelfUpdate": false,
    "AllowFraming": false,
    "WebRTC": false
  },
  "domains": {
    "": {
      "_title": "MeshCentral",
      "_title2": "Example MeshCentral",
      "minify": true,
      "NewAccounts": true,
      "localSessionRecording": true,
      "_userNameIsEmail": true,
      "certUrl": "https://172.22.0.5:443",
      "_TrustedCert": true
    }
  }

Right now with this, MeshCentral does NOT come up and I get a “too many redirects” browser message in Safari

15

TlsOffload

By default this option is set to ‘false’. If set to ‘true’, server will run both web port and the Intel AMT MPS port without TLS with the assumption that a TLS offloading is taking care of this task. For further details, see the “TLS Offloading” section.

This option can also be set to the IP address of the reverse-proxy in order to indicate to MeshCental to only trust HTTP X-Forwarded headers coming from this IP address. See the “Reverse-Proxy Setup” section for an example.

—https://meshcentral.com/docs/MeshCentral2UserGuide.pdf, Section 7.2

This would seem to indicate that you should set TlsOffload to the IP address of Caddy.

To make this work, it is important the server is setup with “–tlsoffload” not with “–notls". This indicates the server that TLS is already being taken care of and MeshCentral does not have to deal with it. MeshCentral will continue to listen to port 80, 443 and 4433.

—https://meshcentral.com/docs/MeshCentral2UserGuide.pdf, Section 10

This would seem to agree.

1 Like
16

UGH! Still no dice, I still get “Too Many Redirects”, even in private browser.

I followed the example for the NGINX Proxy setup on page 31, and changed tlsoffload to the IP of the Caddy container using "TLSOffload": "172.22.0.5". The MeshCentral config file (with commented out options removed) became:

{
  "$schema": "https://raw.githubusercontent.com/Ylianst/MeshCentral/master/meshcentral-config-schema.json",
  "settings": {
    "plugins":{"enabled": true},
    "cert": "mesh.example.com",
    "port": 443,
    "redirPort": 80,
    "AgentPong": 300,
    "TLSOffload": "172.22.0.5",
    "SelfUpdate": false,
    "AllowFraming": false,
    "WebRTC": false
  },
  "domains": {
    "": {
      "minify": true,
      "NewAccounts": true,
      "localSessionRecording": true,
      "certUrl": "https://172.22.0.5:443",
    }
  }

Caddyfile is still:

mesh.example.com {
        reverse_proxy http://meshcentral
}

No errors in Caddy or MeshCentral logs but still no dice! :frowning:

In MeshCentral config I even tried to change (per the NGINX example):

    "port": 4430,
    "RedirPort": 800,

So the logs showed MeshCentral HTTP redirection server running on port 800, and then changed Caddy file to hit meshcentral on port 800 like so:

mesh.example.com {
        reverse_proxy http://meshcentral:800
}

Restarted 1st Caddy then MeshCentral, and still no luck / same issue!

17

As a last ditch attempt I changed the MeshCentral config (showing just relevant entries) to:

 "settings": {
    "plugins":{"enabled": true},
    "cert": "mesh.example.com",
    "port": 443,
    "RedirPort": 80,
    "TLSOffload": "172.22.0.5",
  },
  "domains": {
    "": {
      "certUrl": "https://mesh.example.com",
      "TrustedCert": true
    }
  },

The logs then showed:

MeshCentral HTTP server running on port 443.
MeshCentral HTTP redirection server running on port 80.
MeshCentral v1.1.24, Hybrid (LAN + WAN) mode, Production mode.
Code signed MeshCmd.exe.
Code signed MeshService.exe.
Code signed MeshServiceARM64.exe.
Code signed MeshCmdARM64.exe.
Code signed MeshCmd64.exe.
Code signed MeshService64.exe.
Loaded web certificate from "https://mesh.example.com", host: "mesh.example.com"
  SHA384 cert hash: 9001e831ffe6a0806607c987f3429129094ac087e04e1f86778955242648d7d7059bd5a4f223857689132203fa24b67d

And I got a “Too many redirects page” so I then changed Caddyfile to access MeshCentral on https:

mesh.example.com {
        reverse_proxy https://meshcentral
}

Caddy logs show no errors, but when I access https://mesh.example.com I just get a plain white page, the SSL certificate loads fine for mesh.example.com, but the MeshCentral login interface never loads!

I give up… UGH!

18

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.