Acme DNS-01 Challenge Failing

Help
2

Well it turns out that spending whatever I spent so far + 1h was the key… The problem was that I was “unlucky” by having my first pick of the target service (at 10.183.3.10) be proxmox. Essentially, proxmox does some of its own stuff when it comes to http → https redirects and that is what caused the infinite loops on a valid configuration. As far as I found out, there are two ways to solve this (I modified the format of the caddyfile config a bit because I like the internal routes to be grouped like this):

a. Trust proxmox’s certificate in caddy’s host:

For the certificate:

  1. On your caddy node vim /usr/local/share/ca-certificates/proxmox-ca.crt and paste in what you got from cat /etc/pve/pve-root-ca.pem from your proxmox node.
  2. Do update-ca-certificates on caddy’s node
  3. Restart caddy

Modify Caddyfile:

*.chaos-gang.click {
    tls {
        dns acmedns /etc/caddy/acmedns.json
    }

    @p3x-001 host p3x-001.chaos-gang.click
    handle @p3x-001 {
        reverse_proxy https://10.183.3.10:8006 {
            header_up Host {host}
            header_up X-Forwarded-Proto {scheme}
            header_up X-Forwarded-For {remote}
            header_up X-Real-IP {remote}
        }
    }

    @internalIP not remote_ip 10.183.0.0/16
    handle @internalIP {
        respond "403 Forbidden" 403
    }

    respond "403 Forbidden" 403
}
b. Use tls_insecure_skip_verify

In my case, all of the traffic in this thread (apart from dns lookup) happens on the internal network, so (as far as I am aware) skipping the verification so proxmox stops redirecting should be fine.

*.chaos-gang.click {
    tls {
        dns acmedns /etc/caddy/acmedns.json
    }

    @p3x-001 host p3x-001.chaos-gang.click
    handle @p3x-001 {
        reverse_proxy https://10.183.3.10:8006 {
            transport http {
                tls_insecure_skip_verify
            }
            header_up Host {host}
            header_up X-Forwarded-Proto {scheme}
            header_up X-Forwarded-For {remote}
            header_up X-Real-IP {remote}
        }
    }
}

As far as my understanding goes, the benefit of option a) is that your TLS setup is verified also from caddy → internal service (proxmox). While option b) allows for some mitm attacks and whatnot (though again, this is on the local network).