Moved system, half config works, half fails - HTTP 400 urn:ietf:params:acme:error:malformed - JWS verification error

1. Caddy version (caddy version):

v2.2.1 h1:Q62GWHMtztnvyRU+KPOpw6fNfeCD3SkwH7SfT1Tgt2c=

2. How I run Caddy:

Caddy runs as a Windows Service

a. System environment:

Windows Server 2019

b. Command:

Start Caddy

d. My complete Caddyfile or JSON config:

[redacted] {
tls [redacted] {
	dns cloudflare ******CLOUDFLARE API TOKEN******
}
    root C:\Plex\Caddy\www\Home
    file_server browse
}

[redacted] {
tls [redacted] {
	dns cloudflare ******CLOUDFLARE API TOKEN******
}
    encode gzip
        log {
            output file C:\Plex\Caddy\logs\ombi_access.log {
                roll true                #Rotate logs, enabled by default
                roll_size 5           #Set max size 5 MB
                roll_gzip true           #Whether to compress rolled files
                roll_local_time true     #Use localhost time
                roll_keep 2              #Keep at most 2 log files
                roll_keep_days 7         #Keep log files for 7 days
            }
        }
    reverse_proxy localhost:5000
}

[redacted] {
tls [redacted] {
	dns cloudflare ******CLOUDFLARE API TOKEN******
}
    encode gzip
        log {
            output file C:\Plex\Caddy\logs\plex_access.log {
                roll true                #Rotate logs, enabled by default
                roll_size 5           #Set max size 5 MB
                roll_gzip true           #Whether to compress rolled files
                roll_local_time true     #Use localhost time
                roll_keep 2              #Keep at most 2 log files
                roll_keep_days 7         #Keep log files for 7 days
            }
        }
    reverse_proxy localhost:32400
}

[redacted] {
tls [redacted] {
	dns cloudflare ******CLOUDFLARE API TOKEN******
}
    encode gzip
    root C:\Plex\Caddy\www\Tautulli
        log {
            output file C:\Plex\Caddy\logs\tautulli_access.log {
                roll true                #Rotate logs, enabled by default
                roll_size 5           #Set max size 5 MB
                roll_gzip true           #Whether to compress rolled files
                roll_local_time true     #Use localhost time
                roll_keep 2              #Keep at most 2 log files
                roll_keep_days 7         #Keep log files for 7 days
            }
        }
    reverse_proxy localhost:8181
}

[redacted] {
tls [redacted] {
	dns cloudflare ******CLOUDFLARE API TOKEN******
}
    encode gzip
        log {
            output file C:\Plex\Caddy\logs\radarr_access.log {
                roll true                #Rotate logs, enabled by default
                roll_size 5           #Set max size 5 MB
                roll_gzip true           #Whether to compress rolled files
                roll_local_time true     #Use localhost time
                roll_keep 2              #Keep at most 2 log files
                roll_keep_days 7         #Keep log files for 7 days
            }
        }
    reverse_proxy localhost:7878
}

[redacted] {
tls [redacted] {
	dns cloudflare ******CLOUDFLARE API TOKEN******
}
    encode gzip
        log {
            output file C:\Plex\Caddy\logs\sonarr_access.log {
                roll true                #Rotate logs, enabled by default
                roll_size 5           #Set max size 5 MB
                roll_gzip true           #Whether to compress rolled files
                roll_local_time true     #Use localhost time
                roll_keep 2              #Keep at most 2 log files
                roll_keep_days 7         #Keep log files for 7 days
            }
        }
    reverse_proxy localhost:8989
}

[redacted] {
tls [redacted] {
	dns cloudflare ******CLOUDFLARE API TOKEN******
}
    encode gzip
        log {
            output file C:\Plex\Caddy\logs\qBittorrent_access.log {
                roll true                #Rotate logs, enabled by default
                roll_size_mb 5           #Set max size 5 MB
                roll_gzip true           #Whether to compress rolled files
                roll_local_time true     #Use localhost time
                roll_keep 2              #Keep at most 2 log files
                roll_keep_days 7         #Keep log files for 7 days
            }
        }
    reverse_proxy localhost:9999
}

3. The problem I’m having:

Moved Caddy to a new server, when I go to start caddy, half my domains obtain certs, the other half don’t.
Not really sure what the issue is, as except for the upload domain, I’ve copy and pasted the config for all the others, so they’re exactly the same.
The downloads domain functions, but the request domain does not.

4. Error messages and/or full log output:

C:\Plex\Caddy>2020/11/10 10:11:33.079   ERROR   tls.obtain      will retry      {"error": "[redacted] Obtain: [redacted] creating new order: request to https://acme-v02.api.letsencrypt.org/acme/new-order failed after 1 attempts: HTTP 400 urn:ietf:params:acme:error:malformed - JWS verification error (ca=https://acme-v02.api.letsencrypt.org/directory)", "attempt": 1, "retrying_in": 60, "elapsed": 0.8820097, "max_duration": 2592000}
2020/11/10 10:11:33.514 ERROR   tls.obtain      will retry      {"error": "[redacted] Obtain: [redacted] creating new order: request to https://acme-v02.api.letsencrypt.org/acme/new-order failed after 1 attempts: HTTP 400 urn:ietf:params:acme:error:malformed - JWS verification error (ca=https://acme-v02.api.letsencrypt.org/directory)", "attempt": 1, "retrying_in": 60, "elapsed": 1.2580129, "max_duration": 2592000}
2020/11/10 10:11:33.514 ERROR   tls.obtain      will retry      {"error": "[redacted] Obtain: [redacted] creating new order: request to https://acme-v02.api.letsencrypt.org/acme/new-order failed after 1 attempts: HTTP 400 urn:ietf:params:acme:error:malformed - JWS verification error (ca=https://acme-v02.api.letsencrypt.org/directory)", "attempt": 1, "retrying_in": 60, "elapsed": 1.2410265, "max_duration": 2592000}

5. What I already tried:

Re-downloaded Caddy, no dice.
Caddy Validate my config, all good.
Not sure what else to check, it was working fine on the previous server - Move everything across and suddenly it only half works.

6. Links to relevant resources:

What are the full logs? That might help give a clue.

Sorry for delays, been really busy this week.

Sorry, but what does ubiquiti have to do with this? (This is the first time your post has mentioned it.) And those logs don’t look familiar, the formatting / separators are weird.

I’m not really able to help until your setup is explained more fully, or preferably, reduce the problem down to its minimal parts so it can be reproduced.

At this point I think something is wrong with your setup (rather than a problem in Caddy), but it’s unclear what it is.

How can we reproduce the error?

What is your current, full, unmodified Caddyfile?

I recommend switching to Let’s Encrypt staging for testing, lest you run into rate limits in production.

Well, the original Caddy logs. Caddy does not emit logs like this:

|—|—|—|—|—|

You might find debug mode helpful, please enable that in your global options. It will emit more logs which will be useful.

I suspect that something in your setup is getting in the way of Caddy operating successfully, maybe intercepting logs or bytes on the wire, I dunno.

PS. Run caddy fmt on your Caddyfile, it will make it easier to read.

This issue was resolved by removing the existing setup and re-implementing from scratch.