Apple Pay TLS Cipher Suite Issues

[Jason_Inbody]

1. Caddy version (caddy version):

Caddy v1.0.5 (h1:5B1Hs0UF2x2tggr2X9jL2qOZtDXbIWQb9YLbmlxHSuM=)

2. How I run Caddy:

Loggie

a. System environment:

DigitalOcean VPS Ubuntu 20.04

b. Command:

My Cadddy service file

Description=Caddy HTTP/2 web server
Documentation=https://caddyserver.com/docs
After=network-online.target
Wants=network-online.target systemd-networkd-wait-online.service

; Do not allow the process to be restarted in a tight loop. If the
; process fails to start, something critical needs to be fixed.
StartLimitIntervalSec=14400
StartLimitBurst=10

[Service]
Restart=on-abnormal

; User and group the process will run as.
User=www-data
Group=www-data

; Letsencrypt-issued certificates will be written to this directory.
Environment=CADDYPATH=/etc/ssl/caddy DO_AUTH_TOKEN=<My token> 

; Always set "-root" to something safe in case it gets forgotten in the Caddyfile.
ExecStart=/usr/local/bin/caddy -log stdout -log-timestamps=false -agree=true -conf=/etc/caddy/Caddyfile -root=/var/tmp
ExecReload=/bin/kill -USR1 $MAINPID

; Use graceful shutdown with a reasonable timeout
KillMode=mixed
KillSignal=SIGQUIT
TimeoutStopSec=5s

; Limit the number of file descriptors; see `man systemd.exec` for more limit settings.
LimitNOFILE=1048576
; Unmodified caddy is not expected to use more than that.
LimitNPROC=512

; Use private /tmp and /var/tmp, which are discarded after caddy stops.
PrivateTmp=true
; Use a minimal /dev (May bring additional security if switched to 'true', but it may not work on Raspberry Pi's or other devices, so it has been disabled in this dist.)
PrivateDevices=false
; Hide /home, /root, and /run/user. Nobody will steal your SSH-keys.
ProtectHome=true
; Make /usr, /boot, /etc and possibly some more folders read-only.
ProtectSystem=full
; … except /etc/ssl/caddy, because we want Letsencrypt-certificates there.
;   This merely retains r/w access rights, it does not add any new. Must still be writable on the host!
ReadWritePaths=/etc/ssl/caddy
ReadWriteDirectories=/etc/ssl/caddy

; The following additional security directives only work with systemd v229 or later.
; They further restrict privileges that can be gained by caddy. Uncomment if you like.
; Note that you may have to add capabilities required by any plugins in use.
;CapabilityBoundingSet=CAP_NET_BIND_SERVICE
;AmbientCapabilities=CAP_NET_BIND_SERVICE
;NoNewPrivileges=true

[Install]
WantedBy=multi-user.target

c. Service/unit/compose file:

Service file above

d. My complete Caddyfile or JSON config:

Caddy File

    loggie.app:443 {
      root /var/www/loggie/build/
       tls {
         dns digitalocean
     
       }
     }
     loggie.app:443/api {
         proxy / 127.0.0.1:8000 {
             transparent
         }
     }

3. The problem I’m having:

I’m trying to authenticate my domain with Apple for their Apple Pay verfifcataion process where they have you move a file into Loggie which I have placed in my /var/www/build/.well-known/ dir and the file shows up when I navigate to it. When I go to authenticate with apple they respond with this error.

Correction: When I got my plain domain, load the page then navigate to the apple file it works. But if i navigate there directly (like apple) I get this error
Domain verification failed. Unable to access verification file on server. Confirm that the file is in the correct location, proxies and redirects are not enabled, and the documented Apple Domain Verification IP addresses can access your server.

4. Error messages and/or full log output:

From Apple

Domain verification failed. Review your TLS Certificate configuration to confirm that the certificate is accessible and a supported TLS Cipher Suite is used.

5. What I already tried:

I haven’t tried much because I can’t find much information on Caddy + Apple Pay certification. It seems caddy is missing some TSL types that apple wants but I’m confused on how to set these in the caddy file. I’ve done a TSL cert site and I have TSL 1.2 and 1.3 so I’m not sure what apple is complaining about…

6. Links to relevant resources:

https://developer.apple.com/forums/thread/97031
This talks about TSL issues and possible issues with their IP address not having access.

Also this gives some setup stuff. Apple Developer Documentation How would I add this to my caddy file?

[Jason_Inbody]

Adding

loggie.app:443/.well-known {
  root /var/www/static/.well-known:
  tls {
    dns digitalocean
    cipher TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256
  }
}

To my caddy file gets me past the TSL suite issue and now apple gives an error of

Domain verification failed. Unable to access verification file on server. Confirm that the file is in the correct location, proxies and redirects are not enabled, and the documented Apple Domain Verification IP addresses can access your server.

The url is directly accessible now that I’ve moved it to my static files but apple is some how not able to access it…

[francislavoie]

Caddy v1 is no longer supported. Please upgrade to Caddy v2.

Install - Caddy Documentation

Upgrading to Caddy 2 - Caddy Documentation

How to use DNS provider modules in Caddy 2

Caddy 2 uses a new and improved DNS provider interface for solving the ACME DNS challenge. All you have to do is plug the service provider(s) you need into your build, then add the DNS challenge to your configuration! Getting a DNS provider plugin How you choose to get a custom Caddy build is up to you; we’ll describe two common methods here. Method 1: Go to the Caddy download page. Find your DNS provider in the list of modules (dns.providers.*) and select it. Download your custom Caddy …

[Jason_Inbody]

I’ve upgraded to caddy 2.2.1 and my new caddy file looks like

loggie.app {
  root * /var/www/loggie/build/
  file_server
}
loggie.app:443/api {
  reverse_proxy * 127.0.0.1:8000
}

And my service file is

[Unit]
Description=Caddy
Documentation=https://caddyserver.com/docs/
After=network.target network-online.target
Requires=network-online.target

[Service]
User=caddy
Group=caddy
ExecStart=/usr/bin/caddy run --environ --config /etc/caddy/Caddyfile
ExecReload=/usr/bin/caddy reload --config /etc/caddy/Caddyfile
TimeoutStopSec=5s
LimitNOFILE=1048576
LimitNPROC=512
PrivateTmp=true
ProtectSystem=full
AmbientCapabilities=CAP_NET_BIND_SERVICE

[Install]
WantedBy=multi-user.target

I’ve got the text file apple wants in /var/www/loggie/build/.well-known and when I go to the url I get the text file they want. I’m still getting

Domain verification failed. Review your TLS Certificate configuration to confirm that the certificate is accessible and a supported TLS Cipher Suite is used.

from apple. I was able to see in the older version that the there was a TSL cipher suite miss match between my caddy server and apple’s server from the journalctl but I’m not seeing the mismatch in the journalctl of the new caddy but apple is still throwing errors on their end.

[francislavoie]

I recommend writing your config like this instead:

loggie.app {
	handle /api* {
		reverse_proxy 127.0.0.1:8000
	}

	handle {
		root * /var/www/loggie/build
		file_server
	}
}

I can’t really help with the issues with Apple though, frankly. In the past I’ve seen them have very strange requirements. Caddy’s defaults are ideal for all modern web browsers.

[matt (Matt Holt)]

@Jason_Inbody

from apple. I was able to see in the older version that the there was a TSL cipher suite miss match between my caddy server and apple’s server from the journalctl

Can you paste that log here please?

[Jason_Inbody]

This was from the caddy v1

Nov 28 20:10:52 Loggie caddy[1239]: http: TLS handshake error from 17.179.144.226:7765: tls: no cipher suite supported by both client and server

This was from caddy v2. I’m not seeing any errors pop up in v2. Maybe I don’t the logging properly set in the service to catch it.

loggie@Loggie:~$ sudo journalctl -f -u caddy
– Logs begin at Sun 2020-11-22 00:04:08 UTC. –
Nov 28 21:22:07 Loggie caddy[2297]: JOURNAL_STREAM=9:36021
Nov 28 21:22:07 Loggie caddy[2297]: {“level”:“info”,“ts”:1606598527.1533165,“msg”:“using provided configuration”,“config_file”:“/etc/caddy/Caddyfile”,“config_adapter”:“”}
Nov 28 21:22:07 Loggie caddy[2297]: {“level”:“info”,“ts”:1606598527.1560726,“logger”:“admin”,“msg”:“admin endpoint started”,“address”:“tcp/localhost:2019”,“enforce_origin”:false,“origins”:[“localhost:2019”,“[::1]:2019”,“127.0.0.1:2019”]}
Nov 28 21:22:07 Loggie caddy[2297]: {“level”:“info”,“ts”:1606598527.1565545,“logger”:“tls.cache.maintenance”,“msg”:“started background certificate maintenance”,“cache”:“0xc0002f67e0”}
Nov 28 21:22:07 Loggie caddy[2297]: {“level”:“info”,“ts”:1606598527.1567168,“logger”:“http”,“msg”:“server is listening only on the HTTPS port but has no TLS connection policies; adding one to enable TLS”,“server_name”:“srv0”,“https_port”:443}
Nov 28 21:22:07 Loggie caddy[2297]: {“level”:“info”,“ts”:1606598527.1568086,“logger”:“http”,“msg”:“enabling automatic HTTP->HTTPS redirects”,“server_name”:“srv0”}
Nov 28 21:22:07 Loggie caddy[2297]: {“level”:“info”,“ts”:1606598527.157793,“logger”:“tls”,“msg”:“cleaned up storage units”}
Nov 28 21:22:07 Loggie caddy[2297]: {“level”:“info”,“ts”:1606598527.1581588,“logger”:“http”,“msg”:“enabling automatic TLS certificate management”,“domains”:[“loggie.app”]}
Nov 28 21:22:07 Loggie caddy[2297]: {“level”:“info”,“ts”:1606598527.1683974,“msg”:“autosaved config”,“file”:“/var/lib/caddy/.config/caddy/autosave.json”}
Nov 28 21:22:07 Loggie caddy[2297]: {“level”:“info”,“ts”:1606598527.1686232,“msg”:“serving initial configuration”}

[matt (Matt Holt)]

Thanks! That looks like Apple needs to fix something… surprised they don’t support modern cipher suites?

In Caddy 2, more logs can be visible by enabling debug mode (Global options (Caddyfile) — Caddy Documentation).

[Jason_Inbody]

Turning on debug shows the handshake error on 2.2.1 too now

Nov 29 01:49:38 Loggie caddy[5186]: {“level”:“debug”,“ts”:1606614578.4891794,“logger”:“http.stdlib”,“msg”:“http: TLS handshake error from 17.179.144.226:2356: tls: no cipher suite supported by both client and server”}

These are the list Apple is supposed to support Setting Up Your Server | Apple Developer Documentation and this were the ones one the apple rep said to use in a forum Apple Pay Domain Verification Fail… | Apple Developer Forums.

[francislavoie]

The list of default cipher suites Caddy uses is here:

caddyserver/caddy/blob/master/modules/caddytls/values.go#L52

It looks to me like there’s overlap (first 4 in both lists are the same), so

[Jason_Inbody]

I was never able to get apples servers to work properly. Everything looks fine on the Caddy side and I was out of things to debug on my side. I was using stripe to handle my payment processing and they actually ended up handling the apple merchant side of things for me. Stripe had me also place a file in the same spot ( Loggie) and when stripe checked my site for the proper cipher suites it worked flawlessly. I think the issue is on apple’s side. But this was my work around if anyone else has this issue.