I installed the 2.4.0-beta.1 static binary and then manually ran caddy upgrade. I had in my mind that the latest tweaks to beta.1 would be applied. The upgrade went through successfully. The only issue was that Caddy was downgraded to 2.3.0 
[root@wp-basil /usr/local/bin]# caddy version
v2.4.0-beta.1 h1:Ed/tIaN3p6z8M3pEiXWJL/T8JmCqV62FrSJCHKquW/I=
[root@wp-basil /usr/local/bin]# caddy upgrade
2021/03/09 17:19:29.637 INFO this executable will be replaced {"path": "/usr/local/bin/caddy"}
2021/03/09 17:19:29.637 INFO requesting build {"os": "freebsd", "arch": "amd64", "packages": []}
2021/03/09 17:19:30.381 INFO build acquired; backing up current executable {"current_path": "/usr/local/bin/caddy", "backup_path": "/usr/local/bin/caddy.tmp"}
2021/03/09 17:19:30.381 INFO downloading binary {"source": "https://caddyserver.com/api/download?arch=amd64&os=freebsd", "destination": "/usr/local/bin/caddy"}
2021/03/09 17:19:42.494 INFO download successful; displaying new binary details {"location": "/usr/local/bin/caddy"}
Module versions:
admin.api.load v2.3.0
admin.api.metrics v2.3.0
caddy.adapters.caddyfile v2.3.0
caddy.listeners.tls v2.3.0
caddy.logging.encoders.console v2.3.0
caddy.logging.encoders.filter v2.3.0
caddy.logging.encoders.filter.delete v2.3.0
caddy.logging.encoders.filter.ip_mask v2.3.0
caddy.logging.encoders.json v2.3.0
caddy.logging.encoders.logfmt v2.3.0
caddy.logging.encoders.single_field v2.3.0
caddy.logging.writers.discard v2.3.0
caddy.logging.writers.file v2.3.0
caddy.logging.writers.net v2.3.0
caddy.logging.writers.stderr v2.3.0
caddy.logging.writers.stdout v2.3.0
caddy.storage.file_system v2.3.0
http v2.3.0
http.authentication.hashes.bcrypt v2.3.0
http.authentication.hashes.scrypt v2.3.0
http.authentication.providers.http_basic v2.3.0
http.encoders.gzip v2.3.0
http.encoders.zstd v2.3.0
http.handlers.acme_server v2.3.0
http.handlers.authentication v2.3.0
http.handlers.encode v2.3.0
http.handlers.error v2.3.0
http.handlers.file_server v2.3.0
http.handlers.headers v2.3.0
http.handlers.map v2.3.0
http.handlers.metrics v2.3.0
http.handlers.push v2.3.0
http.handlers.request_body v2.3.0
http.handlers.reverse_proxy v2.3.0
http.handlers.rewrite v2.3.0
http.handlers.static_response v2.3.0
http.handlers.subroute v2.3.0
http.handlers.templates v2.3.0
http.handlers.vars v2.3.0
http.matchers.expression v2.3.0
http.matchers.file v2.3.0
http.matchers.header v2.3.0
http.matchers.header_regexp v2.3.0
http.matchers.host v2.3.0
http.matchers.method v2.3.0
http.matchers.not v2.3.0
http.matchers.path v2.3.0
http.matchers.path_regexp v2.3.0
http.matchers.protocol v2.3.0
http.matchers.query v2.3.0
http.matchers.remote_ip v2.3.0
http.matchers.vars v2.3.0
http.matchers.vars_regexp v2.3.0
http.reverse_proxy.selection_policies.cookie v2.3.0
http.reverse_proxy.selection_policies.first v2.3.0
http.reverse_proxy.selection_policies.header v2.3.0
http.reverse_proxy.selection_policies.ip_hash v2.3.0
http.reverse_proxy.selection_policies.least_conn v2.3.0
http.reverse_proxy.selection_policies.random v2.3.0
http.reverse_proxy.selection_policies.random_choose v2.3.0
http.reverse_proxy.selection_policies.round_robin v2.3.0
http.reverse_proxy.selection_policies.uri_hash v2.3.0
http.reverse_proxy.transport.fastcgi v2.3.0
http.reverse_proxy.transport.http v2.3.0
pki v2.3.0
tls v2.3.0
pki v2.3.0
tls v2.3.0
tls.certificates.automate v2.3.0
tls.certificates.load_files v2.3.0
tls.certificates.load_folders v2.3.0
tls.certificates.load_pem v2.3.0
tls.handshake_match.sni v2.3.0
tls.issuance.acme v2.3.0
tls.issuance.internal v2.3.0
tls.issuance.zerossl v2.3.0
tls.stek.distributed v2.3.0
tls.stek.standard v2.3.0
Version:
v2.3.0 h1:fnrqJLa3G5vfxcxmOH/+kJOcunPLhSBnjgIvjXV/QTA=
2021/03/09 17:19:42.604 INFO upgrade successful; please restart any running Caddy instances {"executable": "/usr/local/bin/caddy"}
[root@wp-basil /usr/local/bin]# service caddy restart
Stopping caddy.
Waiting for PIDS: 76788.
[root@wp-basil /usr/local/bin]# caddy version
v2.3.0 h1:fnrqJLa3G5vfxcxmOH/+kJOcunPLhSBnjgIvjXV/QTA=
In addition, although the output from caddy upgrade suggests that the previous version of Caddy would be backed up as /usr/local/bin/caddy.tmp, the backup didn’t occur.