How to do the challenge with a different port?

To clarify further, both xenolf/lego and Caddy’s flags do effectively the same thing - they set the server to listen to the specified port for that class of traffic. Neither have the capability to instruct LetsEncrypt to change which port to issue a challenge to. For consideration, from the ACME draft specification, emphasis mine:

8.3. HTTP Challenge

[…]
Given a challenge/response pair, the server verifies the client’s control of the domain by verifying that the resource was provisioned as expected.
[…]
3. Dereference the URL using an HTTP GET request. This request MUST be sent to TCP port 80 on the HTTP server.

8.4. TLS with Server Name Indication (TLS SNI) Challenge

[…]
Given a challenge/response pair, the ACME server verifies the client’s control of the domain by verifying that the TLS server was configured appropriately, using these steps:
[…]
2. Open a TLS connection to the domain name being validated, presenting SAN A in the SNI field. This connection MUST be sent to TCP port 443 on the TLS server.