Reverse proxy https to https 502 error

Help
1

1. The problem I’m having:

I’m attempting to set up a reverse proxy to an https host. If allow http access on the server then the address resolves without issue.

If I disallow http access (the default and how I’d prefer to keep it) I get an HTTP 502 error. I also see a ‘ls: failed to verify certificate: x509: certificate signed by unknown authority’ error in the logs.

If I uncomment tls_insecure_skip_verify, then the cert error goes away but I get an HTTP 302 error instead.

Just to try, I tried turning auto_https off or disabling certs and I get a TLS Alert, internal error (592)

2. Error messages and/or full log output:

Output with http access allowed on the smx.lymeinternal.net server. (I apologise for the logs as journalctl -u caddy --no-pager | less +G` shows no entries, a separate problem)

curl -vL https://smx.lymeaccess.net:3445
* Host smx.lymeaccess.net:3445 was resolved.
* IPv6: (none)
* IPv4: 198.55.232.56
*   Trying 198.55.232.56:3445...
* Connected to smx.lymeaccess.net (198.55.232.56) port 3445
* ALPN: curl offers h2,http/1.1
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
*  CAfile: /etc/ssl/certs/ca-certificates.crt
*  CApath: /etc/ssl/certs
* TLSv1.3 (IN), TLS handshake, Server hello (2):
* TLSv1.3 (IN), TLS handshake, Encrypted Extensions (8):
* TLSv1.3 (IN), TLS handshake, Certificate (11):
* TLSv1.3 (IN), TLS handshake, CERT verify (15):
* TLSv1.3 (IN), TLS handshake, Finished (20):
* TLSv1.3 (OUT), TLS change cipher, Change cipher spec (1):
* TLSv1.3 (OUT), TLS handshake, Finished (20):
* SSL connection using TLSv1.3 / TLS_AES_128_GCM_SHA256 / X25519 / id-ecPublicKey
* ALPN: server accepted h2
* Server certificate:
*  subject: CN=smx.lymeaccess.net
*  start date: May  9 18:32:59 2025 GMT
*  expire date: Aug  7 18:32:58 2025 GMT
*  subjectAltName: host "smx.lymeaccess.net" matched cert's "smx.lymeaccess.net"
*  issuer: C=US; O=Let's Encrypt; CN=E6
*  SSL certificate verify ok.
*   Certificate level 0: Public key type EC/prime256v1 (256/128 Bits/secBits), signed using ecdsa-with-SHA384
*   Certificate level 1: Public key type EC/secp384r1 (384/192 Bits/secBits), signed using sha256WithRSAEncryption
*   Certificate level 2: Public key type RSA (4096/152 Bits/secBits), signed using sha256WithRSAEncryption
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
* using HTTP/2
* [HTTP/2] [1] OPENED stream for https://smx.lymeaccess.net:3445/
* [HTTP/2] [1] [:method: GET]
* [HTTP/2] [1] [:scheme: https]
* [HTTP/2] [1] [:authority: smx.lymeaccess.net:3445]
* [HTTP/2] [1] [:path: /]
* [HTTP/2] [1] [user-agent: curl/8.5.0]
* [HTTP/2] [1] [accept: */*]
> GET / HTTP/2
> Host: smx.lymeaccess.net:3445
> User-Agent: curl/8.5.0
> Accept: */*
> 
< HTTP/2 200 
< access-control-allow-origin: *
< alt-svc: h3=":3445"; ma=2592000
< content-security-policy: default-src 'self' data: https://blue.calix.com; script-src 'self' 'unsafe-inline' 'unsafe-eval' data: https://blue.calix.com; connect-src * 'self' data: https://blue.calix.com; img-src data: 'self' https://blue.calix.com; style-src 'self' 'unsafe-inline' data: https://blue.calix.com;
< content-type: text/html; charset=utf-8
< date: Thu, 22 May 2025 17:03:52 GMT
< p3p: CP="IDC DSP COR ADM DEVi TAIi PSA PSD IVAi IVDi CONi HIS OUR IND CNT"
< vary: Accept-Encoding
< via: 1.1 Caddy
< x-content-type-options: nosniff
< x-frame-options: DENY
< 
<!DOCTYPE html>
<html>
<head>
  <script id='meteor-headers' type='application/ejson'>{"token":1747933432280.0332,"headers":{"accept-encoding":"gzip","x-forwarded-proto":"https","x-forwarded-host":"smx.lymeaccess.net:3445","x-forwarded-for":"10.0.1.12","via":"2.0 Caddy","accept":"*/*","host":"smx.lymeaccess.net:3445","connection":"close","x-ip-chain":"10.0.1.12,127.0.0.1"}}</script>
<meta http-equiv="X-UA-Compatible" content="IE=10">
    <div style="text-align:center;position:absolute;left:40%;top:50%" id="initialSmxPageloading"><div class="lds-default"><div></div><div></div><div></div><div></div><div></div><div></div><div></div><div></div><div></div><div></div><div></div><div></div></div> &nbsp;&nbsp;<span style="font-size:14px;font-weight:bold">Loading please wait...</span></div>
  <link rel="stylesheet" type="text/css" class="__meteor-css__" href="/04b06726eaad8387d88a32980b7b2c20d6d2712a.css?meteor_css_resource=true">
<meta charset="utf-8">
    <title>SMx</title>
    <meta name="description" content="">
    <meta name="viewport" content="width=device-width, initial-scale=1.0, maximum-scale=1.0, user-scalable=no">
    <link rel="shortcut icon" href="/images/activate/activate-logo.png" />
    <!--link rel="apple-touch-icon" href="img/splash/sptouch-icon-iphone.png">
    <link rel="apple-touch-icon" sizes="76x76" href="img/splash/touch-icon-ipad.png">
    <link rel="apple-touch-icon" sizes="120x120" href="img/splash/touch-icon-iphone-retina.png">
    <link rel="apple-touch-icon" sizes="152x152" href="img/splash/touch-icon-ipad-retina.png">
    <!-- iOS web-app metas : hides Safari UI Components and Changes Status Bar Appearance -->
    <meta name="apple-mobile-web-app-capable" content="yes">
    <meta name="apple-mobile-web-app-status-bar-style" content="black">
    <!-- Startup image for web apps -->
    <link rel="apple-touch-startup-image" href="img/splash/ipad-landscape.png" media="screen and (min-device-width: 481px) and (max-device-width: 1024px) and (orientation:landscape)">
    <link rel="apple-touch-startup-image" href="img/splash/ipad-portrait.png" media="screen and (min-device-width: 481px) and (max-device-width: 1024px) and (orientation:portrait)">
    <link rel="apple-touch-startup-image" href="img/splash/iphone.png" media="screen and (max-device-width: 320px)" -->
    <script type="text/javascript">
    </script>
</head>
<body>
  <script type="text/javascript">__meteor_runtime_config__ = JSON.parse(decodeURIComponent("%7B%22meteorRelease%22%3A%22METEOR%402.16%22%2C%22gitCommitHash%22%3A%2205015b245173d3a052a61eb5dccd86cd6b311cd5%22%2C%22meteorEnv%22%3A%7B%22NODE_ENV%22%3A%22production%22%2C%22TEST_METADATA%22%3A%22%7B%7D%22%7D%2C%22PUBLIC_SETTINGS%22%3A%7B%22debugMode%22%3Afalse%2C%22pmaainterface%22%3A%22http%3A%2F%2F10.0.1.27%3A4001%2Frestapi.json%22%2C%22pmaaurl%22%3A%2210.0.1.27%3A4001%22%2C%22pmaaRestEndpoint%22%3A%7B%22url%22%3A%22https%3A%2F%2F10.0.1.27%3A18443%2Frest%2Fv1%22%2C%22username%22%3A%22U2FsdGVkX19DbVffva%2F3DE7Hzw93q36L%2BDQXfNhZrN8%3D%22%2C%22password%22%3A%22U2FsdGVkX1%2FYALRNsm75r5ZaJjYRBqDDk6Je0mvUk38%3D%22%2C%22API_PORT_INFO_ZIP_REPORT_DOWNLOAD_IN_CLUSTER_ENV%22%3A%22%3A18443%2Frest%2Fv1%22%7D%2C%22HTTP_NODE_HOST_URL%22%3A%2210.0.1.27%22%2C%22HTTP_NODE_APP_PORT%22%3A5050%2C%22HTTPS_NODE_APP_PORT%22%3A6060%7D%2C%22ROOT_URL%22%3A%22http%3A%2F%2Flocalhost%22%2C%22ROOT_URL_PATH_PREFIX%22%3A%22%22%2C%22reactFastRefreshEnabled%22%3Atrue%2C%22autoupdate%22%3A%7B%22versions%22%3A%7B%22web.browser%22%3A%7B%22version%22%3A%22bc9f08ebe24a7895b078a85e0d624545d33d8911%22%2C%22versionRefreshable%22%3A%22cb466d522a3fa0dbb07db0501e5d9cd6b1554032%22%2C%22versionNonRefreshable%22%3A%22ddebe56116507d6a404c59a811e60db3dc85af02%22%2C%22versionReplaceable%22%3A%22fde72c711cd965fe023eadaedafa4dbf0d783c3d%22%7D%2C%22web.browser.legacy%22%3A%7B%22version%22%3A%220ae0590d32f185cbd0b04f91ff4927c57857cf0b%22%2C%22versionRefreshable%22%3A%22cb466d522a3fa0dbb07db0501e5d9cd6b1554032%22%2C%22versionNonRefreshable%22%3A%22997fd5bda9ab685976974d126f374d7f26eb5e51%22%2C%22versionReplaceable%22%3A%22fde72c711cd965fe023eadaedafa4dbf0d783c3d%22%7D%7D%2C%22autoupdateVersion%22%3Anull%2C%22autoupdateVersionRefreshable%22%3Anull%2C%22autoupdateVersionCordova%22%3Anull%2C%22appId%22%3A%221f6utbarzx8ti1wz7axz%22%7D%2C%22appId%22%3A%221f6utbarzx8ti1wz7axz%22%2C%22isModern%22%3Afalse%7D"))</script>
  <script type="text/javascript" src="/40b167613d9106bbc07984c73c5a30d7208a1855.js?meteor_js_resource=true"></script>

journalctl -f -u caddy 
May 22 12:48:42 caddy systemd[1]: Started caddy.service - Caddy.
May 22 12:48:42 caddy caddy[45518]: {"level":"info","ts":1747932522.8310065,"msg":"serving initial configuration"}
May 22 12:48:42 caddy caddy[45518]: {"level":"info","ts":1747932522.8347163,"logger":"tls","msg":"storage cleaning happened too recently; skipping for now","storage":"FileStorage:/var/lib/caddy/.local/share/caddy","instance":"62fff2c8-6423-494c-b985-4276cd80fc82","try_again":1748018922.8347154,"try_again_in":86399.9999998}
May 22 12:48:42 caddy caddy[45518]: {"level":"info","ts":1747932522.8347795,"logger":"tls","msg":"finished cleaning storage units"}
May 22 12:48:50 caddy caddy[45518]: {"level":"debug","ts":1747932530.0338025,"logger":"events","msg":"event","name":"tls_get_certificate","id":"2f978f2d-46fe-45f4-bdc2-4c421e637064","origin":"tls","data":{"client_hello":{"CipherSuites":[4866,4867,4865,49196,49200,159,52393,52392,52394,49195,49199,158,49188,49192,107,49187,49191,103,49162,49172,57,49161,49171,51,157,156,61,60,53,47,255],"ServerName":"smx.lymeaccess.net","SupportedCurves":[29,23,30,25,24,256,257,258,259,260],"SupportedPoints":"AAEC","SignatureSchemes":[1027,1283,1539,2055,2056,2057,2058,2059,2052,2053,2054,1025,1281,1537,771,769,770,1026,1282,1538],"SupportedProtos":["h2","http/1.1"],"SupportedVersions":[772,771],"RemoteAddr":{"IP":"10.0.1.12","Port":36314,"Zone":""},"LocalAddr":{"IP":"10.0.1.17","Port":3445,"Zone":""}}}}
May 22 12:48:50 caddy caddy[45518]: {"level":"debug","ts":1747932530.0338962,"logger":"tls.handshake","msg":"choosing certificate","identifier":"smx.lymeaccess.net","num_choices":1}
May 22 12:48:50 caddy caddy[45518]: {"level":"debug","ts":1747932530.0339026,"logger":"tls.handshake","msg":"default certificate selection results","identifier":"smx.lymeaccess.net","subjects":["smx.lymeaccess.net"],"managed":true,"issuer_key":"acme-v02.api.letsencrypt.org-directory","hash":"ffbaafe7be4afc2c55eede738333c0b69c5b5df25ad68b9f74783de26ba54d6d"}
May 22 12:48:50 caddy caddy[45518]: {"level":"debug","ts":1747932530.0339093,"logger":"tls.handshake","msg":"matched certificate in cache","remote_ip":"10.0.1.12","remote_port":"36314","subjects":["smx.lymeaccess.net"],"managed":true,"expiration":1754591579,"hash":"ffbaafe7be4afc2c55eede738333c0b69c5b5df25ad68b9f74783de26ba54d6d"}
May 22 12:48:50 caddy caddy[45518]: {"level":"debug","ts":1747932530.0526247,"logger":"http.handlers.reverse_proxy","msg":"selected upstream","dial":"smx.lymeinternal.net:3443","total_upstreams":1}
May 22 12:48:50 caddy caddy[45518]: {"level":"debug","ts":1747932530.056786,"logger":"http.handlers.reverse_proxy","msg":"upstream roundtrip","upstream":"smx.lymeinternal.net:3443","duration":0.004127044,"request":{"remote_ip":"10.0.1.12","remote_port":"36314","client_ip":"10.0.1.12","proto":"HTTP/2.0","method":"GET","host":"smx.lymeaccess.net:3445","uri":"/","headers":{"User-Agent":["curl/8.5.0"],"Accept":["*/*"],"X-Forwarded-For":["10.0.1.12"],"X-Forwarded-Proto":["https"],"X-Forwarded-Host":["smx.lymeaccess.net:3445"],"Via":["2.0 Caddy"]},"tls":{"resumed":false,"version":772,"cipher_suite":4865,"proto":"h2","server_name":"smx.lymeaccess.net"}},"headers":{"Location":["https://smx.lymeaccess.net:3443/"],"Date":["Thu, 22 May 2025 16:48:50 GMT"]},"status":302}
May 22 13:02:37 caddy caddy[45518]: {"level":"debug","ts":1747933357.3790212,"logger":"events","msg":"event","name":"tls_get_certificate","id":"bf3b2a4f-852b-4603-8aa7-34db471f1c4f","origin":"tls","data":{"client_hello":{"CipherSuites":[4866,4867,4865,49196,49200,159,52393,52392,52394,49195,49199,158,49188,49192,107,49187,49191,103,49162,49172,57,49161,49171,51,157,156,61,60,53,47,255],"ServerName":"smx.lymeaccess.net","SupportedCurves":[29,23,30,25,24,256,257,258,259,260],"SupportedPoints":"AAEC","SignatureSchemes":[1027,1283,1539,2055,2056,2057,2058,2059,2052,2053,2054,1025,1281,1537,771,769,770,1026,1282,1538],"SupportedProtos":["h2","http/1.1"],"SupportedVersions":[772,771],"RemoteAddr":{"IP":"10.0.1.12","Port":52580,"Zone":""},"LocalAddr":{"IP":"10.0.1.17","Port":3445,"Zone":""}}}}
May 22 13:02:37 caddy caddy[45518]: {"level":"debug","ts":1747933357.3790817,"logger":"tls.handshake","msg":"choosing certificate","identifier":"smx.lymeaccess.net","num_choices":1}
May 22 13:02:37 caddy caddy[45518]: {"level":"debug","ts":1747933357.3790922,"logger":"tls.handshake","msg":"default certificate selection results","identifier":"smx.lymeaccess.net","subjects":["smx.lymeaccess.net"],"managed":true,"issuer_key":"acme-v02.api.letsencrypt.org-directory","hash":"ffbaafe7be4afc2c55eede738333c0b69c5b5df25ad68b9f74783de26ba54d6d"}
May 22 13:02:37 caddy caddy[45518]: {"level":"debug","ts":1747933357.3791022,"logger":"tls.handshake","msg":"matched certificate in cache","remote_ip":"10.0.1.12","remote_port":"52580","subjects":["smx.lymeaccess.net"],"managed":true,"expiration":1754591579,"hash":"ffbaafe7be4afc2c55eede738333c0b69c5b5df25ad68b9f74783de26ba54d6d"}
May 22 13:02:37 caddy caddy[45518]: {"level":"debug","ts":1747933357.3977475,"logger":"http.handlers.reverse_proxy","msg":"selected upstream","dial":"smx.lymeinternal.net:3443","total_upstreams":1}
May 22 13:02:37 caddy caddy[45518]: {"level":"debug","ts":1747933357.412196,"logger":"http.handlers.reverse_proxy","msg":"upstream roundtrip","upstream":"smx.lymeinternal.net:3443","duration":0.014408559,"request":{"remote_ip":"10.0.1.12","remote_port":"52580","client_ip":"10.0.1.12","proto":"HTTP/2.0","method":"GET","host":"smx.lymeaccess.net:3445","uri":"/","headers":{"User-Agent":["curl/8.5.0"],"Accept":["*/*"],"X-Forwarded-For":["10.0.1.12"],"X-Forwarded-Proto":["https"],"X-Forwarded-Host":["smx.lymeaccess.net:3445"],"Via":["2.0 Caddy"]},"tls":{"resumed":false,"version":772,"cipher_suite":4865,"proto":"h2","server_name":"smx.lymeaccess.net"}},"headers":{"Content-Type":["text/html; charset=utf-8"],"Date":["Thu, 22 May 2025 17:02:37 GMT"],"X-Content-Type-Options":["nosniff"],"P3p":["CP=\"IDC DSP COR ADM DEVi TAIi PSA PSD IVAi IVDi CONi HIS OUR IND CNT\""],"X-Frame-Options":["DENY"],"Content-Security-Policy":["default-src 'self' data: https://blue.calix.com; script-src 'self' 'unsafe-inline' 'unsafe-eval' data: https://blue.calix.com; connect-src * 'self' data: https://blue.calix.com; img-src data: 'self' https://blue.calix.com; style-src 'self' 'unsafe-inline' data: https://blue.calix.com;"],"Vary":["Accept-Encoding"],"Access-Control-Allow-Origin":["*"]},"status":200}
May 22 13:03:52 caddy caddy[45518]: {"level":"debug","ts":1747933432.258722,"logger":"events","msg":"event","name":"tls_get_certificate","id":"a600e97e-c6cd-4ca9-9e67-4ddb3d461c86","origin":"tls","data":{"client_hello":{"CipherSuites":[4866,4867,4865,49196,49200,159,52393,52392,52394,49195,49199,158,49188,49192,107,49187,49191,103,49162,49172,57,49161,49171,51,157,156,61,60,53,47,255],"ServerName":"smx.lymeaccess.net","SupportedCurves":[29,23,30,25,24,256,257,258,259,260],"SupportedPoints":"AAEC","SignatureSchemes":[1027,1283,1539,2055,2056,2057,2058,2059,2052,2053,2054,1025,1281,1537,771,769,770,1026,1282,1538],"SupportedProtos":["h2","http/1.1"],"SupportedVersions":[772,771],"RemoteAddr":{"IP":"10.0.1.12","Port":49628,"Zone":""},"LocalAddr":{"IP":"10.0.1.17","Port":3445,"Zone":""}}}}
May 22 13:03:52 caddy caddy[45518]: {"level":"debug","ts":1747933432.2587845,"logger":"tls.handshake","msg":"choosing certificate","identifier":"smx.lymeaccess.net","num_choices":1}
May 22 13:03:52 caddy caddy[45518]: {"level":"debug","ts":1747933432.2596972,"logger":"tls.handshake","msg":"default certificate selection results","identifier":"smx.lymeaccess.net","subjects":["smx.lymeaccess.net"],"managed":true,"issuer_key":"acme-v02.api.letsencrypt.org-directory","hash":"ffbaafe7be4afc2c55eede738333c0b69c5b5df25ad68b9f74783de26ba54d6d"}
May 22 13:03:52 caddy caddy[45518]: {"level":"debug","ts":1747933432.2597225,"logger":"tls.handshake","msg":"matched certificate in cache","remote_ip":"10.0.1.12","remote_port":"49628","subjects":["smx.lymeaccess.net"],"managed":true,"expiration":1754591579,"hash":"ffbaafe7be4afc2c55eede738333c0b69c5b5df25ad68b9f74783de26ba54d6d"}
May 22 13:03:52 caddy caddy[45518]: {"level":"debug","ts":1747933432.276992,"logger":"http.handlers.reverse_proxy","msg":"selected upstream","dial":"smx.lymeinternal.net:3443","total_upstreams":1}
May 22 13:03:52 caddy caddy[45518]: {"level":"debug","ts":1747933432.2833946,"logger":"http.handlers.reverse_proxy","msg":"upstream roundtrip","upstream":"smx.lymeinternal.net:3443","duration":0.006361476,"request":{"remote_ip":"10.0.1.12","remote_port":"49628","client_ip":"10.0.1.12","proto":"HTTP/2.0","method":"GET","host":"smx.lymeaccess.net:3445","uri":"/","headers":{"X-Forwarded-Proto":["https"],"X-Forwarded-Host":["smx.lymeaccess.net:3445"],"Via":["2.0 Caddy"],"User-Agent":["curl/8.5.0"],"Accept":["*/*"],"X-Forwarded-For":["10.0.1.12"]},"tls":{"resumed":false,"version":772,"cipher_suite":4865,"proto":"h2","server_name":"smx.lymeaccess.net"}},"headers":{"Content-Security-Policy":["default-src 'self' data: https://blue.calix.com; script-src 'self' 'unsafe-inline' 'unsafe-eval' data: https://blue.calix.com; connect-src * 'self' data: https://blue.calix.com; img-src data: 'self' https://blue.calix.com; style-src 'self' 'unsafe-inline' data: https://blue.calix.com;"],"Content-Type":["text/html; charset=utf-8"],"X-Content-Type-Options":["nosniff"],"Access-Control-Allow-Origin":["*"],"P3p":["CP=\"IDC DSP COR ADM DEVi TAIi PSA PSD IVAi IVDi CONi HIS OUR IND CNT\""],"Vary":["Accept-Encoding"],"Date":["Thu, 22 May 2025 17:03:52 GMT"],"X-Frame-Options":["DENY"]},"status":200}

Output with http access disallowed on the smx.lymeinternal.net server.

curl -vL https://smx.lymeaccess.net:3445
* Host smx.lymeaccess.net:3445 was resolved.
* IPv6: (none)
* IPv4: 198.55.232.56
*   Trying 198.55.232.56:3445...
* Connected to smx.lymeaccess.net (198.55.232.56) port 3445
* ALPN: curl offers h2,http/1.1
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
*  CAfile: /etc/ssl/certs/ca-certificates.crt
*  CApath: /etc/ssl/certs
* TLSv1.3 (IN), TLS handshake, Server hello (2):
* TLSv1.3 (IN), TLS handshake, Encrypted Extensions (8):
* TLSv1.3 (IN), TLS handshake, Certificate (11):
* TLSv1.3 (IN), TLS handshake, CERT verify (15):
* TLSv1.3 (IN), TLS handshake, Finished (20):
* TLSv1.3 (OUT), TLS change cipher, Change cipher spec (1):
* TLSv1.3 (OUT), TLS handshake, Finished (20):
* SSL connection using TLSv1.3 / TLS_AES_128_GCM_SHA256 / X25519 / id-ecPublicKey
* ALPN: server accepted h2
* Server certificate:
*  subject: CN=smx.lymeaccess.net
*  start date: May  9 18:32:59 2025 GMT
*  expire date: Aug  7 18:32:58 2025 GMT
*  subjectAltName: host "smx.lymeaccess.net" matched cert's "smx.lymeaccess.net"
*  issuer: C=US; O=Let's Encrypt; CN=E6
*  SSL certificate verify ok.
*   Certificate level 0: Public key type EC/prime256v1 (256/128 Bits/secBits), signed using ecdsa-with-SHA384
*   Certificate level 1: Public key type EC/secp384r1 (384/192 Bits/secBits), signed using sha256WithRSAEncryption
*   Certificate level 2: Public key type RSA (4096/152 Bits/secBits), signed using sha256WithRSAEncryption
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
* using HTTP/2
* [HTTP/2] [1] OPENED stream for https://smx.lymeaccess.net:3445/
* [HTTP/2] [1] [:method: GET]
* [HTTP/2] [1] [:scheme: https]
* [HTTP/2] [1] [:authority: smx.lymeaccess.net:3445]
* [HTTP/2] [1] [:path: /]
* [HTTP/2] [1] [user-agent: curl/8.5.0]
* [HTTP/2] [1] [accept: */*]
> GET / HTTP/2
> Host: smx.lymeaccess.net:3445
> User-Agent: curl/8.5.0
> Accept: */*
> 
< HTTP/2 502 
< alt-svc: h3=":3445"; ma=2592000
< server: Caddy
< content-length: 0
< date: Thu, 22 May 2025 16:45:42 GMT
< 
* Connection #0 to host smx.lymeaccess.net left intact

root@caddy:/# journalctl -f -u caddy 
May 22 12:45:39 caddy caddy[45497]: {"level":"info","ts":1747932339.4033139,"msg":"serving initial configuration"}
May 22 12:45:39 caddy caddy[45497]: {"level":"info","ts":1747932339.408745,"logger":"tls","msg":"storage cleaning happened too recently; skipping for now","storage":"FileStorage:/var/lib/caddy/.local/share/caddy","instance":"62fff2c8-6423-494c-b985-4276cd80fc82","try_again":1748018739.4087439,"try_again_in":86399.99999977}
May 22 12:45:39 caddy caddy[45497]: {"level":"info","ts":1747932339.4088173,"logger":"tls","msg":"finished cleaning storage units"}
May 22 12:45:42 caddy caddy[45497]: {"level":"debug","ts":1747932342.046613,"logger":"events","msg":"event","name":"tls_get_certificate","id":"e3c40d53-5888-4066-9871-209fcff8c225","origin":"tls","data":{"client_hello":{"CipherSuites":[4866,4867,4865,49196,49200,159,52393,52392,52394,49195,49199,158,49188,49192,107,49187,49191,103,49162,49172,57,49161,49171,51,157,156,61,60,53,47,255],"ServerName":"smx.lymeaccess.net","SupportedCurves":[29,23,30,25,24,256,257,258,259,260],"SupportedPoints":"AAEC","SignatureSchemes":[1027,1283,1539,2055,2056,2057,2058,2059,2052,2053,2054,1025,1281,1537,771,769,770,1026,1282,1538],"SupportedProtos":["h2","http/1.1"],"SupportedVersions":[772,771],"RemoteAddr":{"IP":"10.0.1.12","Port":36562,"Zone":""},"LocalAddr":{"IP":"10.0.1.17","Port":3445,"Zone":""}}}}
May 22 12:45:42 caddy caddy[45497]: {"level":"debug","ts":1747932342.0467558,"logger":"tls.handshake","msg":"choosing certificate","identifier":"smx.lymeaccess.net","num_choices":1}
May 22 12:45:42 caddy caddy[45497]: {"level":"debug","ts":1747932342.0467644,"logger":"tls.handshake","msg":"default certificate selection results","identifier":"smx.lymeaccess.net","subjects":["smx.lymeaccess.net"],"managed":true,"issuer_key":"acme-v02.api.letsencrypt.org-directory","hash":"ffbaafe7be4afc2c55eede738333c0b69c5b5df25ad68b9f74783de26ba54d6d"}
May 22 12:45:42 caddy caddy[45497]: {"level":"debug","ts":1747932342.0467715,"logger":"tls.handshake","msg":"matched certificate in cache","remote_ip":"10.0.1.12","remote_port":"36562","subjects":["smx.lymeaccess.net"],"managed":true,"expiration":1754591579,"hash":"ffbaafe7be4afc2c55eede738333c0b69c5b5df25ad68b9f74783de26ba54d6d"}
May 22 12:45:42 caddy caddy[45497]: {"level":"debug","ts":1747932342.0652976,"logger":"http.handlers.reverse_proxy","msg":"selected upstream","dial":"smx.lymeinternal.net:3443","total_upstreams":1}
May 22 12:45:42 caddy caddy[45497]: {"level":"debug","ts":1747932342.0674367,"logger":"http.handlers.reverse_proxy","msg":"upstream roundtrip","upstream":"smx.lymeinternal.net:3443","duration":0.002111685,"request":{"remote_ip":"10.0.1.12","remote_port":"36562","client_ip":"10.0.1.12","proto":"HTTP/2.0","method":"GET","host":"smx.lymeaccess.net:3445","uri":"/","headers":{"User-Agent":["curl/8.5.0"],"Accept":["*/*"],"X-Forwarded-For":["10.0.1.12"],"X-Forwarded-Proto":["https"],"X-Forwarded-Host":["smx.lymeaccess.net:3445"],"Via":["2.0 Caddy"]},"tls":{"resumed":false,"version":772,"cipher_suite":4865,"proto":"h2","server_name":"smx.lymeaccess.net"}},"error":"tls: failed to verify certificate: x509: certificate signed by unknown authority"}
May 22 12:45:42 caddy caddy[45497]: {"level":"error","ts":1747932342.0674784,"logger":"http.log.error","msg":"tls: failed to verify certificate: x509: certificate signed by unknown authority","request":{"remote_ip":"10.0.1.12","remote_port":"36562","client_ip":"10.0.1.12","proto":"HTTP/2.0","method":"GET","host":"smx.lymeaccess.net:3445","uri":"/","headers":{"Accept":["*/*"],"User-Agent":["curl/8.5.0"]},"tls":{"resumed":false,"version":772,"cipher_suite":4865,"proto":"h2","server_name":"smx.lymeaccess.net"}},"duration":0.002247692,"status":502,"err_id":"awux4x0mu","err_trace":"reverseproxy.statusError (reverseproxy.go:1390)"}

Output with http access disallowed on the smx.lymeinternal.net server and tls_insecure_skip_verify uncommented.

curl -vL https://smx.lymeaccess.net:3445
* Host smx.lymeaccess.net:3445 was resolved.
* IPv6: (none)
* IPv4: 198.55.232.56
*   Trying 198.55.232.56:3445...
* Connected to smx.lymeaccess.net (198.55.232.56) port 3445
* ALPN: curl offers h2,http/1.1
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
*  CAfile: /etc/ssl/certs/ca-certificates.crt
*  CApath: /etc/ssl/certs
* TLSv1.3 (IN), TLS handshake, Server hello (2):
* TLSv1.3 (IN), TLS handshake, Encrypted Extensions (8):
* TLSv1.3 (IN), TLS handshake, Certificate (11):
* TLSv1.3 (IN), TLS handshake, CERT verify (15):
* TLSv1.3 (IN), TLS handshake, Finished (20):
* TLSv1.3 (OUT), TLS change cipher, Change cipher spec (1):
* TLSv1.3 (OUT), TLS handshake, Finished (20):
* SSL connection using TLSv1.3 / TLS_AES_128_GCM_SHA256 / X25519 / id-ecPublicKey
* ALPN: server accepted h2
* Server certificate:
*  subject: CN=smx.lymeaccess.net
*  start date: May  9 18:32:59 2025 GMT
*  expire date: Aug  7 18:32:58 2025 GMT
*  subjectAltName: host "smx.lymeaccess.net" matched cert's "smx.lymeaccess.net"
*  issuer: C=US; O=Let's Encrypt; CN=E6
*  SSL certificate verify ok.
*   Certificate level 0: Public key type EC/prime256v1 (256/128 Bits/secBits), signed using ecdsa-with-SHA384
*   Certificate level 1: Public key type EC/secp384r1 (384/192 Bits/secBits), signed using sha256WithRSAEncryption
*   Certificate level 2: Public key type RSA (4096/152 Bits/secBits), signed using sha256WithRSAEncryption
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
* using HTTP/2
* [HTTP/2] [1] OPENED stream for https://smx.lymeaccess.net:3445/
* [HTTP/2] [1] [:method: GET]
* [HTTP/2] [1] [:scheme: https]
* [HTTP/2] [1] [:authority: smx.lymeaccess.net:3445]
* [HTTP/2] [1] [:path: /]
* [HTTP/2] [1] [user-agent: curl/8.5.0]
* [HTTP/2] [1] [accept: */*]
> GET / HTTP/2
> Host: smx.lymeaccess.net:3445
> User-Agent: curl/8.5.0
> Accept: */*
> 
< HTTP/2 302 
< alt-svc: h3=":3445"; ma=2592000
< date: Thu, 22 May 2025 16:48:50 GMT
< via: 1.1 Caddy
< 
* Connection #0 to host smx.lymeaccess.net left intact

journalctl -f -u caddy 
May 22 12:48:42 caddy systemd[1]: Started caddy.service - Caddy.
May 22 12:48:42 caddy caddy[45518]: {"level":"info","ts":1747932522.8310065,"msg":"serving initial configuration"}
May 22 12:48:42 caddy caddy[45518]: {"level":"info","ts":1747932522.8347163,"logger":"tls","msg":"storage cleaning happened too recently; skipping for now","storage":"FileStorage:/var/lib/caddy/.local/share/caddy","instance":"62fff2c8-6423-494c-b985-4276cd80fc82","try_again":1748018922.8347154,"try_again_in":86399.9999998}
May 22 12:48:42 caddy caddy[45518]: {"level":"info","ts":1747932522.8347795,"logger":"tls","msg":"finished cleaning storage units"}
May 22 12:48:50 caddy caddy[45518]: {"level":"debug","ts":1747932530.0338025,"logger":"events","msg":"event","name":"tls_get_certificate","id":"2f978f2d-46fe-45f4-bdc2-4c421e637064","origin":"tls","data":{"client_hello":{"CipherSuites":[4866,4867,4865,49196,49200,159,52393,52392,52394,49195,49199,158,49188,49192,107,49187,49191,103,49162,49172,57,49161,49171,51,157,156,61,60,53,47,255],"ServerName":"smx.lymeaccess.net","SupportedCurves":[29,23,30,25,24,256,257,258,259,260],"SupportedPoints":"AAEC","SignatureSchemes":[1027,1283,1539,2055,2056,2057,2058,2059,2052,2053,2054,1025,1281,1537,771,769,770,1026,1282,1538],"SupportedProtos":["h2","http/1.1"],"SupportedVersions":[772,771],"RemoteAddr":{"IP":"10.0.1.12","Port":36314,"Zone":""},"LocalAddr":{"IP":"10.0.1.17","Port":3445,"Zone":""}}}}
May 22 12:48:50 caddy caddy[45518]: {"level":"debug","ts":1747932530.0338962,"logger":"tls.handshake","msg":"choosing certificate","identifier":"smx.lymeaccess.net","num_choices":1}
May 22 12:48:50 caddy caddy[45518]: {"level":"debug","ts":1747932530.0339026,"logger":"tls.handshake","msg":"default certificate selection results","identifier":"smx.lymeaccess.net","subjects":["smx.lymeaccess.net"],"managed":true,"issuer_key":"acme-v02.api.letsencrypt.org-directory","hash":"ffbaafe7be4afc2c55eede738333c0b69c5b5df25ad68b9f74783de26ba54d6d"}
May 22 12:48:50 caddy caddy[45518]: {"level":"debug","ts":1747932530.0339093,"logger":"tls.handshake","msg":"matched certificate in cache","remote_ip":"10.0.1.12","remote_port":"36314","subjects":["smx.lymeaccess.net"],"managed":true,"expiration":1754591579,"hash":"ffbaafe7be4afc2c55eede738333c0b69c5b5df25ad68b9f74783de26ba54d6d"}
May 22 12:48:50 caddy caddy[45518]: {"level":"debug","ts":1747932530.0526247,"logger":"http.handlers.reverse_proxy","msg":"selected upstream","dial":"smx.lymeinternal.net:3443","total_upstreams":1}
May 22 12:48:50 caddy caddy[45518]: {"level":"debug","ts":1747932530.056786,"logger":"http.handlers.reverse_proxy","msg":"upstream roundtrip","upstream":"smx.lymeinternal.net:3443","duration":0.004127044,"request":{"remote_ip":"10.0.1.12","remote_port":"36314","client_ip":"10.0.1.12","proto":"HTTP/2.0","method":"GET","host":"smx.lymeaccess.net:3445","uri":"/","headers":{"User-Agent":["curl/8.5.0"],"Accept":["*/*"],"X-Forwarded-For":["10.0.1.12"],"X-Forwarded-Proto":["https"],"X-Forwarded-Host":["smx.lymeaccess.net:3445"],"Via":["2.0 Caddy"]},"tls":{"resumed":false,"version":772,"cipher_suite":4865,"proto":"h2","server_name":"smx.lymeaccess.net"}},"headers":{"Location":["https://smx.lymeaccess.net:3443/"],"Date":["Thu, 22 May 2025 16:48:50 GMT"]},"status":302}

3. Caddy version:

caddy version v2.10.0

4. How I installed and ran Caddy:

xcaddy build
–with GitHub - caddy-dns/cloudflare: Caddy module: dns.providers.cloudflare

a. System environment:

Debian 12 systemd

b. Command:

sudo systemctl daemon-reload
sudo systemctl enable --now caddy

c. Service/unit/compose file:

[Unit]
Description=Caddy
Documentation=https://caddyserver.com/docs/
After=network.target network-online.target
Requires=network-online.target

[Service]
Type=notify
User=caddy
Group=caddy
ExecStart=/usr/bin/caddy run --environ --config /etc/caddy/Caddyfile
ExecReload=/usr/bin/caddy reload --config /etc/caddy/Caddyfile --force
TimeoutStopSec=5s
LimitNOFILE=1048576
PrivateTmp=true
ProtectSystem=full
AmbientCapabilities=CAP_NET_ADMIN CAP_NET_BIND_SERVICE

[Install]
WantedBy=multi-user.target

d. My complete Caddy config:

{
        debug
        #       auto_https disable_certs
}
(headers) {
        header {
                -location
        }
}
https://smx.lymeaccess.net:3445 {
        reverse_proxy https://smx.lymeinternal.net:3443 {
                transport http {
                        tls
                        #                       tls_insecure_skip_verify
                        tls_server_name smx.lymeaccess.net
                }
        }
        import headers
}
https://smx.lymeaccess.net:18445 {
        reverse_proxy https://10.0.1.27:18443 {
                transport http {
                        tls
                        tls_insecure_skip_verify
                        tls_server_name smx.lymeaccess.net
                }
        }
}

5. Links to relevant resources:

Settings changed on smx.lymeaccess.net. As of posting http access is disabled.

Y: You are enabling http access and can access SMx with http on port 3000 and https on port 3443
N: You are disabling http access and can access SMx with https on port 3443

2

HTTP 302 isn’t actually an error. It’s just the back-end (https://smx.lymeinternal.net:3443) telling the client where to go next. Any particular reason you’re treating it as an error?

From what I can tell, this is expected behavior. But since you’re removing the Location response header, you’re breaking the redirect. The 302 status tells the client to go somewhere else, and the Location header specifies where. Without it, the redirect can’t work as intended.

2 Likes
3

Ah right, thank you. I removed the location header as the Location was showing up as https://smx.lymeaccess.net:3443 which is not a resolvable address (by design). Wrong header was my first theory on why the link to our server wasn’t resolving, though perhaps simply removing the location header was not the right solution.

4

Try this instead:

https://smx.lymeaccess.net:3445 {
        reverse_proxy https://smx.lymeinternal.net:3443 {
                transport http {
                        tls
                        tls_insecure_skip_verify
                        tls_server_name smx.lymeaccess.net
                        header_up Host {upstream_hostport}
                }
        }
}

It’s possible that your internal site at smx.lymeinternal.net:3443 doesn’t like being accessed as smx.lymeaccess.net:3445, which is the default Host header Caddy sends. That could be why it’s redirecting you to https://smx.lymeaccess.net:3443. Let’s see if this helps.

1 Like
5

Hmm, perhaps not that header precisely, but that seems to be on the right track. Presumably if the location were changed to https://smx.lymeinternalnet:3443 that would work. I’ll give that a try

< HTTP/2 302 
< alt-svc: h3=":3445"; ma=2592000
< date: Fri, 23 May 2025 13:22:06 GMT
< location: https://0.0.0.0:3443/
< via: 1.1 Caddy
< 
* Connection #0 to host smx.lymeaccess.net left intact
* Clear auth, redirects to port from 3445 to 3443
* Issue another request to this URL: 'https://0.0.0.0:3443/'
*   Trying 0.0.0.0:3443...
* connect to 0.0.0.0 port 3443 from 127.0.0.1 port 54368 failed: Connection refused
* Failed to connect to 0.0.0.0 port 3443 after 0 ms: Couldn't connect to server
* Closing connection
curl: (7) Failed to connect to 0.0.0.0 port 3443 after 0 ms: Couldn't connect to server
6

Oh wait or maybe if I try not typoing the word port then it actually works perfectly fine. Thank you!

1 Like