Hosting on GCP domain is on Network Solutions, need SSL certificate automation

1. The problem I’m having:

We are using GKE as our hosting where we have Frontend deployed with service type nodeport and a Caddy deployment with a service type loadbalancer.
Then we have Configmap that has our Caddyfile inside using this file we are doing reverse_proxy to frontend container. So the request follow is like below
User request → Caddy Loadbalancer → Frontend

Our domain is on Google Domains and inside our Caddy Image we are using Caddy module for Google Domains GitHub - caddy-dns/google-domains: Support for ACME DNS challenge through Google Domains. So basically things are working perfectly fine. We have created an A record on Google Domains with the IP of our Caddy Loadbalancer, that give us SSL certificate easily.

Also we are using Caddy module for AWS S3 certmagic GitHub - ss098/certmagic-s3: Certmagic S3-compatible driver written in Go | Caddy 证书存储 Certmagic 接口分布式对象存储实现 to get SSL certificates in S3 bucket. It also work perfectly fine

Problems starts here:
Our Caddy LoadBalancer IP we have given to our Client and they bought domain from Network Solutions. They create an A record there with our Caddy LoadBalancer IP address. When they try to access our front-end application they can’t access it they get SSL issue.
Customer request follow
customerdomain → Caddy Loadbalancer → :x: FrontEnd Application

They can’t reach our FrontEnd because they are not secure. Caddy won’t let’s their pass to the FrontEnd Applicaiton because it’s in between Client and Frontend

When we remove Caddy they can reach our frontend and only on HTTP technically that is correct because Caddy is handling SSL if no caddy no SSL.

We don’t want to eliminate Caddy from the picture we want Caddy handle our SSL and user request.

We want a suggestion from anyone if we Caddy doesn’t have any module, for the party that is our customer is using for domain name in our case Network Solutions. How we can get SSL certificate for our client.

2. Error messages and/or full log output:

We don't have logs ATM because our plan is known our Customer on us for SSL and hosting

3. Caddy version:

FROM caddy:2.7.5-alpine

4. How I installed and ran Caddy:

xcaddy build to install modules

a. System environment:

Docker

b. Command:

c. Service/unit/compose file:

FROM caddy:2.7.5-builder AS builder

# Copy the service account key JSON file into the builder image
COPY service-account-key.json /
RUN chmod 644 /service-account-key.json

# Build Caddy with the Google Cloud DNS module
RUN xcaddy build --with github.com/ss098/certmagic-s3 --with github.com/caddy-dns/google-domains
# RUN xcaddy build --output /tmp/caddy --with github.com/caddy-dns/googleclouddns
# Use the official Caddy runtime image
FROM caddy:2.7.5-alpine

# Copy the built Caddy binary from the builder image
COPY --from=builder /usr/bin/caddy /usr/bin/caddy

# This bring the file at this path /srv
COPY --from=builder /service-account-key.json .
# COPY --from=builder /service-account-key.json /etc/caddy/

# Copy the service account key JSON file into the Docker image
# COPY --from=builder ./srv/service-account-key.json /etc/caddy/service-account-key.json

# Set the GOOGLE_APPLICATION_CREDENTIALS environment variable
ENV GOOGLE_APPLICATION_CREDENTIALS service-account-key.json

# Expose the default Caddy ports
EXPOSE 80 443

d. My complete Caddy config:

  Caddyfile: |
    {
      storage s3 {
          host "s3.amazonaws.com"
          bucket "bucket_name"
          access_id "access_id"
          secret_key "secret_key"
          prefix "ssl"
      }

    }

    (redirect) {
        @http {
            protocol http
        }
        redir @http https://{host}{uri}
    }
    sub.toplevel.com {
        tls {
          dns google_domains access_token_here==
        }
        log {
          output stdout
        }

        import redirect

        encode gzip

        reverse_proxy frontenddefault.svc.cluster.local:3000

        file_server
    }

5. Links to relevant resources:

FYI, Google Domains is shutting down and being transferred to SquareSpace. I suggest you keep close tabs on that, or switch DNS providers. The plugin is no longer maintained and will never receive any updates, because there’s no point supporting a service that’s shutting down.

Sounds like you’re looking for On-Demand TLS:

This is what you should use if you need Caddy to maintain certs for domains outside of your control.

Also, I suggest you get your customers to point a CNAME to one of your domains instead of an A record, because then if you need to change your own IP you only change your own domains and you don’t need to worry about your customer domains because it’ll just follow along.

This doesn’t do anything at all for you. Caddy automatically sets up HTTP->HTTPS redirects. Remove that.

This doesn’t make sense, file_server does nothing in this config because you didn’t set root to anything, and you didn’t set up a matcher to tell Caddy which requests should go to the proxy and which should use the file server, so it only ever uses reverse_proxy.

1 Like

Can you give me example Caddyfile because I tried that at some point but it was also not working below is how I configured. If it’s the write approach let me know I will try to get error logs as well.

  Caddyfile: |
    {
      email arbutt1010@gmail.com
    }

    https://10.0.0.0 { # Here is i have Caddy LB external IP
        tls {
            on_demand
            on_demand_tls {
              ask http://sub.topleveldomain.com
              ask http://mycustomer-topleveldomain.com
            } 
        }
        log {
          output stdout
        }

        encode gzip

        reverse_proxy codi-portal-caddy-version-another.default.svc.cluster.local:3000

        file_server
    }

If above is wrong can you send me example Caddyfile for it please

Remove the IP address. It should only be https://

If you use a hostname there, then Caddy will only match requests with exactly that hostname.

This goes in global options, not inside your site. And you can only have one ask, it needs to be an endpoint that validates that the requesting domain is allowed.

Please re-read the docs and the wiki article the docs link to. It explains in detail.

My Dockerfile for Caddy:

FROM caddy:2.7.5-builder AS builder
RUN xcaddy build 
FROM caddy:2.7.5-alpine
COPY --from=builder /usr/bin/caddy /usr/bin/caddy
EXPOSE 80 443

My Caddyfile:
It’s a configmap manifest that why there are spaces being block for alignment

  Caddyfile: |
    {
      email myemail@gmail.com
      on_demand_tls {
        ask http://mycustomer-topleveldomain.com/check
      } 
    }

    https:// {
        tls {
            on_demand
        }

        log {
          output stdout
        }

        encode gzip

        reverse_proxy frontend-application.default.svc.cluster.local:3000
    }

Inside our front-end application we have a route to go to the backend and that works.
In our backend I have create below script

app.get('/check', (req, res) => {
      const domain = req.query.domain;
    
      // Implement your domain authorization logic here
      const authorized = isDomainAuthorized(domain);
    
      if (authorized) {
        res.status(200).json({ message: `Domain ${domain} is authorized` });
      } else {
        res.status(403).json({ message: `Domain ${domain} is not authorized` });
      }
    });
    
    function isDomainAuthorized(domain) {
      // Return true if authorized, false otherwise
      return domain === 'mycustomer-topleveldomain.com';
    }

When I run my application on localhost and when I search in browser localhost:8080/check?domain=mycustomer-topleveldomain.com
I get this Domain mycustomer-topleveldomain.com is authorized
When I do inspect inside network I get statuscode 200

Request Follow:
Client Request → Caddy LoadBalancer → Reverse_Proxy → Frontend → Backend → /check path is here

Backend Server:
Below are is what I get when I seach my backend-ip/check?domain=mycustomer-topleveldomain.com

General

Request URL: http://ip-address-of-the-backend-where-logic-check-logic-is-running/check?domain=cache-cloud-api.com
Request Method: GET
Status Code: 200 OK
Remote Address: ip-address-of-the-backend-where-logic-check-logic-is-running:80
Referrer Policy: strict-origin-when-cross-origin

Response Header

Access-Control-Allow-Credentials: true
Access-Control-Allow-Headers: Content-Type, Authorization, Token
Access-Control-Allow-Methods: GET, POST, PUT, DELETE
Access-Control-Allow-Origin: undefined
Connection: keep-alive
Content-Length: 54
Content-Type: application/json; charset=utf-8
Date: Thu, 16 Nov 2023 21:43:57 GMT
Etag: W/"sdfgsdfgergvdeg"
Keep-Alive: timeout=5
Vary: Origin
X-Powered-By: Express

Request Headers:

Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.7
Accept-Encoding: gzip, deflate
Accept-Language: en-US,en;q=0.9
Cache-Control: no-cache
Connection: keep-alive
Host: ip-address-of-the-backend-where-logic-check-logic-is-running
Pragma: no-cache
Upgrade-Insecure-Requests: 1
User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/119.0.0.0 Safari/537.36

In Real Development Environment Case:
But now when I run mycustomer-topleveldomain.com in browser I’m not able to get my application
And when I do inspect

General:

Request URL: http://mycustomer-topleveldomain.com/
Referrer Policy: strict-origin-when-cross-origin

Request Header:

Upgrade-Insecure-Requests: 1
User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/119.0.0.0 Safari/537.36

Caddylogs:

{"level":"info","ts":1700173864.8763306,"msg":"using provided configuration","config_file":"/etc/caddy/Caddyfile","config_adapter":"caddyfile"}
{"level":"warn","ts":1700173864.8895411,"msg":"Caddyfile input is not formatted; run 'caddy fmt --overwrite' to fix inconsistencies","adapter":"caddyfile","file":"/etc/caddy/Caddyfile","line":2}
{"level":"info","ts":1700173864.891095,"logger":"admin","msg":"admin endpoint started","address":"localhost:2019","enforce_origin":false,"origins":["//[::1]:2019","//127.0.0.1:2019","//localhost:2019"]}
{"level":"info","ts":1700173864.891349,"logger":"http.auto_https","msg":"enabling automatic HTTP->HTTPS redirects","server_name":"srv0"}
{"level":"info","ts":1700173864.8917487,"logger":"http","msg":"enabling HTTP/3 listener","addr":":443"}
{"level":"info","ts":1700173864.8918686,"msg":"failed to sufficiently increase receive buffer size (was: 208 kiB, wanted: 2048 kiB, got: 416 kiB). See https://github.com/quic-go/quic-go/wiki/UDP-Buffer-Sizes for details."}
{"level":"info","ts":1700173864.8920612,"logger":"http.log","msg":"server running","name":"srv0","protocols":["h1","h2","h3"]}
{"level":"info","ts":1700173864.89213,"logger":"http.log","msg":"server running","name":"remaining_auto_https_redirects","protocols":["h1","h2","h3"]}
{"level":"info","ts":1700173864.8925116,"msg":"autosaved config (load with --resume flag)","file":"/config/caddy/autosave.json"}
{"level":"info","ts":1700173864.8925228,"msg":"serving initial configuration"}
{"level":"info","ts":1700173864.892564,"logger":"tls","msg":"cleaning storage unit","description":"FileStorage:/data/caddy"}
{"level":"info","ts":1700173864.8926146,"logger":"tls","msg":"finished cleaning storage units"}
{"level":"info","ts":1700173864.892686,"logger":"tls.cache.maintenance","msg":"started background certificate maintenance","cache":"0xc00041d980"}

Inside Caddylogs I don’t have any error logs or anything that can help me find out why still not able to get logs

Enable the debug global option and check your logs again.

Are you sure your customer’s domain has the right DNS?

Okay so now I’m getting below logs

{"level":"info","ts":1700227136.8759327,"msg":"using provided configuration","config_file":"/etc/caddy/Caddyfile","config_adapter":"caddyfile"}
{"level":"warn","ts":1700227136.877691,"msg":"Caddyfile input is not formatted; run 'caddy fmt --overwrite' to fix inconsistencies","adapter":"caddyfile","file":"/etc/caddy/Caddyfile","line":2}
{"level":"info","ts":1700227136.8790748,"logger":"admin","msg":"admin endpoint started","address":"localhost:2019","enforce_origin":false,"origins":["//127.0.0.1:2019","//localhost:2019","//[::1]:2019"]}
{"level":"info","ts":1700227136.8794892,"logger":"http.auto_https","msg":"enabling automatic HTTP->HTTPS redirects","server_name":"srv0"}
{"level":"info","ts":1700227136.8795319,"logger":"tls.cache.maintenance","msg":"started background certificate maintenance","cache":"0xc00011da80"}
{"level":"debug","ts":1700227136.8796377,"logger":"http.auto_https","msg":"adjusted config","tls":{"automation":{"policies":[{"on_demand":true}],"on_demand":{"ask":"http://cache-cloud-api.com/check"}}},"http":{"servers":{"remaining_auto_https_redirects":{"listen":[":80"],"routes":[{},{}],"logs":{"default_logger_name":"log0"}},"srv0":{"listen":[":443"],"routes":[{"handle":[{"encodings":{"gzip":{}},"handler":"encode","prefer":["gzip"]},{"handler":"reverse_proxy","upstreams":[{"dial":"ssl.default.svc.cluster.local:3000"}]}]}],"tls_connection_policies":[{}],"automatic_https":{},"logs":{"default_logger_name":"log0"}}}}}
{"level":"info","ts":1700227136.8801842,"logger":"http","msg":"enabling HTTP/3 listener","addr":":443"}
{"level":"info","ts":1700227136.880454,"msg":"failed to sufficiently increase receive buffer size (was: 208 kiB, wanted: 2048 kiB, got: 416 kiB). See https://github.com/quic-go/quic-go/wiki/UDP-Buffer-Sizes for details."}
{"level":"debug","ts":1700227136.8807192,"logger":"http","msg":"starting server loop","address":"[::]:443","tls":true,"http3":true}
{"level":"info","ts":1700227136.8807418,"logger":"http.log","msg":"server running","name":"srv0","protocols":["h1","h2","h3"]}
{"level":"debug","ts":1700227136.8808153,"logger":"http","msg":"starting server loop","address":"[::]:80","tls":false,"http3":false}
{"level":"info","ts":1700227136.8808246,"logger":"http.log","msg":"server running","name":"remaining_auto_https_redirects","protocols":["h1","h2","h3"]}
{"level":"info","ts":1700227136.8811884,"msg":"autosaved config (load with --resume flag)","file":"/config/caddy/autosave.json"}
{"level":"info","ts":1700227136.8812008,"msg":"serving initial configuration"}
{"level":"info","ts":1700227136.8814921,"logger":"tls","msg":"cleaning storage unit","description":"FileStorage:/data/caddy"}
{"level":"info","ts":1700227136.8815575,"logger":"tls","msg":"finished cleaning storage units"}
{"level":"debug","ts":1700227476.7405252,"logger":"http.stdlib","msg":"http: TLS handshake error from 10.128.0.6:41020: tls: client offered only unsupported versions: [302 301]"}
{"level":"debug","ts":1700228307.8627481,"logger":"events","msg":"event","name":"tls_get_certificate","id":"7247fb01-ae44-4103-bb0c-bd3fb92c879a","origin":"tls","data":{"client_hello":{"CipherSuites":[49199,49200,49195,49196,52392,52393,49171,49161,49172,49162,156,157,47,53,49170,10],"ServerName":"","SupportedCurves":[29,23,24,25],"SupportedPoints":"AA==","SignatureSchemes":[1025,1027,1281,1283,513,515],"SupportedProtos":null,"SupportedVersions":[771,770,769],"Conn":{}}}}
{"level":"debug","ts":1700228307.8634315,"logger":"tls.handshake","msg":"no matching certificates and no custom selection logic","identifier":"10.60.2.135"}
{"level":"error","ts":1700228317.8639276,"logger":"tls","msg":"request to 'ask' endpoint failed","ask_endpoint":"http://cache-cloud-api.com/check","domain":"10.60.2.135","error":"error checking http://cache-cloud-api.com/check to determine if certificate for hostname '10.60.2.135' should be allowed: Get \"http://cache-cloud-api.com/check?domain=10.60.2.135\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"}
{"level":"debug","ts":1700228317.8641155,"logger":"http.stdlib","msg":"http: TLS handshake error from 10.128.15.241:28339: certificate is not allowed for server name 10.60.2.135: decision func: error checking http://cache-cloud-api.com/check to determine if certificate for hostname '10.60.2.135' should be allowed: Get \"http://cache-cloud-api.com/check?domain=10.60.2.135\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"}
{"level":"debug","ts":1700228522.9069974,"logger":"http.stdlib","msg":"http: TLS handshake error from 10.60.2.1:3698: tls: client offered only unsupported versions: [302 301]"}
{"level":"info","ts":1700228546.1246722,"logger":"http.log.access.log0","msg":"handled request","request":{"remote_ip":"10.128.15.241","remote_port":"58906","client_ip":"10.128.15.241","proto":"HTTP/1.1","method":"GET","host":"34.136.106.166:80","uri":"/","headers":{"User-Agent":["Mozilla/5.0 (Macintosh; Intel Mac OS X 10.11; rv:47.0) Gecko/20100101 Firefox/47.0"],"Accept":["*/*"],"Connection":["keep-alive"]}},"bytes_read":0,"user_id":"","duration":0.000263191,"size":0,"status":308,"resp_headers":{"Connection":["close"],"Location":["https://34.136.106.166/"],"Content-Type":[],"Server":["Caddy"]}}
{"level":"debug","ts":1700228914.9650488,"logger":"http.stdlib","msg":"http: TLS handshake error from 10.128.0.6:1024: read tcp 10.60.2.135:443->10.128.0.6:1024: read: connection reset by peer"}
{"level":"debug","ts":1700228937.3597276,"logger":"events","msg":"event","name":"tls_get_certificate","id":"7cf6a82f-f934-475f-b336-b4f769e9e523","origin":"tls","data":{"client_hello":{"CipherSuites":[49199,49195,49169,49159,49171,49161,49172,49162,5,47,53,49170,10],"ServerName":"","SupportedCurves":[23,24,25],"SupportedPoints":"AA==","SignatureSchemes":[1025,1027,513,515,1025,1281,1537],"SupportedProtos":null,"SupportedVersions":[771,770,769],"Conn":{}}}}
{"level":"debug","ts":1700228937.359907,"logger":"tls.handshake","msg":"no matching certificates and no custom selection logic","identifier":"10.60.2.135"}
{"level":"error","ts":1700228947.3602724,"logger":"tls","msg":"request to 'ask' endpoint failed","ask_endpoint":"http://cache-cloud-api.com/check","domain":"10.60.2.135","error":"error checking http://cache-cloud-api.com/check to determine if certificate for hostname '10.60.2.135' should be allowed: Get \"http://cache-cloud-api.com/check?domain=10.60.2.135\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"}
{"level":"debug","ts":1700228947.3605545,"logger":"http.stdlib","msg":"http: TLS handshake error from 10.128.0.6:5732: certificate is not allowed for server name 10.60.2.135: decision func: error checking http://cache-cloud-api.com/check to determine if certificate for hostname '10.60.2.135' should be allowed: Get \"http://cache-cloud-api.com/check?domain=10.60.2.135\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"}
{"level":"debug","ts":1700230018.6478019,"logger":"http.stdlib","msg":"http: TLS handshake error from 10.60.2.1:29134: tls: client offered only unsupported versions: [302 301]"}
{"level":"debug","ts":1700230390.9185197,"logger":"http.stdlib","msg":"http: TLS handshake error from 10.60.2.1:42371: tls: first record does not look like a TLS handshake"}
{"level":"info","ts":1700230570.3532252,"logger":"http.log.access.log0","msg":"handled request","request":{"remote_ip":"10.128.0.6","remote_port":"49966","client_ip":"10.128.0.6","proto":"HTTP/1.1","method":"POST","host":"34.136.106.166:80","uri":"/boaform/admin/formLogin","headers":{"Content-Type":["application/x-www-form-urlencoded"],"Content-Length":["29"],"Origin":["http://34.136.106.166:80"],"Accept-Encoding":["gzip, deflate"],"User-Agent":["Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:71.0) Gecko/20100101 Firefox/71.0"],"Accept":["text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8"],"Accept-Language":["en-GB,en;q=0.5"],"Connection":["keep-alive"],"Referer":["http://34.136.106.166:80/admin/login.asp"],"Upgrade-Insecure-Requests":["1"]}},"bytes_read":0,"user_id":"","duration":0.00006985,"size":0,"status":308,"resp_headers":{"Connection":["close"],"Location":["https://34.136.106.166/boaform/admin/formLogin"],"Content-Type":[],"Server":["Caddy"]}}
{"level":"debug","ts":1700231359.805899,"logger":"http.stdlib","msg":"http: TLS handshake error from 10.128.0.6:18336: tls: client offered only unsupported versions: [302 301]"}

This IP is my frontend container IP 10.60.3.135

But the script is in my backend container 10.60.2.129

So the path Caddy is trying to ask is this
Get "http://cache-cloud-api.com/check?domain=10.60.2.135\"

It seems to be it’s looking for it in Front-End container not in the Back-End container

Do you think I should move that check script into front-end container

This reads to me like some HTTP client that’s hitting your server only supports old TLS versions. 302 is TLS 1.1, 301 is TLS 1.0. Caddy only supports TLS 1.2 and 1.3, because older TLS versions are insecure.

Make sure whatever software you’re using to connect to your server is up to date.

Looks like Caddy is failing to connect to cache-cloud-api.com, whatever that is. Make sure it can be reached from Caddy.

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.