On Demand SSL on ports 80,443 with health checks?

1. Caddy version (caddy version):

Version 2

2. How I run Caddy:

On EC2 servers on AWS behind a Load Balancer.

a. System environment:

Ubuntu 18.04

b. My complete Caddyfile or JSON config:

{
  storage file_system {
    root /mnt/efs/fs1
  }
}


:443 {
    reverse_proxy sites.nvssolutions.com:443

        tls moe@nvssolutions.com {
          on_demand
        }
}


*.amazonaws.com:80 {

respond "Working!"
}

3. The problem I’m having:

My main problem is I want on_demand SSL while also needing health checks from the load balancer. (I have several Caddy servers behind a load balancer all sharing a filesystem).

Everything works fine if I use the healthcheck from the load balancer on port 80. However, the rest of the traffic on port 80 is not redirected to 443 and in turn not generating certificates

5. What I already tried:

I tried changing the healthchecks to use Port 81. However, that does’t work and the healthchecks from Amazon always fail.

Here’s the updated Caddyfile I tried with no luck. I tried many variations of this and am hoping someone can point out a proper approach or what I’m doing wrong.

{
  storage file_system {
    root /mnt/efs/fs1
  }
}


:80,:443 {
    reverse_proxy sites.nvssolutions.com:443

        tls moe@nvssolutions.com {
          on_demand
        }
}


*.amazonaws.com:81 {

respond "Working!"
}

I’m also trying to avoid using port 443 for healthchecks since when I did that it was trying to generate certificates for internal load balancer.

Last note, I don’t have an option of choosing the hostname the load balancer uses. This is a restriction with AWS.

Any pointers would be greatly appreciated.

In what manner, specifically, do they always fail? What information does Amazon provide on its health check? It’s really important to get the exact error. Did their check timeout? Was there a protocol error? Is Amazon seeing non-200 responses for the endpoint? The nature of Amazon’s error will tell us what we need to troubleshoot.

1 Like

Unfortunately, Amazon does not supply any sort of information on the type of load balancer I’m using. Simply says “unhealthy”

From my understanding Amazon load balancer is simply asking looking for a 200 success code message. And this does work perfectly on port 80. Just when I try this on port 81 (or any other port) it doesn’t work.

I guess what my bottom line is:
Assuming the issue is let’s say on Amazon, does CaddyServer support having ports 80 and 443 do reverse proxy, while a different port just reponse to the health check.

From my Caddyfile above, am I missing something?

I’m testing for hours different things and it seems very inconsistent (to me at least).

For example, this would work:

    *.amazonaws.com:80 {

    respond "Working!"
    }

But this won't: 

    :80 {

    respond "Working!"
    }

I'm hoping you can shed some more light on this, I'd really appreciate even random ideas of what I can attempt and test.

I’m going to guess that one possibility is that Amazon requires a HTTP endpoint, not a HTTPS endpoint.

*.amazonaws.com:80 will produce a HTTP listener, *.amazonaws.com:81 will produce a self-signed/locally-trusted HTTPS listener with a redirect set up on port 80 (Caddy is HTTPS-first, HTTP only on explicit scheme or default HTTP port).

Try http://*.amazonaws.com:81 and see if that makes a difference.

Thanks @Whitestrake
I just tried that but unfortunately it didn’t work. However it gave me several ideas of things I can try along these lines. I’ll try different ports, protocols and combinations of those. Hopefully if I shoot enough I’ll hit something.

Still no luck for me.

I have another way of asking this. Is it possible to do this:

  1. All port 80 traffic gets routed to HTTPS
  2. All HTTPS traffic get on_demand certificate
  3. Traffic from a specific source, or traffic to a specific endpoint skips the on_demand certificate.

The bottom line for me is that everything works if I set it up on port 80. But I’m trying to avoid certificates being generated for my load-balancer.

If by “routed”, you mean “redirected”, then yes, Caddy automatically sets up HTTP->HTTPS redirects for you via the Automatic HTTPS feature:

I think you’re looking for something like this:

{
	on_demand_tls {
		ask http://example.com/ask
	}
}

*.example.com {
	# your site's directives

	tls {
		on_demand
	}
}

http://for-aws.example.com {
	respond "OK!"
}

Yayyy!! This finally worked. I appreciate it.

I modified the code just a bit as follows:

{
        storage file_system {
         root /mnt/efs/fs1
        }
        on_demand_tls {
                ask https://api.nvssolutions.com/api/domain-checker-base
        }
}

:443 {
    reverse_proxy sites.nvssolutions.io:443

        tls moe@nvssolutions.com {
          on_demand
        }
}


http://*.amazonaws.com {
        respond "OK!"
}

Can I ask one more? Now when I go to a site it doesn’t seem to auto redirect to HTTPS.

If I go to https:// it works fine, but http:// it says: Client sent an HTTP request to an HTTPS server.

This shows error => http://acl1.tagahash.com
This works correctly => https://acl1.tagahash.com

Can you guide me how I can finish this and get it redirected to https?

1 Like

I’m pretty sure that the redirects are only enabled if you use an actual domain as the site label, but I’m not certain.

I think you can add your own redirects like this though:

http:// {
	redir https://{host}{uri}
}

Also just as an additional tip, you can use the caddy fmt command to make your Caddyfile prettier.

It should enable redirects regardless.

Thanks. It’s giving me an error now.

Job for caddy.service failed because the control process exited with error code. See "systemctl status caddy.service" and "journalctl -xe" for details.

Error I"m seeing:

cannot make a TLS automation policy from a server block that has a host-less address when there are other server block addresses lacking a host

This is really where I keep getting stuck. The ondemand SSL is for domains that are not predefined so I must just keep that as host-less.

What’s your entire and unaltered Caddyfile at this point?

@matt thanks for looking at this for me. Great product btw.

#server1

{
        storage file_system {
         root /mnt/efs/fs1
        }
        on_demand_tls {
                ask https://api.nvssolutions.com/api/domain-checker-base
        }
}


:443 {
    reverse_proxy sites.nvssolutions.com:443

        tls moe@nvssolutions.com {
          on_demand
        }
}

http:// {
        redir https://{host}{uri}
}

https://*.amazonaws.com {
        respond "OK!"
}

You can get rid of the http:// server block, that redirect happens automatically.

Thanks. But for some reason it’s still not forwarding my traffic to HTTPS.

When i got to an http site I get this message:

“Client sent an HTTP request to an HTTPS server.”

Your reverse proxy isn’t configured to use TLS. Replace :443 with https:// at the beginning, see how that goes.

Thanks again Matt.

I tried that and still no luck, then it occurred to me that this might be an issue with my AWS Global Accelerator. https://aws.amazon.com/global-accelerator/

Here’s my findings.
If I forward my domain directly to the Caddy server it does forward correctly to HTTPS. However if I forward my domain to the static IP address on my load balancer, that doesn’t seem to forward to https. Clearly not an issue with Caddy.

Here’s a screenshot of the network tab for these requests. I don’t see the “server : Caddy” in the header, I guess that means it’s not even hitting the Caddy server.

Thanks everyone for your help. This ended up mostly being an issue with my configuration on AWS.

All good now.

In case anyone else is building something similar using AWS. Keep in mind you must use a TCP load balancer but since each target group only accepts one type of traffic (http or https) and you can’t redirect on network load balancer, you must create 2 different target groups. One of http and one for https.

2 Likes