How to use enclave's existing PKI certificates for reverse-proxied services in Docker/Podman

1. Output of caddy version:

2.6.2-alpine (docker image from docker hub)

2. How I run Caddy:

I execute a bash shell script that invokes podman to run the image. This script stands up a TLS-secured spring boot service and caddy. Note that the spring security configuration takes the PKI certificate that is presented, and uses a proprietary library to check the certificate against a corporate certificate authority for user authorization tokens. So, any service that is reverse-proxied by caddy must be able to use the certificate on the server that podman is running on. I have this working with nginx, and I would like to convert to caddy, if it will work with our internal (corporate) PKI. The service and caddy are run from this shell script:

podman network create services_network

podman pod create \
 --name services_proxy \
 --network podman,services_network
 --publish 8443:8443

podman pod create \
 --name hello-world-service \
 --network services_network

podman pod create \
 --name hello-world-ui \
 --network services_network

podman run --d \
 --name hello_world_service \
 --pod hello-world-service \
 --secret source=server-pem,target=/certs/server.pem,type=mount \
 --secret source=server-key,target=/certs/server.key,type=mount \
 --volume /local/service-config:/config:Z \
 hello-world-service:3.20.0-SNAPSHOT

podman run --d \
 --name hello_world_ui \
 --pod hello-world-ui \
 --secret source=server-pem,target=/certs/server.pem,type=mount \
 --secret source=server-key,target=/certs/server.key,type=mount \
 --volume /local/service-config:/config:Z \
 hello-world-ui:3.20.0-SNAPSHOT

podman run --d \
 --name services_proxy_caddy \
 --pod services_proxy \
 --secret source=server-pem,target=/certs/server.pem,type=mount \
 --secret source=server-key,target=/certs/server.key,type=mount \
 --volume /local/services/Caddyfile:/etc/caddy/Caddyfile \
 --volume /local/services/caddy-data:/data:Z \
 caddy:2.6.2-alpine

a. System environment:

RHEL 8.6
Podman 4.2.0 (rootless)

b. Command:

# invoke the script that I detailed above:
sh ./podman-stack.sh

c. Service/unit/compose file:

see script contents above

d. My complete Caddy config:

{
    debug
}

:8443 {
    tls /certs/server.pem /certs/server.key

    header {
        Host {upstream_hostport}
    }

    handle /hello-world-service/* {
        reverse_proxy https://hello-world-service
    }

    handle /hello-world-ui/* {
        reverse_proxy https://hello-world-ui
    }
}

3. The problem I’m having:

I have everything working like I expect with nginx, where I can set the proxy SSL/TLS cert/key/password, and the connection is terminated by the proxy and re-established with the configured cert and key. The connection reaches the service correctly, and the app stack is fully able to do what it needs to do on the services network. I am trying to convert this to the Caddy way of doing things with its reverse proxy module.

When trying to reach https://localhost:8443/hello-world-ui/ from a web browser, it doesn’t load a page. It yields a 502 error, saying that localhost is currently unable to handle this request.

4. Error messages and/or full log output:

I am running on an enclave, so I am unable to paste the full line. I think that the direct error message will probably be enough for this, but please let me know if anyone else needs any other information to better diagnose the issue:

x509: certificate is not valid for any names, but wanted to match hello-world-ui

5. What I already tried:

I have also edited the ui handler to skip verification and to handle the header via header_up:

    handle /hello-world-ui/* {
        reverse_proxy https://hello-world-ui {
            header_up Host {remote_hostport}
            transport http {
                tls_insecure_skip_verify
            }
        }
    }

This changed the error to

tls: bad certificate

6. Links to relevant resources:

1 Like

That’s not right, I think you’re looking for {upstream_hostport}.

This will send the Host as hello-world-ui. Does the upstream know how to handle requests for that hostname?

There’s a bunch of TLS options you can play with here:

But it seems like the Go stdlib is rejecting your certificate from upstream as invalid. I couldn’t tell you why. You’ll need to dig deeper.

1 Like

Thank you for the reply. I will try {upstream_hostport}. I have been trying a few of the reverse_proxy options, to little avail. Looking at my nginx config, I also needed to set the tls_server_name to the hostname of the server so that SNI problems won’t be the reason the cert is rejected. Eventually, when I get these basic services working, I will have about ten spring boot apps on that podman network, and they all need to use the server’s cert for various connections that they make on the host’s network. I wish that I could avoid TLS inside of the network, or at least allow caddy to handle the certificates, but there is no possibility that I can influence other services on the network to trust caddy’s certs.

I think that I will need the services on the docker network to respond with their hostname as the physical server’s hostname, so that the certificate that they are using is not rejected. Somehow, either it works this way with nginx, or nginx does not care that a an upstream host named “zxcv” is using a certificate for a host (the physical server that all of this is being hosted on) named “asdf”.

Maybe part of what I am missing is adding tls_trusted_ca_certs /certs/trust.pem in the transport http block.

I realize that I forgot to answer your question. The upstream named hello-world-ui is an OCI container that runs spring boot application that will handle requests made to its context path. It has a context path of hello-world-ui, so if someone opens a browser to https://hello-world-ui/index.html, then that should be forwarded upstream as https://hello-world-ui/hello-world-ui/index.html and the spring boot app can handle that.

1 Like

I’m not sure I understand where you’re at now. Did you figure it out? I’m not sure if there’s still a question here.

I guess I typed a lot of detail, and I can understand why it wasn’t clear if I am still having problems. Where I am located, I just arrived at work, and tried playing with some settings. My configuration is still not working. My updated configuration looks like this:

:8443 {
    tls /certs/server.crt /certs/server.key
    rewrite /hello-world-ui /hello-world-ui/
    map {header.X-ProxiedEntitiesChain} {xpechain} {
        ~^(.+)$ {header.X-ProxiedEntitiesChain}<{tls_client_subject}>
        default <{tls_client_subject}>
    }
    map {header.X-ProxiedIssuersChain} {xpichain} {
        ~^(.+)$ {header.X-ProxiedIssuersChain}<{tls_client_issuer}>
        default <{tls_client_issuer}>
    }
    handle /hello-world-ui/* {
        reverse_proxy https://hello-world-ui {
            header_up Host                   {upstream_hostport}
            header_up X-Real-IP              {remote_host}
            header_up X-Client-Verify        SUCCESS
            header_up X-Client-DN            {tls_client_subject}
            header_up X-SSL-Issuer           {tls_client_issuer}
            header_up X-ProxiedEntities      {xpechain}
            header_up X-ProxiedEntitiesChain {xpechain}
            header_up X-ProxiedIssuersChain  {xpichain}
            transport http {
                tls_insecure_skip_verify
                tls_server_name              my.server.fqdn
                tls_trusted_ca_certs         /certs/trust.crt
            }
        }
    }
    handle /hello-world-service/* {
        reverse_proxy https://hello-world-service
    }
}

I know that the map blocks are problematic, since they don’t seem to be able to get the TLS information at the location where they are defined, but that problem is secondary, at the moment. I am still getting “tls: bad certificate” and an HTTP 502. I have Caddy’s logs turned up to debug, and that’s all I’m getting. How can I get more information?

With the configuration that I have, combined with my nginx config, what you’ve told me, and the documentation for the TLS options, I can’t think of why this isn’t working, or how to dig deeper. I’m missing something, but I can’t figure out what it is.

I was able to reproduce this error in a simplified project that I pushed to my github:

If you have a machine where you can get podman working, this will generate certs and a truststore. After getting podman set up, you can simply mvn clean install spring-boot:build-image, and then cd src/main/resources and execute sh ./podman-up.sh. If you don’t have the time or desire to dig that deeply into this, I completely understand. This has all of the configuration, and I don’t have to worry about errors when I copy things over into this forum from my enclave machine.

I apologize for not being able to provide enough information before since this was all on a private corporate network. Please let me know if this helps to make my misconfiguration somehow more apparent.

Oh - I don’t think you can use {env.*} for some of those, because that must be replaced at runtime. Try using {$ENV} instead, which is replaced at Caddyfile adapt time (before the server runs).

I’m not sure if that’s your issue, but it is one thing that stands out to me in your repo.

But yeah. I don’t had podman, I don’t think I can spend the time setting that and a java environment up (if I’m understanding, I’d need maven too, etc?)

Could you share the actual certificate that’s causing you problems (or a throwaway one generated the same way)?

Maybe you could try using smallstep or mkcert to cert up your certs instead? (Those have good defaults, so it’s much harder to screw up than straight openssl commands).

@francislavoie thank you for your reply. I will try {$ENV} instead. I assume that means {$ENV_VAR_NAME_HERE}. As far as the certificates go, if you have a machine where you can run bash, then you can just use the first 27 lines of this script: podman-caddy-x509/podman-up.sh at main · Steve973/podman-caddy-x509 · GitHub and it will create all of the cert stuff for you to inspect. Otherwise, I can generate them and zip them up for you if you let me know how we can submit files here.

From what you have seen in my caddyfile, does that look reasonable for what I want to do? I haven’t quite spelled it out like this before, but in case it is not obvious: For a bare-metal server that is running docker/podman, I need Caddy and all of the upstream services running in containers (on an overlay network) to use the same corporate managed PKI cert assigned to that bare-metal machine, and they all have to trust that cert.

The error says remote error: tls: bad certificate. What does remote error mean? Is that when Caddy does not like the certificate that it sees during the TLS handshake with the upstream service?

Hey, @francislavoie I found the solution! It was a couple of things. First, when I was signing the cert with my fake CA, I overwrote it directly afterward by creating a self-signed cert. After fixing that up, things started working a little bit better. Then I found the right options for the Caddyfile:

{
    debug
    auto_https off
    ocsp_stapling off
}

:8443 {
    tls /certs/test.crt /certs/test.key {
        client_auth {
            mode                   require_and_verify
            trusted_ca_cert_file   /certs/trust.pem
        }
    }
    handle /greeting-service/* {
        reverse_proxy https://greeting-service:8443 {
            header_up Host                   {$OUTER_HOST}
            transport http {
                tls_server_name              {$OUTER_HOST}
                tls_trusted_ca_certs         /certs/trust.pem
                tls_client_auth              /certs/test.crt /certs/test.key
            }
        }
    }
}

And that did it! I will have some further questions, but this thread is solved. Thank you for your patience and help.

2 Likes

Nice. Glad you figured it out. Sorry I couldn’t have been more help.

I think that you helped me. Plus, my use case is not exactly standard, so that brings its own responsibility with it.

So, I’m assuming that I can use the labels, or whatever it’s called when you use something like “@name” for a block, if I want to add the same stuff that made this work to multiple handle blocks?

Those are named request matchers:

And yes, handle blocks allow you to set up mutually exclusive routes, with a matcher being the condition.

Thanks. I think I was unclear. I would like to keep things a bit more “DRY” with the configuration for each of the matchers. That’s where snippets come into play, right? I thought they were called “labels” but I just checked, and snippets are what I meant.

Yeah, snippets can be used to avoid repetition. You can pass arguments to them as input as well.

Thanks yet again. They work for me if the snippets are outside of any blocks. If I put them inside my :8443 block, they don’t work. So, I assume that means that snippets are global. Unless I missed it in the documentation, that was unclear to me, and I only observed that behavior when trying to use them.

That’s correct. Snippets go top-level. See the structure diagram at the top of this page in the docs:

You’re right, that’s somewhat ambiguous in the “Snippets” section on that page.

1 Like

Ok, my fault then, since I missed that. It’s a good diagram and it makes things quite clear. By the way, those map blocks that in included in an earlier Caddyfile that I posted (for putting some of the TLS info into headers) are also working. Those are all of the concerns I had before moving forward with Caddy. I am pretty confident that I can get all of this working on our enclave, so I will be replacing nginx with Caddy on Monday. Thanks again for all of your help, and I really appreciate it!

2 Likes

This topic was automatically closed after 30 days. New replies are no longer allowed.