Need to understand what caddy tries to do here

Valen · June 26, 2021, 7:13pm

1. Caddy version (`caddy version`):

v2.4.2 h1:chB106RlsIaY4mVEyq9OQM5g/9lHYVputo/LAX2ndFg=

2. How I run Caddy:

sudo /usr/bin/caddy run --environ --config /etc/caddy/Caddyfile

a. System environment:

Debian 9
Official Deb package
Using cli for troubleshooting, same symtoms with systemd

b. Command:

sudo /usr/bin/caddy run --environ --config /etc/caddy/Caddyfile

c. Service/unit/compose file:

sudo /usr/bin/caddy run --environ --config /etc/caddy/Caddyfile
nothing else

d. My complete Caddyfile or JSON config:

CaddyFile

(logstash_info) {
}

{
	import logstash_info
}

import http.d/*.conf

Imported FIle

sub.freeipa.domain:443 {

	tls {
		issuer acme https://ipa-ca.freeipa.domain/acme/directory {
			email xx@yyy.zz
		}
	}


	@internal {
		remote_ip 192.168.0.0/16
	}



        encode gzip

		handle @internal {
			reverse_proxy /* localhost:1234 {

			header_up Host localhost


			}
		}


	respond 403
}

3. The problem I’m having:

I want to use the new acme functionality of my freeipa installation. It uses dogtag pki acme responder. I cannot say if this is a caddy problem or a problem with freeipa/dogtag. Maybe someone can point me in the right direction.

4. Error messages and/or full log output:


2021/06/26 18:57:30.161	ERROR	tls.obtain	could not get certificate from issuer	{"identifier": "sub.freeipa.domain", "issuer": "ipa-ca.freeipa.domain-acme-directory", "error": "registering account [mailto:xx@yyy.zz] with server: request to https://ipa.freeipa.domain/acme/new-account failed after 3 attempts: HTTP 500: <!doctype html><html lang=\"en\"><head><title>HTTP Status 500 – Internal Server Error</title><style type=\"text/css\">body {font-family:Tahoma,Arial,sans-serif;} h1, h2, h3, b {color:white;background-color:#525D76;} h1 {font-size:22px;} h2 {font-size:16px;} h3 {font-size:14px;} p {font-size:12px;} a {color:black;} .line {height:1px;background-color:#525D76;border:none;}</style></head><body><h1>HTTP Status 500 – Internal Server Error</h1><hr class=\"line\" /><p><b>Type</b> Exception Report</p><p><b>Message</b> java.lang.Exception: Unsupported JWS algorithm: ES256</p><p><b>Description</b> The server encountered an unexpected condition that prevented it from fulfilling the request.</p><p><b>Exception</b></p><pre>org.jboss.resteasy.spi.UnhandledException: java.lang.Exception: Unsupported JWS algorithm: ES256\n\torg.jboss.resteasy.core.ExceptionHandler.handleApplicationException(ExceptionHandler.java:78)\n\torg.jboss.resteasy.core.ExceptionHandler.handleException(ExceptionHandler.java:222)\n\torg.jboss.resteasy.core.SynchronousDispatcher.writeException(SynchronousDispatcher.java:179)\n\torg.jboss.resteasy.core.SynchronousDispatcher.invoke(SynchronousDispatcher.java:422)\n\torg.jboss.resteasy.core.SynchronousDispatcher.invoke(SynchronousDispatcher.java:213)\n\torg.jboss.resteasy.plugins.server.servlet.ServletContainerDispatcher.service(ServletContainerDispatcher.java:228)\n\torg.jboss.resteasy.plugins.server.servlet.HttpServletDispatcher.service(HttpServletDispatcher.java:56)\n\torg.jboss.resteasy.plugins.server.servlet.HttpServletDispatcher.service(HttpServletDispatcher.java:51)\n\tjavax.servlet.http.HttpServlet.service(HttpServlet.java:741)\n\tsun.reflect.GeneratedMethodAccessor41.invoke(Unknown Source)\n\tsun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)\n\tjava.lang.reflect.Method.invoke(Method.java:498)\n\torg.apache.catalina.security.SecurityUtil$1.run(SecurityUtil.java:282)\n\torg.apache.catalina.security.SecurityUtil$1.run(SecurityUtil.java:279)\n\tjava.security.AccessController.doPrivileged(Native Method)\n\tjavax.security.auth.Subject.doAsPrivileged(Subject.java:549)\n\torg.apache.catalina.security.SecurityUtil.execute(SecurityUtil.java:314)\n\torg.apache.catalina.security.SecurityUtil.doAsPrivilege(SecurityUtil.java:170)\n\tjava.security.AccessController.doPrivileged(Native Method)\n\torg.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:53)\n\tsun.reflect.GeneratedMethodAccessor42.invoke(Unknown Source)\n\tsun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)\n\tjava.lang.reflect.Method.invoke(Method.java:498)\n\torg.apache.catalina.security.SecurityUtil$1.run(SecurityUtil.java:282)\n\torg.apache.catalina.security.SecurityUtil$1.run(SecurityUtil.java:279)\n\tjava.security.AccessController.doPrivileged(Native Method)\n\tjavax.security.auth.Subject.doAsPrivileged(Subject.java:549)\n\torg.apache.catalina.security.SecurityUtil.execute(SecurityUtil.java:314)\n\torg.apache.catalina.security.SecurityUtil.doAsPrivilege(SecurityUtil.java:253)\n</pre><p><b>Root Cause</b></p><pre>java.lang.Exception: Unsupported JWS algorithm: ES256\n\torg.dogtagpki.acme.server.ACMEEngine.validateJWS(ACMEEngine.java:679)\n\torg.dogtagpki.acme.server.ACMENewAccountService.createNewAccount(ACMENewAccountService.java:55)\n\tsun.reflect.GeneratedMethodAccessor67.invoke(Unknown Source)\n\tsun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)\n\tjava.lang.reflect.Method.invoke(Method.java:498)\n\torg.jboss.resteasy.core.MethodInjectorImpl.invoke(MethodInjectorImpl.java:140)\n\torg.jboss.resteasy.core.ResourceMethodInvoker.invokeOnTarget(ResourceMethodInvoker.java:295)\n\torg.jboss.resteasy.core.ResourceMethodInvoker.invoke(ResourceMethodInvoker.java:249)\n\torg.jboss.resteasy.core.ResourceMethodInvoker.invoke(ResourceMethodInvoker.java:236)\n\torg.jboss.resteasy.core.SynchronousDispatcher.invoke(SynchronousDispatcher.java:406)\n\torg.jboss.resteasy.core.SynchronousDispatcher.invoke(SynchronousDispatcher.java:213)\n\torg.jboss.resteasy.plugins.server.servlet.ServletContainerDispatcher.service(ServletContainerDispatcher.java:228)\n\torg.jboss.resteasy.plugins.server.servlet.HttpServletDispatcher.service(HttpServletDispatcher.java:56)\n\torg.jboss.resteasy.plugins.server.servlet.HttpServletDispatcher.service(HttpServletDispatcher.java:51)\n\tjavax.servlet.http.HttpServlet.service(HttpServlet.java:741)\n\tsun.reflect.GeneratedMethodAccessor41.invoke(Unknown Source)\n\tsun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)\n\tjava.lang.reflect.Method.invoke(Method.java:498)\n\torg.apache.catalina.security.SecurityUtil$1.run(SecurityUtil.java:282)\n\torg.apache.catalina.security.SecurityUtil$1.run(SecurityUtil.java:279)\n\tjava.security.AccessController.doPrivileged(Native Method)\n\tjavax.security.auth.Subject.doAsPrivileged(Subject.java:549)\n\torg.apache.catalina.security.SecurityUtil.execute(SecurityUtil.java:314)\n\torg.apache.catalina.security.SecurityUtil.doAsPrivilege(SecurityUtil.java:170)\n\tjava.security.AccessController.doPrivileged(Native Method)\n\torg.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:53)\n\tsun.reflect.GeneratedMethodAccessor42.invoke(Unknown Source)\n\tsun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)\n\tjava.lang.reflect.Method.invoke(Method.java:498)\n\torg.apache.catalina.security.SecurityUtil$1.run(SecurityUtil.java:282)\n\torg.apache.catalina.security.SecurityUtil$1.run(SecurityUtil.java:279)\n\tjava.security.AccessController.doPrivileged(Native Method)\n\tjavax.security.auth.Subject.doAsPrivileged(Subject.java:549)\n\torg.apache.catalina.security.SecurityUtil.execute(SecurityUtil.java:314)\n\torg.apache.catalina.security.SecurityUtil.doAsPrivilege(SecurityUtil.java:253)\n</pre><p><b>Note</b> The full stack trace of the root cause is available in the server logs.</p><hr class=\"line\" /><h3>Apache Tomcat/9.0.30</h3></body></html>"}

5. What I already tried:

I think the point is:

java.lang.Exception: Unsupported JWS algorithm: ES256

Maybe can someone explain to me what happened here? Maybe I have to ask questions at some freeipa support forum, but I am not sure what caddy tries to do here.

I tested my freeipa/dogtag installation with certbot succesfully. So there seems to be some difference between certbot and caddy on how they register with an acme server.

Thank you!

matt · June 27, 2021, 5:19pm

Hmm something must be wrong:

$ curl -v "https://ipa-ca.freeipa.domain/acme/directory"
* Could not resolve host: ipa-ca.freeipa.domain
* Closing connection 0
curl: (6) Could not resolve host: ipa-ca.freeipa.domain

Please follow the help template instructions and make sure we can reproduce the issue first.

Valen · June 27, 2021, 7:14pm

Hi! Thanks for your answer!

The domains are all within a VPN net and will not be reachable from the outside. You will not be able to reproduce this from your point. Maybe the Help forum is the wrong section?

I circled the problem into the account creation phase of the ACME stuff caddy does. It seems to communicate differently than certbot with the ACME server. I would like to understand what the difference is and what component is at fault.

Valen · June 29, 2021, 4:22pm

Getting a automatic certificate using traefik and certbot. There seems to be something different when caddy request a certificate. I am totally out of my depth here. I can post how to re-create the setup locally (freeipa has a docker container now) but that would definitely be quite some effort to reproduce this. Anyone willing to do that?

francislavoie · June 29, 2021, 10:19pm

You’ll need to ask the Dogtag project (I think?) to add support for that JWS signature algorithm. (I think this is the relevant docs ACME Responder · dogtagpki/pki Wiki · GitHub)

Caddy uses that signature algorithm when the private key is a P-256 ECC key.

Or maybe you can configure Caddy to use a different key type with the key_type option (see tls (Caddyfile directive) — Caddy Documentation)

github.com

mholt/acmez/blob/6519778db58880455f6332c2024e4b15603c3f8e/acme/jws.go#L244


      
          // jwsHasher indicates suitable JWS algorithm name and a hash function
          // to use for signing a digest with the provided key.
          // It returns ("", 0) if the key is not supported.
          func jwsHasher(pub crypto.PublicKey) (string, crypto.Hash) {
          	switch pub := pub.(type) {
          	case *rsa.PublicKey:
          		return "RS256", crypto.SHA256
          	case *ecdsa.PublicKey:
          		switch pub.Params().Name {
          		case "P-256":
          			return "ES256", crypto.SHA256
          		case "P-384":
          			return "ES384", crypto.SHA384
          		case "P-521":
          			return "ES512", crypto.SHA512
          		}
          	}
          	return "", 0
          }
          
          // jwkThumbprint creates a JWK thumbprint out of pub

Remove the /* here. It’s redundant/slower. Redundant because all requests begin with /, and marginally slower because it makes Caddy make an extra tautological comparison on every request.

You can shorten this to this (and I also recommend moving it to just before your handle so it’s closer together, for readability):

@internal remote_ip 192.168.0.0/16

Also, use the caddy fmt command to clean up your Caddyfile’s formatting. It’ll look prettier.

Valen · July 1, 2021, 9:23pm

This looks like its going in the right direction. But I get a:

Error during parsing: unrecognized ACME issuer property: key_type

Well, does the issuer live in acmeissuer.go? The switch statement sadly does not contain key_type.

francislavoie · July 1, 2021, 11:01pm

Ah, I think the docs have key_type in the wrong place. It’s in the top-level of tls, or as a global option, not part of issuer configuration.

Fixed the docs, will be updated next time they’re pushed to the live site:

https://github.com/caddyserver/website/commit/56e1517c0f1e2b7ca838ac49c8f6916cab9d8934

Valen · July 2, 2021, 7:24am

Thank you. Caddy now starts, but it still seems to use the same signature algorithm. Even with key_type rsa4096. Is that expected?

francislavoie · July 2, 2021, 2:57pm

You might need to delete the contents of Caddy’s storage to make it forget your existing set of keys. It’s probably trying to reuse the keys you already have on disk.

Valen · July 2, 2021, 6:30pm

I cleared all traces of caddys data on disk. Still getting the same result

system · July 26, 2021, 7:13pm

This topic was automatically closed after 30 days. New replies are no longer allowed.