Need help with setting up Caddy, error while trying to get cert from ACME

st3fus · May 16, 2023, 11:02am

1. The problem I’m having:

I’m just trying to set up a simple web server for a website that has frontend and backend reverse proxy and a certificate obtain/renewing. Also I double checked the cPanel of my domain provider, everything is properly configured and I created an A record pointing to public IP of my EC2 instance

2. Error messages and/or full log output:

caddy             | {"level":"error","ts":1684234642.087624,"logger":"tls.obtain","msg":"could not get certificate from issuer","identifier":"parkovins.rs","issuer":"acme-v02.api.letsencrypt.org-directory","error":"HTTP 403 urn:ietf:params:acme:error:unauthorized - Cannot negotiate ALPN protocol \"acme-tls/1\" for tls-alpn-01 challenge"}
caddy             | {"level":"info","ts":1684234642.8903217,"logger":"tls.issuance.zerossl","msg":"generated EAB credentials","key_id":"NzGCWJjJU8XXbK037n9FpA"}
caddy             | {"level":"info","ts":1684234643.624006,"logger":"http","msg":"waiting on internal rate limiter","identifiers":["parkovins.rs"],"ca":"https://acme.zerossl.com/v2/DV90","account":"office@datadrill.io"}
caddy             | {"level":"info","ts":1684234643.6240368,"logger":"http","msg":"done waiting on internal rate limiter","identifiers":["parkovins.rs"],"ca":"https://acme.zerossl.com/v2/DV90","account":"office@datadrill.io"}
caddy             | {"level":"info","ts":1684234644.146425,"logger":"http.acme_client","msg":"trying to solve challenge","identifier":"parkovins.rs","challenge_type":"http-01","ca":"https://acme.zerossl.com/v2/DV90"}
caddy             | {"level":"error","ts":1684234649.780045,"logger":"http.acme_client","msg":"challenge failed","identifier":"parkovins.rs","challenge_type":"http-01","problem":{"type":"","title":"","detail":"","instance":"","subproblems":[]}}
caddy             | {"level":"error","ts":1684234649.7800872,"logger":"http.acme_client","msg":"validating authorization","identifier":"parkovins.rs","problem":{"type":"","title":"","detail":"","instance":"","subproblems":[]},"order":"https://acme.zerossl.com/v2/DV90/order/74SZaGau22bHakgHcRZWOw","attempt":1,"max_attempts":3}
caddy             | {"level":"error","ts":1684234649.7801142,"logger":"tls.obtain","msg":"could not get certificate from issuer","identifier":"parkovins.rs","issuer":"acme.zerossl.com-v2-DV90","error":"HTTP 0  - "}
caddy             | {"level":"error","ts":1684234649.7801986,"logger":"tls.obtain","msg":"will retry","error":"[parkovins.rs] Obtain: [parkovins.rs] solving challenge: parkovins.rs: [parkovins.rs] authorization failed: HTTP 0  -  (ca=https://acme.zerossl.com/v2/DV90)","attempt":1,"retrying_in":60,"elapsed":12.01381492,"max_duration":2592000}
caddy             | {"level":"info","ts":1684234709.7810264,"logger":"tls.obtain","msg":"obtaining certificate","identifier":"parkovins.rs"}
caddy             | {"level":"info","ts":1684234710.874321,"logger":"http.acme_client","msg":"trying to solve challenge","identifier":"parkovins.rs","challenge_type":"tls-alpn-01","ca":"https://acme-staging-v02.api.letsencrypt.org/directory"}
caddy             | {"level":"error","ts":1684234711.835971,"logger":"http.acme_client","msg":"challenge failed","identifier":"parkovins.rs","challenge_type":"tls-alpn-01","problem":{"type":"urn:ietf:params:acme:error:unauthorized","title":"","detail":"Cannot negotiate ALPN protocol \"acme-tls/1\" for tls-alpn-01 challenge","instance":"","subproblems":[]}}
caddy             | {"level":"error","ts":1684234711.8366144,"logger":"http.acme_client","msg":"validating authorization","identifier":"parkovins.rs","problem":{"type":"urn:ietf:params:acme:error:unauthorized","title":"","detail":"Cannot negotiate ALPN protocol \"acme-tls/1\" for tls-alpn-01 challenge","instance":"","subproblems":[]},"order":"https://acme-staging-v02.api.letsencrypt.org/acme/order/102374194/8773327814","attempt":1,"max_attempts":3}
caddy             | {"level":"info","ts":1684234713.1591616,"logger":"http.acme_client","msg":"trying to solve challenge","identifier":"parkovins.rs","challenge_type":"http-01","ca":"https://acme-staging-v02.api.letsencrypt.org/directory"}
caddy             | {"level":"error","ts":1684234714.9210389,"logger":"http.acme_client","msg":"challenge failed","identifier":"parkovins.rs","challenge_type":"http-01","problem":{"type":"urn:ietf:params:acme:error:unauthorized","title":"","detail":"2a01:4f8:252:601f::2: Invalid response from http://parkovins.rs/.well-known/acme-challenge/8c6tMGfYJaJuRN0LkvDIS9HDFxk4j7lhiesFCNybAkU: 404","instance":"","subproblems":[]}}
caddy             | {"level":"error","ts":1684234714.9217007,"logger":"http.acme_client","msg":"validating authorization","identifier":"parkovins.rs","problem":{"type":"urn:ietf:params:acme:error:unauthorized","title":"","detail":"2a01:4f8:252:601f::2: Invalid response from http://parkovins.rs/.well-known/acme-challenge/8c6tMGfYJaJuRN0LkvDIS9HDFxk4j7lhiesFCNybAkU: 404","instance":"","subproblems":[]},"order":"https://acme-staging-v02.api.letsencrypt.org/acme/order/102374194/8773328104","attempt":2,"max_attempts":3}
caddy             | {"level":"error","ts":1684234714.9221172,"logger":"tls.obtain","msg":"could not get certificate from issuer","identifier":"parkovins.rs","issuer":"acme-v02.api.letsencrypt.org-directory","error":"HTTP 403 urn:ietf:params:acme:error:unauthorized - 2a01:4f8:252:601f::2: Invalid response from http://parkovins.rs/.well-known/acme-challenge/8c6tMGfYJaJuRN0LkvDIS9HDFxk4j7lhiesFCNybAkU: 404"}
caddy             | {"level":"info","ts":1684234715.5698996,"logger":"http.acme_client","msg":"trying to solve challenge","identifier":"parkovins.rs","challenge_type":"http-01","ca":"https://acme.zerossl.com/v2/DV90"}
caddy             | {"level":"error","ts":1684234721.1209114,"logger":"http.acme_client","msg":"challenge failed","identifier":"parkovins.rs","challenge_type":"http-01","problem":{"type":"","title":"","detail":"","instance":"","subproblems":[]}}
caddy             | {"level":"error","ts":1684234721.120953,"logger":"http.acme_client","msg":"validating authorization","identifier":"parkovins.rs","problem":{"type":"","title":"","detail":"","instance":"","subproblems":[]},"order":"https://acme.zerossl.com/v2/DV90/order/4z2ubEf8GELYZm0n1CQj-w","attempt":1,"max_attempts":3}
caddy             | {"level":"error","ts":1684234721.1209805,"logger":"tls.obtain","msg":"could not get certificate from issuer","identifier":"parkovins.rs","issuer":"acme.zerossl.com-v2-DV90","error":"HTTP 0  - "}
caddy             | {"level":"error","ts":1684234721.121013,"logger":"tls.obtain","msg":"will retry","error":"[parkovins.rs] Obtain: [parkovins.rs] solving challenge: parkovins.rs: [parkovins.rs] authorization failed: HTTP 0  -  (ca=https://acme.zerossl.com/v2/DV90)","attempt":2,"retrying_in":120,"elapsed":83.354629337,"max_duration":2592000}

3. Caddy version:

4. How I installed and ran Caddy:

Running it dockerized with docker compose

a. System environment:

OS is Ubuntu 22.04 (EC2 Instance on AWS)

b. Command:

PASTE OVER THIS, BETWEEN THE ``` LINES.
Please use the preview pane to ensure it looks nice.

c. Service/unit/compose file:

version: "3.8"
services:
  caddy:
    image: caddy:latest
    container_name: caddy
    restart: unless-stopped
    ports:
      - "80:80"
      - "443:443"
    volumes:
      - ./Caddyfile:/etc/caddy/Caddyfile
      - /data/caddy/data:/data # Optional
      - /data/caddy/config:/config # Optional
    networks:
      - net
  parkovi_backend:
    container_name: parkovi_backend
    image: datadrill/parkovi-backend:latest
    environment:
      ACCESS_TOKEN_SECRET: ${ACCESS_TOKEN_SECRET}
      AWS_S3_ACCESS_KEY_ID: ${AWS_S3_ACCESS_KEY_ID}
      AWS_S3_BUCKET_NAME: ${AWS_S3_BUCKET_NAME}
      AWS_S3_REGION: ${AWS_S3_REGION}
      AWS_S3_SECRET_KEY: ${AWS_S3_SECRET_KEY}
      DB_HOST: ${DB_HOST}
      DB_NAME: ${DB_NAME}
      DB_PASSWORD: ${DB_PASSWORD}
      DB_PORT: ${DB_PORT}
      DB_USER: ${DB_USER}
      GOOGLE_API_KEY: ${GOOGLE_API_KEY}
      HOST: ${HOST}
      PORT: ${PORT}
      REFRESH_TOKEN_SECRET: ${REFRESH_TOKEN_SECRET}
    command:
      - /bin/bash
      - -c
      - |
        npx sequelize-cli db:migrate
        npm start
    networks:
      - net
    volumes:
      - ".:/parkovi_backend_vol"
      - /parkovi_backend_vol/node_modules
    ports:
      - "8000:8000"
  parkovi_frontend:
    container_name: parkovi_frontend
    image: datadrill/parkovi-frontend:latest
    environment:
      NEXT_PUBLIC_API_URL: ${NEXT_PUBLIC_API_URL}
      NEXT_PUBLIC_GOOGLE_API_KEY: ${NEXT_PUBLIC_GOOGLE_API_KEY}
      # NEXT_PUBLIC_S3_DOMAIN: ${NEXT_PUBLIC_S3_DOMAIN}
      NEXT_PUBLIC_S3_URL: ${NEXT_PUBLIC_S3_URL}
    volumes:
      - ".:/parkovi_frontend_vol"
      - /parkovi_frontend_vol/node_modules
    ports:
      - "3000:3000"
    networks:
      - net
networks:
  net:

d. My complete Caddy config:

{
	email office@datadrill.io
}

parkovins.rs {
	reverse_proxy parkovi_frontend:3000
	handle_path /api* {
		reverse_proxy parkovi_backend:8000
	}
}

5. Links to relevant resources:

francislavoie · May 16, 2023, 2:29pm

Both of these imply that you’re not hitting Caddy, and instead are hitting another server which is responding with errors when being requested to solve the ACME challenges.

Make sure DNS is correct, make sure your firewall is allowing connections on port 80/443, ports are forwarded, make sure you don’t have another proxy in between, e.g. turn off Cloudflare’s strict mode if you have that on.

These aren’t really optional, you can remove that comment.

You should add - "443:443/udp" as well for HTTP/3 support. See Keep Caddy Running — Caddy Documentation

If you only need Caddy to proxy to these, you don’t need to publish these ports. In fact, publishing them risks other clients making direct requests to these, bypassing TLS.

I would adjust it to this:

	handle_path /api* {
		reverse_proxy parkovi_backend:8000
	}

	handle {
		reverse_proxy parkovi_frontend:3000
	}

A bit more idiomatic. Same thing, just reads nicer.

st3fus · May 17, 2023, 10:12am

I modified everything as you’ve said, I’ve double checked my AWS instance, everything is opened, acutally I’ve opened all ports on the security group for the instance:

Port range Protocol Source
All	           All	   0.0.0.0/0

It’s using some local, serbian cloud provider (Not my choice, client already bought it there) and there’s nothing related to strict in their options. They’re using cPanel to ZoneEdit and I’ve added everything there, including the A record targeting my Elastic IP but it’s still not working, can’t obtain the certificate.

st3fus · May 17, 2023, 10:34am

I’ve removed my domain from that cPanel and created a Hosted Zone on AWS, I think I got too many denies for this hour, so I’ll be waiting another hour to try doing it from AWS

st3fus · May 17, 2023, 11:05am

Getting these errors now, I’ve already added A record and CNAME record on AWS Hosted Zone, does this mean I still need to wait for DNS to propagate? And for how long?

caddy             | {"level":"error","ts":1684321482.3438954,"logger":"http.acme_client","msg":"challenge failed","identifier":"parkovins.rs","challenge_type":"tls-alpn-01","problem":{"type":"urn:ietf:params:acme:error:dns","title":"","detail":"DNS problem: SERVFAIL looking up A for parkovins.rs - the domain's nameservers may be malfunctioning; DNS problem: SERVFAIL looking up AAAA for parkovins.rs - the domain's nameservers may be malfunctioning","instance":"","subproblems":[]}}
caddy             | {"level":"error","ts":1684321482.3446069,"logger":"http.acme_client","msg":"validating authorization","identifier":"parkovins.rs","problem":{"type":"urn:ietf:params:acme:error:dns","title":"","detail":"DNS problem: SERVFAIL looking up A for parkovins.rs - the domain's nameservers may be malfunctioning; DNS problem: SERVFAIL looking up AAAA for parkovins.rs - the domain's nameservers may be malfunctioning","instance":"","subproblems":[]},"order":"https://acme-v02.api.letsencrypt.org/acme/order/1112405227/182741783387","attempt":1,"max_attempts":3}
caddy             | {"level":"info","ts":1684321483.67847,"logger":"http.acme_client","msg":"trying to solve challenge","identifier":"parkovins.rs","challenge_type":"http-01","ca":"https://acme-v02.api.letsencrypt.org/directory"}
caddy             | {"level":"error","ts":1684321489.5259798,"logger":"http.acme_client","msg":"challenge failed","identifier":"parkovins.rs","challenge_type":"http-01","problem":{"type":"urn:ietf:params:acme:error:dns","title":"","detail":"DNS problem: SERVFAIL looking up A for parkovins.rs - the domain's nameservers may be malfunctioning; DNS problem: SERVFAIL looking up AAAA for parkovins.rs - the domain's nameservers may be malfunctioning","instance":"","subproblems":[]}}
caddy             | {"level":"error","ts":1684321489.5266995,"logger":"http.acme_client","msg":"validating authorization","identifier":"parkovins.rs","problem":{"type":"urn:ietf:params:acme:error:dns","title":"","detail":"DNS problem: SERVFAIL looking up A for parkovins.rs - the domain's nameservers may be malfunctioning; DNS problem: SERVFAIL looking up AAAA for parkovins.rs - the domain's nameservers may be malfunctioning","instance":"","subproblems":[]},"order":"https://acme-v02.api.letsencrypt.org/acme/order/1112405227/182741800387","attempt":2,"max_attempts":3}
caddy             | {"level":"error","ts":1684321489.5271711,"logger":"tls.obtain","msg":"could not get certificate from issuer","identifier":"parkovins.rs","issuer":"acme-v02.api.letsencrypt.org-directory","error":"HTTP 400 urn:ietf:params:acme:error:dns - DNS problem: SERVFAIL looking up A for parkovins.rs - the domain's nameservers may be malfunctioning; DNS problem: SERVFAIL looking up AAAA for parkovins.rs - the domain's nameservers may be malfunctioning"}
caddy             | {"level":"info","ts":1684321489.5280194,"logger":"http","msg":"waiting on internal rate limiter","identifiers":["parkovins.rs"],"ca":"https://acme.zerossl.com/v2/DV90","account":"office@datadrill.io"}
caddy             | {"level":"info","ts":1684321489.5284057,"logger":"http","msg":"done waiting on internal rate limiter","identifiers":["parkovins.rs"],"ca":"https://acme.zerossl.com/v2/DV90","account":"office@datadrill.io"}
caddy             | {"level":"info","ts":1684321490.005547,"logger":"http.acme_client","msg":"trying to solve challenge","identifier":"parkovins.rs","challenge_type":"http-01","ca":"https://acme.zerossl.com/v2/DV90"}
caddy             | {"level":"error","ts":1684321495.5036893,"logger":"http.acme_client","msg":"challenge failed","identifier":"parkovins.rs","challenge_type":"http-01","problem":{"type":"","title":"","detail":"","instance":"","subproblems":[]}}
caddy             | {"level":"error","ts":1684321495.5043035,"logger":"http.acme_client","msg":"validating authorization","identifier":"parkovins.rs","problem":{"type":"","title":"","detail":"","instance":"","subproblems":[]},"order":"https://acme.zerossl.com/v2/DV90/order/o4XHsQD5DhUK0rr17Ie7rw","attempt":1,"max_attempts":3}
caddy             | {"level":"error","ts":1684321495.5047207,"logger":"tls.obtain","msg":"could not get certificate from issuer","identifier":"parkovins.rs","issuer":"acme.zerossl.com-v2-DV90","error":"HTTP 0  - "}
caddy             | {"level":"error","ts":1684321495.5047638,"logger":"tls.obtain","msg":"will retry","error":"[parkovins.rs] Obtain: [parkovins.rs] solving challenge: parkovins.rs: [parkovins.rs] authorization failed: HTTP 0  -  (ca=https://acme.zerossl.com/v2/DV90)","attempt":1,"retrying_in":60,"elapsed":20.072356346,"max_duration":2592000}

st3fus · May 17, 2023, 1:21pm

Okay, managed to get the certificate, it was just a matter of waiting for DNS and A record to propagate. Now, I’m getting a 404 error for Backend… Frontend is displayed but I can’t get anything from backend thats running on port 8000

https://parkovins.rs/api/park?lng=cyr&isTour=true

This is the request in Networks tab, once I click on Preview I get this:

Cannot GET /park

And it’s same for others /blog and etc

But all those are behind /api/ which I already proxied in my Caddyfile, aren’t they?

matt · May 17, 2023, 6:59pm

Does your backend support being proxied in a subfolder (/api/)?

francislavoie · May 17, 2023, 7:38pm

If you meant to proxy /api/park without stripping /api, use handle instead of handle_path.

st3fus · May 18, 2023, 10:35am

That might be it, I’ll give it a try

st3fus · May 18, 2023, 4:12pm

So managed to solve that for the other app, thanks a bunch for the help.

Now I have another app to solve I’m hoping I’ll explain this correctly since I’m fresh to this, I wanna see if Caddy can handle this since I’ve been reading about Caddy vs Nginx but most of the blogs/threads are outdated.

I have an app that has Frontend, AdminPanel and Backend. I’ll refer to them as FE, AP and BE.
FE, AP are in Next.js and BE is in Django/Python.

On my current staging EC2 instance, I have everything running on the same machine, FE on port 3000, AP on port 3001 and BE on port 8000

I want to have FE being accessed on domain(.)com, AP being accessed on admin(.)domain(.)com and BE is just in background.

I’m wondering if I can do the reverse proxying with Caddy, so when I access domain(.)com it redirects me to frontend:3000 and when I access admin(.)domain(.)com it redirects me to adminpanel:3001. BE remains on the same for both apps since they’re pulling data from it. Is it possible to solve this with Caddy? I have a basic idea how would i do it

PS Sorry for the brackets, forum doesn’t allow me to post links as it seems.

francislavoie · May 18, 2023, 9:43pm

Yep.

example.com {
	reverse_proxy localhost:3000
}

admin.example.com {
	reverse_proxy localhost:3001
}

And your two apps can just make requests to localhost:8000 to hit the backend as normal.

system · June 17, 2023, 9:44pm

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.