Help converting nginx config to caddy

thebetterjort · May 3, 2021, 8:35pm

1. Caddy version (`caddy version`):

caddy v2 latest docker

2. How I run Caddy:

dev.arwd.circ8.dcn {
 tls /root/certs/example.crt /root/certs/example.key
 reverse_proxy /calendar/* http://calendarservice
 reverse_proxy /fcsf/* http://signup-api:1984
 reverse_proxy /auth/* http://jenieauth
 reverse_proxy /hasura/* http://graphql-engine:8080
 reverse_proxy /graphql/* http://travelvouchers-server:3000
 reverse_proxy /employees/* http://employee-sync:3004
 reverse_proxy /directory-api/* http://directory-api:4005
}

a. System environment:

b. Command:

docker-compose up -d --build 
docker-compose restart

c. Service/unit/compose file:

version: "3.7"

services:
  caddy:
    image: caddy:latest
    restart: unless-stopped
    ports:
      - "80:80"
      - "443:443"
    volumes:
      - ./Caddyfile:/etc/caddy/Caddyfile
      - ./certs:/root/certs
      - caddy_data:/data
      - caddy_config:/config
    networks:
      - backend

volumes:
  caddy_data:
  caddy_config:

networks:
  backend:
   external: true

d. My complete Caddyfile or JSON config:

dev.arwd.circ8.dcn {
 tls /root/certs/example.crt /root/certs/example.key
 reverse_proxy /calendar/* http://calendarservice
 reverse_proxy /fcsf/* http://signup-api:1984
 reverse_proxy /auth/* http://jenieauth
 reverse_proxy /hasura/* http://graphql-engine:8080
 reverse_proxy /graphql/* http://travelvouchers-server:3000
 reverse_proxy /employees/* http://employee-sync:3004
 reverse_proxy /directory-api/* http://directory-api:4005
}

3. The problem I’m having:

I want to move away from Nginx and start using Caddy.

Trying to run caddy in front of several applications. Most of importantly is a nextjs app. I get a 502 error when trying to access the backend services. I have this setup on a different server that works well, but it only has one service.

2021/05/03 20:21:02 http: TLS handshake error from 156.127.60.152:55571: remote error: tls: unknown certificate authority.

I chaulk this to some type of browser not liking the cert – either because its now being served under caddy instead of nginx, but I’m not sure. I don’t get this error on my other server that I’m running my own certs on.

4. Error messages and/or full log output:

2021/05/03 19:54:35 http: TLS handshake error from 156.127.60.152:53052: read tcp 172.31.0.3:443->156.127.60.152:53052: read: connection timed out
2021/05/03 19:55:27 http: TLS handshake error from 156.127.60.152:60446: remote error: tls: unknown certificate authority
2021/05/03 19:55:27 http: TLS handshake error from 156.127.60.152:60447: remote error: tls: unknown certificate authority
2021/05/03 19:55:27 http: TLS handshake error from 156.127.60.152:60448: remote error: tls: unknown certificate authority
2021/05/03 19:55:27 http: TLS handshake error from 156.127.60.152:60451: remote error: tls: unknown certificate authority
2021/05/03 19:55:27 http: TLS handshake error from 156.127.60.152:60449: remote error: tls: unknown certificate authority
2021/05/03 19:55:27 http: TLS handshake error from 156.127.60.152:60450: remote error: tls: unknown certificate authority
2021/05/03 19:55:27 http: TLS handshake error from 156.127.60.152:60452: remote error: tls: unknown certificate authority

5. What I already tried:

I need help debugging why my services aren’t showing up on the paths. NextJs app should be on /fcsf and forwarded to the signup-api (dockerhostname) on port 1984 – But, I don’t see any foot traffic on the nextjs logs. I’ve confirmed the docker network names and that is carried over from nginx. So, I’m confident that the docker containers are talking to each other.

I’ve tried to disable http->https redirect to see if its an issue with the certs. I added this
{
auto_https off
}
to the top of the Caddyfile, but I still get redirects in firefox.

6. Links to relevant resources: nginx.conf

proxy_cache_path /var/cache/nginx levels=1:2 keys_zone=STATIC:10m inactive=7d use_temp_path=off;

upstream fcsf-api {
  server signup-api:1984;
}

upstream internal-site {
  server www.arwd.circ8.dcn;
}


server {
    listen 443 ssl http2 default_server;
    server_name _;

        ssl_certificate /etc/nginx/ssl/self.cer;
        ssl_certificate_key /etc/nginx/ssl/self-ssl.key;


    ssl_protocols TLSv1.2 TLSv1.3;
    ssl_ciphers ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY130
5:ECDHE-RSA-CHACHA20-POLY1305:DHE-RSA-AES128-GCM-SHA256:DHE-RSA-AES256-GCM-SHA384;
    ssl_prefer_server_ciphers off;

        ssl_ecdh_curve secp384r1;
        ssl_session_timeout 5m;
        ssl_session_cache shared:SSL:5m;
        ssl_session_tickets off;
        ssl_stapling off;
        ssl_stapling_verify off;
        add_header Strict-Transport-Security "max-age=63072000; includeSubdomains";
        add_header X-Frame-Options DENY;
        add_header X-Content-Type-Options nosniff;
        #access_log  /var/log/nginx/host.access.log  main;

   location / {
     proxy_pass https://internal-site/;
     proxy_set_header X-Real-IP $remote_addr;
}


    location /fcsf {
    proxy_pass http://fcsf-api/fcsf/;
    proxy_redirect  off;
    proxy_set_header X-Real-IP $remote_addr;
  }

    location /fcsf/_next/static/ {
    proxy_cache STATIC;
    proxy_pass http://fcsf-api/fcsf/_next/static/;

    # For testing cache - remove before deploying to production
    #add_header X-Cache-Status $upstream_cache_status;
  }

  location /fcsf/static/ {
    proxy_cache STATIC;
    proxy_ignore_headers Cache-Control;
    proxy_cache_valid 60m;
    proxy_pass http://fcsf-api/fcsf/static/;

    # For testing cache - remove before deploying to production
    #add_header X-Cache-Status $upstream_cache_status;
  }

    location /fcsf2 {
    proxy_pass         http://fcsf-api/;
    proxy_redirect     off;
    proxy_set_header   Host             $host;
    proxy_set_header   X-Real-IP        $remote_addr;
    proxy_set_header   X-Forwarded-For  $proxy_add_x_forwarded_for;
  }



    location /listen/ {
      proxy_pass http://icecast:8000/;
      proxy_redirect     off;
      proxy_set_header   Host             $host;
      proxy_set_header   X-Real-IP        $remote_addr;
      proxy_set_header   X-Forwarded-For  $proxy_add_x_forwarded_for;
    }

    location /calendar/ {
        proxy_pass http://calendarservice/;
     }

    location /jenieauth/ {
        proxy_pass http://jenieauth/;
      }

   location /hasura/ {
       proxy_pass http://graphql-engine:8080/;
    }

   location /graphql/ {
       proxy_pass http://travelvouchers-server:3000/;
    }

   location /employees/ {
           proxy_pass http://employee-sync:3004/;
    }

   location /directory-api/ {
     proxy_pass http://directory-api:4005/;
   }
    error_page   500 502 503 504  /50x.html;
    location = /50x.html {
        root   /usr/share/nginx/html;
    }


}

francislavoie · May 3, 2021, 9:37pm

Which version exactly? Run docker-compose exec caddy caddy version to find out. You should be at v2.3.0 or later.

I think this is because your certificate doesn’t contain the full chain, but just the leaf certificate. You need to append the CA’s intermediate certificate to your /root/certs/example.crt. For example: https://www.digicert.com/kb/ssl-support/pem-ssl-creation.htm

thebetterjort · May 4, 2021, 4:06am

Thank you @francislavoie

I changed my docker-compose version to caddy:2.3.0 and I added the entire chain to the .pem file per your instructions and the tls error went away!! Progress!

The error only went away when I updated the version and not when I updated the .pem file.
I’m able to get to a service at the root dev.arwd.circ8.dcn, but not dev.arwd.circ8.dcn/fcsf
fcsf is a nextjs app on port 1984, I get the nextjs default 404 error.
anything on dev.arwd.circ8.dcn/fcsf gets a 404 error, strangely css/js assets gets loaded dev.arwd.circ8.dcn/_next and returns 200 –
How can I rewrite the rules in Caddyfile?

francislavoie · May 4, 2021, 4:19am

You’re likely running into this:

I don’t know what you mean. Please be more specific.

thebetterjort · May 4, 2021, 4:32am

I’ve added /fcsf to basePath in my nextjs build. Still no dice.

Thanks for the link, I’m not sure it applies.

If I just start with Hasura, which according to their documents works out of the box with Caddy.
Enable HTTPS | Hasura GraphQL Docs

hasura.arwd.circ8.dcn {
 tls /root/certs/example.pem /root/certs/example.key
 reverse_proxy /hasura/* http://graphql-engine:8080
}

The problem now is that ACME/ZeroSSL fire up … and the auto https starts up… but I’m trying to pass my own cert. The other issue is I’m trying to have 1 cert that covers https for the domain and that is why I prefer the arwd.circ8.dcn/path – if that make sense.

Attackers might be trying to steal your information from hasura.dev.arwd.circ8.dcn (for example, passwords, messages, or credit cards). Learn more

NET::ERR_CERT_COMMON_NAME_INVALID

francislavoie · May 4, 2021, 4:46am

If the certificate doesn’t have hasura.dev.arwd.circ8.dcn in it, then you can’t use it for that domain. Because of that, Caddy tries to fetch a certificate for you to make it work.

You still haven’t given me any more information to help you otherwise.

thebetterjort · May 4, 2021, 4:49am

I’m able to resolve /fcsf by

setup buildDir to include /fcsf
reverse proxy * http://signup-api:1984
This resolves to dev.arwd.circ8.dcn/fcsf

My question is hasura starts by default on dev.arwd.circ8.dcn/hasura or at least thats expected, but nextjs keeps getting in the way (I think) because I get the nextjs 404 error page.

greenpau · May 4, 2021, 6:52am

@thebetterjort , based on you saying “hasura starts by default on dev.arwd.circ8.dcn/hasur” … I think you need to rewrite of /hasura such that the backend, i.e. http://graphql-engine:8080, would receive / (instead of /hasura).

It would be something along the lines of:

route /hasura* {
  uri strip_prefix /hasura
  reverse_proxy http://graphql-engine:8080
}

thebetterjort · May 4, 2021, 7:03am

Ok, interesting. I’ve given that a go and hasura redirects to dev.arwd.circ8.dcn/console… I’m trying to wrap my head around this. We are talking to hasura at some level…
dev.arwd.circ8.dcn/hasura/console is the admin page usually.

EDIT: I was able to get hasura working like this, but not with the /hasura path that I need.
reverse_proxy http://graphql-engine:8080

Whitestrake · May 4, 2021, 7:23am

If it’s redirecting to /console instead of /hasura/console, it sounds like the backend app is ignorant of the base URL.

It’s worth reiterating that based on this behaviour, the wiki page I wrote that @francislavoie linked to a little further up this thread is going to be pertinent information.

As outlined in that wiki entry, your options would be:

Make the backend app aware of your base URL (may not be possible, I’m unfamiliar with that app)
Use a subdomain instead of a subfolder (you’ve indicated this is undesirable)
Manually filter any links and redirects to impose the base URL wherever applicable (can get incredibly complex)

thebetterjort · May 4, 2021, 7:26am

@Whitestrake Thank you for your response.

I was able to get hasura going with all of your help. I’m starting to pick up more on what is happening on the Caddy side. One question, if I want hasura to be on /hasura instead of the default /console … how would I make that change? I think this is a dumb question, but I’m confused on this. Seems like I can only get it work if I name the route exactly what the path of the http service is.

dev.arwd.circ8.dcn {
 tls /root/certs/example.pem /root/certs/example.key
  route /console {
    reverse_proxy http://graphql-engine:8080
  }
 route / {
  reverse_proxy http://signup-api:1984
 }
}

Whitestrake · May 4, 2021, 8:10am

Yes. I apologise for repeatedly hammering this point, but I did write this post specifically to explain exactly what’s going on so people in your situation can understand it and make an informed decision on how to proceed - again, this is the post previously linked in this thread:

I would excerpt the pertinent points, but truly, the entire post is pertinent and - I hope - pretty informative.

thebetterjort · May 4, 2021, 8:17am

Thank you for your wiki entry. It’s helpful, but I have several services and its unlikely that all of them suffer the same painful issues that you describe.

EDIT: NextJS proxy server is butting heads with caddy. I’ve added /fcsf to the buildPath as per your wiki entry, but I need a better way of writing out the configuration so that reverse_proxy * to get NextJS away from the root path.

Whitestrake · May 4, 2021, 8:19am

Yes, many apps can be written with entirely relative links such that this problem never ever arises in the first place.

Some apps have complexity that precludes universally relative linking.

I gather from your posts that Hasura, specifically, is the app you’re having this trouble with. Thus, Hasura is the app you will need to handle specially.

thebetterjort · May 4, 2021, 8:44am

Here is the solution:

dev.arwd.circ8.dcn {

  tls /root/certs/example.pem /root/certs/example.key

  reverse_proxy http://calendarservice
  reverse_proxy http://jenieauth
  reverse_proxy http://employee-sync:3004
  reverse_proxy http://directory-api:4005
  reverse_proxy http://graphql-engine:8080
  reverse_proxy http://travelvouchers-server:3000

  route /listen* {
  reverse_proxy http://icecast:8000
  }
  route /fcsf* {
  reverse_proxy http://signup-api:1984
  }

}

system · June 2, 2021, 8:35pm

This topic was automatically closed after 30 days. New replies are no longer allowed.