Rails application behing regular proxy

Hello,

1. Caddy version (caddy version):

not certain. built it from source from the repo so I don’t have the exact version number, but the syntax seems to be v2.

2. How I run Caddy:

Systemd

a. System environment:

Ubuntu 18.04

b. Command:

sudo systemctl restart caddy.service

c. Service/unit/compose file:

[Unit]
Description=Caddy Web Server
Documentation=https://caddyserver.com/docs/
After=network.target

[Service]
User=caddy
Group=caddy
ExecStart=/usr/local/bin/caddy -conf /etc/caddy/Caddyfile -email "pierre.depaz@gmail.com" -agree
ExecReload=/usr/bin/pkill -USR1 caddy
TimeoutStopSec=5s
LimitNOFILE=1048576
LimitNPROC=512
PrivateTmp=true
#ProtectSystem=full
CapabilityBoundingSet=CAP_NET_BIND_SERVICE 
AmbientCapabilities=CAP_NET_BIND_SERVICE
NoNewPrivileges=true

[Install]
WantedBy=multi-user.target

d. My complete Caddyfile or JSON config:

wishes.enframed.net {
  proxy / localhost:3003
  header Host {http.request.host}
  header X-Forwarded-For {http.request.remote.host}
  header X-Forwarded-Port {http.request.port}
  header X-Forwarded-Proto {http.request.scheme} 
  log /var/log/caddy.access.log
  gzip
}

3. The problem I’m having:

I’m using Caddy as a proxy to a Rails application, to which I make POST requests through AJAX. All other requests work fine.

Rails sees the POST request as forwarded by Caddy as coming for http://localhost (wrong scheme+domain), instead of seeing for https://wishes.enframed.net

After some research, it seems that I cannot properly configure caddy to forward this information correctly

4. Error messages and/or full log output:

this is from the rails app log:

W, [2020-11-27T13:07:46.879115 #3990628]  WARN -- : [6d0e97ac-2c10-4a2f-9d51-6acf29c4847b] HTTP Origin header (https://wishes.enframed.net) didn't match request.base_url (http://localhost:3003)                                                              
I, [2020-11-27T13:07:46.879445 #3990628]  INFO -- : [6d0e97ac-2c10-4a2f-9d51-6acf29c4847b] Completed 422 Unprocessable Entity in 0ms (Allocations: 128)                                                                                                        
F, [2020-11-27T13:07:46.880174 #3990628] FATAL -- : [6d0e97ac-2c10-4a2f-9d51-6acf29c4847b]                                                                                                                                                                     
[6d0e97ac-2c10-4a2f-9d51-6acf29c4847b] ActionController::InvalidAuthenticityToken (ActionController::InvalidAuthenticityToken):   

5. What I already tried:

A variety of combinations of the following:

header Host {http.request.host}
header X-Forwarded-For {http.request.remote.host}
header X-Forwarded-Port {http.request.port}
header X-Forwarded-Proto {http.request.scheme}

or

header Host {host}
header X-Enable-Ssl on
header X-Forwarded-For {host}
header X-Forwarded-Port {port}
header X-Forwarded-Proto {scheme}

as well as setting it as a reverse proxy, upon which caddy crashes.

including the post found here

6. Links to relevant resources:

This is the relevant issue in the Rails repo, which I’m unable to properly understand/find the correct syntax to fix: https://github.com/rails/rails/issues/22965

Thanks for the help!

That’s Caddy v1 syntax. In v2, you use reverse_proxy instead.

Please try to figure out exactly which version you’re running. If you’re on v1, please upgrade to v2. Caddy v1 is no longer supported.

In v2, this is probably all you need:

wishes.enframed.net {
	encode gzip
	reverse_proxy localhost:3003
	log {
		output file /var/log/caddy.access.log
	}
}

Hi @francislavoie,

Thanks for your reply. I’ve upgraded to caddy v2.2.1 and migrated my config file with global block and the directives you suggested.

(and now I’m having issues running it properly from the caddy user, since it keeps on asking the Let’s Encrypt servers for certificates, even though it already has a .caddy/acme/sites folder, which then leads to a timeout; but this is not the point of the current issue, I’ll figure it out.)

For now, I can run as sudo caddy run --config /etc/caddy/Caddyfile

So, for the rails app issue, I have copied exactly the config block you provided, along with the file server directive and a change to the log path:

wishes.enframed.net {
  encode gzip
  reverse_proxy / localhost:3003
  root * /var/www/wish-generator/public
  file_server
  log {
    output file /var/log/caddy/access.log
  }
}

Any GET to https://wishes.enframed.net/ completes successfully.

2020/11/28 11:14:27.711 info    http.log.access.log2    handled request {"request": {"remote_addr": "109.190.253.16:54386", "proto": "HTTP/2.0", "method": "GET", "host": "wishes.enframed.net", "uri": "/", "headers": {"Te": ["trailers"], "Accept-Language": ["en-US,en;q=0.5"], "Accept-Encoding": ["gzip, deflate, br"], "Cookie": ["_wish_generator_session=xmmb%2FA%2Bt7aneHUXRRQAIfKYm1zTAfsT%2F0%2FKmjHpazcdAoF%2FdF54k%2Ftl4tsnBUts0ukpRvXce4%2F3d16EHG0e9awAQrbEZ5FY7%2BEEb60D8aspC8xPtxheQOBtUohtMlh4j8Fo8l%2FeRKziv8SiWL%2B4NLd%2BJn2PSf3VwX4a6pSDSWWjrAXy3ToZxBCnV8TiPkaexCB%2B3ana268vAUWO2CsWstxL6zZeo7Ibyi0yOgC%2BlMMTvg2OWALW%2FBoplDG8jf4s6wXAHjClAmya8RnMTpkqCxMS%2F8LhmAXoeGQXd3psTdQ%3D%3D--aPkDRbU%2FBYL%2F3hTO--xSmyn6EO%2BU60HqjR8APPMg%3D%3D"], "Pragma": ["no-cache"], "Cache-Control": ["no-cache"], "User-Agent": ["Mozilla/5.0 (X11; Linux x86_64; rv:84.0) Gecko/20100101 Firefox/84.0"], "Accept": ["text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8"], "Dnt": ["1"], "Upgrade-Insecure-Requests": ["1"]}, "tls": {"resumed": false, "version": 772, "cipher_suite": 4865, "proto": "h2", "proto_mutual": true, "server_name": "wishes.enframed.net"}}, "common_log": "109.190.253.16 - - [28/Nov/2020:06:14:27 -0500] \"GET / HTTP/2.0\" 200 585", "duration": 0.005321212, "size": 585, "status": 200, "resp_headers": {"X-Xss-Protection": ["1; mode=block"], "Server": ["Caddy"], "X-Download-Options": ["noopen"], "X-Permitted-Cross-Domain-Policies": ["none"], "X-Request-Id": ["62cbbe3e-27d3-4349-b3f6-35c228245ce6"], "Vary": ["Accept-Encoding"], "Referrer-Policy": ["strict-origin-when-cross-origin"], "X-Content-Type-Options": ["nosniff"], "X-Runtime": ["0.003256"], "Content-Type": ["text/html; charset=utf-8"], "Etag": ["W/\"7f04bda298869a37ddaa52f901163fab\""], "Set-Cookie": ["_wish_generator_session=TaEfWvdw1yVS8fOR16zX4MurUh1QoS1J20LMiHuirrvNDQ%2BrfgQGSWoufQp6knptqgkW%2BOUGMIlnC8xpTapyurOED7ourODjPd9E%2FwgCWsb1%2BWeVNP5E1StCfN3x%2FWGWobs9lTMjNJt5xLe6RdAKFhL9VB%2BR6eWyA20vqa5Nx8eX572gMRkQVHNbCEEmKd0mgvM9qKHfpKB5G%2FdTe1m5Su68KIhumrHqMijJ9z6Ns1ITG6Pr6fG3eAT0L2Y78JywoMScScjPvC86hmfaK64KLuu0OjJaIgF%2FRHbZvG4leA%3D%3D--mdVesLHw8vZBaMwJ--3ic0nOXhAmjcBvqR3dQKKA%3D%3D; path=/; HttpOnly"], "X-Frame-Options": ["SAMEORIGIN"], "Cache-Control": ["max-age=0, private, must-revalidate"], "Content-Encoding": ["gzip"]}}

Any GET to any other path fails with the following (here, /wishes/new):

2020/11/28 11:08:19.205 error   http.log.access.log2    handled request {"request": {"remote_addr": "109.190.253.16:54310", "proto": "HTTP/2.0", "method": "GET", "host": "wishes.enframed.net", "uri": "/wishes/new", "headers": {"Upgrade-Insecure-Requests": ["1"], "Te": ["trailers"], "User-Agent": ["Mozilla/5.0 (X11; Linux x86_64; rv:84.0) Gecko/20100101 Firefox/84.0"], "Referer": ["https://wishes.enframed.net/"], "Dnt": ["1"], "Cookie": ["_wish_generator_session=xmmb%2FA%2Bt7aneHUXRRQAIfKYm1zTAfsT%2F0%2FKmjHpazcdAoF%2FdF54k%2Ftl4tsnBUts0ukpRvXce4%2F3d16EHG0e9awAQrbEZ5FY7%2BEEb60D8aspC8xPtxheQOBtUohtMlh4j8Fo8l%2FeRKziv8SiWL%2B4NLd%2BJn2PSf3VwX4a6pSDSWWjrAXy3ToZxBCnV8TiPkaexCB%2B3ana268vAUWO2CsWstxL6zZeo7Ibyi0yOgC%2BlMMTvg2OWALW%2FBoplDG8jf4s6wXAHjClAmya8RnMTpkqCxMS%2F8LhmAXoeGQXd3psTdQ%3D%3D--aPkDRbU%2FBYL%2F3hTO--xSmyn6EO%2BU60HqjR8APPMg%3D%3D"], "Accept": ["text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8"], "Accept-Language": ["en-US,en;q=0.5"], "Accept-Encoding": ["gzip, deflate, br"]}, "tls": {"resumed": false, "version": 772, "cipher_suite": 4865, "proto": "h2", "proto_mutual": true, "server_name": "wishes.enframed.net"}}, "common_log": "109.190.253.16 - - [28/Nov/2020:06:08:19 -0500] \"GET /wishes/new HTTP/2.0\" 404 0", "duration": 0.000157395, "size": 0, "status": 404, "resp_headers": {"Server": ["Caddy"]}}

The logs from the rails app only show a 200 response to the / route, and nothing to /wishes/new.

I’m still reading up on the documentation and making sure caddy can run as a user, but any leads would be appreciated!

Thanks for your help,

In v2, path matching is exact, so with reverse_proxy / you’re telling Caddy to only proxy on requests to exactly / (and not things like /foo). You can remove the / to make Caddy proxy on all requests, or change it to something like /api* to only proxy on requests that start with /api.

Also, it doesn’t make sense to both have a file server and a proxy enabled at the same time without having matchers to decide when either should be used. When do you need static files to be served vs when do you need to proxy? You need to tell Caddy what to do.

1 Like

Sweet, thanks.

This is the first time that I’m developing a Rails application, so I wasn’t quite sure who was in charge of what, and it looked like, by default, Rails doesn’t serve static files (with recommendations to do so via ngnix or apache). I’ve commented out the root * and file_server directives and it only serves all pages when I explicitly configure Rails to do so.

Anyways, this did fix my POST request issue, thanks! Apologies for the simple problem, but I was a little confused by the v1 vs. v2 syntax.

Any leads for the fact that the caddy user cannot access the .caddy/acme contents? Or should I start a separate thread?

Here is some info just in case:

global options

{
  email redacted.redacted@gmail.com
}

v2 service file (taken from the repo)

# 
# Make sure the ExecStart and ExecReload commands are correct
# for your installation.
#
# See https://caddyserver.com/docs/install for instructions.
# 
# WARNING: This service does not use the --resume flag, so if you
# use the API to make changes, they will be overwritten by the
# Caddyfile next time the service is restarted. If you intend to
# use Caddy's API to configure it, add the --resume flag to the
# `caddy run` command or use the caddy-api.service file instead.
                        
[Unit]
Description=Caddy
Documentation=https://caddyserver.com/docs/
After=network.target network-online.target
Requires=network-online.target
  
[Service]
User=caddy
Group=caddy
ExecStart=/usr/bin/caddy run --environ --config /etc/caddy/Caddyfile
ExecReload=/usr/bin/caddy reload --config /etc/caddy/Caddyfile
TimeoutStopSec=5s
LimitNOFILE=1048576
LimitNPROC=512  
PrivateTmp=true         
ProtectSystem=full      
AmbientCapabilities=CAP_NET_BIND_SERVICE
        
[Install]
WantedBy=multi-user.target

My caddy user:

caddy:x:998:998:Caddy web server:/var/lib/caddy:/bin/bash

The caddy home dir

pierre@enframed:~$ ls -la /var/lib/caddy/
total 52
drwxr-xr-x  6 caddy caddy  4096 Nov 28 06:36 .
drwxr-xr-x 48 root  root   4096 Jul 21 16:25 ..
-rw-------  1 caddy caddy  1639 Nov 28 06:45 .bash_history
-rw-r--r--  1 caddy caddy   220 Apr  8  2014 .bash_logout
-rw-r--r--  1 caddy caddy  3771 May 16  2017 .bashrc
drwxr-xr-x  5 caddy caddy 12288 Nov 27 05:36 .caddy
-rw-r--r--  1 caddy caddy     0 Nov 25  2018 .cloud-locale-test.skip
drwx------  3 caddy caddy  4096 Nov 28 03:40 .config
drwx------  3 caddy caddy  4096 Nov 28 05:30 .local
-rw-r--r--  1 caddy caddy   655 May 16  2017 .profile
drwx------  2 caddy caddy  4096 Nov 28 03:37 .step
-rw-------  1 caddy caddy  1313 Nov 28 06:36 .viminfo

contents of .caddy/

pierre@enframed:~$ ls -la /var/lib/caddy/.caddy/acme/
total 24
drwxr-xr-x 4 caddy caddy  4096 Apr 10  2020 .
drwxr-xr-x 5 caddy caddy 12288 Nov 27 05:36 ..
drwxr-xr-x 4 caddy caddy  4096 Dec 30  2017 acme-v01.api.letsencrypt.org
drwxr-xr-x 5 caddy caddy  4096 Apr 10  2020 acme-v02.api.letsencrypt.org

when I try caddy run

I get

2020/11/28 12:07:42.469 INFO    tls.obtain      lock acquired   {"identifier": "rules.enframed.net"}
2020/11/28 12:07:42.831 ERROR   tls.obtain      will retry      {"error": "[pierredepaz.net] Obtain: registering account with server: request to https://acme-v02.api.letsencrypt.org/acme/new-acct failed after 1 attempts: HTTP 429 urn:ietf:params:acme:error:rateLimited - Error creating new account :: too many registrations for this IP: see https://letsencrypt.org/docs/rate-limits/", "attempt": 1, "retrying_in": 60, "elapsed": 0.372432192, "max_duration": 2592000}

Thanks!

You’re looking at the old storage location for Caddy v1. In v2, it’s in /var/lib/caddy/.local/share/caddy

I doubt it’s a storage issue, because Caddy checks beforehand if it can write where it needs to before starting cert maintenance, and the lock acquired message is confirming that.

You’ve been rate limited by Let’s Encrypt, probably because of too many failed attempts. You’ll need to wait it out. It’s unclear why it failed though, without full logs of your previous attempts.

Back to the file_server, which paths contain static files that you need to serve? You could do something like this:

@static path /css/* /js/*
handle @static {
	root * /var/www/wish-generator/public
	file_server
}

handle {
	reverse_proxy localhost:3003
}

Basically this will handle requests to those paths using the file server, otherwise fallback to proxying. There’s lots you can do here, you just need to figure out what you need.

1 Like

Sweet, thanks, I didn’t notice the change in storage location. I’ve copied all my certificates to it and now all domains are served correctly.

Got it, yes, there are only two folders I need to serve through the webserver, I’ve changed the Caddyfile and it works great.

Thank you for your help!

This topic was automatically closed after 30 days. New replies are no longer allowed.