Optional reverse proxy with file system fallback

1. Caddy version (caddy version):

╰─ caddy version
v2.4.6 h1:HGkGICFGvyrodcqOOclHKfvJC0qTU7vny/7FhYp9hNw=

2. How I run Caddy:

via a Caddyfile with the following


cdn.company.local {
  uri strip_prefix /staging
  @isIn0 file {
      root /path/to/phase0/dist
  }
  handle @isIn0 {
      root * /path/to/phase0/dist
      file_server
  }
  @isIn1 file {
      root /path/to/phase1/dist
  }
  handle @isIn1 {
      root * /path/to/phase1/dist
      file_server
  }
  
  handle {
      rewrite * /dist/{path}
      reverse_proxy http://localhost:8092 {
          lb_policy first
          health_status 200
          unhealthy_status 502
          health_uri /
          fail_duration 1s
      }
  }
  
  handle_errors {
      uri strip_prefix /staging
      root * /path/to/phase2/dist
      file_server
  }
}

a. System environment:

Running directly on my m1 macbook pro and also for some people on x86 macbook pro and for others on ubuntu 21.04 in windows WSL

b. Command:

caddy run

c. Service/unit/compose file:

N/A

d. My complete Caddyfile or JSON config:

same as (2)

3. The problem I’m having:

The configuration works as intended.

If the path is in phase0 on disk it serves from that location, if it’s from phase1 on disk it serves from that location, otherwise if my localhost:8092 is up it serves from there, otherwise it serves from phase2 on disk

I just want to do it in a way that doesn’t make caddy keep complaining localhost:8092 isn’t running when it isn’t cause it’s normal for that situation and it creates noise I don’t want.

4. Error messages and/or full log output:

2022/02/15 01:45:15.679 INFO    using adjacent Caddyfile
2022/02/15 01:45:15.686 WARN    input is not formatted with 'caddy fmt' {"adapter": "caddyfile", "file": "Caddyfile", "line": 2}
2022/02/15 01:45:15.688 INFO    admin   admin endpoint started  {"address": "tcp/localhost:2019", "enforce_origin": false, "origins": ["localhost:2019", "[::1]:2019", "127.0.0.1:2019"]}
2022/02/15 01:45:15.688 INFO    tls.cache.maintenance   started background certificate maintenance      {"cache": "0x140003be770"}
2022/02/15 01:45:15.688 INFO    http    server is listening only on the HTTPS port but has no TLS connection policies; adding one to enable TLS {"server_name": "srv0", "https_port": 443}
2022/02/15 01:45:15.688 INFO    http    enabling automatic HTTP->HTTPS redirects        {"server_name": "srv0"}
2022/02/15 01:45:15.691 INFO    http.handlers.reverse_proxy.health_checker.active       HTTP request failed     {"host": "localhost:8092", "error": "Get \"http://localhost:8092/\": dial tcp [::1]:8092: connect: connection refused"}
2022/02/15 01:45:15.784 INFO    pki.ca.local    root certificate is already trusted by system   {"path": "storage:pki/authorities/local/root.crt"}
2022/02/15 01:45:15.785 INFO    http    enabling automatic TLS certificate management   {"domains": ["cdn.company.local"]}
2022/02/15 01:45:15.785 INFO    tls     cleaning storage unit   {"description": "FileStorage:/Users/stephenm/Library/Application Support/Caddy"}
2022/02/15 01:45:15.785 INFO    autosaved config (load with --resume flag)      {"file": "/Users/stephenm/Library/Application Support/Caddy/autosave.json"}
2022/02/15 01:45:15.785 INFO    serving initial configuration
2022/02/15 01:45:15.786 INFO    tls.obtain      acquiring lock  {"identifier": "cdn.company.local"}
2022/02/15 01:45:15.790 INFO    tls     finished cleaning storage units
2022/02/15 01:45:15.807 INFO    tls.obtain      lock acquired   {"identifier": "cdn.company.local"}
2022/02/15 01:45:15.808 INFO    tls.obtain      certificate obtained successfully       {"identifier": "cdn.company.local"}
2022/02/15 01:45:15.808 INFO    tls.obtain      releasing lock  {"identifier": "cdn.company.local"}
2022/02/15 01:45:15.808 WARN    tls     stapling OCSP   {"error": "no OCSP stapling for [cdn.company.local]: no OCSP server specified in certificate"}
2022/02/15 01:45:45.693 INFO    http.handlers.reverse_proxy.health_checker.active       HTTP request failed     {"host": "localhost:8092", "error": "Get \"http://localhost:8092/\": dial tcp [::1]:8092: connect: connection refused"}
2022/02/15 01:46:15.693 INFO    http.handlers.reverse_proxy.health_checker.active       HTTP request failed     {"host": "localhost:8092", "error": "Get \"http://localhost:8092/\": dial tcp [::1]:8092: connect: connection refused"}
2022/02/15 01:46:45.692 INFO    http.handlers.reverse_proxy.health_checker.active       HTTP request failed     {"host": "localhost:8092", "error": "Get \"http://localhost:8092/\": dial tcp [::1]:8092: connect: connection refused"}

5. What I already tried:

I’ve tried understanding how to set logging options to ignore this particular warning but I’m finding it very confusing and haven’t got far with that.

2022/02/15 01:46:45.692 INFO    http.handlers.reverse_proxy.health_checker.active       HTTP request failed     {"host": "localhost:8092", "error": "Get \"http://localhost:8092/\": dial tcp [::1]:8092: connect: connection refused"}

As far as I don’t really have options directly in the Caddyfile to tell it to not care if the localhost:8092 is not there?

You probably don’t need the health checking at all. It’s probably only useful for you if you actually expect high uptime or have more than one backend that you could fallback to.

In your case, I think you can probably just set a pretty low dial_timeout on the http transport, something like 500ms or even 1s is probably plenty for a service running on the same machine. So when the timeout is reached it should probably fallback to your handle_errors.

But what you are trying is pretty unique, I’ve not seen anyone else try this kind of “intentional downtime with fallback” thing for reverse_proxy yet.

1 Like

Hi,

It seems I didn’t have notifications set for this forum so I missed that reply. oops.

Just gave it a shot and it works, except it has worse error logs when the yarn hmr isn’t running (I find yarn hmr destroys cpu and memory after a while, especially on the old versions of things it’s stuck on)

2022/02/16 11:57:06.594 ERROR   http.log.error  dial tcp [::1]:8092: connect: connection refused        {"request": {"remote_addr": "127.0.0.1:63942", "proto": "HTTP/2.0", "
method": "GET", "host": "cdn.company.local", "uri": "/staging/website-ui-staging.css", "headers": {"Sec-Fetch-User": ["?1"], "Sec-Fetch-Dest": ["document"], "Cache-Control"
: ["max-age=0"], "Upgrade-Insecure-Requests": ["1"], "Sec-Gpc": ["1"], "Sec-Fetch-Mode": ["navigate"], "Accept-Language": ["en-GB,en-US;q=0.9,en;q=0.8"], "User-Agent": ["Moz
illa/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/98.0.4758.87 Safari/537.36"], "Accept": ["text/html,application/xhtml+xml,applicat
ion/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.9"], "Sec-Fetch-Site": ["none"], "Accept-Encoding": ["gzip, deflate, br"]}, "tl
s": {"resumed": false, "version": 772, "cipher_suite": 4865, "proto": "h2", "proto_mutual": true, "server_name": "cdn.company.local"}}, "duration": 0.000814958, "status": 5
02, "err_id": "g5ygfdj7q", "err_trace": "reverseproxy.statusError (reverseproxy.go:886)"}
2022/02/16 11:57:07.036 ERROR   http.log.error  error handling handler error    {"request": {"remote_addr": "127.0.0.1:63942", "proto": "HTTP/2.0", "method": "GET", "host":
"cdn.company.local", "uri": "/favicon.ico", "headers": {"Sec-Gpc": ["1"], "Sec-Fetch-Site": ["same-origin"], "Sec-Fetch-Dest": ["image"], "Referer": ["https://cdn.company.
local/staging/website-ui-staging.css"], "Accept-Encoding": ["gzip, deflate, br"], "Accept-Language": ["en-GB,en-US;q=0.9,en;q=0.8"], "User-Agent": ["Mozilla/5.0 (Macintosh;
Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/98.0.4758.87 Safari/537.36"], "Accept": ["image/avif,image/webp,image/apng,image/svg+xml,image/*,*/*;q=
0.8"], "Sec-Fetch-Mode": ["no-cors"]}, "tls": {"resumed": false, "version": 772, "cipher_suite": 4865, "proto": "h2", "proto_mutual": true, "server_name": "cdn.company.loca
l"}}, "duration": 0.00389475, "error": "{id=43374atig} fileserver.(*FileServer).notFound (staticfiles.go:510): HTTP 404", "first_error": {"msg": "dial tcp [::1]:8092: connec
t: connection refused", "status": 502, "err_id": "3g6uxq8jr", "err_trace": "reverseproxy.statusError (reverseproxy.go:886)"}}

Using

      handle {
          rewrite * /dist/{path}
          reverse_proxy http://localhost:8092 {
              transport http {
                  dial_timeout 400ms
              }
          }
      }

      handle_errors {
          uri strip_prefix /staging
          root * /path/to/phase2/dist
          file_server
      }

I suppose I could make some tiny go program that does it’s own proxy…

This essentially means you got a 404 from file_server cause it doesn’t have the file. You could use the file matcher beforehand to just respond with some text, or rewrite to a file that does exist (like a 404.html or whatever, depending).

What’s weird though is that the file is there and it served it up!

And I tried it again and now it only has the first error. In my defence it was nearly midnight when I tried last night :stuck_out_tongue:

I still get that first 502 error though. Can I make that error silent?

Currently there’s no mechanism for that, but I’ll look into seeing if we can quiet errors that get handled handle_errors by changing the log level to DEBUG instead for those (so that they don’t appear when the log level is INFO, i.e. the default).

1 Like

nice, thanks :slight_smile:

1 Like

This topic was automatically closed after 30 days. New replies are no longer allowed.