Caddy Stripping QueryString?

claycle · February 15, 2024, 5:27am

1. The problem I’m having:

I am serving a static website (built with Hugo) on a “simple” caddy server in AWS Lightsail (Ubuntu). All has been going swimmingly until I wanted to add a search function to my website. This is handled by sending a QueryString “?q=foobar” to a search URI on the website (mysite/search?q=foobar).

This all worked just fine on localhost, but then I deployed to my AWS server with caddy and searches failed (nothing was found). I spent many hours scratching my head, trying to figure out what I was (or Hugo was) doing wrong and searching the web for answers. In my gyrations, I noted that the URI was being stripped of the querystring.

I finally though to try hosting the site on another webserver I had, that has a basic deployment of Apache on it.

And it worked perfectly. The querystring was not stripped from the URI and the search page functioned as expected.

Unfortunately, I don’t know why Caddy is doing this (presumably it’s Caddy), what magic words I need to invoke in searching the web to find documentation on it, or what I need to add or change in my configuration to make it work.

So I am asking for a clue for $100, Alex.

2. Error messages and/or full log output:

There are no errors. Caddy is just stripping the querystring off the URI???

3. Caddy version:

ubuntu@ip-172-26-6-157:~$ caddy version
v2.7.6 h1:w0NymbG2m9PcvKWsrXO6EEkY9Ru4FJK8uQbYcev1p3A=

4. How I installed and ran Caddy:

Ubuntu apt and systemctl.

a. System environment:

AWL Lightsail Ubuntu 20.x

Ubuntu 20.04.6 LTS (GNU/Linux 5.15.0-1053-aws x86_64)

b. Command:

PASTE OVER THIS, BETWEEN THE ``` LINES.
Please use the preview pane to ensure it looks nice.

c. Service/unit/compose file:

PASTE OVER THIS, BETWEEN THE ``` LINES.
Please use the preview pane to ensure it looks nice.

d. My complete Caddy config:

www.foundry.deadlymoon.info {
# Set this path to your site's directory.
# root * /usr/share/caddy
root * /home/ubuntu/www

# Enable the static file server.
file_server {
hide 404.html 
}
handle_errors {
@404 {
expression {http.error.status_code} == 404
}
rewrite @404 /404.html
file_server
}

encode zstd gzip

# Another common task is to set up a reverse proxy:
# reverse_proxy localhost:8080

# Or serve a PHP site through php-fpm:
#php_fastcgi localhost:9000
php_fastcgi unix//run/php/php-fpm.sock

# Prevent access to dot-files, except .well-known
@dotFiles {
path */.*
not path /.well-known/*
}
}

# Refer to the Caddy docs for more information:
# https://caddyserver.com/docs/caddyfile

foundry.deadlymoon.info {
# PROXY ALL REQUEST TO PORT 30000
reverse_proxy localhost:30000
encode zstd gzip
}

5. Links to relevant resources:

francislavoie · February 15, 2024, 8:02am

Your config is messy to read, it looks like all whitespace got lost. Please run caddy fmt -w to clean it up.

Please enable the debug global option, then make a request with curl -v and show your logs.

We need to see evidence of the problem.

claycle · February 15, 2024, 11:48am

That was just bad markup.

For the command:

curl -v "https://www.foundry.deadlymoon.info/search/?q=zola" > ~/curl.txt

# The Caddyfile is an easy way to configure your Caddy web server.
#
# Unless the file starts with a global options block, the first
# uncommented line is always the address of your site.
#
# To use your own domain name (with automatic HTTPS), first make
# sure your domain's A/AAAA DNS records are properly pointed to
# this machine's public IP, then replace ":80" below with your
# domain name.

www.foundry.deadlymoon.info {
        # Set this path to your site's directory.
        # root * /usr/share/caddy
        root * /home/ubuntu/www

        # Enable the static file server.
        file_server {
                hide 404.html
        }
        handle_errors {
                @404 {
                        expression {http.error.status_code} == 404
                }
                rewrite @404 /404.html
                file_server
        }

        encode zstd gzip

        # Another common task is to set up a reverse proxy:
        # reverse_proxy localhost:8080

        # Or serve a PHP site through php-fpm:
        #php_fastcgi localhost:9000
        php_fastcgi unix//run/php/php-fpm.sock

        # Prevent access to dot-files, except .well-known
        @dotFiles {
                path */.*
                not path /.well-known/*
        }
}

# Refer to the Caddy docs for more information:
# https://caddyserver.com/docs/caddyfile

foundry.deadlymoon.info {
        # PROXY ALL REQUEST TO PORT 30000
        reverse_proxy localhost:30000
        encode zstd gzip
}

curl -v

ubuntu@ip-172-26-6-157:/var/log/caddy$ curl -v "https://www.foundry.deadlymoon.info/search/?q=zola" > ~/curl.txt
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0*   Trying 3.136.224.250:443...
* TCP_NODELAY set
* Connected to www.foundry.deadlymoon.info (3.136.224.250) port 443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* successfully set certificate verify locations:
*   CAfile: /etc/ssl/certs/ca-certificates.crt
  CApath: /etc/ssl/certs
} [5 bytes data]
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
} [512 bytes data]
* TLSv1.3 (IN), TLS handshake, Server hello (2):
{ [122 bytes data]
* TLSv1.3 (IN), TLS handshake, Encrypted Extensions (8):
{ [15 bytes data]
* TLSv1.3 (IN), TLS handshake, Certificate (11):
{ [2407 bytes data]
* TLSv1.3 (IN), TLS handshake, CERT verify (15):
{ [79 bytes data]
* TLSv1.3 (IN), TLS handshake, Finished (20):
{ [36 bytes data]
* TLSv1.3 (OUT), TLS change cipher, Change cipher spec (1):
} [1 bytes data]
* TLSv1.3 (OUT), TLS handshake, Finished (20):
} [36 bytes data]
* SSL connection using TLSv1.3 / TLS_AES_128_GCM_SHA256
* ALPN, server accepted to use h2
* Server certificate:
*  subject: CN=www.foundry.deadlymoon.info
*  start date: Feb 12 04:11:40 2024 GMT
*  expire date: May 12 04:11:39 2024 GMT
*  subjectAltName: host "www.foundry.deadlymoon.info" matched cert's "www.foundry.deadlymoon.info"
*  issuer: C=US; O=Let's Encrypt; CN=R3
*  SSL certificate verify ok.
* Using HTTP2, server supports multi-use
* Connection state changed (HTTP/2 confirmed)
* Copying HTTP/2 data in stream buffer to connection buffer after upgrade: len=0
} [5 bytes data]
* Using Stream ID: 1 (easy handle 0x563bd49490e0)
} [5 bytes data]
> GET /search/?q=zola HTTP/2
> Host: www.foundry.deadlymoon.info
> user-agent: curl/7.68.0
> accept: */*
> 
{ [5 bytes data]
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
{ [122 bytes data]
* Connection state changed (MAX_CONCURRENT_STREAMS == 250)!
} [5 bytes data]
< HTTP/2 200 
< accept-ranges: bytes
< alt-svc: h3=":443"; ma=2592000
< content-type: text/html; charset=utf-8
< etag: "s8wb6w4ke"
< last-modified: Thu, 15 Feb 2024 11:42:32 GMT
< server: Caddy
< content-length: 5918
< date: Thu, 15 Feb 2024 11:46:26 GMT
< 
  0  5918    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0{ [5 bytes data]
100  5918  100  5918    0     0  50581      0 --:--:-- --:--:-- --:--:-- 50152
* Connection #0 to host www.foundry.deadlymoon.info left intact

curl.txt

ubuntu@ip-172-26-6-157:/var/log/caddy$ cat ~/curl.txt 
<!DOCTYPE html>
<html lang="en-us" dir="ltr">

<head>
  <meta charset="utf-8">
<meta name="viewport" content="width=device-width">

<title>Search | Song of the Woodpecker</title>




      <script src="https://www.foundry.deadlymoon.info/js/main.882bd7acc809fb3fe602db0c7a1b9cb98576aca7f2a11377550d9020ef850dfc.js" integrity="sha256-iCvXrMgJ&#43;z/mAtsMehucuYV2rKfyoRN3VQ2QIO&#43;FDfw=" crossorigin="anonymous"></script>





<link rel="stylesheet" href="https://www.foundry.deadlymoon.info/css/styles.min.css" media="screen">
<link rel='shortcut icon' type='image/x-icon' href='/favicon.ico' />

</head>

<body>
  <header>
    

<nav class="navbar navbar-expand-lg navbar-dark bg-dark m-0">
  <div class="container-fluid m-0">
    
    <a class="navbar-brand" href="/">
    <img src="https://www.foundry.deadlymoon.info/images/wp-shield.webp" alt="" width="40" height="auto" class="d-inline-block align-text-middle">
    Song of the Woodpecker
    </a>
  
    <button class="navbar-toggler" type="button" data-bs-toggle="collapse" data-bs-target="#navbarSupportedContent" aria-controls="navbarSupportedContent" aria-expanded="false" aria-label="Toggle navigation">
      <span class="navbar-toggler-icon"></span>
    </button>
    
    <div class="collapse navbar-collapse" id="navbarSupportedContent">
    <ul class="navbar-nav me-auto mb-2 mb-lg-0" style="width:100%">
    
    <li class="nav-item">
      <a class="nav-link" href="/about/">About</a>
    </li>
    
    <li class="nav-item">
      <a class="nav-link" href="/stories/">Stories</a>
    </li>
    
    <li class="nav-item">
      <a class="nav-link" href="/lore/">Lore</a>
    </li>
    
    <li class="nav-item">
      <a class="nav-link" href="/rules/">Rules</a>
    </li>
    
    <li class="nav-item dropdown">
      <a class="nav-link dropdown-toggle"
          id="navbarDropdown5" role="button" data-bs-toggle="dropdown" aria-expanded="false" href="http://chaosium.com"><span class="runes">G</span></a>
        <ul class="dropdown-menu" aria-labelledby="navbarDropdown5">
    <li>
      <a class="dropdown-item" href="http://chaosium.com">Chaosium</a>
    </li>
        
    <li><hr class="dropdown-divider"></li>
    
    <li>
      <a class="dropdown-item" href="https://wellofdaliath.chaosium.com/">Well of Deliath</a>
    </li>
    
        </ul>
    </li>
    </ul>
    

    
    <script>
      var SITE = "https://www.deadlymoon.info";
      
      function fireSearch(f) {
        alert(f); 
        if (!f) return false;
        
        
        var query = new URLSearchParams();
        console.log("F", f.elements.q.name, f.elements.q.value);
        
        query.append(f.elements.q.name, f.elements.q.value);
        var url = SITE + "/search?" + query.toString();
        url = /search?" + query.toString();
        alert(url);
        //window.location.assign("http://www.w3schools.com");
        window.location.assign(url);
        return false;
      }
    </script>
    
    <form class="d-flex"  action="/search" method="GET"
      XonSubmit="return fireSearch(this);">
      <input class="form-control me-2" type="search" placeholder="Search" aria-label="Search" name="q" id="search-query" >
      <button class="btn btn-outline-success" type="submit"
        XonCLick = "fireSearch(this.form)">Search</button>
    </form>
  </div>
</nav>



    
    
            

    <div class="container-fluid g-0 mt-0 p-0 mb-4">
        <div class="ratio" style="--bs-aspect-ratio: 28.125%;">
            <img class="object-fit-cover" src="../cover.webp"/>
        </div>
        <img src="https://www.foundry.deadlymoon.info/images/rune-banner.webp" class="rune-banner" />
    </div>


<div class="container">
    <nav id="breadcrumbs" aria-label="breadcrumb" style="--bs-breadcrumb-divider: '/';">
      <ol class="breadcrumb small">
          <li class="breadcrumb-item small"><a href="/">Home</a></li>
          
          <li class="breadcrumb-item active small " aria-current="page"><a href="/search/">Search</a></li>
      </ol>
    </nav>
</div>




  </header>
  <main>
    

<div class="container">
  <div class="row">
    <div class="col">

      <div class="content-wrapper">
          <div id="search-results"></div>
          <div class="search-loading">Searching...</div>
        
          <script id="search-result-template" type="text/x-js-template">
          <div id="summary-${key}">
              <h3><a href="${link}">${title}</a></h3>
              <p>${snippet}</p>
              <p>
                  <small>
                      ${ isset categories }Categories: ${categories}<br>${ end }
                  </small>
              </p>
          </div>
          </script>
        
          <script src="https://cdnjs.cloudflare.com/ajax/libs/fuse.js/3.2.0/fuse.min.js"></script>
          <script src="https://cdnjs.cloudflare.com/ajax/libs/mark.js/8.11.1/mark.min.js"></script>
          <script src="/search.js"></script>
      </div>
    
    
      
    </div>
  </div>
</div>







  </main>
  <footer>
    <div class="container mt-5">
    <div class="row">
        <hr class="ke-rule" />
        <div class="col text-left text-md-center">
            <p>
                
               <div class="small">Original material is Copyright &copy; 2024 by <a href="bWFpbHRvOjNseHRkMTJnQGR1Y2suY29t" data-scramble data-scramble-html="no">Claycle</a>. All rights reserved.</div>
                <div class="small">This is site is intended to document a ongoing private game for the players of that game and anyone who might be curious.</div>                
                <div class="small">Some material may be derived from or subject to Chaosium's copyright, and no claims are made over any of that material.</div>
            </p>
        </div>
    </div>
</div>




<script src="https://www.foundry.deadlymoon.info/js/bundle.min.js" defer></script>

</script>

  </footer>
  

</body>

</html>

caddy debug log:

ubuntu@ip-172-26-6-157:/var/log/caddy$ sudo cat www.foundry-access.log
{"level":"info","ts":1707997586.9689636,"logger":"http.log.access.log0","msg":"handled request","request":{"remote_ip":"3.136.224.250","remote_port":"47190","client_ip":"3.136.224.250","proto":"HTTP/2.0","method":"GET","host":"www.foundry.deadlymoon.info","uri":"/search/?q=zola","headers":{"User-Agent":["curl/7.68.0"],"Accept":["*/*"]},"tls":{"resumed":false,"version":772,"cipher_suite":4865,"proto":"h2","server_name":"www.foundry.deadlymoon.info"}},"bytes_read":0,"user_id":"","duration":0.013212755,"size":5918,"status":200,"resp_headers":{"Server":["Caddy"],"Alt-Svc":["h3=\":443\"; ma=2592000"],"Etag":["\"s8wb6w4ke\""],"Content-Type":["text/html; charset=utf-8"],"Last-Modified":["Thu, 15 Feb 2024 11:42:32 GMT"],"Accept-Ranges":["bytes"],"Content-Length":["5918"]}}

Finally, you can compare the sites, with identical deployment bases, and see how they respond differently:

Caddy Version: https://www.foundry.deadlymoon.info
Apache Version: https://www.deadlymoon.info

Using the search box on the caddy version will see the querystring stripped, but not on the apache version.

Image:

(These are not the same host)

francislavoie · February 15, 2024, 11:56am

That’s the access logs, not the debug logs.

The debug logs will show exactly what Caddy is sending to your PHP backend.

claycle · February 15, 2024, 11:58am

Where are the debug logs, then?

claycle · February 15, 2024, 11:59am

Also, there is no PHP involved. This is static.

francislavoie · February 15, 2024, 12:01pm

You have this hard-coded in your page’s JS. Are you sure that’s correct?

francislavoie · February 15, 2024, 12:02pm

As I wrote above, and as described in the help topic template, you need to add the debug global option to your config. They get written out to stdout by default (so, to the log journal).

claycle · February 15, 2024, 12:02pm

That is NOT being used. If you look at the form code, those are “commented” out. The form is just submitting directly, not going through any code.

Those are artifacts of trying to figure out what was going on (using console.log and alert to show me what was or wasn’t occuring).

claycle · February 15, 2024, 12:04pm

The config is before I added the logging directives. When command was run, I had added:

ubuntu@ip-172-26-6-157:/etc/caddy$ cat Caddyfile
# The Caddyfile is an easy way to configure your Caddy web server.
#
# Unless the file starts with a global options block, the first
# uncommented line is always the address of your site.
#
# To use your own domain name (with automatic HTTPS), first make
# sure your domain's A/AAAA DNS records are properly pointed to
# this machine's public IP, then replace ":80" below with your
# domain name.
{
        debug
}

www.foundry.deadlymoon.info {
        # Set this path to your site's directory.
        # root * /usr/share/caddy
        root * /home/ubuntu/www

        log {
                output file /var/log/caddy/www.foundry-access.log
        }
...

francislavoie · February 15, 2024, 12:07pm

Ah, I see what’s going on.

Caddy’s file server is performing a canonicalization redirect from /search to /search/ (adding the trailing slash) because search is a directory. That ends up dropping the query.

Make sure to use /search/ when building your URLs instead of /search so that you make the request to the right file from the start.

claycle · February 15, 2024, 12:12pm

Log excerpt at:

https://filedn.com/lghoT7mSAQwSJ1ej9qTGJxL/caddydebug.txt

claycle · February 15, 2024, 12:16pm

Gadzooks, that was driving me nuts yesterday. I couldn’t sleep…

Thank you.

francislavoie · February 15, 2024, 12:21pm

It’s a bug… it shouldn’t drop the query when redirecting. I’m looking to fix it. But yeah, for now just use the canonical form, with the trailing slash.

francislavoie · February 15, 2024, 12:28pm

github.com/caddyserver/caddy

fileserver: Preserve query during canonicalization redirect

caddyserver:master ← caddyserver:canon-preserve-query

opened 12:23PM - 15 Feb 24 UTC

francislavoie

+4 -0

Replaces https://github.com/caddyserver/caddy/pull/5254 Context: https://caddy.…community/t/caddy-stripping-querystring/22754 When a canonicalization redirect happens, it's only taking the path and adding a `/`, which loses the original query if there was one. This preserves it safely, without making additional modifications to the URL.