502 error on get request to external HTTPS domains

Help
1

1. Caddy version (caddy version):

v2.5.1

2. How I run Caddy:

On a Hetzner cloud server.

a. System environment:

Hetzner cloud server (CPX11)
Ubuntu 20.04.4 LTS
Project stack: React, MongoDB, Express, Node.js
Domain from namecheap

b. Command:

cd ~/projects/dgg-emote-profiles/client
caddy start Caddyfile

c. Service/unit/compose file:

n/a

d. My complete Caddyfile or JSON config:

www.zelig.dev {

    handle /* {
        root * /root/projects/dgg-emote-profiles/client/build
        file_server
        try_files {path} /index.html
    }

    handle /dggers {
        reverse_proxy localhost:5000
    }

    log {
        level debug
        output file /root/projects/dgg-emote-profiles/client/caddy_test.log
    }
}

3. The problem I’m having:

I am trying to make a get request to https://www.random.org/integers/?num=1&min=1&max=10&col=1&base=10&format=plain&rnd=new just to test to see if I can even get a get request to work from the server-side.

4. Error messages and/or full log output:

Error from browser:

{
    "message": "Request failed with status code 502",
    "name": "AxiosError",
    "config": {
        "transitional": {
            "silentJSONParsing": true,
            "forcedJSONParsing": true,
            "clarifyTimeoutError": false
        },
        "transformRequest": [
            null
        ],
        "transformResponse": [
            null
        ],
        "timeout": 0,
        "xsrfCookieName": "XSRF-TOKEN",
        "xsrfHeaderName": "X-XSRF-TOKEN",
        "maxContentLength": -1,
        "maxBodyLength": -1,
        "env": {
            "FormData": null
        },
        "headers": {
            "Accept": "application/json, text/plain, */*",
            "Content-Type": "application/json"
        },
        "method": "post",
        "url": "https://www.zelig.dev/dggers",
        "data": "{\"username\":\"sdfasdf\"}"
    },
    "code": "ERR_BAD_RESPONSE",
    "status": 502
}

On start:

2022/06/08 22:49:32.404 INFO    using adjacent Caddyfile
2022/06/08 22:49:32.405 WARN    Caddyfile input is not formatted; run the 'caddy fmt' command to fix inconsistencies    {"adapter": "caddyfile", "file": "Caddyfile", "line": 2}
2022/06/08 22:49:32.405 INFO    admin   admin endpoint started  {"address": "tcp/localhost:2019", "enforce_origin": false, "origins": ["//localhost:2019", "//[::1]:2019", "//127.0.0.1:2019"]}
2022/06/08 22:49:32.406 INFO    http    server is listening only on the HTTPS port but has no TLS connection policies; adding one to enable TLS {"server_name": "srv0", "https_port": 443}
2022/06/08 22:49:32.406 INFO    http    enabling automatic HTTP->HTTPS redirects        {"server_name": "srv0"}
2022/06/08 22:49:32.406 INFO    tls.cache.maintenance   started background certificate maintenance      {"cache": "0xc0002613b0"}
2022/06/08 22:49:32.406 INFO    http    enabling automatic TLS certificate management   {"domains": ["www.zelig.dev"]}
2022/06/08 22:49:32.407 INFO    autosaved config (load with --resume flag)      {"file": "/root/.config/caddy/autosave.json"}
2022/06/08 22:49:32.407 INFO    serving initial configuration
2022/06/08 22:49:32.406 INFO    tls     cleaning storage unit   {"description": "FileStorage:/root/.local/share/caddy"}
2022/06/08 22:49:32.407 INFO    tls     finished cleaning storage units
Successfully started Caddy (pid=237605) - Caddy is running in the background

Making the get request:

2022/06/08 22:45:06.607     ERROR   http.log.error.log0     dial tcp [::1]:5000: connect: connection refused      {"request": {"remote_ip": "72.48.226.138", "remote_port": "43462", "proto": "HTTP/2.0", "method": "POST", "host": "www.zelig.dev", "uri": "/dggers", "headers": {"Sec-Fetch-Site": ["same-origin"], "Sec-Fetch-Dest": ["empty"], "Sec-Ch-Ua": ["\" Not A;Brand\";v=\"99\", \"Chromium\";v=\"101\", \"Google Chrome\";v=\"101\""], "Referer": ["https://www.zelig.dev/"], "Content-Length": ["21"], "Sec-Ch-Ua-Mobile": ["?0"], "Accept-Encoding": ["gzip, deflate, br"], "Sec-Fetch-Mode": ["cors"], "Accept-Language": ["en-US,en;q=0.9"], "Accept": ["application/json, text/plain, */*"], "Content-Type": ["application/json"], "User-Agent": ["Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/101.0.4951.64 Safari/537.36"], "Sec-Ch-Ua-Platform": ["\"Linux\""], "Origin": ["https://www.zelig.dev"]}, "tls": {"resumed": true, "version": 772, "cipher_suite": 4865, "proto": "h2", "server_name": "www.zelig.dev"}}, "duration": 0.000661579, "status": 502, "err_id": "p3qyqkpei", "err_trace": "reverseproxy.statusError (reverseproxy.go:1196)"}

5. What I already tried:

I tried searching around for other users making requests to HTTPS on this forum but were only able to find users who wanted to make a request to internal domains that they control. I tried to see if it was an issue with my server, but I was able to make the get request with no issue in a separate project without Caddy. I tried making the get request from the client side but I got a CORS error, and was advised that I should be making these requests from the server-side instead.

2

“Connection refused” means that Caddy couldn’t connect to your server at localhost:5000. So make sure it’s running and accessible to Caddy.

3

I’m able to connect to my server fine without the GET request to the external website inserted into my back-end. It isn’t until I add that request to the external website and request /dggers that everything crashes, and then if I try to make another request to /dggers, it refuses to connect.

4

For some more context, getDggers() is the get request associated with /dggers, and createDgger() is the post request when you insert a new user at /dggers.

import DggerEmoteList from "../models/dggerEmoteList.js";
import http from 'http';

export const getDggers = async (req, res) => {
    try {
        const dggerEmotes = await DggerEmoteList.find();
        return res.status(200).json(dggerEmotes);
    } catch (error) {
        return res.status(404).json({ message: error.message });
    }
}

export const createDgger = async (req, res) => {
    var dgger = req.body;
    const newDgger = new DggerEmoteList(dgger)

    var http = require('http');

    var options = {
          host: 'www.random.org',
          path: '/integers/?num=1&min=1&max=10&col=1&base=10&format=plain&rnd=new'
    };

    callback = function(response) {
          var str = '';

      response.on('data', function (chunk) {
              str += chunk;
            });
    response.on('end', function () {
          console.log(str);
        });
    }

    http.request(options, callback).end();

    try {
        await newDgger.save();
        return res.status(201).json(newDgger);
    } catch (error) {
        return res.status(409).json({message: error.message()});
    }
}

Here is index.js from the server-side

import express from 'express';
import bodyParser from 'body-parser';
import mongoose from 'mongoose';
import cors from 'cors';
import dotenv from 'dotenv';

import postRoutes from './routes/dggers.js';

const app = express();
dotenv.config();

app.use(bodyParser.json({ limit: "30mb", extended: true }))
app.use(bodyParser.urlencoded({ limit: "30mb", extended: true }))
app.use(cors());

// Express middleware
app.use('/dggers', postRoutes);

const PORT = process.env.PORT || 5000;



// Connect to database
mongoose.connect(process.env.CONNECTION_URL).then(
    () => app.listen(PORT, () => console.log(`Server running on port ${PORT}`)),
    err => console.log(`${err} did not connect`)
);
5

Since you’re running on Ubuntu, I strongly recommend you run Caddy as a systemd service for reliability, instead of using caddy start.

Keep in mind that path matching in Caddy is exact, so this won’t match requests like /dggers/foo for example.

What do you mean by “everything crashes”?

This doesn’t sound like a problem with Caddy.

6

Well, I feel like a moron, but I found out what my issue was somewhat, although I still don’t fully understand some of the stuff going on regarding when I need to fully restart my backend (but this is just my lack of knowledge regarding how Nodejs works I think, not Caddy related). I’ll write this for some more context to help other noobs like me perhaps get a better direction if they run into this, but hopefully some of you can also point me in the right direction towards some other resources.

fuser -k 5000/tcp
cd backend
yarn start

Yep… that’s all I needed to do in order to get my placeholder get request to other websites working. I thought that for some reason that starting and stopping of my backend was handled by my caddyfile, but no, as I mentioned, I only setup my caddyfile to handle some static files and reverse proxy to my backend… I still need to restart my backend when doing certain changes.

Alas, not an issue with Caddy. You all are great, thanks for the project :smiley: .

2 Likes
7

Great - thanks for following up with your solution!

8

This topic was automatically closed after 30 days. New replies are no longer allowed.