Caddy not logging request to file from cloudflared Argo Tunnel

Help
1

1. Caddy version (caddy version):

v2.3.0 h1:fnrqJLa3G5vfxcxmOH/+kJOcunPLhSBnjgIvjXV/QTA=

2. How I run Caddy:

a. System environment:

LXC Container on Proxmox
Linux caddy 5.4.106-1-pve #1 SMP PVE 5.4.106-1 (Fri, 19 Mar 2021 11:08:47 +0100) x86_64 x86_64 x86_64 GNU/Linux
Distributor ID: Ubuntu
Description: Ubuntu 20.04.2 LTS
Release: 20.04
Codename: focal

b. Command:

caddy run --config /etc/caddy/Caddyfile

c. Service/unit/compose file:

# caddy.service
#
# For using Caddy with a config file.
#
# Make sure the ExecStart and ExecReload commands are correct
# for your installation.
#
# See https://caddyserver.com/docs/install for instructions.
#
# WARNING: This service does not use the --resume flag, so if you
# use the API to make changes, they will be overwritten by the
# Caddyfile next time the service is restarted. If you intend to
# use Caddy's API to configure it, add the --resume flag to the
# `caddy run` command or use the caddy-api.service file instead.

[Unit]
Description=Caddy
Documentation=https://caddyserver.com/docs/
After=network.target network-online.target
Requires=network-online.target

[Service]
User=caddy
Group=caddy
ExecStart=/usr/bin/caddy run --environ --config /etc/caddy/Caddyfile
ExecReload=/usr/bin/caddy reload --config /etc/caddy/Caddyfile
TimeoutStopSec=5s
LimitNOFILE=1048576
LimitNPROC=512
PrivateTmp=true
ProtectSystem=full
AmbientCapabilities=CAP_NET_BIND_SERVICE

[Install]
WantedBy=multi-user.target

d. My complete Caddyfile or JSON config:

http://9cq4.domain.tld:10000 {
        root * /mnt/nas/9cq4
        file_server {
                hide Thumbs.db
        }
        log {
                output file /var/log/caddy/access-9cq4.log
                format single_field common_log
        }
}

Cloudflared config:

tunnel: x
credentials-file: x
origincert: x
logfile: /var/log/cloudflared-domain.log

ingress:
  - hostname: 9cq4.domain.tld
    service: http://127.0.0.1:10000
    originRequest:
        httpHostHeader: 9cq4.domain.tld
  - service: http_status:404

3. The problem I’m having:

Caddy isn’t logging proxied traffic from cloudflared argo tunnel to the specified file, only logs direct traffic locally e.g. curl --resolve '9cq4.domain.tld:10000:127.0.0.1' http://9cq4.domain.tld:10000/. Traffic log from cloudflared reverse proxy are in journalctl, not in the file. Only direct traffic is logged to the file.

4. Error messages and/or full log output:

2021/04/28 02:30:58.688 INFO    using provided configuration    {"config_file": "Caddyfile", "config_adapter": ""}
2021/04/28 02:30:58.690 INFO    admin   admin endpoint started  {"address": "tcp/localhost:2019", "enforce_origin": false, "origins": ["localhost:2019", "[::1]:2019", "127.0.0.1:2019"]}
2021/04/28 02:30:58.691 INFO    tls.cache.maintenance   started background certificate maintenance      {"cache": "0xc0002ea620"}
2021/04/28 02:30:58.691 INFO    autosaved config        {"file": "/root/.config/caddy/autosave.json"}
2021/04/28 02:30:58.691 INFO    serving initial configuration
2021/04/28 02:30:58.691 INFO    tls     cleaned up storage units
2021/04/28 02:32:24.314 INFO    http.log.access handled request {"request": {"remote_addr": "127.0.0.1:41728", "proto": "HTTP/1.1", "method": "GET", "host": "9cq4.domain.tld", "uri": "/", "headers": {"Accept-Encoding": ["gzip"], "Cf-Connecting-Ip": ["<redacted>"], "X-Forwarded-Proto": ["https"], "Cdn-Loop": ["cloudflare"], "Cf-Ray": ["<redacted>"], "Cf-Request-Id": ["<redacted>"], "Cf-Warp-Tag-Id": ["<redacted>"], "X-Forwarded-For": ["<redacted>"], "Cf-Visitor": ["{\"scheme\":\"https\"}"], "Connection": ["keep-alive"], "User-Agent": ["curl/7.68.0"], "Accept": ["*/*"], "Cf-Ipcountry": ["<redacted>"]}}, "common_log": "127.0.0.1 - - [28/Apr/2021:10:32:24 +0800] \"GET / HTTP/1.1\" 200 13602", "duration": 0.00257954, "size": 13602, "status": 200, "resp_headers": {"Server": ["Caddy"], "Etag": ["\"qs8510ahu\""], "Content-Type": ["text/html; charset=utf-8"], "Last-Modified": ["Tue, 27 Apr 2021 13:19:48 GMT"], "Accept-Ranges": ["bytes"], "Content-Length": ["13602"]}}
^C2021/04/28 02:32:41.047       INFO    shutting down   {"signal": "SIGINT"}
2021/04/28 02:32:41.548 INFO    tls.cache.maintenance   stopped background certificate maintenance      {"cache": "0xc0002ea620"}
2021/04/28 02:32:42.048 INFO    admin   stopped previous server
2021/04/28 02:32:42.048 INFO    shutdown done   {"signal": "SIGINT"}

I ran two request, the first one directly using the above command curl --resolve '9cq4.domain.tld:10000:127.0.0.1' http://9cq4.domain.tld:10000/, the second one requesting through cloudflare proxy curl https://9cq4.domain.tld/. I understand that I did not use the correct user agent for curl in the second request and was asked for captcha by cloudflare, but the request still went through, same with using a browser.
This is the first request, it is in the log file
127.0.0.1 - - [28/Apr/2021:10:32:05 +0800] "GET / HTTP/1.1" 200 13602

This is the log from the cloudflare traffic
2021/04/28 02:32:24.314 INFO http.log.access handled request {"request": {"remote_addr": "127.0.0.1:41728", "proto": "HTTP/1.1", "method": "GET", "host": "9cq4.domain.tld", "uri": "/", "headers": {"Accept-Encoding": ["gzip"], "Cf-Connecting-Ip": ["<redacted>"], "X-Forwarded-Proto": ["https"], "Cdn-Loop": ["cloudflare"], "Cf-Ray": ["<redacted>"], "Cf-Request-Id": ["<redacted>"], "Cf-Warp-Tag-Id": ["<redacted>"], "X-Forwarded-For": ["<redacted>"], "Cf-Visitor": ["{\"scheme\":\"https\"}"], "Connection": ["keep-alive"], "User-Agent": ["curl/7.68.0"], "Accept": ["*/*"], "Cf-Ipcountry": ["<redacted>"]}}, "common_log": "127.0.0.1 - - [28/Apr/2021:10:32:24 +0800] \"GET / HTTP/1.1\" 200 13602", "duration": 0.00257954, "size": 13602, "status": 200, "resp_headers": {"Server": ["Caddy"], "Etag": ["\"qs8510ahu\""], "Content-Type": ["text/html; charset=utf-8"], "Last-Modified": ["Tue, 27 Apr 2021 13:19:48 GMT"], "Accept-Ranges": ["bytes"], "Content-Length": ["13602"]}}

this part should be in the log file
"common_log": "127.0.0.1 - - [28/Apr/2021:10:32:24 +0800] \"GET / HTTP/1.1\" 200 13602"

So I did another request, one without the hostname directly to localhost:10000
curl http://127.0.0.1:10000/
It ends up in journalctl too, not the log file, so I think there is a wrong configuration between these two that separates the log

5. What I already tried:

Previously, this does not exist in the cloudflared configuration file. I added it but there is no difference.

    originRequest:
        httpHostHeader: 9cq4.domain.tld

I also tried removing the hostname in caddy file so it would be like this

http://:10000 {

No difference, same result with the same test

6. Links to relevant resources:

https://developers.cloudflare.com/cloudflare-one/connections/connect-apps/configuration/ingress

2

You can turn on debug mode to find out more about what Caddy’s doing. Add this to the top of your Caddyfile:

{
	debug
}

I’ve not played around with Argo so I’m not so sure what to suggest. Maybe you could intercept the TCP traffic with something like Wireshark and inspect what it looks like before it reaches Caddy? See what’s different between curl and argo.

3

I found the problem, it is in the Host header as I previously thought. I think it is my misunderstanding that Host does not contain port when it actually does.
I did a diff between the two requests JSON log and it turns out in that the request from curl, the Host header contains the port "host": "9cq4.domain.tld:10000", while the request from cloudflared doesn’t "host": "9cq4.domain.tld". That is why the other hosts works with cloudflared, because they use standard port which does not require it to be explicitly specified.
This is what I changed in the cloudflared file:

    originRequest:
---        httpHostHeader: 9cq4.domain.tld
+++        httpHostHeader: 9cq4.domain.tld:10000

It works now! Thanks.

1 Like
4

This topic was automatically closed after 30 days. New replies are no longer allowed.