Caddy 2.6.x stop WebDAV client to finish uploading

Help
1

First, my public IP and FQDN are all redacted for security reasons and both are not the problem.

I use a banking software (Banking4i) that backup it’s data file to my WebDAV (SFTPgo) server. Caddy is used as a Reverse Proxy. Everything is a common dockerized setup:
Client Software (iOS or macOS) → Caddy Reverse Proxy (TLS) (Docker) → WebDAV Server (no TLS) (Docker)

  • Caddy v2.5.2 and earlier work as expected.
  • Caddy v2.6.0-beta.3 crashes Caddy
  • Caddy v2.6.0-beta.5 and later breaks WebDAV client to finish uploading

I use an own Caddy Docker image with some modules, but behavior is the same with your pure Caddy Docker image from hub.docker.com

Debug output of Caddy 2.5.2:

{
    "level": "debug",
    "ts": 1670177114.3610454,
    "logger": "http.handlers.reverse_proxy",
    "msg": "upstream roundtrip",
    "upstream": "192.168.1.3:6061",
    "duration": 0.014817828,
    "request": {
        "remote_ip": "2.2.2.2",
        "remote_port": "55725",
        "proto": "HTTP/1.1",
        "method": "PUT",
        "host": "webdav.example.com",
        "uri": "/banking4i/Meine%20Banken.sub",
        "headers": {
            "X-Forwarded-Host": [
                "webdav.example.com"
            ],
            "X-Forwarded-Port": [
                "443"
            ],
            "Expect": [
                "100-continue"
            ],
            "Authorization": [],
            "Content-Type": [
                "application/octet-stream"
            ],
            "X-Real-Ip": [
                "2.2.2.2"
            ],
            "Content-Length": [
                "196608"
            ],
            "X-Forwarded-For": [
                "2.2.2.2"
            ],
            "X-Forwarded-Proto": [
                "https"
            ],
            "User-Agent": [
                "Mozilla/5.0 (Subsembly.Interweb)"
            ]
        },
        "tls": {
            "resumed": false,
            "version": 771,
            "cipher_suite": 49195,
            "proto": "",
            "server_name": "webdav.example.com"
        }
    },
    "headers": {
        "Etag": [
            "\"172da93c8d59d66430000\""
        ],
        "Date": [
            "Sun, 04 Dec 2022 18:05:14 GMT"
        ],
        "Content-Length": [
            "7"
        ],
        "Content-Type": [
            "text/plain; charset=utf-8"
        ]
    },
    "status": 201
}

Debug output of Caddy 2.6.0-beta.3:

{"level":"debug","ts":1670177345.0879314,"logger":"http.handlers.reverse_proxy","msg":"selected upstream","dial":"192.168.1.3:6061","total_upstreams":1}
panic: invalid WriteHeader code 0
goroutine 188 [running]:
net/http.checkWriteHeaderCode(...)
	net/http/server.go:1116
net/http.(*response).WriteHeader(0xc00067c2a0, 0x0)
	net/http/server.go:1150 +0x9db
github.com/caddyserver/caddy/v2/modules/caddyhttp.(*responseRecorder).WriteHeader(0xc0006f9650?, 0x0?)
	github.com/caddyserver/caddy/v2@v2.6.0-beta.3/modules/caddyhttp/responsewriter.go:178 +0xbd
github.com/caddyserver/caddy/v2/modules/caddyhttp.(*responseRecorder).WriteHeader(0xc0006f97a0?, 0x0?)
	github.com/caddyserver/caddy/v2@v2.6.0-beta.3/modules/caddyhttp/responsewriter.go:178 +0xbd
github.com/caddyserver/caddy/v2/modules/caddyhttp.(*responseRecorder).WriteHeader(0xc0009d4d08?, 0xc0009d4cb0?)
	github.com/caddyserver/caddy/v2@v2.6.0-beta.3/modules/caddyhttp/responsewriter.go:178 +0xbd
github.com/caddyserver/caddy/v2/modules/caddyhttp/reverseproxy.(*Handler).reverseProxy.func1(0xc00061b920?, 0xc0005bc900?)
	github.com/caddyserver/caddy/v2@v2.6.0-beta.3/modules/caddyhttp/reverseproxy/reverseproxy.go:783 +0x5d
net/http.(*persistConn).readResponse(0xc000995680, {{}, 0xc0005bc900, {0xc0005bc800}, 0xc000258c00, 0x1, 0xc000258b40, 0xc000258ba0}, 0xc000637580)
	net/http/transport.go:2313 +0x20c
net/http.(*persistConn).readLoop(0xc000995680)
	net/http/transport.go:2108 +0x3aa
created by net/http.(*Transport).dialConn
	net/http/transport.go:1751 +0x173e

Debug output of Caddy 2.6.0-beta.5 and later (2.6.2 and also :latest Docker image from today):

{
    "level": "debug",
    "ts": 1670177188.2818708,
    "logger": "http.handlers.reverse_proxy",
    "msg": "upstream roundtrip",
    "upstream": "192.168.1.3:6061",
    "duration": 0.020302583,
    "request": {
        "remote_ip": "2.2.2.2",
        "remote_port": "55749",
        "proto": "HTTP/1.1",
        "method": "PUT",
        "host": "webdav.example.com",
        "uri": "/banking4i/Meine%20Banken.sub",
        "headers": {
            "X-Forwarded-For": [
                "2.2.2.2"
            ],
            "X-Forwarded-Host": [
                "webdav.example.com"
            ],
            "User-Agent": [
                "Mozilla/5.0 (Subsembly.Interweb)"
            ],
            "Expect": [
                "100-continue"
            ],
            "Authorization": [],
            "Content-Type": [
                "application/octet-stream"
            ],
            "X-Real-Ip": [
                "2.2.2.2"
            ],
            "Content-Length": [
                "196608"
            ],
            "X-Forwarded-Proto": [
                "https"
            ],
            "X-Forwarded-Port": [
                "443"
            ]
        },
        "tls": {
            "resumed": false,
            "version": 771,
            "cipher_suite": 49195,
            "proto": "",
            "server_name": "webdav.example.com"
        }
    },
    "headers": {
        "Date": [
            "Sun, 04 Dec 2022 18:06:28 GMT"
        ],
        "Content-Length": [
            "7"
        ],
        "Content-Type": [
            "text/plain; charset=utf-8"
        ],
        "Etag": [
            "\"172da94dc2fec6e530000\""
        ]
    },
    "status": 201
}

My own Caddy image includes module caddy-trace.

Output of caddy-trace Caddy 2.5.2:

{
    "level": "debug",
    "time": "2022-11-29T12:36:46.185Z",
    "msg": "debugging request",
    "request_id": "7cee08d6-e386-45a8-92f0-2e7024b8262c",
    "direction": "incoming",
    "tag": "",
    "method": "PUT",
    "proto": "HTTP/1.1",
    "host": "webdav.example.com",
    "uri": "/banking4i/Meine%20Banken.sub",
    "remote_addr_port": "2.2.2.2:55350",
    "remote_addr": "2.2.2.2",
    "remote_port": 55350,
    "content_length": 4520448,
    "cookie_count": 0,
    "user_agent": "Mozilla/5.0 (Subsembly.Interweb)",
    "referer": "",
    "cookies": [],
    "query_params": {},
    "headers": {
        "Authorization": "Basic xx ",
        "Content-Length": "4520448",
        "Content-Type": "application/octet-stream",
        "Expect": "100-continue",
        "User-Agent": "Mozilla/5.0 (Subsembly.Interweb)"
    },
    "form": {}
}
{
    "level": "debug",
    "time": "2022-11-29T12:36:46.315Z",
    "msg": "debugging response",
    "request_id": "7cee08d6-e386-45a8-92f0-2e7024b8262c",
    "direction": "outgoing",
    "tag": "",
    "status_code": 201,
    "response_size": 7,
    "buffer_size": 7,
    "response_headers": {
        "Content-Length": [
            "7"
        ],
        "Content-Type": [
            "text/plain; charset=utf-8"
        ],
        "Date": [
            "Tue, 29 Nov 2022 12:36:46 GMT"
        ],
        "Etag": [
            "\"172c0e6913785c5944fa00\""
        ],
        "Server": [
            "Caddy"
        ],
        "Strict-Transport-Security": [
            "max-age=63072000; includeSubDomains; preload"
        ]
    }
}

Output of caddy-trace Caddy 2.6.2:

{
    "level": "debug",
    "time": "2022-11-29T12:38:26.518Z",
    "msg": "debugging request",
    "request_id": "3498c983-b106-440d-a5e9-8169b8647311",
    "direction": "incoming",
    "tag": "",
    "method": "PUT",
    "proto": "HTTP/1.1",
    "host": "webdav.example.com",
    "uri": "/banking4i/Meine%20Banken.sub",
    "remote_addr_port": "2.2.2.2:55714",
    "remote_addr": "2.2.2.2",
    "remote_port": 55714,
    "content_length": 4520448,
    "cookie_count": 0,
    "user_agent": "Mozilla/5.0 (Subsembly.Interweb)",
    "referer": "",
    "cookies": [],
    "query_params": {},
    "headers": {
        "Authorization": "Basic xx ",
        "Content-Length": "4520448",
        "Content-Type": "application/octet-stream",
        "Expect": "100-continue",
        "User-Agent": "Mozilla/5.0 (Subsembly.Interweb)"
    },
    "form": {}
}
{
    "level": "debug",
    "time": "2022-11-29T12:38:26.650Z",
    "msg": "debugging response",
    "request_id": "3498c983-b106-440d-a5e9-8169b8647311",
    "direction": "outgoing",
    "tag": "",
    "status_code": 201,
    "response_size": 7,
    "buffer_size": 7,
    "response_headers": {
        "Alt-Svc": [
            "h3=\":443\"; ma=2592000"
        ],
        "Content-Length": [
            "7"
        ],
        "Content-Type": [
            "text/plain; charset=utf-8"
        ],
        "Date": [
            "Tue, 29 Nov 2022 12:38:26 GMT"
        ],
        "Etag": [
            "\"172c0e806fcbcbc544fa00\""
        ],
        "Strict-Transport-Security": [
            "max-age=63072000; includeSubDomains; preload"
        ]
    }
}

2.2.2.2 is placeholder for my Internet public IP address.

I can’t find anything what could be wrong from all this debug output, but it feels there is an incompatibility (or bug?) introduced after Caddy 2.6.0

Client Software is Banking4:
https://subsembly.com/banking4.html

It doesn’t matter if I use iOS or macOS version. With Caddy 2.5.2 both platform upload their backup files, after Caddy 2.6.0 upload hangs. Other WebDAV software works, also download from WebDAV always works with Banking4!
With macOS I have tried different Banking4 versions, behave all the same: with Caddy 2.5.2 it works, after 2.6.0 the upload dialog window of Banking4 hangs and only closes after reloading (not stopping) Caddy.

2

Please post your config, and how you build and run Caddy, as requested by the help topic template. The template is required to be filled, as per the forum rules. You can find it by clicking on New Topic, choosing the Help category. You can copy it into a new reply to this topic.

Have you tried with Caddy v2.6.2, the latest version? I’m not clear on that from your post.

It’s hard to say what’s going wrong without more details of your setup.

3

Hello @francislavoie

thanks your your answer, I have tried all Caddy 2.6.x versions, up to 2.6.2 and also 2.6:latest, but looks like it’s 2.6.2

I’ve only tried Caddy 2.6 beta versions to narrow down when the problem starts, but then see the crash in 2.6 beta 3.

Here is the filled template:

1. Output of caddy version:

v2.5.2 h1:eCJdLyEyAGzuQTa5Mh3gETnYWDClo1LjtQm2q9RNZrs=
v2.6.2 h1:wKoFIxpmOJLGl3QXoo6PNbYvGW4xLEgo32GPBEjWL8o=

2. How I run Caddy:

a. System environment:

Latest Ubuntu 22.04 with Docker 20.10.21. Caddy runs in Docker Container.

b. Command:

caddy run --config /etc/caddy/Caddyfile --adapter caddyfile

c. Service/unit/compose file:

  caddy:
    container_name: caddy
    image: caddy-proxy:2.5.2
    #image: caddy-proxy:2.6.2
    restart: always 
    env_file:
      - caddy.env
    volumes:
      - /etc/localtime:/etc/localtime:ro
      - /etc/timezone:/etc/timezone:ro
      - /docker/conf/caddy:/etc/caddy
      - /docker/data/caddy/config:/config
      - /docker/data/caddy/data:/data
      - /docker/data/letsencrypt:/letsencrypt
    networks:
      - caddy 
    ports:
      - 80:80/tcp
      - 443:443/tcp
      - 443:443/udp
      - 2019:2019/tcp
      - 8448:8448/tcp
      - 8448:8448/udp

d. My complete Caddy config:

# Caddyfile

{
    servers :443
    servers :8448
    #debug
    admin :2019
}
import snippets/*.conf
import sites/*.conf

# wildcard.conf

(wildcard) {
    tls /letsencrypt/certificates/_.example.com.crt /letsencrypt/certificates/_.example.com.key
}

# client_headers.conf

(client_headers) {
    header Strict-Transport-Security "max-age=63072000; includeSubDomains; preload"
    header -Server
}

# backend_headers.conf

(backend_headers) {
    header_up X-Real-IP {remote_host}
    header_up X-Forwarded-Port {args.0}
}

# webdav.conf

webdav.example.com:443 {
    import wildcard
    import client_headers

    reverse_proxy http://192.168.1.3:6061 {
        import backend_headers 443
    }
}

3. The problem I’m having:

See my first post.

4. Error messages and/or full log output:

See my first post.

5. What I already tried:

See my first post.

6. Links to relevant resources:

See my first post.

4

How are you building this image? I want to see the command you used to build Caddy, and which Go version you’re using.

You can open an issue on Github so we can keep track of this.

5

This is how I build it:

FROM caddy:2.6.2-builder-alpine AS builder

RUN xcaddy build \
    --with github.com/greenpau/caddy-trace \
    --with github.com/ueffel/caddy-basic-auth-filter \
    --with github.com/lucaslorentz/caddy-docker-proxy/v2 \
    --with github.com/caddyserver/transform-encoder

FROM caddy:2.6.2-alpine
COPY --from=builder /usr/bin/caddy /usr/bin/caddy

But this doesn’t matter. In my first post I wrote it also happens with “pure” or “original” Caddy image from Dockerhub:

 I use an own Caddy Docker image with some modules, **but behavior is the same** with your pure Caddy Docker image from [hub.docker.com](http://hub.docker.com)

To understand it better:

  • caddy 2.5.2 from dockerhub: works
  • caddy 2.5.2 self build (see my Dockerfile): works
  • caddy 2.6.2 from dockerhub: doesn’t work
  • caddy 2.6.2 self build (see my Dockerfile): doesn’t work
  • caddy 2.6.0 beta 3 from dockerhub: crash
  • caddy 2.6.0 beta 5 from dockerhub: doesn’t work

2.6.0 beta was only used for testing and my hope, that my problem doesn’t exist with an early 2.6.x version.

You can open an issue on Github so we can keep track of this.

Can I reference to this thread?

6

How do you add webdav server to banking4? I’m trying to reproduce it on mac. Is it not available in demo version?

1 Like
7

Thank you for this question! Now I’m really completely confused. To try without a license I used my 3rd very old Mac, where a very old Banking4 version was still installed. With this Banking4 version it uploads to Caddy 2.6.2 without any problems!

Now my current state is completely bad for me: There is the newest Banking4 version, which can’t upload when Caddy 2.6.2 runs, but works with Caddy 2.5.2 and there is an old version of Banking4 from 2020-12, which works with Caddy 2.5.2 and 2.6.2.

OMG … now I’m lost. Now you say it’s Subsemblys fault and Subsembly told me (I pinged them first) I should ask Webserver vendor…

Banking4 7.5.2 (2020-12) with Caddy 2.5.2: works
Banking4 7.5.2 (2020-12) with Caddy 2.6.2: works

Banking4 7.9.1 (2022-09) with Caddy 2.5.2: works
Banking4 7.9.1 (2022-09) with Caddy 2.6.2: upload window hangs

Sorry. To answer you question: you can create an offline account with the demo version (30 day trial), close the account window and then choose in the menubar “webdav” as an upload destination.

1 Like
8

Can you provide some screenshots, I added an offline account and some records in it. There is no webdav menu anywhere.

1 Like
9

I have tested with an unregistered 30d trial version from scratch. WebDAV setup was possible after creating a container, some accounts, then closing the “app” window and open the “cloud” menu while the “sync/login” window is again visible:

Bildschirm­foto 2022-12-08 um 07.18.08
Bildschirm­foto 2022-12-08 um 07.18.081156×458 197 KB

10

image
image2274×706 84.6 KB

I added some random records and then clicked Cloud Upload. It says file in cloud up-to-date.

Edit: never mind, my caddy server is behind cloudflare.

1 Like
11

Thanks for testing! What WebDAV Server do you use? I use SFTPgo:

For me, the upload process window never closes with this combination:
Banking4 7.9.1 (2022-09) with Caddy 2.6.2: upload window hangs

I need to reload Caddy first.

never mind, my caddy server is behind cloudflare.

I use Caddy as a Reverse Proxy (not Cloudflare) and SFTPgo as a WebDAV server:

Client Apps (eg. Banking4, Enpass) — Internet —> Caddy (Reverse Proxy) —> SFTPgo

12

I’m using sftpgo as you do, the problem exists.

13

What do you think how to debug this problem properly to find the cause? I feel lost, I don’t find some more debug options and I’m not a developer. Also cause can be Banking4 because an older version of Banking4 works with Caddy 2.6.2

14

It’s related to 103 early hints. Golang http.Server already supports 100 expect continue. But commit introduced write 1xx status to client.

Gotta look up how rfc says about multiple 100 expect continue.

1 Like
15

Thanks for this finding. How did you found it from the logs? This is one of my main guess after reading several times the changelogs, but don’t found anything to disable it.

16

By reading git commits after 2.5.2 release, it can’t be disabled from configuration I’m afraid. You can try if caddy compiles with golang 1.18, that version 1xx response code won’t be written to client.

17

I believe our 1xx handling is correct, but it’s new so maybe there’s a bug. I dunno. Is it something about this client that can’t handle 1xx responses?

18

Is it something about this client that can’t handle 1xx responses?

I’m just the user of this App, but in touch with their support. I’ve forwarded them the link to this thread.

19

It’s about client receiving 100 status twice. Once from stdlib, another from reverse proxy.

20

FYI @dunglas just so you’re in the loop on this (possible bug with 1xx status code handling)