Caddy2 reverse proxy for Harbor: docker push EOF

[SakuraYuyuko]

1. Caddy version (caddy version):

2.1.1

2. How I run Caddy:

Using docker, version 19.03.8, build afacb8b
Using Docker image: caddy:2.1.1

a. System environment:

System: Centos7
Docker version 19.03.8, build afacb8b
Using Docker image: caddy:2.1.1

b. Command:

docker-compose up -d

c. Service/unit/compose file:

version: "2"
networks:
  portainer_default:
    external:
      name: portainer_default

services:
  caddy-entry:
    image: caddy:2.1.1
    #environment:
    restart: always
    networks:
      - portainer_default
    hostname: caddy-entry
    domainname: docker.network
    container_name: caddy-entry
    volumes:
      - "/data/caddy/config:/etc/caddy"
      - "/data/nginx/config/ssl:/etc/caddy/ssl"
      - "/data/nginx/wwwroot:/wwwroot"
      - "/data/caddy/logs:/var/logs/caddy"
    ports:
      - "80:80"
      - "443:443"

d. My complete Caddyfile or JSON config:

:443 {
    encode zstd gzip
    tls /etc/caddy/ssl/*.meikai.network/fullchain.cer /etc/caddy/ssl/*.meikai.network/*.meikai.network.key
}
docker-registry.uuzdream.cn {
    reverse_proxy * {
        to http://10.198.12.12:4000
        header_up X-Forwarded-Proto "https"
        flush_interval -1
        transport http {
            write_buffer 0
            read_buffer 0
        }
    }
}

3. The problem I’m having:

I use caddy for Harbor(docker image registry). I tried to push docker image in to my registry but the docker said: Retrying in 4 seconds then EOF

4. Error messages and/or full log output:

The caddy didn’t say anything
The docker deamon said:

level=error msg="Upload failed, retrying: EOF"
level=error msg="Upload failed, retrying: EOF"
level=error msg="Upload failed, retrying: EOF"
level=error msg="Not continuing with push after error: context canceled"

Harbor didn’t say anything.

5. What I already tried:

I use nginx before and I try to set

header_up X-Forwarded-Proto "https"
header_up X-Forwarded-For {host}
header_up Host {host}

websocket(caddy v1)
header_up Connection "upgrade"

but it didn’t work.

6. Links to relevant resources:

Caddy V2 how to proxy websoket (v2ray websocket + tls)

Do you need a websocket, or do you need an unix socket? There is an example of an unix socket given here: Conventions — Caddy Documentation As for web sockets, the documentation for matchers shows an entire websocket proxy: Caddyfile Concepts — Caddy Documentation

Docker registry behind caddy as reverse proxy

Hey, I’m running a docker registry inside a docker container behind caddy. Pushing and pulling works just fine, but building with drone fails because of some authentication stuff. 172.17.0.21 - [19/Jul/2016:22:10:11 +0000] "GET /v2/ HTTP/1.1" 401 17 172.17.0.21 - [19/Jul/2016:22:10:11 +0000] "GET /v1/_ping HTTP/1.1" 401 17 172.17.0.21 - [19/Jul/2016:22:10:11 +0000] "POST /v1/users/ HTTP/1.1" 401 17 172.17.0.21 - [19/Jul/2016:22:10:11 +0000] "GET /v1/users/ HTTP/1.1" 404 14 Now there’s an offi…

(other I forgot sorry)

[francislavoie]

Hmm. Does the docker registry communication happen over TCP rather than HTTP? If that’s the case then using this approach to proxy wouldn’t work. Just a guess.

What are in your Caddy logs? You omitted those. You can see them with docker-compose logs caddy-entry

[SakuraYuyuko]

This is the log when I was pushing the image

(When I enable the debug mode…)

(Sorry about the image is not very clear. You can open these image in a new tab to get a clearer view)

[SakuraYuyuko]

I tried to add

header_up Host {http.reverse_proxy.upstream.hostport}

and it works!
But it seem like caddy doesn’t let the http basic auth info pass to the backend. (I already login in to my registry by using the command docker login docker-registry.uuzdream.cn)

The push refers to repository [docker-registry.uuzdream.cn/library/gradle]
87e25f9e8a42: Pushing [==================================================>]    116MB/116MB
2de5ae7bc7e4: Pushing [>                                                  ]  533.5kB/141.4MB
f2411f32c29f: Pushing [==================================================>]  413.2kB
ab222698d644: Pushing [==================================================>]  109.7MB/109.7MB
60d7a9dafbc1: Pushing [==================================================>]  35.68MB/35.68MB
8682f9a74649: Waiting
d3a6da143c91: Waiting
83f4287e1f04: Waiting
7ef368776582: Waiting
unauthorized: unauthorized to access repository: library/gradle, action: push: unauthorized to access repository: library/gradle, action: push

[francislavoie]

@SakuraYuyuko

and it works!

Great!

@SakuraYuyuko

But it seem like caddy doesn’t let the http basic auth info pass to the backend.

Hmm, it looks like it does based on the debug logs you screenshotted above, it shows the Authorization header is passed through.

The error message there looks like a permissions issue. Are you sure your user is correctly configured to have access to that repo?

[SakuraYuyuko]

Yes, I’m sure that I have correct permission of registry.
I tested this by pull an image from a private project.

The push log:

[SakuraYuyuko]

Update config:

docker-registry.uuzdream.cn {
    reverse_proxy * http://10.198.12.12:4000 {
        header_up Host {http.reverse_proxy.upstream.hostport}
        header_up X-Real-IP {http.request.remote}
        header_up X-Forwarded-For {http.request.remote}
        header_up X-Forwarded-Port {http.request.port}
        header_up X-Forwarded-Proto {http.request.scheme}
    }
}

[francislavoie]

@SakuraYuyuko

header_up X-Forwarded-For {http.request.remote}
header_up X-Forwarded-Proto {http.request.scheme}

These two should not be necessary, Caddy will set those by itself.

Other than that… I’m at a loss

I went digging through some documentation for docker registry and some sample nginx configs to try and compare and I’m not noticing anything that looks relevant here.

[SakuraYuyuko]

Thanks for your hard working :). I went digging into different docs, github issues, too.
I try to compare caddy and nginx(works for me) config file.

location / {
		client_max_body_size 2000m;
		proxy_buffering off;
		proxy_ssl_verify off;
		proxy_set_header X-Forwarded-Proto $scheme;
		proxy_request_buffering off;
		proxy_send_timeout 900;
	    proxy_read_timeout 900;
		proxy_pass http://10.198.12.12:4000;
}

(This nginx config works)

docker-registry.uuzdream.cn {
    reverse_proxy * http://10.198.12.12:4000 {
        header_up Host {http.reverse_proxy.upstream.hostport}
        header_up X-Real-IP {http.request.remote}
        header_up X-Forwarded-Port {http.request.port}
        flush_interval -1
        transport http {
            read_buffer 0
            write_buffer 0
        }
    }
}

(Caddy config / not working)

and I found some differences in the progress bar of docker push command between nginx and caddy:

• nginx: the progress bar is like: 10% → 20% → 40%
• caddy: the progress bar just rush! 0% → 100% and unauthorized: unauthorized to access repository

I also found something in the harbor-registry.log

Aug  6 15:23:02 172.29.0.1 registry[2146]: time="2020-08-06T07:23:02.303921124Z" level=error msg="response completed with error" auth.user.name="harbor_registry_user" err.code="blob unknown" err.detail=sha256:7595c8c21622ea8a8b9778972e26dbbe063f7a1c4b0a28a80a34ebb3d343b586 err.message="blob unknown to registry" go.version=go1.14.5 http.request.host="10.198.12.12:4000" http.request.id=53a4e652-1b14-472f-9e05-5846995a1c01 http.request.method=HEAD http.request.remoteaddr=10.198.12.12 http.request.uri="/v2/ci-env/gradle/blobs/sha256:7595c8c21622ea8a8b9778972e26dbbe063f7a1c4b0a28a80a34ebb3d343b586" http.request.useragent="docker/19.03.8 go/go1.12.17 git-commit/afacb8b kernel/5.5.13-1.el7.elrepo.x86_64 os/linux arch/amd64 UpstreamClient(Docker-Client/19.03.8 \(linux\))" http.response.contenttype="application/json; charset=utf-8" http.response.duration=11.835536ms http.response.status=404 http.response.written=157 vars.digest="sha256:7595c8c21622ea8a8b9778972e26dbbe063f7a1c4b0a28a80a34ebb3d343b586" vars.name="ci-env/gradle"

[francislavoie]

@SakuraYuyuko

proxy_ssl_verify off;

Hmm this one seems peculiar in the nginx config. Since the proxy is over http:// that shouldn’t be necessary, right?

I did some digging around about blob unknown to registry and it seemed to point to issues with the X-Forwarded-Proto header (needing to be https) but we can see that that’s correctly being sent in your debug logs.

[SakuraYuyuko]

Yeah… I think so… This problem is so strange