Caddy2 reverse proxy for Harbor: docker push EOF

1. Caddy version (caddy version):


2. How I run Caddy:

Using docker, version 19.03.8, build afacb8b
Using Docker image: caddy:2.1.1

a. System environment:

System: Centos7
Docker version 19.03.8, build afacb8b
Using Docker image: caddy:2.1.1

b. Command:

docker-compose up -d

c. Service/unit/compose file:

version: "2"
      name: portainer_default

    image: caddy:2.1.1
    restart: always
      - portainer_default
    hostname: caddy-entry
    container_name: caddy-entry
      - "/data/caddy/config:/etc/caddy"
      - "/data/nginx/config/ssl:/etc/caddy/ssl"
      - "/data/nginx/wwwroot:/wwwroot"
      - "/data/caddy/logs:/var/logs/caddy"
      - "80:80"
      - "443:443"

d. My complete Caddyfile or JSON config:

:443 {
    encode zstd gzip
    tls /etc/caddy/ssl/* /etc/caddy/ssl/**
} {
    reverse_proxy * {
        header_up X-Forwarded-Proto "https"
        flush_interval -1
        transport http {
            write_buffer 0
            read_buffer 0

3. The problem I’m having:

I use caddy for Harbor(docker image registry). I tried to push docker image in to my registry but the docker said: Retrying in 4 seconds then EOF

4. Error messages and/or full log output:

The caddy didn’t say anything
The docker deamon said:

level=error msg="Upload failed, retrying: EOF"
level=error msg="Upload failed, retrying: EOF"
level=error msg="Upload failed, retrying: EOF"
level=error msg="Not continuing with push after error: context canceled"

Harbor didn’t say anything.

5. What I already tried:

I use nginx before and I try to set

header_up X-Forwarded-Proto "https"
header_up X-Forwarded-For {host}
header_up Host {host}
websocket(caddy v1)
header_up Connection "upgrade"

but it didn’t work.

6. Links to relevant resources:

(other I forgot sorry)

Hmm. Does the docker registry communication happen over TCP rather than HTTP? If that’s the case then using this approach to proxy wouldn’t work. Just a guess.

What are in your Caddy logs? You omitted those. You can see them with docker-compose logs caddy-entry

This is the log when I was pushing the image

(When I enable the debug mode…)

(Sorry about the image is not very clear. You can open these image in a new tab to get a clearer view)

I tried to add

header_up Host {http.reverse_proxy.upstream.hostport}

and it works!
But it seem like caddy doesn’t let the http basic auth info pass to the backend. (I already login in to my registry by using the command docker login

The push refers to repository []
87e25f9e8a42: Pushing [==================================================>]    116MB/116MB
2de5ae7bc7e4: Pushing [>                                                  ]  533.5kB/141.4MB
f2411f32c29f: Pushing [==================================================>]  413.2kB
ab222698d644: Pushing [==================================================>]  109.7MB/109.7MB
60d7a9dafbc1: Pushing [==================================================>]  35.68MB/35.68MB
8682f9a74649: Waiting
d3a6da143c91: Waiting
83f4287e1f04: Waiting
7ef368776582: Waiting
unauthorized: unauthorized to access repository: library/gradle, action: push: unauthorized to access repository: library/gradle, action: push


Hmm, it looks like it does based on the debug logs you screenshotted above, it shows the Authorization header is passed through.

The error message there looks like a permissions issue. Are you sure your user is correctly configured to have access to that repo?

Yes, I’m sure that I have correct permission of registry.
I tested this by pull an image from a private project.

The push log:

Update config: {
    reverse_proxy * {
        header_up Host {http.reverse_proxy.upstream.hostport}
        header_up X-Real-IP {http.request.remote}
        header_up X-Forwarded-For {http.request.remote}
        header_up X-Forwarded-Port {http.request.port}
        header_up X-Forwarded-Proto {http.request.scheme}

These two should not be necessary, Caddy will set those by itself.

Other than that… I’m at a loss :man_shrugging:

I went digging through some documentation for docker registry and some sample nginx configs to try and compare and I’m not noticing anything that looks relevant here. :frowning_face:

Thanks for your hard working :). I went digging into different docs, github issues, too.
I try to compare caddy and nginx(works for me) config file.

location / {
		client_max_body_size 2000m;
		proxy_buffering off;
		proxy_ssl_verify off;
		proxy_set_header X-Forwarded-Proto $scheme;
		proxy_request_buffering off;
		proxy_send_timeout 900;
	    proxy_read_timeout 900;

(This nginx config works) {
    reverse_proxy * {
        header_up Host {http.reverse_proxy.upstream.hostport}
        header_up X-Real-IP {http.request.remote}
        header_up X-Forwarded-Port {http.request.port}
        flush_interval -1
        transport http {
            read_buffer 0
            write_buffer 0

(Caddy config / not working)

and I found some differences in the progress bar of docker push command between nginx and caddy:

  • nginx: the progress bar is like: 10% → 20% → 40%
  • caddy: the progress bar just rush! 0% → 100% and unauthorized: unauthorized to access repository

I also found something in the harbor-registry.log

Aug  6 15:23:02 registry[2146]: time="2020-08-06T07:23:02.303921124Z" level=error msg="response completed with error""harbor_registry_user" err.code="blob unknown" err.detail=sha256:7595c8c21622ea8a8b9778972e26dbbe063f7a1c4b0a28a80a34ebb3d343b586 err.message="blob unknown to registry" go.version=go1.14.5"" http.request.method=HEAD http.request.remoteaddr= http.request.uri="/v2/ci-env/gradle/blobs/sha256:7595c8c21622ea8a8b9778972e26dbbe063f7a1c4b0a28a80a34ebb3d343b586" http.request.useragent="docker/19.03.8 go/go1.12.17 git-commit/afacb8b kernel/5.5.13-1.el7.elrepo.x86_64 os/linux arch/amd64 UpstreamClient(Docker-Client/19.03.8 \(linux\))" http.response.contenttype="application/json; charset=utf-8" http.response.duration=11.835536ms http.response.status=404 http.response.written=157 vars.digest="sha256:7595c8c21622ea8a8b9778972e26dbbe063f7a1c4b0a28a80a34ebb3d343b586""ci-env/gradle"

Hmm this one seems peculiar in the nginx config. Since the proxy is over http:// that shouldn’t be necessary, right?

I did some digging around about blob unknown to registry and it seemed to point to issues with the X-Forwarded-Proto header (needing to be https) but we can see that that’s correctly being sent in your debug logs. :thinking:

Yeah… I think so… This problem is so strange

This topic was automatically closed after 30 days. New replies are no longer allowed.