1. My Caddy version (caddy version):
Via docker (caddy:latest): v2.0.0-rc.3 h1:z2H/QnaRscip6aZJxwTbghu3zhC88Vo8l/K57WUce4Q=
2. How I run Caddy:
a. System environment:
Server:
Ubuntu Server 20.04
Docker version 19.03.8, build afacb8b7f0
Pushing from client:
Pop_OS 20.04 (basically Ubuntu 20.04)
Docker version 19.03.8, build afacb8b7f0
b. Command:
I run all my docker containers via docker-compose:
docker-compose up -d
c. Service/unit/compose file:
The docker registry is also started via a Docker compose file:
version: '3'
networks:
web:
external: true
registry:
image: registry:2
restart: always
volumes:
- "./registry:/var/lib/registry"
networks:
- default
- web
version: '3'
networks:
web:
external: true
services:
caddy:
image: caddy
restart: always
ports:
- "80:80"
- "443:443"
volumes:
- "./Caddyfile:/etc/caddy/Caddyfile:ro"
- "./www:/www:ro"
- "./config:/config"
- "./data:/data"
networks:
- web
d. My complete Caddyfile or JSON config:
(auth) {
basicauth {
admin redacted
}
}
registry.example.com {
import auth
reverse_proxy /v2* http://registry_registry_1:5000
reverse_proxy http://registry_registry-ui_1
}
3. The problem I’m having:
I’m trying to push a docker image to my private registry, like this:
docker push registry.example.com/test
This keeps failing, however. (Details below)
Note that this used to work fine when I was using Traefik instead of Caddy, which is why I’m fairly positive this is an issue with Caddy, somehow. (Or my configuration, of course.)
4. Error messages and/or full log output:
❯ docker push registry.example.com/test
The push refers to repository [registry.example.com/test]
7fdb0a6602d3: Pushing [==================================================>] 3.703MB/3.703MB
EOF
Before it says “EOF”, it tells me “Retrying in X seconds” until eventually it gives up.
Running dockerd --debug in another window when trying to do the same thing doesn’t show me much interesting information either:
DEBU[2020-05-04T22:08:09.445125407+02:00] Calling HEAD /_ping
DEBU[2020-05-04T22:08:09.449518419+02:00] Calling POST /v1.40/images/registry.example.com/test/push?tag=
DEBU[2020-05-04T22:08:09.453669652+02:00] hostDir: /etc/docker/certs.d/registry.example.com
DEBU[2020-05-04T22:08:09.453776652+02:00] Trying to push registry.example.com/test to https://registry.example.com v2
DEBU[2020-05-04T22:08:09.468914587+02:00] Pushing repository: registry.example.com/test:latest
DEBU[2020-05-04T22:08:09.469253576+02:00] Pushing layer: sha256:7fdb0a6602d30a4309a789b993cf5c363ba2a0262cfc5a449015682546943beb
DEBU[2020-05-04T22:08:09.643584506+02:00] Assembling tar data for 9657298eea8b858d1a08a87db8051a0259a7cd11fad3e7ba4039b33df41dfef2
ERRO[2020-05-04T22:08:09.653373300+02:00] Upload failed, retrying: EOF
The caddy log itself shows nothing when this happens.
The registry log shows this, which doesn’t look abnormal to me:
registry_1 | time="2020-05-04T20:08:09.640562023Z" level=info msg="response completed" go.version=go1.11.2 http.request.host=registry.example.com http.request.id=c1c61318-92d3-4ffd-9f45-65ebd380efd5 http.request.method=POST http.request.remoteaddr=redacted http.request.uri="/v2/ip/blobs/uploads/" http.request.useragent="docker/19.03.8 go/go1.13.8 git-commit/afacb8b7f0 kernel/5.4.0-7626-generic os/linux arch/amd64 UpstreamClient(Docker-Client/19.03.8 \(linux\))" http.response.duration=55.810875ms http.response.status=202 http.response.written=0
registry_1 | 172.18.0.2 - - [04/May/2020:20:08:09 +0000] "POST /v2/ip/blobs/uploads/ HTTP/1.1" 202 0 "" "docker/19.03.8 go/go1.13.8 git-commit/afacb8b7f0 kernel/5.4.0-7626-generic os/linux arch/amd64 UpstreamClient(Docker-Client/19.03.8 \\(linux\\))"
5. What I already tried:
I tried setting REGISTRY_STORAGE_REDIRECT_DISABLE=true in the registry configuration as recommended here. This made no difference.
I tried pushing both a very small image (3.7 MB) and a big image (567 MB) with the same result.
6. Links to relevant resources:
- Registry on Docker Hub: Docker Hub
Ps: Apologies for redacting my domain name to example.com as well as my IP address, I assume that’s fine.