1. Caddy version (caddy version):
I am using caddy:latest from Docker hub which points to version: 2.4.6 at the time of writing.
2. How I run Caddy:
I have Caddy bound to port 80 & 443 inside a Docker Swarm to handle proxying traffic to other containers within the same swarm. I am hosting a dozen different websites with Caddy & they are all working as expected (Aside form the bug I will be talking about here).
a. System environment:
The Docker Swarm nodes are all running either Ubuntu 18.04 or 20.04. On the client-side, I have tried performing the clone from Windows 10, 11 & Ubuntu 20.04. All environments are having the same issue.
b. Command:
N/A
c. Service/unit/compose file:
Caddy:
# proxy stack
version: '3.6'
services:
caddy:
image: caddy:2.4.6
ports:
- 80:80
- 443:443
networks:
- network
networks:
network: # "proxy_network" externally
driver: overlay
The following compose files are the different Git hosting environments I have tried connecting. Only one would be active at a time & they all result in the same error upon clone.
Gitlab (HTTP):
# gitlab stack
version: '3.6'
services:
service: # "gitlab_service" externally
image: gitlab/gitlab-ee:latest
environment:
GITLAB_OMNIBUS_CONFIG: |
external_url 'https://gitlab.psdcitywide.com'
nginx['listen_port'] = 80
nginx['listen_https'] = false
networks:
- proxy_network
networks:
proxy_network:
external: true
Gitlab (HTTPS):
# gitlab stack
version: '3.6'
services:
service:
image: gitlab/gitlab-ee:latest
environment:
GITLAB_OMNIBUS_CONFIG: |
external_url 'https://gitlab.psdcitywide.com'
networks:
- proxy_network
networks:
proxy_network:
external: true
Gitea:
# gitea stack
version: '3.6'
services:
gitea: # "gitea_gitea" externally
image: gitea/gitea:1.15.4
networks:
- proxy_network
networks:
proxy_network:
external: true
d. My complete Caddyfile or JSON config:
These are the respective Caddyfile snippits for each Git environment I tried to host. Only one of these are active at a time.
Gitlab (HTTP):
gitlab.psdcitywide.com {
reverse_proxy gitlab_service:80
}
Gitlab (HTTPS):
gitlab.psdcitywide.com {
reverse_proxy https://gitlab_service:443 {
transport http {
tls_insecure_skip_verify
}
}
}
Gitlab (HTTP + HTTPS):
http://gitlab.psdcitywide.com, https://gitlab.psdcitywide.com {
reverse_proxy gitlab_service:80
}
Gitea:
gitea.psdcitywide.com {
reverse_proxy gitea_gitea:3000
}
3. The problem I’m having:
I am trying to host GitLab using Caddy to provide TLS. Although I was able to get everything running correctly including the web interface over HTTPS; I was unable to clone projects using the same HTTPS URL. Each time I tried, regardless of environment, configuration or Git provider (I tried both GitLab & Gitea), it would result in the same error.
4. Error messages and/or full log output:
Everything I tried resulted in the same error.
Gitlab (HTTP, HTTPS & HTTP + HTTPS configs):
$ git clone https://gitlab.psdcitywide.com/ztimson/test.git
Cloning into 'test'...
fatal: unable to access 'https://gitlab.psdcitywide.com/ztimson/test.git/': server certificate verification failed. CAfile: none CRLfile: none
Gitea:
$ git clone http://gitea.psdcitywide.com/ztimson/test.git
Cloning into 'test'...
fatal: unable to access 'http://gitea.psdcitywide.com/ztimson/test.git/': server certificate verification failed. CAfile: none CRLfile: none
5. What I already tried:
As I mentioned, I have been trying to setup GitLab on our servers using Caddy to provide Letsencrypt certificates.
My first attempt was to turn off GitLab’s HTTPS completely and use Caddy to force HTTPS & forward all traffic to GitLab’s HTTP port. Although the site worked cloning did not. See GitLab (HTTP) configurations.
My second attempt was to use GitLab’s HTTPS port instead since it was complaining about a no cerficate being found however it resulted in the same thing. See GitLab (HTTPS) configurations. Even though both Caddy & Gitlab have their own cert, Git isn’t seeing either.
With neither working over HTTPS, I setup Caddy to disable forwarding & use GitLabs HTTP port so I could try cloning without TLS. This time it worked, however clonging over HTTPS would still confirming the problem was with the certficates/proxy. See GitLab (HTTP + HTTPS) configurations.
At this point I thought I had some certificate or header problem between GitLab & Caddy so I tried a ton of things but they pretty much all resulted in either an invalid cerficate or GitLab failing to work completely. The configs I have included are the only “working” setups that gave access to the websites with HTTPS.
Lastly I decided to try hosting a different minamilist Git program to test if the problem was with Caddy + Git or GitLab/My configuration. A colleague had a working setup of NGINX + Letsencrypt + Gitea so we coppied the configuration replacing NGINX + Letsencrypt for Caddy. This also fails in the same way as GitLab, see the Gitea configurations.
In conclusion, I can get the website to work & clone over HTTP (and SSH which bypasses Caddy) but regardless of which Git server & client I use I can’t clone over HTTPS leading me to conclude the issue is with the only common denominator, Caddy & Git, or I am missing a key part of the configuration.
Note: Turning off Git SSL verification works but this isn’t a viable long term solution for me: git config http.sslVerify false
Thank you in advance for the help!
6. Links to relevant resources:
I have left the Gitea instance up & so that you can inspect it. As you can see everything seems to be in order until you try cloning something over HTTPS.
Feel free to create a user account & repo to try cloning. My test repo is also public so you can test right away with: git clone https://gitea.psdcitywide.com/ztimson/test