'caddy reload' vails with error 'connection refused' when running in Gitlab CI environment

Help
1

1. The problem I’m having:

When trying to run caddy reload in a Gitlab CI container, I encounter an error:

Error: sending configuration to instance: performing request: Post "http://localhost:2019/load": dial tcp 127.0.0.1:2019: connect: connection refused

This error does not occur when creating the same container locally.

We use a custom caddy image like so:

ARG CADDY_VERSION=2.8
FROM caddy:${CADDY_VERSION}-builder-alpine AS builder

RUN xcaddy build \
    --with github.com/caddyserver/transform-encoder \
    --with github.com/hairyhenderson/caddy-teapot-module \
    --with github.com/caddy-dns/cloudflare \
    --with github.com/mholt/caddy-l4 \
    --with github.com/abiosoft/caddy-json-schema

FROM caddy:${CADDY_VERSION}-alpine

RUN apk add curl jq

COPY --from=builder /usr/bin/caddy /usr/bin/caddy
COPY Caddyfile /etc/caddy/Caddyfile

With the following Caddyfile:

{
	admin 0.0.0.0:2019
	email operations@example.com
	log access-json {
		format json
		include http.log.access.access-logs
		output file /var/log/caddy/access-json.log
		format filter {
			request>headers>Cookie delete
			resp_headers delete
		}
	}
	log access-common {
		format transform `{request>headers>X-Forwarded-For>[0]:request>remote_ip} - {user_id} [{ts}] {request>headers>X-Forwarded-Host:request>host} "{request>method} {request>uri} {request>proto}" {status} {size} "{request>headers>Referer>[0]}" "{request>headers>User-Agent>[0]}"` {
			time_format iso8601
		}
		include http.log.access.access-logs
		output file /var/log/caddy/access-common.log
	}
}
(base-rm-resp-headers) {
	header -server
	header -x-commerce-core
	header -x-drupal-cache
	header -x-drupal-dynamic-cache
	header -x-generator
	header -x-powered-by
}
(base-transport-tls-insecure) {
	transport http {
		tls_insecure_skip_verify
	}
}

import Caddyfile.d/*.caddyfile

In out Gitlab CI stage, we simply do the following:

  1. Use the caddy image built from the above Dockerfile
  2. Copy the Caddyfile from our repo into /etc/caddy/Caddyfile - the Caddyfile in our repo is 100% identical to the Caddyfile baked into the container image, so this step is not relevant at this point in time.
  3. Copy a JSON file with mholt/caddy-l4 server definitions from our repo into /etc/caddy/layer4-ssh-proxies.json
  4. cd /etc/caddy
  5. caddy version
  6. caddy reload

At this point the pipeline fails with the error above.

When running the container locally, mounting the repo at /builds/operations/ingress-router/, and executing the same steps, the caddy reload command works as expected:

docker run -v $PWD/ingress-router:/builds/operations/ingress-router gitlab-reg.example.com/operations/ops-oci-images/infra/caddy:2.8-alpine
{"level":"info","ts":1722918322.7366016,"msg":"using config from file","file":"/etc/caddy/Caddyfile"}
{"level":"info","ts":1722918322.7370393,"msg":"adapted config to JSON","adapter":"caddyfile"}
{"level":"info","ts":1722918322.7374763,"logger":"admin","msg":"admin endpoint started","address":"0.0.0.0:2019","enforce_origin":false,"origins":["//0.0.0.0:2019"]}
{"level":"warn","ts":1722918322.737484,"logger":"admin","msg":"admin endpoint on open interface; host checking disabled","address":"0.0.0.0:2019"}
{"level":"info","ts":1722918322.7377057,"msg":"autosaved config (load with --resume flag)","file":"/config/caddy/autosave.json"}
{"level":"info","ts":1722918322.737711,"msg":"serving initial configuration"}

and then connecting via docker exec and running the CI steps manually:

docker exec -it caddy sh
/srv # cp -av /builds/operations/ingress-router/caddy/Caddyfile /etc/caddy/Caddyfile
'/builds/operations/ingress-router/caddy/Caddyfile' -> '/etc/caddy/Caddyfile'
/srv # cp -av /builds/operations/ingress-router/caddy/layer4-ssh-proxies.json /etc/caddy/layer4-ssh-proxies.json
'/builds/operations/atech-ingress-router/caddy/layer4-ssh-proxies.json' -> '/etc/caddy/layer4-ssh-proxies.json'
/srv # cd /etc/caddy/
/etc/caddy # ls -lah
total 16K    
drwxr-xr-x    1 root     root          76 Aug  6 04:01 .
drwxr-xr-x    1 root     root          10 Aug  6 03:57 ..
-rw-r--r--    1 1000     1000         945 Aug  5 06:22 Caddyfile
-rw-r--r--    1 1000     1000        8.8K Aug  6 01:27 layer4-ssh-proxies.json
/etc/caddy # caddy version
v2.8.4 h1:q3pe0wpBj1OcHFZ3n/1nl4V4bxBrYoSoab7rL9BMYNk=
/etc/caddy # caddy reload
2024/08/06 04:02:00.143	INFO	using adjacent Caddyfile
2024/08/06 04:02:00.143	WARN	No files matching import glob pattern	{"pattern": "Caddyfile.d/*.caddyfile"}
2024/08/06 04:02:00.143	INFO	adapted config to JSON	{"adapter": "caddyfile"}

2. Error messages and/or full log output:

Full output of Gitlab CI failure.

Running with gitlab-runner 17.2.0 (6428c288)
  on gitlab-ci-ldr-02 BawuXMRm, system ID: s_0975065499d9
Preparing the "docker" executor
00:05
Using Docker executor with image gitlab-reg.example.com/operations/ops-oci-images/infra/caddy:2.8-alpine ...
Authenticating with credentials from job payload (GitLab Registry)
Pulling docker image gitlab-reg.example.com/operations/ops-oci-images/infra/caddy:2.8-alpine ...
Using docker image sha256:bfec4a400ec0efa80c1ba5254dbd379f69a89e4a18ce4e2416ce9311a7816ddb for gitlab-reg.example.com/operations/ops-oci-images/infra/caddy:2.8-alpine with digest gitlab-reg.atech.host/operations/ops-oci-images/infra/caddy@sha256:1b3d4e6fd487227d248a71c0e524ec3b86bc5bdc5fb5b05b44029a0a55ee78c6 ...
Preparing environment
00:01
Running on runner-bawuxmrm-project-217-concurrent-1 via gitlab-ci-ldr-02...
Getting source from Git repository
00:01
Fetching changes with git depth set to 20...
Reinitialized existing Git repository in /builds/operations/ingress-router/.git/
Checking out 88cae7ef as detached HEAD (ref is main)...
Skipping Git submodules setup
Executing "step_script" stage of the job script
00:01
Using docker image sha256:bfec4a400ec0efa80c1ba5254dbd379f69a89e4a18ce4e2416ce9311a7816ddb for gitlab-reg.example.com/operations/ops-oci-images/infra/caddy:2.8-alpine with digest gitlab-reg.example.com/operations/ops-oci-images/infra/caddy@sha256:1b3d4e6fd487227d248a71c0e524ec3b86bc5bdc5fb5b05b44029a0a55ee78c6 ...
$ echo "Test configuration of Caddy L4 Proxy JSON."
Test configuration of Caddy L4 Proxy JSON.
$ cp -av /builds/operations/ingress-router/caddy/Caddyfile /etc/caddy/Caddyfile
'/builds/operations/ingress-router/caddy/Caddyfile' -> '/etc/caddy/Caddyfile'
$ cp -av /builds/operations/ingress-router/caddy/layer4-ssh-proxies.json /etc/caddy/layer4-ssh-proxies.json
'/builds/operations/ingress-router/caddy/layer4-ssh-proxies.json' -> '/etc/caddy/layer4-ssh-proxies.json'
$ cd /etc/caddy
$ ls -lah
total 16K    
drwxr-xr-x    1 root     root          60 Aug  6 03:05 .
drwxr-xr-x    1 root     root          19 Aug  6 03:05 ..
-rw-rw-rw-    1 root     root         945 Aug  6 02:49 Caddyfile
-rw-rw-rw-    1 root     root        8.8K Aug  6 02:49 layer4-ssh-proxies.json
$ caddy version
v2.8.4 h1:q3pe0wpBj1OcHFZ3n/1nl4V4bxBrYoSoab7rL9BMYNk=
$ caddy reload
{"level":"info","ts":1722913515.415652,"msg":"using adjacent Caddyfile"}
{"level":"warn","ts":1722913515.417704,"msg":"No files matching import glob pattern","pattern":"Caddyfile.d/*.caddyfile"}
{"level":"info","ts":1722913515.4183245,"msg":"adapted config to JSON","adapter":"caddyfile"}
Error: sending configuration to instance: performing request: Post "http://localhost:2019/load": dial tcp 127.0.0.1:2019: connect: connection refused
Cleaning up project directory and file based variables
00:01
ERROR: Job failed: exit code 1

3. Caddy version:

v2.8.4 h1:q3pe0wpBj1OcHFZ3n/1nl4V4bxBrYoSoab7rL9BMYNk=

4. How I installed and ran Caddy:

Custom docker image, as above.

a. System environment:

Ubuntu 22.04 with Docker 24.0.7

b. Command:

Gitlab CI pipeline, as descibed above. The pipeline fails at caddy reload inside the container image.

c. Service/unit/compose file:

Dockerfile

ARG CADDY_VERSION=2.8
FROM caddy:${CADDY_VERSION}-builder-alpine AS builder

RUN xcaddy build \
    --with github.com/caddyserver/transform-encoder \
    --with github.com/hairyhenderson/caddy-teapot-module \
    --with github.com/caddy-dns/cloudflare \
    --with github.com/mholt/caddy-l4 \
    --with github.com/abiosoft/caddy-json-schema

FROM caddy:${CADDY_VERSION}-alpine

RUN apk add curl jq

COPY --from=builder /usr/bin/caddy /usr/bin/caddy
COPY Caddyfile /etc/caddy/Caddyfile

d. My complete Caddy config:

{
	admin 0.0.0.0:2019
	email operations@example.com
	log access-json {
		format json
		include http.log.access.access-logs
		output file /var/log/caddy/access-json.log
		format filter {
			request>headers>Cookie delete
			resp_headers delete
		}
	}
	log access-common {
		format transform `{request>headers>X-Forwarded-For>[0]:request>remote_ip} - {user_id} [{ts}] {request>headers>X-Forwarded-Host:request>host} "{request>method} {request>uri} {request>proto}" {status} {size} "{request>headers>Referer>[0]}" "{request>headers>User-Agent>[0]}"` {
			time_format iso8601
		}
		include http.log.access.access-logs
		output file /var/log/caddy/access-common.log
	}
}
(base-rm-resp-headers) {
	header -server
	header -x-commerce-core
	header -x-drupal-cache
	header -x-drupal-dynamic-cache
	header -x-generator
	header -x-powered-by
}
(base-transport-tls-insecure) {
	transport http {
		tls_insecure_skip_verify
	}
}

import Caddyfile.d/*.caddyfile
2

Hi,

This can be closed as resolved.

It turns out the the Gitlab CI environment creates its own entrypoint (/bin/sh) rather than using the container’s built entrypoint; and so in our case caddy was not already running.

1 Like
3

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.