Caddy (caddy-docker-proxy) returning 0 status

Help
1

1. Output of caddy version:

v2.5.2 h1:eCJdLyEyAGzuQTa5Mh3gETnYWDClo1LjtQm2q9RNZrs=

2. How I run Caddy:

docker-compose home built dockerfile for caddy-docker-proxy

FROM caddy:builder AS builder

RUN caddy-builder github.com/caddy-dns/cloudflare github.com/lucaslorentz/caddy-docker-proxy/plugin

FROM caddy:2-alpine

COPY --from=builder /usr/bin/caddy /usr/bin/caddy

CMD ["caddy", "docker-proxy"]

REverse proxy specifications are used as labels from containers that wish to be reverse proxied.
Eg settings for nextcloud

labels:
      - com.centurylinklabs.watchtower.enable=false
      - caddy=*.drogo-internal.example.com
      - caddy.@nextcloud=host nextcloud.drogo-internal.example.com
      - caddy.reverse_proxy=@nextcloud "{{ upstreams 80 }}"

a. System environment:

Ubuntu LTS 20.04

Client:
 Version:           18.06.1-ce
 API version:       1.38
 Go version:        go1.10.3
 Git commit:        e68fc7a
 Built:             Tue Aug 21 17:25:03 2018
 OS/Arch:           linux/amd64
 Experimental:      true

Server:
 Engine:
  Version:          18.06.1-ce
  API version:      1.38 (minimum version 1.12)
  Go version:       go1.10.3
  Git commit:       e68fc7a
  Built:            Tue Aug 21 17:23:27 2018
  OS/Arch:          linux/amd64
  Experimental:     false

b. Command:

n/a docker compose

Paste command here.

c. Service/unit/compose file:

services:
  caddy:
    container_name: caddy
    restart: always
    mem_limit: 75m
    labels:
      - com.centurylinklabs.watchtower.enable=false
    build:
      context: "/hdd/docker-data/network_access/caddy/docker/"
      dockerfile: dockerfile
    environment:
      - ACME_AGREE=true
      - LETSENCRYPT_EMAIL=letsencrypt@example.com
      - CLOUDFLARE_EMAIL=cloudflare@example.com
      - CLOUDFLARE_API_KEY=${CLOUDFLARE_API_KEY}
      - CADDY_INGRESS_NETWORKS=caddy-proxy
      - CADDY_DOCKER_CADDYFILE_PATH=/etc/caddy/Caddyfile
    networks:
      - caddy-proxy
    ports:
      - 2019:2019
      - 443:443
      - 8480:8480
      - 8443:8443
    dns:
      - 192.168.10.1
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock
      - "/hdd/docker-data/network_access/caddy/:/etc/caddy/"
      - caddy-data:/data

networks:
  caddy-proxy:
    driver: bridge
    external: true

volumes:
  caddy-data:

d. My complete Caddy config:

{
	"admin": {
		"listen": "tcp/localhost:2019"
	},
	"apps": {
		"http": {
			"servers": {
				"srv0": {
					"listen": [
						":443"
					],
					"logs": {},
					"routes": [
						{
							"match": [
								{
									"host": [
										"drogo-internal.example.com"
									]
								}
							],
							"terminal": true
						},
						{
							"match": [
								{
									"host": [
										"internal.example.com"
									]
								}
							],
							"terminal": true
						},
						{
							"handle": [
								{
									"handler": "subroute",
									"routes": [
										{
											"handle": [
												{
													"handler": "reverse_proxy",
													"upstreams": [
														{
															"dial": "172.26.0.8:80"
														}
													]
												}
											],
											"match": [
												{
													"host": [
														"nextcloud.drogo-internal.example.com"
													]
												}
											]
										},
										{
											"handle": [
												{
													"handler": "reverse_proxy",
													"upstreams": [
														{
															"dial": "172.26.0.6:2342"
														}
													]
												}
											],
											"match": [
												{
													"host": [
														"photos.drogo-internal.example.com"
													]
												}
											]
										},
										{
											"handle": [
												{
													"handler": "reverse_proxy",
													"upstreams": [
														{
															"dial": "172.26.0.7:7878"
														}
													]
												}
											],
											"match": [
												{
													"host": [
														"radarr.drogo-internal.example.com"
													]
												}
											]
										},
										{
											"handle": [
												{
													"handler": "reverse_proxy",
													"upstreams": [
														{
															"dial": "172.26.0.5:8989"
														}
													]
												}
											],
											"match": [
												{
													"host": [
														"sonarr.drogo-internal.example.com"
													]
												}
											]
										},
										{
											"handle": [
												{
													"handler": "reverse_proxy",
													"upstreams": [
														{
															"dial": "172.26.0.4:8082"
														}
													]
												}
											],
											"match": [
												{
													"host": [
														"traccar.drogo-internal.example.com"
													]
												}
											]
										}
									]
								}
							],
							"match": [
								{
									"host": [
										"*.drogo-internal.example.com"
									]
								}
							],
							"terminal": true
						},
						{
							"handle": [
								{
									"handler": "subroute",
									"routes": [
										{
											"handle": [
												{
													"handler": "reverse_proxy",
													"upstreams": [
														{
															"dial": "172.26.0.3:80"
														}
													]
												}
											],
											"match": [
												{
													"host": [
														"reader.internal.example.com"
													]
												}
											]
										}
									]
								}
							],
							"match": [
								{
									"host": [
										"*.internal.example.com"
									]
								}
							],
							"terminal": true
						}
					]
				}
			}
		},
		"tls": {
			"automation": {
				"policies": [
					{
						"issuers": [
							{
								"challenges": {
									"dns": {
										"provider": {
											"api_token": "REDACTED",
											"name": "cloudflare"
										}
									}
								},
								"email": "letsencrypt@example.com",
								"module": "acme"
							},
							{
								"challenges": {
									"dns": {
										"provider": {
											"api_token": "REDACTED",
											"name": "cloudflare"
										}
									}
								},
								"email": "letsencrypt@example.com",
								"module": "zerossl"
							}
						],
						"subjects": [
							"drogo-internal.example.com",
							"internal.example.com",
							"*.drogo-internal.example.com",
							"*.internal.example.com"
						]
					}
				]
			}
		}
	}
}

3. The problem I’m having:

Using Cloudflare argo I can directly talk to my nextcloud instance via http.
But wanting to change host cloudflare argo talks to my nextcloud instance to use https via caddy.

If I go to nextcloud.drogo-internal.andc.nz caddy works as advertised.

{
	"level": "error",
	"ts": 1661999159.5069156,
	"logger": "http.log.access",
	"msg": "handled request",
	"request": {
		"remote_ip": "192.168.10.147",
		"remote_port": "61669",
		"proto": "HTTP/2.0",
		"method": "GET",
		"host": "nextcloud.drogo-internal.example.com",
		"uri": "/js/core/merged-template-prepend.js?v=cd3d4599-0",
		"headers": {
			"Referer": [
				"https://nextcloud.drogo-internal.example.com/"
			],
			"Accept-Language": [
				"en-NZ,en;q=0.9"
			],
			"Cookie": [],
			"Sec-Ch-Ua": [
				"\"Chromium\";v=\"104\", \" Not A;Brand\";v=\"99\", \"Microsoft Edge\";v=\"104\""
			],
			"Dnt": [
				"1"
			],
			"Sec-Ch-Ua-Mobile": [
				"?0"
			],
			"Sec-Ch-Ua-Platform": [
				"\"Windows\""
			],
			"Sec-Fetch-Site": [
				"same-origin"
			],
			"User-Agent": [
				"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/104.0.5112.102 Safari/537.36 Edg/104.0.1293.63"
			],
			"Accept": [
				"*/*"
			],
			"Sec-Fetch-Mode": [
				"no-cors"
			],
			"Sec-Fetch-Dest": [
				"script"
			],
			"Accept-Encoding": [
				"gzip, deflate, br"
			]
		},
		"tls": {
			"resumed": true,
			"version": 772,
			"cipher_suite": 4865,
			"proto": "h2",
			"server_name": "nextcloud.drogo-internal.example.com"
		}
	},
	"user_id": "",
	"duration": 0.080608104,
	"size": 9236,
	"status": 400,
	"resp_headers": {
		"Server": [
			"Caddy",
			"nginx/1.18.0"
		],
		"Content-Type": [
			"text/html; charset=UTF-8"
		],
		"Expires": [
			"Thu, 19 Nov 1981 08:52:00 GMT"
		],
		"Content-Security-Policy": [
			"default-src 'self'; script-src 'self' 'nonce-cnRxTmJyNVZJSnljSlhhWjlYcWJBODBGQUtiTWNEcGNZY3BKQVVycmFOND06NTRuNEJJdCtVdmFvYUFYeGxpbTBhSnQxVmQ3K0JHb29HSjg1TmhPbk81bz0='; style-src 'self' 'unsafe-inline'; frame-src *; img-src * data: blob:; font-src 'self' data:; media-src *; connect-src *; object-src 'none'; base-uri 'self';"
		],
		"Pragma": [
			"no-cache"
		],
		"Date": [
			"Thu, 01 Sep 2022 02:25:59 GMT"
		],
		"Cache-Control": [
			"no-store, no-cache, must-revalidate"
		]
	}
}

But via argo, I get 0 as status and bytes…And cant see anything wrong with hosts or paths.

{
	"level": "info",
	"ts": 1661998314.4685634,
	"logger": "http.log.access",
	"msg": "handled request",
	"request": {
		"remote_ip": "172.26.0.1",
		"remote_port": "53596",
		"proto": "HTTP/1.1",
		"method": "GET",
		"host": "cloud2.example.com",
		"uri": "/login",
		"headers": {
			"Cookie": [],
			"Sec-Fetch-Dest": [
				"document"
			],
			"X-Forwarded-For": [
				"222.222.222.222"
			],
			"Accept-Encoding": [
				"gzip"
			],
			"Sec-Ch-Ua": [
				"\"Chromium\";v=\"104\", \" Not A;Brand\";v=\"99\", \"Microsoft Edge\";v=\"104\""
			],
			"Cdn-Loop": [
				"cloudflare"
			],
			"Cf-Ipcountry": [
				"NZ"
			],
			"Sec-Fetch-User": [
				"?1"
			],
			"Cf-Visitor": [
				"{\"scheme\":\"https\"}"
			],
			"Connection": [
				"keep-alive"
			],
			"Sec-Ch-Ua-Mobile": [
				"?0"
			],
			"Sec-Fetch-Mode": [
				"navigate"
			],
			"Cache-Control": [
				"max-age=0"
			],
			"Cf-Connecting-Ip": [
				"222.222.222.222"
			],
			"Dnt": [
				"1"
			],
			"Sec-Fetch-Site": [
				"none"
			],
			"Upgrade-Insecure-Requests": [
				"1"
			],
			"User-Agent": [
				"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/104.0.5112.102 Safari/537.36 Edg/104.0.1293.63"
			],
			"Accept-Language": [
				"en-NZ,en;q=0.9"
			],
			"X-Forwarded-Proto": [
				"https"
			],
			"Cf-Ray": [
				"743a7a58dae0a94f-SYD"
			],
			"Accept": [
				"text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.9"
			],
			"Cf-Warp-Tag-Id": [
				"5a920fb2532a"
			],
			"Priority": [
				"u=0"
			],
			"Sec-Ch-Ua-Platform": [
				"\"Windows\""
			]
		},
		"tls": {
			"resumed": false,
			"version": 772,
			"cipher_suite": 4865,
			"proto": "",
			"server_name": "nextcloud.drogo-internal.example.com"
		}
	},
	"user_id": "",
	"duration": 0.000007201,
	"size": 0,
	"status": 0,
	"resp_headers": {
		"Server": [
			"Caddy"
		]
	}
}

4. Error messages and/or full log output:

Paste logs/commands/output here.
USE THE PREVIEW PANE TO MAKE SURE IT LOOKS NICELY FORMATTED.

5. What I already tried:

referred to Caddy reverse_proxy always returns 0 size

6. Links to relevant resources:

2

I think i’ve worked it out myself. Essentially cloudflare is reverse proxying for me. And I had to set teh headers passed from cloudflare to what caddy was expecting.

Not sure what in the logs told me this or how I would have worked that out. It was only by changing stuff and playing I got it going.

I need to look at how to return errors rather than nothing in my caddy config.

3

I think the problem is the Host header was not something Caddy was configured for, so it handles the request with “nothing to do” because no matcher matched the request. So the behaviour you see is an empty response because “Caddy worked as configured”.

4

How would I normally diagnose that and work it out?

These new logs I’m not that keen on, they’re very hard to read. But there’s nothing (that I can understand) that tells me caddy didn’t know what to do. Apart from the status 0.

What would I put in my caddy file to generate and error if caddy didn’t know what to do with a request?

5

Re reading your response. Host head was Nextcloud.drogo-internal etc. which should have had a matched.

6

I’m seeing something different here.

7

I had both cloud and cloud2 at one point while I migrated.

But does that get passed all the way through to caddy? As I could see the host being picked up as the drogo-internal one.

8

It doesn’t help that you’re redacting your domains. It’s essentially impossible for us to trust the information in your logs as being accurate and consistent. So I don’t know what to tell you.

9

Yeah but that makes a problem for publicly accessible sites or domains where domain registration may include address information.

I get that you guys would prefer it unedited. But then users do was some privacy as well. I don’t know what the solution is.

10

Whois information is redacted these days. And domains can still be correlated to owners/hosts:

Point is, we are both confused and unable to help without specifics. Sorry.

11

My post issue, was probably unrelated to the domain redacting than it was choosing the wrong log entry, as they’re so long and unwieldy that picking up errors, info, what hosts, paths etc is almost impossible.

I much preferred the single line from the caddy v1 - and even more so when you could add to the format, the host/matcher that the log line applied to.

These caddy 2 lines make for issues as @francislavoie discovered while I was A/B testing some scenarios to get something that wasn’t working to go.

12

This topic was automatically closed after 30 days. New replies are no longer allowed.