Python's urllib.request fails but standard curl works

Help
1

1. The problem I’m having:

I am trying to request an access token from one of my containers through python. Using a curl command in the terminal works, I get a data return and caddy logs but using python I get a 403 error in the Python terminal and zero caddy logs.

I have changed the url to avoid the need of credentials but the same errors occur regardless, data is returned with a curl command but not in Python (403 error).

2. Error messages and/or full log output:

In Python:

HTTP Error: 403 Forbidden

In Caddy: No logs produced

3. Caddy version:

v2.9.1 h1:OEYiZ7DbCzAWVb6TNEkjRcSCRGHVoZsJinoDR/n9oaY=

4. How I installed and ran Caddy:

through a docker-compose file

a. System environment:

Ubuntu 24.04 LTS. Docker-Compose

b. Command:

curl -X POST https://gramps.malakan.co.uk

import json
from urllib.error import HTTPError, URLError
from urllib.request import Request, urlopen

url = "https://gramps.malakan.co.uk"

req = Request(url)

try:
    with urlopen(req) as response:
        # Read the response
        response_data = response.read().decode("utf-8")
        print(f"Response: {response_data}")
except HTTPError as e:
    print(f"HTTP Error: {e.code} {e.reason}")
except URLError as e:
    print(f"URL Error: {e.reason}")

c. Service/unit/compose file:

docker-compose.yml

include:
  - ./helloworld.yml
  - ./gramps.yml

networks:
  caddy_net:
    external: false
    name: caddy_net

services:
  caddy:
    container_name: caddy
    image: caddy
    ports:
      - "80:80"
      - "443:443"
    networks:
      caddy_net:
    volumes:
      - ./caddy/data/:/data/
      - ./caddy/config/:/config/
      - ./Caddyfile:/etc/caddy/Caddyfile

gramps.yml

services:
  grampsweb: &grampsweb
    container_name: grampsweb_server
    image: ghcr.io/gramps-project/grampsweb:latest
    restart: always
    environment: &grampsweb-env
      GRAMPSWEB_TREE: "Gramps Web"  # will create a new tree if not exists
      VIRTUAL_PORT: "5000"
      VIRTUAL_HOST: gramps.malakan.co.uk  # e.g. gramps.mydomain.com
      GRAMPSWEB_CELERY_CONFIG__broker_url: "redis://grampsweb_redis:6379/0"
      GRAMPSWEB_CELERY_CONFIG__result_backend: "redis://grampsweb_redis:6379/0"
      GRAMPSWEB_RATELIMIT_STORAGE_URI: redis://grampsweb_redis:6379/1
    depends_on:
      - grampsweb_redis
    volumes:
      - gramps_users:/app/users  # persist user database
      - gramps_index:/app/indexdir  # persist search index
      - gramps_thumb_cache:/app/thumbnail_cache  # persist thumbnails
      - gramps_cache:/app/cache  # persist export and report caches
      - gramps_secret:/app/secret  # persist flask secret
      - gramps_db:/root/.gramps/grampsdb  # persist Gramps database
      - gramps_media:/app/media  # persist media files
      - gramps_tmp:/tmp
    networks:
      caddy_net:
      grampsweb_net:
    
  grampsweb_celery:
    <<: *grampsweb  # YAML merge key copying the entire grampsweb service config
    ports: []
    container_name: grampsweb_celery
    depends_on:
      - grampsweb_redis
    command: celery -A gramps_webapi.celery worker --loglevel=INFO --concurrency=2
    networks:
      grampsweb_net:

  grampsweb_redis:
    image: docker.io/library/redis:7.2.4-alpine
    container_name: grampsweb_redis
    restart: always
    networks:
      grampsweb_net:

networks:
  grampsweb_net:
    external: false
    name: grampsweb_net

volumes:
  gramps_users:
  gramps_index:
  gramps_thumb_cache:
  gramps_cache:
  gramps_secret:
  gramps_db:
  gramps_media:
  gramps_tmp:

d. My complete Caddy config:

{
	debug
}

malakan.co.uk {
	reverse_proxy http://helloworld:8000
}

gramps.malakan.co.uk {
	reverse_proxy http://grampsweb:5000

	log {
		format json
		level ERROR
	}
}

test.malakan.co.uk {
	reverse_proxy http://helloworld:8000

	log {
		format json
		level INFO
	}

	@restricted path /restricted/*
	respond @restricted 403
}

5. Links to relevant resources:

2

Hi @Malakan,

And as a test can you change the URL to use HTTP and not HTTPS?
Trying to rule out TLS (SSL) issues such as TLS versions and cipher suite differences between curl and python.

3 
curl -X POST http://gramps.malakan.co.uk

seemingly has no return but does produce a log entries:

caddy | {“level”:“info”,“ts”:1736885540.789267,“logger”:“http.log.access”,“msg”:“handled request”,“request”:{“remote_ip”:“172.70.85.52”,“remote_port”:“52928”,“client_ip”:“172.70.85.52”,“proto”:“HTTP/1.1”,“method”:“POST”,“host”:“gramps.malakan.co.uk”,“uri”:“/”,“headers”:{“Cf-Ipcountry”:[“GB”],“User-Agent”:[“curl/7.81.0”],“Connection”:[“Keep-Alive”],“Cf-Ray”:[“902047c58d307741-LHR”],“X-Forwarded-For”:[“212.127.0.96”],“Cdn-Loop”:[“cloudflare; loops=1”],“Accept-Encoding”:[“gzip”],“Accept”:[“/”],“X-Forwarded-Proto”:[“http”],“Cf-Visitor”:[“{"scheme":"http"}”],“Cf-Connecting-Ip”:[“212.127.0.96”]}},“bytes_read”:0,“user_id”:“”,“duration”:0.000036776,“size”:0,“status”:308,“resp_headers”:{“Content-Type”:,“Server”:[“Caddy”],“Connection”:[“close”],“Location”:[“https://gramps.malakan.co.uk/”]}}

For whatever reason this does seem to hang sometimes but canceling and rerunning always runs.

In python, using the below url, I still get the 403 error and no caddy logs

url = "http://gramps.malakan.co.uk"

The python script seems to error out at the below line:

with urlopen(req) as response: