Python's urllib.request fails but standard curl works

Malakan · January 14, 2025, 4:15pm

1. The problem I’m having:

I am trying to request an access token from one of my containers through python. Using a curl command in the terminal works, I get a data return and caddy logs but using python I get a 403 error in the Python terminal and zero caddy logs.

I have changed the url to avoid the need of credentials but the same errors occur regardless, data is returned with a curl command but not in Python (403 error).

2. Error messages and/or full log output:

In Python:

HTTP Error: 403 Forbidden

In Caddy: No logs produced

3. Caddy version:

v2.9.1 h1:OEYiZ7DbCzAWVb6TNEkjRcSCRGHVoZsJinoDR/n9oaY=

4. How I installed and ran Caddy:

through a docker-compose file

a. System environment:

Ubuntu 24.04 LTS. Docker-Compose

b. Command:

curl -X POST https://gramps.malakan.co.uk

import json
from urllib.error import HTTPError, URLError
from urllib.request import Request, urlopen

url = "https://gramps.malakan.co.uk"

req = Request(url)

try:
    with urlopen(req) as response:
        # Read the response
        response_data = response.read().decode("utf-8")
        print(f"Response: {response_data}")
except HTTPError as e:
    print(f"HTTP Error: {e.code} {e.reason}")
except URLError as e:
    print(f"URL Error: {e.reason}")

c. Service/unit/compose file:

docker-compose.yml

include:
  - ./helloworld.yml
  - ./gramps.yml

networks:
  caddy_net:
    external: false
    name: caddy_net

services:
  caddy:
    container_name: caddy
    image: caddy
    ports:
      - "80:80"
      - "443:443"
    networks:
      caddy_net:
    volumes:
      - ./caddy/data/:/data/
      - ./caddy/config/:/config/
      - ./Caddyfile:/etc/caddy/Caddyfile

gramps.yml

services:
  grampsweb: &grampsweb
    container_name: grampsweb_server
    image: ghcr.io/gramps-project/grampsweb:latest
    restart: always
    environment: &grampsweb-env
      GRAMPSWEB_TREE: "Gramps Web"  # will create a new tree if not exists
      VIRTUAL_PORT: "5000"
      VIRTUAL_HOST: gramps.malakan.co.uk  # e.g. gramps.mydomain.com
      GRAMPSWEB_CELERY_CONFIG__broker_url: "redis://grampsweb_redis:6379/0"
      GRAMPSWEB_CELERY_CONFIG__result_backend: "redis://grampsweb_redis:6379/0"
      GRAMPSWEB_RATELIMIT_STORAGE_URI: redis://grampsweb_redis:6379/1
    depends_on:
      - grampsweb_redis
    volumes:
      - gramps_users:/app/users  # persist user database
      - gramps_index:/app/indexdir  # persist search index
      - gramps_thumb_cache:/app/thumbnail_cache  # persist thumbnails
      - gramps_cache:/app/cache  # persist export and report caches
      - gramps_secret:/app/secret  # persist flask secret
      - gramps_db:/root/.gramps/grampsdb  # persist Gramps database
      - gramps_media:/app/media  # persist media files
      - gramps_tmp:/tmp
    networks:
      caddy_net:
      grampsweb_net:
    
  grampsweb_celery:
    <<: *grampsweb  # YAML merge key copying the entire grampsweb service config
    ports: []
    container_name: grampsweb_celery
    depends_on:
      - grampsweb_redis
    command: celery -A gramps_webapi.celery worker --loglevel=INFO --concurrency=2
    networks:
      grampsweb_net:

  grampsweb_redis:
    image: docker.io/library/redis:7.2.4-alpine
    container_name: grampsweb_redis
    restart: always
    networks:
      grampsweb_net:

networks:
  grampsweb_net:
    external: false
    name: grampsweb_net

volumes:
  gramps_users:
  gramps_index:
  gramps_thumb_cache:
  gramps_cache:
  gramps_secret:
  gramps_db:
  gramps_media:
  gramps_tmp:

d. My complete Caddy config:

{
	debug
}

malakan.co.uk {
	reverse_proxy http://helloworld:8000
}

gramps.malakan.co.uk {
	reverse_proxy http://grampsweb:5000

	log {
		format json
		level ERROR
	}
}

test.malakan.co.uk {
	reverse_proxy http://helloworld:8000

	log {
		format json
		level INFO
	}

	@restricted path /restricted/*
	respond @restricted 403
}

5. Links to relevant resources:

Bruce5051 · January 14, 2025, 5:43pm

Hi @Malakan,

And as a test can you change the URL to use HTTP and not HTTPS?
Trying to rule out TLS (SSL) issues such as TLS versions and cipher suite differences between curl and python.

Malakan · January 14, 2025, 8:10pm

curl -X POST http://gramps.malakan.co.uk

seemingly has no return but does produce a log entries:

caddy | {“level”:“info”,“ts”:1736939369.2940955,“logger”:“http.log.access”,“msg”:“handled request”,“request”:{“remote_ip”:“141.101.98.153”,“remote_port”:“43786”,“client_ip”:“141.101.98.153”,“proto”:“HTTP/1.1”,“method”:“POST”,“host”:“gramps.malakan.co.uk”,“uri”:“/”,“headers”:{“Accept-Encoding”:[“gzip”],“Cf-Visitor”:[“{"scheme":"http"}”],“X-Forwarded-Proto”:[“http”],“Cf-Connecting-Ip”:[“212.127.0.96”],“Connection”:[“Keep-Alive”],“Cf-Ray”:[“902569f1bb8c48cd-LHR”],“X-Forwarded-For”:[“212.127.0.96”],“Cdn-Loop”:[“cloudflare; loops=1”],“Accept”:[“/”],“Cf-Ipcountry”:[“GB”],“User-Agent”:[“curl/7.81.0”]}},“bytes_read”:0,“user_id”:“”,“duration”:0.000036595,“size”:0,“status”:308,“resp_headers”:{“Server”:[“Caddy”],“Connection”:[“close”],“Location”:[“Gramps Web”:}}

For whatever reason this does seem to hang sometimes but canceling and rerunning always runs.

In python, using the below url, I still get the 403 error and no caddy logs

url = "http://gramps.malakan.co.uk"

The python script seems to error out at the below line:

with urlopen(req) as response:

Bruce5051 · January 15, 2025, 4:26pm

At this point kindly wait for a more knowledgeable community member to assist.

matt · January 15, 2025, 5:59pm

The request isn’t even hitting Caddy… it’s hitting Cloudflare, but it’s not showing Caddy. Especially if there’s no Caddy logs at all. So, maybe check your DNS settings / network configuration.

Malakan · January 16, 2025, 7:14am

Looking at my cloudflare settings: I have A records for all DNS addresses, including gramps.malakan.co.uk. They are proxied and TTL is set to auto. SSL/TLS encryption mode is set to Full (strict).

As this is self-hosted, my router has all 80 and 443 traffic pointing to the server for internal and external ports for TCP/UDP.

Are there any other DNS settings / network configurations I need to look into?

Edit:

The issue appears to be the proxied setting on Cloudflare, disabling allowed the python script to get a server response.

both http and https had the same server response:

Response: <!doctype html><html lang="en"><head><meta charset="utf-8"><meta name="viewport" content="width=device-width,initial-scale=1,viewport-fit=cover"><meta name="description" content="Gramps Web - research &amp; organize your family tree together"><base href="/"><link 
rel="stylesheet" href="fonts/fonts.css"><link rel="stylesheet" href="tippy.css"><link rel="icon" href="images/favicon.ico"><link rel="manifest" href="manifest.json" crossorigin="use-credentials"><meta name="theme-color" content="#6D4C41"><meta name="mobile-web-app-capable" 
content="yes"><meta name="application-name" content="Gramps"><meta name="apple-mobile-web-app-capable" content="yes"><meta name="apple-mobile-web-app-status-bar" content="#6D4C41"><meta name="apple-mobile-web-app-title" content="Gramps"><link rel="apple-touch-icon" sizes="192x192" href="images/icon192.png"><style>body,html{margin:0;padding:0;height:100%}</style><title>Gramps Web</title><link rel="preload" href="./2ea051d6.js" as="script" crossorigin="anonymous"></head><body><gramps-js></gramps-js><script src="config.js"></script><script type="module" src="./2ea051d6.js"></script><script>"serviceWorker"in navigator&&window.addEventListener("load",(function(){navigator.serviceWorker.register("sw.js").then((function(){console.log("ServiceWorker registered.")})).catch((function(e){console.log("ServiceWorker registration failed: ",e)}))}))</script></body></html>

Edit:

While the simple http/https url (gramps.malakan.co.uk) gets a response, I now get a new error from my desired url: gramps.malakan.co.uk/api/token.

HTTP Error: 308 Permanent Redirect

One step closer to the solution!

This url requires some extra data to be sent, but here is the code I use with the password removed:

url = "http://malakan.co.uk/api/token"

data = {
        "username": "Malakan",
        "password": "[REDACTED]"
    }

json_data = json.dumps(data).encode("utf-8")

headers = {"content-Type": "application/json"}

req = Request(url, data=json_data, headers=headers)

try:
    with urlopen(req) as response:
        # Read the response
        response_data = response.read().decode("utf-8")
        print(f"Response: {response_data}")
except HTTPError as e:
    print(f"HTTP Error: {e.code} {e.reason}")

Malakan · January 16, 2025, 4:14pm

I’ve just tested it within the application I am trying to run and it works successfully now with the proxy setting off. Still get the 308 error with my own test python script…

Mohammed90 · January 16, 2025, 8:04pm

308 is not an error. It’s a redirection status telling the client to try at the URL specified in the Location header

Use https

Malakan · January 17, 2025, 6:39pm

I was also missing a / on the end of the url.

Going a google search, I found some info about headers. Setting my headers to below appears to solve the 308 and I can get my token in my Python script with the proxy OFF, how can I get this working with it ON?

headers = {
    "User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/114.0.0.0 Safari/537.36",
    "Content-Type": "application/json"
}

system · February 16, 2025, 6:40pm

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.