(Sometimes) 503 Errors with DocuSeal in Docker Setup & Database Compatibility Issue

Help
1

1. The problem I’m having:

I've been wrestling with this issue since 8 AM, and now it's 4 AM the next day.

I’m running Docuseal inside a container, along with Caddy and PostgreSQL containers for support. Docuseal is accessible, but I’m hitting a 503 Error about 30% of the time. I suspect that Caddy isn’t properly utilizing the Caddyfile, which includes a reverse proxy setup for redirecting certain requests to Apache2.

Additionally, in a moment of desperation during this troubleshooting marathon, I decided to reinstall MySQL. That’s when the errors started to escalate, leading me to believe I might have botched something with the databases, making them incompatible. Everything was running smoothly with the Caddyfile setup until my attempted MySQL “fix.” . I installed, deinstalled and tampered with a lot of Settings until I realized that I broke something in the system. Any insights or advice on where I might have gone wrong would be greatly appreciated.

When I run docker ps i get the following:

CONTAINER ID   IMAGE                      COMMAND                  CREATED       STATUS                    PORTS                                                                                                                       NAMES
b2ed4da4a542   caddy:latest               "caddy run --config …"   3 hours ago   Up 50 minutes             0.0.0.0:80->80/tcp, :::80->80/tcp, 0.0.0.0:443->443/tcp, :::443->443/tcp, 0.0.0.0:443->443/udp, :::443->443/udp, 2019/tcp   docuseal_caddy_1
2cb7162fdf8a   docuseal/docuseal:latest   "/app/bin/rails serv…"   3 hours ago   Up 46 minutes             0.0.0.0:3000->3000/tcp, :::3000->3000/tcp                                                                                   docuseal_app_1
51b44314af0c   postgres:15                "docker-entrypoint.s…"   3 hours ago   Up 50 minutes (healthy)   5432/tcp                                                                                                                    docuseal_postgres_1

2. Error messages and/or full log output:

Firstly the log file from the caddy docker:

{"level":"info","ts":1709857499.6908562,"msg":"using provided configuration","config_file":"/etc/caddy/Caddyfile","config_adapter":"caddyfile"}
{"level":"warn","ts":1709857499.6920455,"msg":"Caddyfile input is not formatted; run 'caddy fmt --overwrite' to fix inconsistencies","adapter":"caddyfile","file":"/etc/caddy/Caddyfile","line":2}
{"level":"info","ts":1709857499.6928616,"logger":"admin","msg":"admin endpoint started","address":"localhost:2019","enforce_origin":false,"origins":["//localhost:2019","//[::1]:2019","//127.0.0.1:2019"]}
{"level":"info","ts":1709857499.6930435,"logger":"http.auto_https","msg":"server is listening only on the HTTPS port but has no TLS connection policies; adding one to enable TLS","server_name":"srv0","https_port":443}
{"level":"info","ts":1709857499.6930964,"logger":"http.auto_https","msg":"enabling automatic HTTP->HTTPS redirects","server_name":"srv0"}
{"level":"info","ts":1709857499.6931388,"logger":"tls.cache.maintenance","msg":"started background certificate maintenance","cache":"0xc000459700"}
{"level":"info","ts":1709857499.6934469,"logger":"tls","msg":"cleaning storage unit","description":"FileStorage:/data/caddy"}
{"level":"info","ts":1709857499.6934643,"logger":"http.log","msg":"server running","name":"remaining_auto_https_redirects","protocols":["h1","h2","h3"]}
{"level":"info","ts":1709857499.6935391,"logger":"http","msg":"enabling HTTP/3 listener","addr":":443"}
{"level":"info","ts":1709857499.6936135,"msg":"failed to sufficiently increase receive buffer size (was: 208 kiB, wanted: 2048 kiB, got: 416 kiB). See https://github.com/quic-go/quic-go/wiki/UDP-Buffer-Sizes for details."}
{"level":"info","ts":1709857499.6938457,"logger":"http.log","msg":"server running","name":"srv0","protocols":["h1","h2","h3"]}
{"level":"info","ts":1709857499.6938586,"logger":"http","msg":"enabling automatic TLS certificate management","domains":["freepbx.staffconnect.agency","docuseal.staffconnect.agency"]}
{"level":"info","ts":1709857499.6943157,"logger":"tls","msg":"finished cleaning storage units"}
{"level":"info","ts":1709857499.695106,"msg":"autosaved config (load with --resume flag)","file":"/config/caddy/autosave.json"}
{"level":"info","ts":1709857499.6951191,"msg":"serving initial configuration"}
{"level":"error","ts":1709857507.8034132,"logger":"http.log.error","msg":"dial tcp: lookup to on 127.0.0.11:53: server misbehaving","request":{"remote_ip":"95.91.242.170","remote_port":"50634","client_ip":"95.91.242.170","proto":"HTTP/2.0","method":"GET","host":"docuseal.staffconnect.agency","uri":"/templates/26","headers":{"Vnd.prefetch":["true"],"Sec-Fetch-Dest":["empty"],"Accept-Language":["de-DE,de;q=0.9,en-US;q=0.8,en;q=0.7"],"Cookie":[],"Sec-Ch-Ua":["\"Chromium\";v=\"122\", \"Not(A:Brand\";v=\"24\", \"Google Chrome\";v=\"122\""],"Sec-Ch-Ua-Mobile":["?0"],"Sec-Fetch-Site":["same-origin"],"Sec-Fetch-Mode":["cors"],"If-None-Match":["W/\"863823ebbd2b9d56879cc97a54f835f2\""],"Accept":["text/html, application/xhtml+xml"],"User-Agent":["Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/122.0.0.0 Safari/537.36"],"Sec-Ch-Ua-Platform":["\"Windows\""],"Referer":["https://docuseal.staffconnect.agency/"],"Accept-Encoding":["gzip, deflate, br, zstd"]},"tls":{"resumed":false,"version":772,"cipher_suite":4865,"proto":"h2","server_name":"docuseal.staffconnect.agency"}},"duration":0.001531383,"status":502,"err_id":"9m72bptd6","err_trace":"reverseproxy.statusError (reverseproxy.go:1248)"}
{"level":"error","ts":1709857508.4188697,"logger":"http.log.error","msg":"dial tcp: lookup to on 127.0.0.11:53: server misbehaving","request":{"remote_ip":"95.91.242.170","remote_port":"50634","client_ip":"95.91.242.170","proto":"HTTP/2.0","method":"GET","host":"docuseal.staffconnect.agency","uri":"/","headers":{"Sec-Ch-Ua":["\"Chromium\";v=\"122\", \"Not(A:Brand\";v=\"24\", \"Google Chrome\";v=\"122\""],"Referer":["https://docuseal.staffconnect.agency/"],"Cookie":[],"If-None-Match":["W/\"0986e8a65bf27fa8f219edccd21e53b9\""],"Accept":["text/html, application/xhtml+xml"],"Sec-Ch-Ua-Mobile":["?0"],"Sec-Fetch-Mode":["cors"],"Accept-Encoding":["gzip, deflate, br, zstd"],"Accept-Language":["de-DE,de;q=0.9,en-US;q=0.8,en;q=0.7"],"User-Agent":["Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/122.0.0.0 Safari/537.36"],"Sec-Ch-Ua-Platform":["\"Windows\""],"Sec-Fetch-Site":["same-origin"],"Sec-Fetch-Dest":["empty"]},"tls":{"resumed":false,"version":772,"cipher_suite":4865,"proto":"h2","server_name":"docuseal.staffconnect.agency"}},"duration":0.001125914,"status":502,"err_id":"sujkzytqa","err_trace":"reverseproxy.statusError (reverseproxy.go:1248)"}
{"level":"error","ts":1709857512.4713018,"logger":"http.log.error","msg":"dial tcp: lookup to on 127.0.0.11:53: server misbehaving","request":{"remote_ip":"95.91.242.170","remote_port":"50634","client_ip":"95.91.242.170","proto":"HTTP/2.0","method":"GET","host":"docuseal.staffconnect.agency","uri":"/","headers":{"Accept":["text/html, application/xhtml+xml"],"User-Agent":["Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/122.0.0.0 Safari/537.36"],"Referer":["https://docuseal.staffconnect.agency/templates/26"],"Accept-Language":["de-DE,de;q=0.9,en-US;q=0.8,en;q=0.7"],"Sec-Fetch-Site":["same-origin"],"Accept-Encoding":["gzip, deflate, br, zstd"],"Cookie":[],"If-None-Match":["W/\"f8d91c5628ac134973f8014e8185a19a\""],"Sec-Ch-Ua":["\"Chromium\";v=\"122\", \"Not(A:Brand\";v=\"24\", \"Google Chrome\";v=\"122\""],"Sec-Ch-Ua-Mobile":["?0"],"Sec-Fetch-Dest":["empty"],"Vnd.prefetch":["true"],"Sec-Ch-Ua-Platform":["\"Windows\""],"Sec-Fetch-Mode":["cors"]},"tls":{"resumed":false,"version":772,"cipher_suite":4865,"proto":"h2","server_name":"docuseal.staffconnect.agency"}},"duration":0.001351834,"status":502,"err_id":"s8d3rbejf","err_trace":"reverseproxy.statusError (reverseproxy.go:1248)"}
{"level":"error","ts":1709857512.701204,"logger":"http.log.error","msg":"dial tcp: lookup to on 127.0.0.11:53: server misbehaving","request":{"remote_ip":"95.91.242.170","remote_port":"50634","client_ip":"95.91.242.170","proto":"HTTP/2.0","method":"GET","host":"docuseal.staffconnect.agency","uri":"/","headers":{"Sec-Fetch-Site":["same-origin"],"Referer":["https://docuseal.staffconnect.agency/templates/26"],"Accept-Encoding":["gzip, deflate, br, zstd"],"Cookie":[],"Sec-Ch-Ua":["\"Chromium\";v=\"122\", \"Not(A:Brand\";v=\"24\", \"Google Chrome\";v=\"122\""],"Sec-Ch-Ua-Mobile":["?0"],"Sec-Fetch-Dest":["document"],"Accept-Language":["de-DE,de;q=0.9,en-US;q=0.8,en;q=0.7"],"Sec-Fetch-Mode":["navigate"],"Sec-Fetch-User":["?1"],"Sec-Ch-Ua-Platform":["\"Windows\""],"Upgrade-Insecure-Requests":["1"],"User-Agent":["Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/122.0.0.0 Safari/537.36"],"Accept":["text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.7"]},"tls":{"resumed":false,"version":772,"cipher_suite":4865,"proto":"h2","server_name":"docuseal.staffconnect.agency"}},"duration":0.001232025,"status":502,"err_id":"bcxi3axqm","err_trace":"reverseproxy.statusError (reverseproxy.go:1248)"}
{"level":"error","ts":1709857515.1383793,"logger":"http.log.error","msg":"dial tcp: lookup to on 127.0.0.11:53: server misbehaving","request":{"remote_ip":"95.91.242.170","remote_port":"50634","client_ip":"95.91.242.170","proto":"HTTP/2.0","method":"GET","host":"docuseal.staffconnect.agency","uri":"/","headers":{"Upgrade-Insecure-Requests":["1"],"Sec-Fetch-Mode":["navigate"],"Accept-Encoding":["gzip, deflate, br, zstd"],"Sec-Fetch-Dest":["document"],"Referer":["https://docuseal.staffconnect.agency/templates/26"],"Accept-Language":["de-DE,de;q=0.9,en-US;q=0.8,en;q=0.7"],"Cookie":[],"Cache-Control":["max-age=0"],"Sec-Ch-Ua":["\"Chromium\";v=\"122\", \"Not(A:Brand\";v=\"24\", \"Google Chrome\";v=\"122\""],"Sec-Fetch-Site":["same-origin"],"Accept":["text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.7"],"Sec-Fetch-User":["?1"],"Sec-Ch-Ua-Mobile":["?0"],"Sec-Ch-Ua-Platform":["\"Windows\""],"User-Agent":["Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/122.0.0.0 Safari/537.36"]},"tls":{"resumed":false,"version":772,"cipher_suite":4865,"proto":"h2","server_name":"docuseal.staffconnect.agency"}},"duration":0.001545654,"status":502,"err_id":"yzk9yq4aw","err_trace":"reverseproxy.statusError (reverseproxy.go:1248)"}
{"level":"error","ts":1709857516.7814813,"logger":"http.log.error","msg":"dial tcp: lookup to on 127.0.0.11:53: server misbehaving","request":{"remote_ip":"95.91.242.170","remote_port":"50634","client_ip":"95.91.242.170","proto":"HTTP/2.0","method":"GET","host":"docuseal.staffconnect.agency","uri":"/templates/25","headers":{"Accept-Encoding":["gzip, deflate, br, zstd"],"Sec-Fetch-Dest":["empty"],"Sec-Ch-Ua":["\"Chromium\";v=\"122\", \"Not(A:Brand\";v=\"24\", \"Google Chrome\";v=\"122\""],"Sec-Ch-Ua-Mobile":["?0"],"User-Agent":["Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/122.0.0.0 Safari/537.36"],"Sec-Fetch-Mode":["cors"],"Vnd.prefetch":["true"],"Sec-Fetch-Site":["same-origin"],"Accept-Language":["de-DE,de;q=0.9,en-US;q=0.8,en;q=0.7"],"Cookie":[],"Accept":["text/html, application/xhtml+xml"],"Sec-Ch-Ua-Platform":["\"Windows\""],"Referer":["https://docuseal.staffconnect.agency/"]},"tls":{"resumed":false,"version":772,"cipher_suite":4865,"proto":"h2","server_name":"docuseal.staffconnect.agency"}},"duration":0.001406184,"status":502,"err_id":"0yxyh52ww","err_trace":"reverseproxy.statusError (reverseproxy.go:1248)"}
{"level":"error","ts":1709857516.922504,"logger":"http.log.error","msg":"dial tcp: lookup to on 127.0.0.11:53: server misbehaving","request":{"remote_ip":"95.91.242.170","remote_port":"50634","client_ip":"95.91.242.170","proto":"HTTP/2.0","method":"GET","host":"docuseal.staffconnect.agency","uri":"/templates/21","headers":{"Sec-Fetch-Mode":["cors"],"Referer":["https://docuseal.staffconnect.agency/"],"Accept-Encoding":["gzip, deflate, br, zstd"],"Accept-Language":["de-DE,de;q=0.9,en-US;q=0.8,en;q=0.7"],"Cookie":[],"If-None-Match":["W/\"bc4753070d1c1e195c1383f803b4c6b8\""],"Sec-Ch-Ua":["\"Chromium\";v=\"122\", \"Not(A:Brand\";v=\"24\", \"Google Chrome\";v=\"122\""],"Vnd.prefetch":["true"],"Sec-Fetch-Site":["same-origin"],"Sec-Fetch-Dest":["empty"],"User-Agent":["Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/122.0.0.0 Safari/537.36"],"Sec-Ch-Ua-Platform":["\"Windows\""],"Accept":["text/html, application/xhtml+xml"],"Sec-Ch-Ua-Mobile":["?0"]},"tls":{"resumed":false,"version":772,"cipher_suite":4865,"proto":"h2","server_name":"docuseal.staffconnect.agency"}},"duration":0.001235384,"status":502,"err_id":"wa1sjfbef","err_trace":"reverseproxy.statusError (reverseproxy.go:1248)"}
{"level":"error","ts":1709857518.1394527,"logger":"http.log.error","msg":"dial tcp: lookup to on 127.0.0.11:53: server misbehaving","request":{"remote_ip":"95.91.242.170","remote_port":"50634","client_ip":"95.91.242.170","proto":"HTTP/2.0","method":"GET","host":"docuseal.staffconnect.agency","uri":"/submissions/27","headers":{"Sec-Ch-Ua-Mobile":["?0"],"Sec-Ch-Ua-Platform":["\"Windows\""],"Sec-Ch-Ua":["\"Chromium\";v=\"122\", \"Not(A:Brand\";v=\"24\", \"Google Chrome\";v=\"122\""],"Sec-Fetch-Site":["same-origin"],"Accept-Language":["de-DE,de;q=0.9,en-US;q=0.8,en;q=0.7"],"Cookie":[],"If-None-Match":["W/\"cc085c1e5304546d52585ade90cceb76\""],"Vnd.prefetch":["true"],"User-Agent":["Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/122.0.0.0 Safari/537.36"],"Sec-Fetch-Mode":["cors"],"Sec-Fetch-Dest":["empty"],"Accept-Encoding":["gzip, deflate, br, zstd"],"Accept":["text/html, application/xhtml+xml"],"Referer":["https://docuseal.staffconnect.agency/templates/21"]},"tls":{"resumed":false,"version":772,"cipher_suite":4865,"proto":"h2","server_name":"docuseal.staffconnect.agency"}},"duration":0.001306194,"status":502,"err_id":"a9q5wbug2","err_trace":"reverseproxy.statusError (reverseproxy.go:1248)"}
{"level":"error","ts":1709857519.5850296,"logger":"http.log.error","msg":"dial tcp: lookup to on 127.0.0.11:53: server misbehaving","request":{"remote_ip":"95.91.242.170","remote_port":"50634","client_ip":"95.91.242.170","proto":"HTTP/2.0","method":"GET","host":"docuseal.staffconnect.agency","uri":"/templates/27","headers":{"User-Agent":["Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/122.0.0.0 Safari/537.36"],"Sec-Ch-Ua-Platform":["\"Windows\""],"Referer":["https://docuseal.staffconnect.agency/"],"Accept-Language":["de-DE,de;q=0.9,en-US;q=0.8,en;q=0.7"],"Accept":["text/html, application/xhtml+xml"],"Accept-Encoding":["gzip, deflate, br, zstd"],"Sec-Fetch-Mode":["cors"],"Sec-Ch-Ua":["\"Chromium\";v=\"122\", \"Not(A:Brand\";v=\"24\", \"Google Chrome\";v=\"122\""],"Sec-Ch-Ua-Mobile":["?0"],"Vnd.prefetch":["true"],"Sec-Fetch-Site":["same-origin"],"Sec-Fetch-Dest":["empty"],"Cookie":[]},"tls":{"resumed":false,"version":772,"cipher_suite":4865,"proto":"h2","server_name":"docuseal.staffconnect.agency"}},"duration":0.001267675,"status":502,"err_id":"f2eek8u15","err_trace":"reverseproxy.statusError (reverseproxy.go:1248)"}

There is a lot more to the docker caddy logs but I suppose those are consequent errors.

3. Caddy version:

v2.7.6 h1:w0NymbG2m9PcvKWsrXO6EEkY9Ru4FJK8uQbYcev1p3A=

4. How I installed and ran Caddy:

as a dockerwith a docker-compose file

a. System environment:

Ubuntu 22.04. Linux VPS

b. Command:

docker-compose up -d

c. Service/unit/compose file:

version: '3'

services:
  app:
    depends_on:
      postgres:
        condition: service_healthy
    image: docuseal/docuseal:latest
    ports:
      - 3000:3000
    volumes:
      - .:/data
    environment:
      FORCE_SSL: 'true'
      DATABASE_URL: postgresql://postgres:postgres@postgres:5432/docuseal

  postgres:
    image: postgres:15
    volumes:
      - './pg_data:/var/lib/postgresql/data'
    environment:
      POSTGRES_USER: postgres
      POSTGRES_PASSWORD: postgres
      POSTGRES_DB: docuseal
    healthcheck:
      test: ["CMD-SHELL", "pg_isready -U postgres"]
      interval: 5s
      timeout: 5s
      retries: 5

  caddy:
    image: caddy:latest
    ports:
      - 80:80
      - 443:443
      - 443:443/udp
    volumes:
      - ./data:/data
      - ./config:/config
      - ./Caddyfile:/etc/caddy/Caddyfile:ro
    environment:
      - HOST=docuseal.staffconnect.agency
      - CADDY_HTTP_TIMEOUT=30s     # Adjust the HTTP timeout as needed
      - CADDY_READ_TIMEOUT=30s     # Adjust the read timeout as needed
      - CADDY_WRITE_TIMEOUT=30s
    extra_hosts:
      - "host.docker.internal:host-gateway"

The Timeouts were set because I read somehwhere that this fixes the DNS 503 Error issue.

d. My complete Caddy config:

freepbx.staffconnect.agency {
  reverse_proxy host.docker.internal:50000
}

docuseal.staffconnect.agency {
  reverse_proxy to app:3000
}

Apache2 is listening in Port 50000. This setup used to work earlier today.

5. Links to relevant resources:

2

Howdy @Leon, welcome to the Caddy community!

Is this verbatim in your Caddyfile?

Caddy will interpret to here as another host and attempt to round-robin load balance between http://to:80 and http://app:3000 with this config. If to isn’t a valid host you’re intending to route to, that would explain the consistent intermittent failure.

See examples: reverse_proxy (Caddyfile directive) — Caddy Documentation

1 Like
3

Hey Whitestrake! thanks for the nice welcoming and fast response <3

so i would just delete the “to” ? I can try it but this configuration used to work before my setting tampering^^

4

Yeah, to shouldn’t be there unless there’s literally a backend host called to that you’re meant to be round-robin load balancing to. Definitely remove it unless you know specifically that you want it there.

1 Like
5

I ran docker-compose up again but the 503 error persists unfortunately :confused:

6

Looking at the errors in your log:

"msg": "dial tcp: lookup to on 127.0.0.11:53: server misbehaving",

Here it’s quite literally telling you that whenever Caddy tries to resolve the hostname to (in lookup to on 127.0.0.1:53, where to is literally what Caddy is trying to look up), your DNS server is chucking a fit because it seems to have no idea wtf to do with that request, haha.

It’s a strange error to see - normally you’d see NXDOMAIN or something like that, but in this case, I guess it’s warranted because .to is a country-code top level domain, so Caddy is getting a kind of DNS response for the TLD lookup that it’s not expecting, and thus Caddy throws the 502 because it can’t route this backend request. It would throw a 502 for NXDOMAIN too, so the end result is the same, it’s just a mildly interesting thing to note.

In these cases, though, it’s throwing "status": 502, rather than 503 errors. Do you have any 503s in the rest of the logs that weren’t posted here?

1 Like
7

caddy fmt --override did the trick for me now it works without causing those errors. Now i just need to get it to work again routing to apache2 as it was before. Because right now when i want to reach freepbx.staffconnect.agency it throws an Error 502 :confused:

8

Try adding debug do your Caddyfile global options and capture those logs to see exactly what Caddy is sending upstream and getting back from upstream.

1 Like
9

I figured it out it works now! :slight_smile: Thanks a lot :slight_smile:

2 Likes
10

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.