Intermittent 502 gateway errors using caddy docker proxy (docker swarm services)

1. The problem I’m having:

I just started having intermittent 502 gateway errors when using caddy docker proxy on my docker swarm cluster. This happened once before (on older version of docker caddy proxy) and a reboot of servers helped (This does NOT help). I recently upgraded docker AND caddy which seems to be possible related? I am having about 200 of these out of 2.5 million requests.

2. Error messages and/or full log output:

docker service logs phppointofsale_caddy_server -t --since 4m | egrep -i ‘502|no such host’

NOTE: is always references php-pos-web-3 server for any of these errors which is where caddy controller is. And this type of 502 error is only happening on phppointofsale_staging_php_point_of_sale_app service only (Other 502 errors are connection reset). This no such host is only on web3 and is still intermitent on this service.

phppointofsale_caddy_server.0.d2s0daer2mkv@docker-php-pos-web-3
{
“level”: “error”,
“ts”: 1755261077.3105364,
“logger”: “http.log.error”,
“msg”: “dial tcp: lookup phppointofsale_staging_php_point_of_sale_app on 127.0.0.11:53: no such host”,
“request”: {
“remote_ip”: “10.0.0.25”,
“remote_port”: “15140”,
“client_ip”: “10.0.0.25”,
“proto”: “HTTP/1.1”,
“method”: “GET”,
“host”: “``demo.phppointofsalestaging.com``”,
“uri”: “/”,
“headers”: {
“Cache-Control”: [
“no-cache”
],
“Cdn-Loop”: [
“cloudflare; loops=1”
],
“Accept”: [
/
],
“X-Forwarded-Proto”: [
“https”
],
“Cf-Ipcountry”: [
“LU”
],
“X-Forwarded-Port”: [
“443”
],
“Accept-Encoding”: [
“gzip, br”
],
“User-Agent”: [
“Site24x7”
],
“X-Forwarded-For”: [
“172.255.250.4, 162.158.110.146”
],
“Cf-Visitor”: [
“{"scheme":"https"}”
],
“Cf-Connecting-Ip”: [
“172.255.250.4”
],
“X-Amzn-Trace-Id”: [
“Root=1-689f2895-02b1eb0819a612bc5dd29641”
],
“Cf-Ray”: [
“96f8b542d85e18af-FRA”
]
}
},
“duration”: 0.003625796,
“status”: 502,
“err_id”: “tdymgt72x”,
“err_trace”: “reverseproxy.statusError (reverseproxy.go:1390)”
}

phppointofsale_caddy_server.0.d2s0daer2mkv@docker-php-pos-web-3
{
“level”: “error”,
“ts”: 1755261077.9752274,
“logger”: “http.log.error”,
“msg”: “dial tcp: lookup phppointofsale_staging_php_point_of_sale_app on 127.0.0.11:53: no such host”,
“request”: {
“remote_ip”: “10.0.0.3”,
“remote_port”: “16744”,
“client_ip”: “10.0.0.3”,
“proto”: “HTTP/1.1”,
“method”: “GET”,
“host”: “``demo.phppointofsalestaging.com``”,
“uri”: “/”,
“headers”: {
“X-Amzn-Trace-Id”: [
“Root=1-689f2895-28844db239fbd4ba65bdffee”
],
“Cf-Connecting-Ip”: [
“5.188.95.23”
],
“X-Forwarded-For”: [
“5.188.95.23, 172.70.153.176”
],
“X-Forwarded-Port”: [
“443”
],
“Cache-Control”: [
“no-cache”
],
“Cdn-Loop”: [
“cloudflare; loops=1”
],
“Accept-Encoding”: [
“gzip, br”
],
“User-Agent”: [
“Site24x7”
],
“X-Forwarded-Proto”: [
“https”
],
“Cf-Ray”: [
“96f8b545bf24d0ed-TLV”
],
“Cf-Visitor”: [
“{"scheme":"https"}”
],
“Accept”: [
/
],
“Cf-Ipcountry”: [
“IL”
]
}
},
“duration”: 0.001099402,
“status”: 502,
“err_id”: “91pahukrz”,
“err_trace”: “reverseproxy.statusError (reverseproxy.go:1390)”
}

3. Caddy version:

Caddy Version v2.10.0 h1:fonubSaQKF1YANl8TXqGcn4IbIRUDdfAkpcsfI/vX5U=
Caddy Docker Proxy v2.10.0

4. How I installed and ran Caddy:

docker stack deploy -c docker-compose-us-east-1.yml phppointofsale --with-registry-auth

On swarm cluster with 5 nodes, 3 fronted nodes and 2 background nodes

via docker hub image https://hub.docker.com/r/lucaslorentz/caddy-docker-proxy

a. System environment:

Ubuntu 22.04.5 LTS (Jammy Jellyfish)
Docker version 28.3.3, build 980b856

b. Command:

docker stack deploy -c docker-compose-us-east-1.yml phppointofsale --with-registry-auth

c. Service/unit/compose file:

The file is below

version: '3.9'
services:
  production_healthcheck:
    logging:
      driver: "local"
    deploy:
      restart_policy:
        condition: any
      mode: global
      labels:
        caddy: "http://"
        caddy.reverse_proxy.trusted_proxies: "private_ranges"
        caddy.reverse_proxy: "{{upstreams}}"
    image: phppointofsale/production-healthcheck
    build:
      context: "production_healthcheck"
    restart: always
    networks:
      - app_network
      - mail
  production_php_point_of_sale_app:
    logging:
      driver: "local"
    deploy:
      mode: global
      restart_policy:
        condition: any
      placement:
        constraints: [node.labels.phppos_app == true]
      labels:
        caddy: "http://*.phppointofsale.com, http://*.phppos.com, http://*.phppointofsale.lk"
        caddy.reverse_proxy.trusted_proxies: "private_ranges"
        caddy.reverse_proxy: "{{upstreams}}"
    image: phppointofsale/production-app
    build:
      context: "production_php_point_of_sale_app"
    restart: always
    env_file:
      - production_php_point_of_sale_app/.env
      - .env
    networks:
      - app_network
      - mail
  production_php_point_of_sale_app_gardenview:
    logging:
      driver: "local"
    deploy:
      mode: global
      restart_policy:
        condition: any
      placement:
        constraints: [node.labels.phppos_app == true]
      labels:
        caddy: "http://gardenview.phppointofsale.com, http://gardenview.phppos.com"
        caddy.reverse_proxy.trusted_proxies: "private_ranges"
        caddy.reverse_proxy: "{{upstreams}}"
    image: phppointofsale/gardenview-pos
    build:
      context: "production_php_point_of_sale_app_gardenview"
    restart: always
    env_file:
      - production_php_point_of_sale_app_gardenview/.env
      - .env
    networks:
      - app_network
      - mail  
  production_php_point_of_sale_app_pinelane:
    logging:
      driver: "local"
    deploy:
      mode: global
      restart_policy:
        condition: any
      placement:
        constraints: [node.labels.phppos_app == true]
      labels:
        caddy: "http://pinelane.phppointofsale.com, http://pinelane.phppos.com"
        caddy.reverse_proxy.trusted_proxies: "private_ranges"
        caddy.reverse_proxy: "{{upstreams}}"
    image: phppointofsale/pinelane-pos
    build:
      context: "production_php_point_of_sale_app_pinelane"
    restart: always
    env_file:
      - production_php_point_of_sale_app_pinelane/.env
      - .env
    networks:
      - app_network
      - mail
  production_php_point_of_sale_app_zayyar:
    logging:
      driver: "local"
    deploy:
      mode: global
      restart_policy:
        condition: any
      placement:
        constraints: [node.labels.phppos_app == true]
      labels:
        caddy: "http://zayyar.phppointofsale.com, http://zayyar.phppos.com"
        caddy.reverse_proxy.trusted_proxies: "private_ranges"
        caddy.reverse_proxy: "{{upstreams}}"
    image: phppointofsale/zayyar-pos
    build:
      context: "production_php_point_of_sale_app_zayyar"
    restart: always
    env_file:
      - production_php_point_of_sale_app_zayyar/.env
      - .env
    networks:
      - app_network
      - mail
  production_php_point_of_sale_app_background:
    logging:
      driver: "local"
    deploy:
      mode: global
      restart_policy:
        condition: any
      placement:
        constraints: [node.labels.phppos_background == true]
      labels:
        caddy: "http://*.phppointofsale.com/index.php/ecommerce/cron, http://*.phppos.com/index.php/ecommerce/cron, http://*.phppointofsale.com/index.php/ecommerce/manual_sync, http://*.phppos.com/index.php/ecommerce/manual_sync, http://*.phppointofsale.com/index.php/cron/*, http://*.phppos.com/index.php/cron/*, http://*.phppointofsale.com/PHP-Point-Of-Sale-Prev/index.php/ecommerce/cron, http://*.phppos.com/PHP-Point-Of-Sale-Prev/index.php/ecommerce/cron, http://*.phppointofsale.com/PHP-Point-Of-Sale-Prev/index.php/ecommerce/manual_sync, http://*.phppos.com/PHP-Point-Of-Sale-Prev/index.php/ecommerce/manual_sync, http://*.phppointofsale.com/PHP-Point-Of-Sale-Prev/index.php/cron/*, http://*.phppos.com/PHP-Point-Of-Sale-Prev/index.php/cron/*"
        caddy.reverse_proxy.trusted_proxies: "private_ranges"
        caddy.reverse_proxy: "{{upstreams}}"
    image: phppointofsale/production-app
    build:
      context: "production_php_point_of_sale_app"
    entrypoint: /bin/sh -c
    command: [ "/var/www/html/docker-entry-point-background.sh" ]
    restart: always
    env_file:
      - production_php_point_of_sale_app/.env
      - .env
    networks:
      - app_network
      - mail
  production_php_point_of_sale_app_customs:
    logging:
      driver: "local"
    deploy:
      mode: global
      restart_policy:
        condition: any
      placement:
        constraints: [node.labels.phppos_app == true]
      labels:
        caddy: http://custom.phppointofsale.com, http://custom.phppos.com
        caddy.reverse_proxy.trusted_proxies: "private_ranges"
        caddy.reverse_proxy: "{{upstreams}}"
    image: phppointofsale/production-app-custom
    build:
      context: "production_php_point_of_sale_app_customs"
    restart: always
    env_file: .env
    networks:
      - app_network
      - mail
  production_php_point_of_sale_help_website:
    logging:
      driver: "local"
    deploy:
      mode: global
      restart_policy:
        condition: any
      placement:
        constraints: [node.labels.phppos_website == true]
      labels:
        caddy: http://help.phppointofsale.com, http://help.phppos.com
        caddy.reverse_proxy.trusted_proxies: "private_ranges"
        caddy.reverse_proxy: "{{upstreams}}"
    image: phppointofsale/help-website
    build:
      context: "production_php_point_of_sale_help_website"
    restart: always
    networks:
      - app_network
      - mail
  production_php_point_of_sale_website:
    logging:
      driver: "local"
    deploy:
      mode: global
      restart_policy:
        condition: any
      placement:
        constraints: [node.labels.phppos_website == true]
      labels:
        caddy: http://www.phppointofsale.com, http://www.phppos.com, http://phppointofsale.com, http://phppos.com
        caddy.reverse_proxy.trusted_proxies: "private_ranges"
        caddy.reverse_proxy: "{{upstreams}}"
    image: phppointofsale/production-website
    build:
      context: "production_php_point_of_sale_website"
    restart: always
    env_file:
      - production_php_point_of_sale_website/.env
      - .env
    networks:
      - app_network
      - mail
  staging_php_point_of_sale_app_background:
    logging:
      driver: "local"
    deploy:
      mode: global
      restart_policy:
        condition: any
      placement:
        constraints: [node.labels.phppos_background == true]
      labels:
        caddy: "http://*.phppointofsalestaging.com/index.php/ecommerce/cron, http://*.phppointofsalestaging.com/index.php/ecommerce/manual_sync, http://*.phppointofsalestaging.com/index.php/cron/*, http://*.phppointofsalestaging.com/PHP-Point-Of-Sale-Prev/index.php/ecommerce/cron, http://*.phppointofsalestaging.com/PHP-Point-Of-Sale-Prev/index.php/ecommerce/manual_sync, http://*.phppointofsalestaging.com/PHP-Point-Of-Sale-Prev/index.php/cron/*"
        caddy.reverse_proxy.trusted_proxies: "private_ranges"
        caddy.reverse_proxy: "{{upstreams}}"
    image: phppointofsale/staging-app
    build:
      context: "staging_php_point_of_sale_app"
    entrypoint: /bin/sh -c
    command: [ "/var/www/html/docker-entry-point-background.sh" ]
    restart: always
    env_file:
      - staging_php_point_of_sale_app/.env
      - .env
    networks:
      - app_network
      - mail
  staging_php_point_of_sale_app:
    logging:
      driver: "local"
    deploy:
      mode: global
      restart_policy:
        condition: any
      placement:
        constraints: [node.labels.phppos_app == true]
      labels:
        caddy: "http://*.phppointofsalestaging.com"
        caddy.reverse_proxy.trusted_proxies: "private_ranges"
        caddy.reverse_proxy: "{{upstreams}}"
    image: phppointofsale/staging-app
    build:
      context: "staging_php_point_of_sale_app"
    restart: always
    env_file:
      - staging_php_point_of_sale_app/.env
      - .env
    networks:
      - app_network
      - mail
  staging_php_point_of_sale_website:
    logging:
      driver: "local"
    deploy:
      mode: global
      restart_policy:
        condition: any
      placement:
        constraints: [node.labels.phppos_website == true]
      labels:
        caddy: http://www.phppointofsalestaging.com, http://phppointofsalestaging.com
        caddy.reverse_proxy.trusted_proxies: "private_ranges"
        caddy.reverse_proxy: "{{upstreams}}"
    image: phppointofsale/staging-website
    build:
      context: "staging_php_point_of_sale_website"
    restart: always
    env_file:
      - staging_php_point_of_sale_website/.env
      - .env
    networks:
      - app_network
      - mail
  newrelic_daemon:
    logging:
      driver: "local"
    deploy:
      restart_policy:
        condition: any
      mode: global
    image: phppointofsale/newrelic-php-daemon
    restart: always
    build:
      context: "newrelic_daemon"
    networks:
      - app_network
  agent:
    logging:
      driver: "local"
    deploy:
      restart_policy:
        condition: any
      mode: global
    image: phppointofsale/newrelic-agent
    build:
      context: "newrelic_infra"
    cap_add:
      - SYS_PTRACE
    network_mode: host
    pid: host
    privileged: true
    volumes:
      - "/:/host:ro"
      - "/var/run/docker.sock:/var/run/docker.sock"
    restart: unless-stopped

  caddy_server:
    image: lucaslorentz/caddy-docker-proxy
    ports:
      - 80:80
    networks:
      - caddy_controller
      - app_network
    environment:
      - CADDY_DOCKER_MODE=server
      - CADDY_CONTROLLER_NETWORK=10.200.200.0/24
    volumes:
      - caddy_data:/data
    deploy:
      restart_policy:
        condition: any
      mode: global
      labels:
        caddy_controlled_server:

  caddy_controller:
    image: lucaslorentz/caddy-docker-proxy
    networks:
      - caddy_controller
      - app_network
    environment:
      - CADDY_DOCKER_MODE=controller
      - CADDY_CONTROLLER_NETWORK=10.200.200.0/24
      - CADDY_DOCKER_PROXY_SERVICE_TASKS=false
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock
    deploy:
      restart_policy:
        condition: any
      placement:
        constraints: [node.role == manager]
  postfix:
    logging:
      driver: "local"
    build:
      context: "postfix"
    deploy:
      restart_policy:
        condition: any
      mode: global
    image: phppointofsale/postfix
    networks:
      - mail
    hostname: postfix
    restart: always
  background_jobs_us_east_1:
    logging:
      driver: "local"
    image: phppointofsale/job-scheduler-us-east-1
    deploy:
      replicas: 1
      restart_policy:
        condition: any
      placement:
        constraints: [node.labels.phppos_background == true]
    restart: always
    env_file:
      - job_scheduler/.env
      - .env
    build:
      context: "job_scheduler"
      args:
          CRONFILE: crontab-us-east-1
  php_my_admin_us_east_1:
    logging:
      driver: "local"
    image: phppointofsale/phpmyadmin-us-east-1
    deploy:
      restart_policy:
        condition: any
      replicas: 1
      labels:
        caddy: http://mysql.phppointofsale.com, http://mysql.phppos.com
        caddy.reverse_proxy.trusted_proxies: "private_ranges"
        caddy.reverse_proxy: "{{upstreams}}"
    restart: always
    build:
      context: "phpmyadmin-us-east-1"
    networks:
      - app_network
      - mail
  pt-kill-db-1:
    logging:
      driver: "local"
    image: phppointofsale/percona-toolkit
    deploy:
      restart_policy:
        condition: any
      replicas: 1
    restart: always
    build:
      context: "percona-toolkit" 
    command: /usr/bin/pt-kill --rds --match-command Query --victims all --match-user phppoint --busy-time 60 --kill --print h=php-pos-db.phppointofsale.com,u=master,p=PASSWORD,P=3306
  pt-kill-db-2:
    logging:
      driver: "local"
    image: phppointofsale/percona-toolkit
    deploy:
      restart_policy:
        condition: any
      replicas: 1
    restart: always
    build:
      context: "percona-toolkit"  
    command: /usr/bin/pt-kill --rds --match-command Query --victims all --match-user phppoint --busy-time 60 --kill --print h=php-pos-db-2.phppointofsale.com,u=master,p=PASSWORD,P=3306
networks:
  caddy_controller:
    driver: overlay
    ipam:
      driver: default
      config:
        - subnet: "10.200.200.0/24"
  app_network:
        driver: overlay
  mail:
      driver: overlay
volumes:
  caddy_data: {}

d. My complete Caddy config:

The file is below

{
  "admin": {
    "listen": "tcp/10.200.200.10:2019"
  },
  "apps": {
    "http": {
      "servers": {
        "srv0": {
          "listen": [
            ":80"
          ],
          "routes": [
            {
              "handle": [
                {
                  "handler": "subroute",
                  "routes": [
                    {
                      "handle": [
                        {
                          "handler": "reverse_proxy",
                          "trusted_proxies": [
                            "192.168.0.0/16",
                            "172.16.0.0/12",
                            "10.0.0.0/8",
                            "127.0.0.1/8",
                            "fd00::/8",
                            "::1"
                          ],
                          "upstreams": [
                            {
                              "dial": "phppointofsale_production_php_point_of_sale_app_gardenview:80"
                            }
                          ]
                        }
                      ]
                    }
                  ]
                }
              ],
              "match": [
                {
                  "host": [
                    "gardenview.phppointofsale.com",
                    "gardenview.phppos.com"
                  ]
                }
              ],
              "terminal": true
            },
            {
              "handle": [
                {
                  "handler": "subroute",
                  "routes": [
                    {
                      "handle": [
                        {
                          "handler": "reverse_proxy",
                          "trusted_proxies": [
                            "192.168.0.0/16",
                            "172.16.0.0/12",
                            "10.0.0.0/8",
                            "127.0.0.1/8",
                            "fd00::/8",
                            "::1"
                          ],
                          "upstreams": [
                            {
                              "dial": "phppointofsale_staging_php_point_of_sale_website:80"
                            }
                          ]
                        }
                      ]
                    }
                  ]
                }
              ],
              "match": [
                {
                  "host": [
                    "www.phppointofsalestaging.com",
                    "phppointofsalestaging.com"
                  ]
                }
              ],
              "terminal": true
            },
            {
              "handle": [
                {
                  "handler": "subroute",
                  "routes": [
                    {
                      "handle": [
                        {
                          "handler": "reverse_proxy",
                          "trusted_proxies": [
                            "192.168.0.0/16",
                            "172.16.0.0/12",
                            "10.0.0.0/8",
                            "127.0.0.1/8",
                            "fd00::/8",
                            "::1"
                          ],
                          "upstreams": [
                            {
                              "dial": "phppointofsale_production_php_point_of_sale_app_pinelane:80"
                            }
                          ]
                        }
                      ]
                    }
                  ]
                }
              ],
              "match": [
                {
                  "host": [
                    "pinelane.phppointofsale.com",
                    "pinelane.phppos.com"
                  ]
                }
              ],
              "terminal": true
            },
            {
              "handle": [
                {
                  "handler": "subroute",
                  "routes": [
                    {
                      "handle": [
                        {
                          "handler": "reverse_proxy",
                          "trusted_proxies": [
                            "192.168.0.0/16",
                            "172.16.0.0/12",
                            "10.0.0.0/8",
                            "127.0.0.1/8",
                            "fd00::/8",
                            "::1"
                          ],
                          "upstreams": [
                            {
                              "dial": "phppointofsale_production_php_point_of_sale_app_customs:80"
                            }
                          ]
                        }
                      ]
                    }
                  ]
                }
              ],
              "match": [
                {
                  "host": [
                    "custom.phppointofsale.com",
                    "custom.phppos.com"
                  ]
                }
              ],
              "terminal": true
            },
            {
              "handle": [
                {
                  "handler": "subroute",
                  "routes": [
                    {
                      "handle": [
                        {
                          "handler": "reverse_proxy",
                          "trusted_proxies": [
                            "192.168.0.0/16",
                            "172.16.0.0/12",
                            "10.0.0.0/8",
                            "127.0.0.1/8",
                            "fd00::/8",
                            "::1"
                          ],
                          "upstreams": [
                            {
                              "dial": "phppointofsale_production_php_point_of_sale_app_zayyar:80"
                            }
                          ]
                        }
                      ]
                    }
                  ]
                }
              ],
              "match": [
                {
                  "host": [
                    "zayyar.phppointofsale.com",
                    "zayyar.phppos.com"
                  ]
                }
              ],
              "terminal": true
            },
            {
              "handle": [
                {
                  "handler": "subroute",
                  "routes": [
                    {
                      "handle": [
                        {
                          "handler": "reverse_proxy",
                          "trusted_proxies": [
                            "192.168.0.0/16",
                            "172.16.0.0/12",
                            "10.0.0.0/8",
                            "127.0.0.1/8",
                            "fd00::/8",
                            "::1"
                          ],
                          "upstreams": [
                            {
                              "dial": "phppointofsale_php_my_admin_us_east_1:80"
                            }
                          ]
                        }
                      ]
                    }
                  ]
                }
              ],
              "match": [
                {
                  "host": [
                    "mysql.phppointofsale.com",
                    "mysql.phppos.com"
                  ]
                }
              ],
              "terminal": true
            },
            {
              "handle": [
                {
                  "handler": "subroute",
                  "routes": [
                    {
                      "handle": [
                        {
                          "handler": "reverse_proxy",
                          "trusted_proxies": [
                            "192.168.0.0/16",
                            "172.16.0.0/12",
                            "10.0.0.0/8",
                            "127.0.0.1/8",
                            "fd00::/8",
                            "::1"
                          ],
                          "upstreams": [
                            {
                              "dial": "phppointofsale_production_php_point_of_sale_help_website:80"
                            }
                          ]
                        }
                      ]
                    }
                  ]
                }
              ],
              "match": [
                {
                  "host": [
                    "help.phppointofsale.com",
                    "help.phppos.com"
                  ]
                }
              ],
              "terminal": true
            },
            {
              "handle": [
                {
                  "handler": "subroute",
                  "routes": [
                    {
                      "handle": [
                        {
                          "handler": "reverse_proxy",
                          "trusted_proxies": [
                            "192.168.0.0/16",
                            "172.16.0.0/12",
                            "10.0.0.0/8",
                            "127.0.0.1/8",
                            "fd00::/8",
                            "::1"
                          ],
                          "upstreams": [
                            {
                              "dial": "phppointofsale_production_php_point_of_sale_website:80"
                            }
                          ]
                        }
                      ]
                    }
                  ]
                }
              ],
              "match": [
                {
                  "host": [
                    "www.phppointofsale.com",
                    "www.phppos.com",
                    "phppointofsale.com",
                    "phppos.com"
                  ]
                }
              ],
              "terminal": true
            },
            {
              "handle": [
                {
                  "handler": "subroute",
                  "routes": [
                    {
                      "handle": [
                        {
                          "handler": "reverse_proxy",
                          "trusted_proxies": [
                            "192.168.0.0/16",
                            "172.16.0.0/12",
                            "10.0.0.0/8",
                            "127.0.0.1/8",
                            "fd00::/8",
                            "::1"
                          ],
                          "upstreams": [
                            {
                              "dial": "phppointofsale_staging_php_point_of_sale_app_background:80"
                            }
                          ]
                        }
                      ]
                    }
                  ]
                }
              ],
              "match": [
                {
                  "host": [
                    "*.phppointofsalestaging.com"
                  ],
                  "path": [
                    "/index.php/ecommerce/cron"
                  ]
                },
                {
                  "host": [
                    "*.phppointofsalestaging.com"
                  ],
                  "path": [
                    "/index.php/ecommerce/manual_sync"
                  ]
                },
                {
                  "host": [
                    "*.phppointofsalestaging.com"
                  ],
                  "path": [
                    "/index.php/cron/*"
                  ]
                },
                {
                  "host": [
                    "*.phppointofsalestaging.com"
                  ],
                  "path": [
                    "/PHP-Point-Of-Sale-Prev/index.php/ecommerce/cron"
                  ]
                },
                {
                  "host": [
                    "*.phppointofsalestaging.com"
                  ],
                  "path": [
                    "/PHP-Point-Of-Sale-Prev/index.php/ecommerce/manual_sync"
                  ]
                },
                {
                  "host": [
                    "*.phppointofsalestaging.com"
                  ],
                  "path": [
                    "/PHP-Point-Of-Sale-Prev/index.php/cron/*"
                  ]
                }
              ],
              "terminal": true
            },
            {
              "handle": [
                {
                  "handler": "subroute",
                  "routes": [
                    {
                      "handle": [
                        {
                          "handler": "reverse_proxy",
                          "trusted_proxies": [
                            "192.168.0.0/16",
                            "172.16.0.0/12",
                            "10.0.0.0/8",
                            "127.0.0.1/8",
                            "fd00::/8",
                            "::1"
                          ],
                          "upstreams": [
                            {
                              "dial": "phppointofsale_staging_php_point_of_sale_app:80"
                            }
                          ]
                        }
                      ]
                    }
                  ]
                }
              ],
              "match": [
                {
                  "host": [
                    "*.phppointofsalestaging.com"
                  ]
                }
              ],
              "terminal": true
            },
            {
              "handle": [
                {
                  "handler": "subroute",
                  "routes": [
                    {
                      "handle": [
                        {
                          "handler": "reverse_proxy",
                          "trusted_proxies": [
                            "192.168.0.0/16",
                            "172.16.0.0/12",
                            "10.0.0.0/8",
                            "127.0.0.1/8",
                            "fd00::/8",
                            "::1"
                          ],
                          "upstreams": [
                            {
                              "dial": "phppointofsale_production_php_point_of_sale_app_background:80"
                            }
                          ]
                        }
                      ]
                    }
                  ]
                }
              ],
              "match": [
                {
                  "host": [
                    "*.phppointofsale.com",
                    "*.phppos.com"
                  ],
                  "path": [
                    "/index.php/ecommerce/cron"
                  ]
                },
                {
                  "host": [
                    "*.phppointofsale.com",
                    "*.phppos.com"
                  ],
                  "path": [
                    "/index.php/ecommerce/manual_sync"
                  ]
                },
                {
                  "host": [
                    "*.phppointofsale.com",
                    "*.phppos.com"
                  ],
                  "path": [
                    "/index.php/cron/*"
                  ]
                },
                {
                  "host": [
                    "*.phppointofsale.com",
                    "*.phppos.com"
                  ],
                  "path": [
                    "/PHP-Point-Of-Sale-Prev/index.php/ecommerce/cron"
                  ]
                },
                {
                  "host": [
                    "*.phppointofsale.com",
                    "*.phppos.com"
                  ],
                  "path": [
                    "/PHP-Point-Of-Sale-Prev/index.php/ecommerce/manual_sync"
                  ]
                },
                {
                  "host": [
                    "*.phppointofsale.com",
                    "*.phppos.com"
                  ],
                  "path": [
                    "/PHP-Point-Of-Sale-Prev/index.php/cron/*"
                  ]
                }
              ],
              "terminal": true
            },
            {
              "handle": [
                {
                  "handler": "subroute",
                  "routes": [
                    {
                      "handle": [
                        {
                          "handler": "reverse_proxy",
                          "trusted_proxies": [
                            "192.168.0.0/16",
                            "172.16.0.0/12",
                            "10.0.0.0/8",
                            "127.0.0.1/8",
                            "fd00::/8",
                            "::1"
                          ],
                          "upstreams": [
                            {
                              "dial": "phppointofsale_production_php_point_of_sale_app:80"
                            }
                          ]
                        }
                      ]
                    }
                  ]
                }
              ],
              "match": [
                {
                  "host": [
                    "*.phppointofsale.com",
                    "*.phppos.com",
                    "*.phppointofsale.lk"
                  ]
                }
              ],
              "terminal": true
            },
            {
              "handle": [
                {
                  "handler": "subroute",
                  "routes": [
                    {
                      "handle": [
                        {
                          "handler": "reverse_proxy",
                          "trusted_proxies": [
                            "192.168.0.0/16",
                            "172.16.0.0/12",
                            "10.0.0.0/8",
                            "127.0.0.1/8",
                            "fd00::/8",
                            "::1"
                          ],
                          "upstreams": [
                            {
                              "dial": "phppointofsale_production_healthcheck:80"
                            }
                          ]
                        }
                      ]
                    }
                  ]
                }
              ],
              "terminal": true
            }
          ]
        }
      }
    }
  }
}

5. Links to relevant resources:

I do not have any of this


127.0.0.11 is your Docker’s internal DNS, and the errors is telling you that your Docker’s internal DNS knows nothing about phppointofsale_staging_php_point_of_sale_app.

Is there a container with such name running?

1 Like

Yes it is running and see no signs of the container rebooting on any server. It is very intermittent.

What I am going to try is take a backup of a server without issue and replace that server with issue and see what happens. If I still have issue then will come back

Actually going to try this first

docker node update --availability drain docker-php-pos-web-3
docker swarm leave --force
sudo systemctl stop docker
sudo rm -rf /var/lib/docker/network/files/*
sudo systemctl start docker

rejoin swarm, then:

docker node update --availability active docker-php-pos-web-3

1 Like

Leaving the swarm, stopping docker, removing network files, starting docker, then re-join swarm fixed it