Https subpath routing issues on azure container instance (docker)

janvda · September 27, 2021, 12:31pm

1. Caddy version (`caddy version`):

docker hub version: caddy:2.4.5-alpine

/srv # caddy version
v2.4.5 h1:P1mRs6V2cMcagSPn+NWpD+OEYUYLIf6ecOa48cFGeUg=

2. How I run Caddy:

I am running caddy on azure as a docker container in my azure container instance.

a. System environment:

Azure Container Instance (Docker)

b. Command:

The following azure command is used to create my 3 containers (one of them is caddy) on azure.

az container create --resource-group test_jan --file niamondo1.yaml

c. Service/unit/compose file:

My azure yaml file niamondo1.yaml (I have masked/replaced the irrelevant part ) =

api-version: 2021-03-01
location: westeurope
name: niamondo1
properties:
  containers:
    - name: niamondo
      properties:
        image: janvda/serviceA
        #ports:
        #  - port: 1880
        #    protocol: TCP
        resources:
          requests:
            cpu: 0.4
            memoryInGB: 0.4
          limits:
            cpu: 1
            memoryInGB: 1        
    - name: niamondo2
      properties:
        image: janvda/serviceB
        ports:
          - port: 1881
            protocol: TCP
        resources:
          requests:
            cpu: 0.4
            memoryInGB: 0.4
          limits:
            cpu: 1
            memoryInGB: 1        
    - name: caddy
      properties:
        image: caddy:2.4.5-alpine
        #command: [ 'caddy', 'run' ]
        ports:
          - port: 80
            protocol: TCP
          - port: 443
            protocol: TCP
        resources:
          requests:
            cpu: 0.2
            memoryInGB: 0.2
          limits:
            cpu: 0.5
            memoryInGB: 0.5
        volumeMounts:
         - name: caddyfile-volume
           mountPath: /etc/caddy
  ipAddress:
    ports:
      - port: 80
        protocol: TCP
      - port: 443
        protocol: TCP
    type: Public
    dnsNameLabel: niamondo100
  osType: Linux
  volumes:
   - name: caddyfile-volume
     secret:
       Caddyfile: bmlhbW9uZG8xMDAud2VzdGV1cm9wZS5henVyZWNvbnRhaW5lci5pbyB7CiAgICBoYW5kbGVfcGF0aCAvb25lLyogewogICAgICAgcmV2ZXJzZV9wcm94eSBsb2NhbGhvc3Q6MTg4MAogICAgfQogICAgaGFuZGxlX3BhdGggL3R3by8qIHsKICAgICAgIHJldmVyc2VfcHJveHkgbG9jYWxob3N0OjE4ODEKICAgIH0KfQo=
tags: null
type: Microsoft.ContainerInstance/containerGroups

d. My complete Caddyfile or JSON config:

nia....westeurope.azurecontainer.io {
        handle_path /one/* {
                reverse_proxy localhost:1880
        }
        handle_path /two/* {
                reverse_proxy localhost:1881
        }
}

3. The problem I’m having:

I have 3 containers in my azure container instance:

caddy
serviceA (niamondo)
serviceB (niamondo2)

The Caddyfile assure that

URLs like https://nia....westeurope.azurecontainer.io/one/* are properly routed to serviceA
URLs like https://nia....westeurope.azurecontainer.io/two/* are properly routed to serviceB

but I have following routing issues (maybe all having the same root cause):

https://nia....westeurope.azurecontainer.io/one won’t get routed to serviceA. I would like that https://nia....westeurope.azurecontainer.io/one gets me to same page as https://nia....westeurope.azurecontainer.io/one/.
https://nia....westeurope.azurecontainer.io/one/worldmap doesn’t get me to the right page. I need to add a / to get me to the right page. Note that this terminating / was not needed when I access serviceA using a Caddyfile that has the reverse_proxy part without a handle_path
serviceA is returning a page with relative links but these links don’t have the /one/ subpath. So when clicking on these links I navigate to a non-existing page.

I am also wondering if there is a command that allows to show the current configuration (Caddyfile or Json) that caddy is using.

4. Error messages and/or full log output:

none

5. What I already tried:

I also tried following Caddyfile but this didn’t solve any of the problems.

nia....westeurope.azurecontainer.io {
    handle_path /one* {
       reverse_proxy localhost:1880
    }
    handle_path /two* {
       reverse_proxy localhost:1881
    }
}

6. Links to relevant resources:

none

francislavoie · September 27, 2021, 4:10pm

This is a problem fundamental to proxying apps that are not configured to be proxied under a subpath. See this article, which explains:

The solution is to use a subdomain for each of the services.

janvda · September 27, 2021, 6:34pm

@francislavoie , thanks a lot for the response.

The article mentions also another solution based on using header and reverse_proxy directive.
I would like to give that approach a shot but I don’t know exactly how I must configure it. Can you help me with that ?

I am not specialist in this domain but I think that all my problems are related to redirection of pages to another page.

So in the example below you see that /one/worldmap becomes redirected to /worldmap/ while it should be redirected to /one/worldmap/

Jans-MacBook-Pro:~ jan$ curl -I https://niamondo100.westeurope.azurecontainer.io/one/worldmap
HTTP/2 301 
access-control-allow-origin: *
content-security-policy: default-src 'none'
content-type: text/html; charset=UTF-8
date: Mon, 27 Sep 2021 18:23:37 GMT
location: /worldmap/
server: Caddy
vary: Accept-Encoding
x-content-type-options: nosniff
x-powered-by: Express
content-length: 183

Jans-MacBook-Pro:~ jan$

francislavoie · September 27, 2021, 10:24pm

You’ll just be playing whack-a-mole with different path issues if you take that approach.

The fundamental problem is that your upstream application doesn’t know that it’s supposed to be running under a subpath, so it triggers redirects.

The best/easiest fix is to run each app under each their own subdomains, so they can use the paths they want.

Another option, if you have control of the code, then the upstream app should be modified/configured such that it knows that it’s running under a different base path. This will make sure that all URLs it creates are prefixed before being written to the response (like links in HTML, Location headers, etc)

Whitestrake · September 27, 2021, 11:26pm

You will need to familiarize yourself with the usage of:

Then you will need to carefully inspect your application, identify every instance of incorrect links, and manually configure each replacement such that Caddy corrects all the invalid asset links.

@francislavoie is not exaggerating when he says you will be playing whack-a-mole.

I’m afraid that in general, this is not the kind of problem that someone is going to be able to give you a simple answer for - unless they are exceedingly generous with their time and expertise, or you’re paying them to solve the problem for you.

janvda · September 28, 2021, 7:02am

I understand that caddy configuration is fairly simply in case of subdomains but the problem with that is that on azure cloud I cannot define subdomains. This means that I need to create/register my own domain, create subdomains for it and assure that it resolves to the IP address of my azure container instance. The thing is that the IP addres of my azure container instance might change as I am planning to run it only when needed. In other words to assure these subdomains work, I need to implement an automatic update of my domain name IP address that becomes at least triggered when redeploying my container instance. So I prefer not to go with the subdomain solution due to the extra effort and complexity introduced by this automatic IP address update.

I understand that for some applications it might not be an easy problem but in my case the number of redirects seem very limited and once I land on the correct page from then onwards the application is working as expected. So I am not convinced I am playing whack-a-mole.

I have read those pages several times but they don’t provide actual examples of how they can be used to fix redirection issues when using subpaths. So some help with that would be welcome - especially as this is not my domain of expertise.
I also tried to fix some of the issues through redir, but redir works for the subpaths /one and /one/docs but doesn’t work for subpaths /one/worldmap and /one/mui. It is very strange that it doesn’t work for /one/mui but does work for /one as I am redirecting to exactly the same path = /one/mui/. I don’t understand why this is not working - can you help me with that ?

My Caddyfile:

niamondo100.westeurope.azurecontainer.io {
    redir /one          /one/mui/
    redir /one/worldmap /one/worldmap/
    redir /one/mui      /one/mui/
    redir /one/docs     /one/docs/index.md
    handle_path /one/* {
       reverse_proxy localhost:1880 
    }

    redir /two          /two/mui/
    redir /two/worldmap /two/worldmap/
    redir /two/mui      /two/mui/
    redir /two/docs     /two/docs/index.md
    handle_path /two/* {
       reverse_proxy localhost:1881 
    }
}

Whitestrake · September 28, 2021, 7:22am

That’s because there is no specific usage for fixing redirection issues with subpaths. This use case falls under the general use of those tools.

For replace-response, you need to find instances of the incorrect links in the response, and replace them with the correct link. That means: inspecting the HTML of the response, determining what the incorrect part looks like, and configuring Caddy to replace it in-flight with the correct response. The general example is sufficient for what you need.

It won’t work for location headers, so you need to fix that with reverse-proxy, using header manipulation: reverse_proxy (Caddyfile directive) — Caddy Documentation. That means the same as above, but inspecting the response headers, rather than the response body itself, and correcting the contents of the (typically) Location header. The specification in the Syntax section is what you need:

    # header manipulation
    header_up   [+|-]<field> [<value|regexp> [<replacement>]]
    header_down [+|-]<field> [<value|regexp> [<replacement>]]

janvda · September 28, 2021, 7:36am

janvda:

I also tried to fix some of the issues through redir, but redir works for the subpaths /one and /one/docs but doesn’t work for subpaths /one/worldmap and /one/mui. It is very strange that it doesn’t work for /one/mui but does work for /one as I am redirecting to exactly the same path = /one/mui/. I don’t understand why this is not working - can you help me with that ?

My Caddyfile:
niamondo100.westeurope.azurecontainer.io {
    redir /one          /one/mui/
    redir /one/worldmap /one/worldmap/
    redir /one/mui      /one/mui/
    redir /one/docs     /one/docs/index.md
    handle_path /one/* {
       reverse_proxy localhost:1880 
    }

    redir /two          /two/mui/
    redir /two/worldmap /two/worldmap/
    redir /two/mui      /two/mui/
    redir /two/docs     /two/docs/index.md
    handle_path /two/* {
       reverse_proxy localhost:1881 
    }
}

Before diving into header manipulation, I would like first to understand why the redir is not working for the /one/mui path and is working for the /one path while they both redirect to exactly the same path /one/mui/ which is also confirmed by bellow curl commands.

Jans-MBP:~ jan$ curl -I https://niamondo100.westeurope.azurecontainer.io/one
HTTP/2 302 
location: /one/mui/
server: Caddy
date: Tue, 28 Sep 2021 07:35:00 GMT

Jans-MBP:~ jan$ curl -I https://niamondo100.westeurope.azurecontainer.io/one/mui
HTTP/2 302 
location: /one/mui/
server: Caddy
date: Tue, 28 Sep 2021 07:35:03 GMT

Jans-MBP:~ jan$

So when entering https://niamondo100.westeurope.azurecontainer.io/one/mui it seems that redir is not working as browser tries to show non existing page https://niamondo100.westeurope.azurecontainer.io/mui/ =

janvda · September 28, 2021, 8:17am

It seems that redir is not working in browser if the new path is just the original path with / added at the end. To fix this I have changed the original path (so instead of /one/worldmap I have taken /one/map which will be redirected to /one/worldmap/).

Here the working Caddyfile

niamondo100.westeurope.azurecontainer.io {
    redir /one          /one/mui/
    redir /one/map      /one/worldmap/
    redir /one/game     /one/mui/
    redir /one/docs     /one/docs/index.md
    handle_path /one/* {
       reverse_proxy localhost:1880 
    }

    redir /two          /two/mui/
    redir /two/map      /two/worldmap/
    redir /two/game     /two/mui/
    redir /two/docs     /two/docs/index.md
    handle_path /two/* {
       reverse_proxy localhost:1881 
    }
}

system · October 27, 2021, 12:31pm

This topic was automatically closed after 30 days. New replies are no longer allowed.