How to handle sub-subdomains and an ssh-tunnel?

neuronflow · March 29, 2020, 9:17am

1. My Caddy version (`caddy version`):

latest v2 alpine docker

2. How I run Caddy:

docker-compose

a. System environment:

v2 latest alpine docker

b. Command:

CURRENT_UID=$(id -u):$(id -g) docker-compose up

c. Service/unit/compose file:

# run this with: "  CURRENT_UID=$(id -u):$(id -g) docker-compose up ""
version: "3"
services:
  caddykraken:
    image: caddy_kraken
    container_name: caddykraken
    hostname: caddykraken
    user: root
    # user: ${CURRENT_UID}
    ports:
      - 443:443
      - 80:80
      # - 2020:2020
    volumes:
      # Just a note - as of the latest caddy/caddy images, these locations are now /config/caddy and /data/caddy. See the (new!) docs for some details: https://github.com/caddyserver/caddy-docker#️-a-note-about-persisted-data 1
      - "./caddy_secrets/data_lets_encrypt_storage:/data"
      - "./caddy_secrets/config_storage:/config"

d. My complete Caddyfile or JSON config:

{	
    email   zournyque@gmail.com
    # acme_ca https://acme-staging-v02.api.letsencrypt.org/directory
}

translatum.xyz {
    respond "let the neurons flow - translatum.xyz!"
}

# perception studies
perception.translatum.xyz {
    reverse_proxy {
        to perceptionist:8888
    }
}


glio.translatum.xyz {
    reverse_proxy {
        to bratsstarz:80
    }
}


pneumonia.translatum.xyz {
    reverse_proxy {
        to pneustarz:80
    }
}


# kraken
corona.translatum.xyz {
    reverse_proxy {
        to coronakraken:80
    }
}

kraken.translatum.xyz {
    respond "high 8 - long live the kraken on translatum.xyz!"
}

corona.kraken.translatum.xyz {
    reverse_proxy {
        to localhost:8080
    }
}

redis.kraken.translatum.xyz {
    reverse_proxy {
        to localhost:6379
    }
}

aioh.kraken.translatum.xyz {
    reverse_proxy {
        to localhost:8889
    }
}

formidable.kraken.translatum.xyz {
    reverse_proxy {
        to localhost:3020
    }
}

3. The problem I’m having:

Hi,

my backend servers, are running at a server with no public ip to be accessed from the internet. therefore I have setup an ssh-tunnel to my frontend server, which has this connection.
The various backend servers should all be reachable with sub-subdomains of kraken.translatum.xyz, e.g. redis.translatum.xyz. However caddy is complaining about missing dns entries. Unfortunately at my domain provider (strato.de ) I can only setup A and AAAA records for domains and subdomains, not sub-subdomains.

4. Error messages and/or full log output:

percy            | 2020/03/29 08:35:53 [INFO] [aioh.kraken.translatum.xyz] acme: Could not find solver for: tls-alpn-01
percy            | 2020/03/29 08:35:53 [INFO] [aioh.kraken.translatum.xyz] acme: use http-01 solver
percy            | 2020/03/29 08:35:53 [INFO] [aioh.kraken.translatum.xyz] acme: Trying to solve HTTP-01
percy            | 2020/03/29 08:35:53 [INFO] [formidable.kraken.translatum.xyz] acme: Obtaining bundled SAN certificate given a CSR
percy            | 2020/03/29 08:35:53 [INFO] [formidable.kraken.translatum.xyz] AuthURL: https://acme-staging-v02.api.letsencrypt.org/acme/authz-v3/46136226
percy            | 2020/03/29 08:35:53 [INFO] [formidable.kraken.translatum.xyz] acme: Could not find solver for: tls-alpn-01
percy            | 2020/03/29 08:35:53 [INFO] [formidable.kraken.translatum.xyz] acme: use http-01 solver
percy            | 2020/03/29 08:35:53 [INFO] [formidable.kraken.translatum.xyz] acme: Trying to solve HTTP-01
percy            | 2020/03/29 08:35:53 [INFO] Deactivating auth: https://acme-staging-v02.api.letsencrypt.org/acme/authz-v3/46136225
percy            | 2020/03/29 08:35:53 [INFO] Unable to deactivate the authorization: https://acme-staging-v02.api.letsencrypt.org/acme/authz-v3/46136225
percy            | 2020/03/29 08:35:53 [ERROR] acme: Error -> One or more domains had a problem:
percy            | [aioh.kraken.translatum.xyz] acme: error: 400 :: urn:ietf:params:acme:error:dns :: DNS problem: NXDOMAIN looking up A for aioh.kraken.translatum.xyz - check that a DNS record exists for this domain, url:
percy            |  (challenge=http-01 remaining=[])
percy            | 2020/03/29 08:35:54 [INFO] [corona.kraken.translatum.xyz] acme: Obtaining bundled SAN certificate given a CSR
percy            | 2020/03/29 08:35:54 [INFO] [corona.kraken.translatum.xyz] AuthURL: https://acme-staging-v02.api.letsencrypt.org/acme/authz-v3/46136230
percy            | 2020/03/29 08:35:54 [INFO] [corona.kraken.translatum.xyz] acme: Could not find solver for: tls-alpn-01
percy            | 2020/03/29 08:35:54 [INFO] [corona.kraken.translatum.xyz] acme: use http-01 solver
percy            | 2020/03/29 08:35:54 [INFO] [corona.kraken.translatum.xyz] acme: Trying to solve HTTP-01
percy            | 2020/03/29 08:35:55 [ERROR] attempt 2: [redis.kraken.translatum.xyz] Obtain: [redis.kraken.translatum.xyz] acme: Error -> One or more domains had a problem:
percy            | [redis.kraken.translatum.xyz] acme: error: 400 :: urn:ietf:params:acme:error:dns :: DNS problem: NXDOMAIN looking up A for redis.kraken.translatum.xyz - check that a DNS record exists for this domain, url:
percy            |  - retrying in 2m0s (1m17.674394657s/720h0m0s elapsed)...
percy            | 2020/03/29 08:35:55 [INFO] Deactivating auth: https://acme-staging-v02.api.letsencrypt.org/acme/authz-v3/46136230
percy            | 2020/03/29 08:35:55 [INFO] Unable to deactivate the authorization: https://acme-staging-v02.api.letsencrypt.org/acme/authz-v3/46136230
percy            | 2020/03/29 08:35:55 [ERROR] acme: Error -> One or more domains had a problem:
percy            | [corona.kraken.translatum.xyz] acme: error: 400 :: urn:ietf:params:acme:error:dns :: DNS problem: NXDOMAIN looking up A for corona.kraken.translatum.xyz - check that a DNS record exists for this domain, url:
percy            |  (challenge=http-01 remaining=[])
percy            | 2020/03/29 08:35:55 [ERROR] attempt 2: [aioh.kraken.translatum.xyz] Obtain: [aioh.kraken.translatum.xyz] acme: Error -> One or more domains had a problem:
percy            | [aioh.kraken.translatum.xyz] acme: error: 400 :: urn:ietf:params:acme:error:dns :: DNS problem: NXDOMAIN looking up A for aioh.kraken.translatum.xyz - check that a DNS record exists for this domain, url:
percy            |  - retrying in 2m0s (1m18.487564881s/720h0m0s elapsed)...
percy            | 2020/03/29 08:35:57 [ERROR] attempt 2: [corona.kraken.translatum.xyz] Obtain: [corona.kraken.translatum.xyz] acme: Error -> One or more domains had a problem:
percy            | [corona.kraken.translatum.xyz] acme: error: 400 :: urn:ietf:params:acme:error:dns :: DNS problem: NXDOMAIN looking up A for corona.kraken.translatum.xyz - check that a DNS record exists for this domain, url:
percy            |  - retrying in 2m0s (1m20.251783503s/720h0m0s elapsed)...
percy            | 2020/03/29 08:35:58 [INFO] Deactivating auth: https://acme-staging-v02.api.letsencrypt.org/acme/authz-v3/46136226
percy            | 2020/03/29 08:35:58 [INFO] Unable to deactivate the authorization: https://acme-staging-v02.api.letsencrypt.org/acme/authz-v3/46136226
percy            | 2020/03/29 08:35:58 [ERROR] acme: Error -> One or more domains had a problem:
percy            | [formidable.kraken.translatum.xyz] acme: error: 400 :: urn:ietf:params:acme:error:dns :: DNS problem: NXDOMAIN looking up A for formidable.kraken.translatum.xyz - check that a DNS record exists for this domain, url:
percy            |  (challenge=http-01 remaining=[])
percy            | 2020/03/29 08:36:00 [ERROR] attempt 2: [formidable.kraken.translatum.xyz] Obtain: [formidable.kraken.translatum.xyz] acme: Error -> One or more domains had a problem:

5. What I already tried:

I have no clue what I should try…maybe I could change my backend servers to things like…
kraken.translatum.xyz/redis ? but ideally I would like to stick to the sub-subdomain scheme.

6. Links to relevant resources:

neuronflow · March 29, 2020, 12:40pm

I managed to setup subdomains on my provider now and the certificates seem to be working as far as I can tell. However forwarding through the tunnel does not.

On server A I am running caddy and reverse proxy requests to corona.kraken.translatum.xyz to localhost:8080.

From server B I ssh tunnel to server A and map 8080:localhost8080 , I also run a docker with nginx serving on 8080.

Requests to corona.kraken.translatum.xyz give me a 502 bad gateway:

percy            | {"level":"error","ts":1585485590.7365258,"logger":"http.log.error","msg":"dial tcp 127.0.0.1:8080: connect: connection refused","request":{"method":"GET","uri":"/","proto":"HTTP/1.1","remote_addr":"62.216.209.86:44298","host":"corona.kraken.translatum.xyz","headers":{"Accept":["*/*"],"Cache-Control":["no-cache"],"Postman-Token":["c881aba2-65f7-4c62-8e92-c5d335af5ed5"],"Accept-Encoding":["gzip, deflate, br"],"Connection":["keep-alive"],"User-Agent":["PostmanRuntime/7.24.0"]},"tls":{"resumed":false,"version":771,"ciphersuite":49196,"proto":"","proto_mutual":true,"server_name":"corona.kraken.translatum.xyz"}},"status":502,"err_id":"k7937tb03","err_trace":"reverseproxy.(*Handler).ServeHTTP (reverseproxy.go:363)"}```

francislavoie · March 29, 2020, 5:23pm

Is localhost what you actually want here? Are these all services you’re running inside the same container as Caddy? If not, you should set the upstream to your other container names, using the DNS that docker-compose sets up for you.

Also, minor thing, but it’s unnecessary to use the to subdirective, you can turn those all into one-liners.

reverse_proxy other-container:8080 for example.

neuronflow · March 29, 2020, 6:11pm

Hi,
thanks for the response. Yes, my caddy resides inside a docker on server A and maps services to other dockers running on it using the service id. Server B is connected with an ssh tunnel towards A. The four dockers from server B ,which has no direct access, are running on ports 8080, 6370, 8889, 3020. Therefore I established an ssh tunnel for these ports.

I can access the dockers on the host system with curl localhost:8080 for example. But you might be right, one reason I cannot reach those might be that caddy resides inside a docker?

I believe this might also be relevant here? Is there a replacement for transparent in v2 already?

francislavoie · March 29, 2020, 6:20pm

You set up ssh tunnels inside the Docker container or on the host? “localhost” inside the container will only see things that are also running inside that container.

transparent is implicit in v2. See the reverse_proxy documentation

neuronflow · March 29, 2020, 7:25pm

You set up ssh tunnels inside the Docker container or on the host? “localhost” inside the container will only see things that are also running inside that container.

This is probably my problem. My ssh tunnel is setup on the host. How should I reference addresses on the host or is there a way to ssh tunnel inside the docker container?

Is it possible to proxy towards docker containers running on another machine?

francislavoie · March 29, 2020, 7:54pm

At this point this is more of a question for how to use Docker than Caddy. I’m not sure I’ll be able to help you find the appropriate solution.

One option is to use host networking, but doing that throws away a lot of the advantages of Docker.

Another option might be to run the ssh tunnel in another Docker container and proxy through that container. You’ll need to do your own research on that.

neuronflow · March 29, 2020, 9:27pm

Thank you!
I will try the things suggested in this SO:

francislavoie · March 29, 2020, 9:42pm

So apparently things have changed since I last looked into this. Apparently in the next major version of Docker, i.e. 20.04 I think, there should be support for adding this option to containers:

extra_hosts:
  - "host.docker.internal:host-gateway"

Then instead of localhost for your proxy, you could use host.docker.internal:3020 for example.

See the following:

https://github.com/moby/moby/pull/40007

neuronflow · March 29, 2020, 9:50pm

I am trying to implement the solution based on this atm:

# inspiration: https://stackoverflow.com/a/38753971/3485363
export DOCKERHOST=$(ifconfig | grep -E "([0-9]{1,3}\.){3}[0-9]{1,3}" | grep -v 127.0.0.1 | awk '{ print $2 }' | cut -f2 -d: | head -n1)
echo DOCKERHOST: $DOCKERHOST
export CURRENT_UID=$(id -u):$(id -g)
echo CURRENT_UID: $CURRENT_UID
sudo -E docker-compose up

then in docker-compose:

myapp:
  build: .
  ports:
    - "80:80"
  extra_hosts:
    - "dockerhost:$DOCKERHOST"

If this does not work I will try this:

neuronflow · March 29, 2020, 10:21pm

hmmm my tunnel seems to be reachable from within the caddy docker now using the above technique:

/etc # cat hosts
127.0.0.1	localhost
::1	localhost ip6-localhost ip6-loopback
fe00::0	ip6-localnet
ff00::0	ip6-mcastprefix
ff02::1	ip6-allnodes
ff02::2	ip6-allrouters
172.24.0.1	dockerhost
172.24.0.2	percy
/etc # curl 172.24.0.1:8080
<!DOCTYPE html><html lang=en><head><meta charset=utf-8><meta http-equiv=X-UA-Compatible content="IE=edge"><meta name=viewport content="width=device-width,initial-scale=1"><link rel=icon href=/favicon.ico><link rel=stylesheet href=https://cdn.materialdesignicons.com/2.5.94/css/materialdesignicons.min.css><title>corona_kraken</title><link href=/js/about.835a0559.js rel=prefetch><link href=/css/app.88a1ad02.css rel=preload as=style><link href=/css/chunk-vendors.f1f97ad3.css rel=preload as=style><link href=/js/app.3ae49eac.js rel=preload as=script><link href=/js/chunk-vendors.f22c4265.js rel=preload as=script><link href=/css/chunk-vendors.f1f97ad3.css rel=stylesheet><link href=/css/app.88a1ad02.css rel=stylesheet></head><body><noscript><strong>We're sorry but corona_kraken doesn't work properly without JavaScript enabled. Please enable it to continue.</strong></noscript><div id=app></div><script src=/js/chunk-vendors.f22c4265.js></script><script src=/js/app.3ae49eac.js></script></body></html>/etc #help

however caddy still blocks requests with 502 what am I doing wrong?
Caddyfile:

corona.kraken.translatum.xyz {
    reverse_proxy {
        to dockerhost:8080
    }
}

output:

percy            | {"level":"error","ts":1585520029.8451645,"logger":"http.log.error","msg":"dial tcp 127.0.0.1:8080: connect: connection refused","request":{"method":"GET","uri":"/","proto":"HTTP/2.0","remote_addr":"62.216.209.86:44305","host":"corona.kraken.translatum.xyz","headers":{"User-Agent":["Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_4) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/80.0.3987.149 Safari/537.36"],"Accept":["text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.9"],"Sec-Fetch-User":["?1"],"Accept-Encoding":["gzip, deflate, br"],"Accept-Language":["en-US,en;q=0.9,de-DE;q=0.8,de;q=0.7"],"Upgrade-Insecure-Requests":["1"],"Sec-Fetch-Dest":["document"],"Sec-Fetch-Site":["cross-site"],"Sec-Fetch-Mode":["navigate"],"Dnt":["1"],"Cache-Control":["max-age=0"]},"tls":{"resumed":false,"version":772,"ciphersuite":4865,"proto":"h2","proto_mutual":true,"server_name":"corona.kraken.translatum.xyz"}},"status":502,"err_id":"smdud2qrv","err_trace":"reverseproxy.(*Handler).ServeHTTP (reverseproxy.go:363)"}

neuronflow · March 29, 2020, 10:34pm

I solved it by adding http:// in front of dockerhost

system · April 28, 2020, 10:34pm

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.