ReversProxy in Docker Compose not resolving container/service

Kai_Amrhein · November 25, 2020, 9:25am

1. Caddy version (`caddy version`): 2.1.1

2. How I run Caddy:

Caddy is running as a service inside a docker-compose setup of two containers.

a. System environment:

Docker Container group on the Azure Cloud

b. Command:

This is the command to get the containers created from the yaml file below:
az container create --resource-group resGrpName --name caddyContainerGroup --ports 443 80 8080 8181 4848 --file ./dockerfiles/docker-compose_azure.yml

c. Service/unit/compose file:

this is the yaml file i am using for the Azure service to create the containers:

The dash container contains a payara app server application listening on port 8181.

location: someLocation
name: caddyContainerGroup
properties:
  containers:
  - name: caddy
    properties:
      image: registry_blurred.io/caddy:latest
      resources:
        requests:
          cpu: 1
          memoryInGb: 0.5
      ports:
      - port: 80
      - port: 443
    
  - name: dash
    properties:
      image: registry_blurred.io/dashboard:latest
      resources:
        requests:
          cpu: 2
          memoryInGb: 2
      ports:
      - port: 8080
      - port: 8181
      
  osType: Linux
  ipAddress:
    type: Public
    ports:
    - protocol: tcp
      port: 80
    - protocol: tcp
      port: 8080
    - protocol: tcp
      port: 443
    - protocol: tcp
      port: 8181
    dnsNameLabel: caddydash
  imageRegistryCredentials:
    - server: registry_blurred.io
      username: name
      password: pwd

After this command is executed, both Services are up and running:

I can access the Dashboard app on port 8181 directly with no issues.

d. My complete Caddyfile or JSON config:

I tried this with 2 different files:
First one where I thought that docker is resolving the name of Dash:

caddydash.someRegion.azurecontainer.io:443 {
		reverse_proxy dash:8181	
	}

Second one where I went for the fully qualified name:

caddydash.someRegion.azurecontainer.io:443 {
		reverse_proxy caddydash.someRegion.azurecontainer.io:8181	
	}

3. The problem I’m having:

the revers-proxying from the caddy to dash container is not working. Caddy throws the error(s) listed below when using the 2 different Caddyfiles:

4. Error messages and/or full log output:

Error 1 when trying to resolve container by docker service name (“dash”):
{“level”:“error”,“ts”:1606293917.5539818,“logger”:“http.log.error”,“msg”:“dial tcp: lookup dash on 168.63.129.16:53: no such host”,“request”:{“method”:“GET”,“uri”:"/login.xhtml",“proto”:“HTTP/2.0”,“remote_addr”:“10.240.255.55:33293”,“host”:“caddydash.someRegion.azurecontainer.io”,“headers”:{“Upgrade-Insecure-Requests”:[“1”],“User-Agent”:[“Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/87.0.4280.66 Safari/537.36”],“Accept”:[“text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,/;q=0.8,application/signed-exchange;v=b3;q=0.9”],“Cookie”:[“JSESSIONID=e81c327c2a9dbfe3fe5fc6437868”],“Cache-Control”:[“max-age=0”],“Sec-Fetch-Site”:[“cross-site”],“Sec-Fetch-Mode”:[“navigate”],“Sec-Fetch-User”:["?1"],“Sec-Fetch-Dest”:[“document”],“Accept-Encoding”:[“gzip, deflate, br”],“Accept-Language”:[“de-DE,de;q=0.9,en-US;q=0.8,en;q=0.7”]},“tls”:{“resumed”:false,“version”:772,“ciphersuite”:4865,“proto”:“h2”,“proto_mutual”:true,“server_name”:“caddydash.someRegion.azurecontainer.io”}},“duration”:0.034422681,“status”:502,“err_id”:“k8251dk35”,“err_trace”:“reverseproxy.(*Handler).ServeHTTP (reverseproxy.go:411)”}

Error 2 when trying to resolve container by FQDN (“caddydash.someRegion.azurecontainer.io”):
{“level”:“error”,“ts”:1606295980.588889,“logger”:“http.log.error”,“msg”:“EOF”,“request”:{“method”:“GET”,“uri”:"/login.xhtml",“proto”:“HTTP/2.0”,“remote_addr”:“10.240.255.56:33368”,“host”:“caddydash.someRegion.azurecontainer.io”,“headers”:{“Accept”:[“text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,/;q=0.8,application/signed-exchange;v=b3;q=0.9”],“Sec-Fetch-Site”:[“none”],“Upgrade-Insecure-Requests”:[“1”],“User-Agent”:[“Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/87.0.4280.66 Safari/537.36”],“Sec-Fetch-Mode”:[“navigate”],“Sec-Fetch-User”:["?1"],“Sec-Fetch-Dest”:[“document”],“Accept-Encoding”:[“gzip, deflate, br”],“Accept-Language”:[“de-DE,de;q=0.9,en-US;q=0.8,en;q=0.7”],“Cookie”:[“JSESSIONID=e81c327c2a9dbfe3fe5fc6437868”],“Cache-Control”:[“max-age=0”]},“tls”:{“resumed”:false,“version”:772,“ciphersuite”:4865,“proto”:“h2”,“proto_mutual”:true,“server_name”:“caddydash.someRegion.azurecontainer.io”}},“duration”:0.020348627,“status”:502,“err_id”:“yuipi2krx”,“err_trace”:“reverseproxy.(*Handler).ServeHTTP (reverseproxy.go:411)”}

And also, the caddy log contains a lot of other, strange things…
#1: Lots of these TLS handshake errors:
2020/11/25 08:53:26 http: TLS handshake error from 10.240.255.55:58240: EOF
2020/11/25 08:53:32 http: TLS handshake error from 10.240.255.55:58332: EOF
2020/11/25 08:53:38 http: TLS handshake error from 10.240.255.55:58427: EOF
2020/11/25 08:53:44 http: TLS handshake error from 10.240.255.55:58517: EOF
2020/11/25 08:53:50 http: TLS handshake error from 10.240.255.55:58601: EOF
2020/11/25 08:53:56 http: TLS handshake error from 10.240.255.55:58691: EOF
2020/11/25 08:54:02 http: TLS handshake error from 10.240.255.55:58785: EOF
2020/11/25 08:54:08 http: TLS handshake error from 10.240.255.55:58873: EOF
2020/11/25 08:54:14 http: TLS handshake error from 10.240.255.55:58960: EOF
2020/11/25 08:54:20 http: TLS handshake error from 10.240.255.55:59068: EOF
2020/11/25 08:54:26 http: TLS handshake error from 10.240.255.55:59171: EOF
2020/11/25 08:54:32 http: TLS handshake error from 10.240.255.55:59288: EOF
2020/11/25 08:54:38 http: TLS handshake error from 10.240.255.55:59382: EOF
2020/11/25 08:54:44 http: TLS handshake error from 10.240.255.55:59462: EOF
2020/11/25 08:54:50 http: TLS handshake error from 10.240.255.55:59564: EOF
2020/11/25 08:54:56 http: TLS handshake error from 10.240.255.55:59664: EOF
2020/11/25 08:55:02 http: TLS handshake error from 10.240.255.55:59752: EOF
2020/11/25 08:55:08 http: TLS handshake error from 10.240.255.55:59831: EOF

#2: Errors for unresolved requests that I did not make:
{“level”:“error”,“ts”:1606294510.916728,“logger”:“http.log.error”,“msg”:“dial tcp: lookup dash on 168.63.129.16:53: no such host”,“request”:{“method”:“GET”,“uri”:"/",“proto”:“HTTP/2.0”,“remote_addr”:“10.240.255.55:40712”,“host”:“caddydash.someRegion.azurecontainer.io”,“headers”:{“Sec-Fetch-Site”:[“none”],“Accept-Encoding”:[“gzip, deflate, br”],“Pragma”:[“no-cache”],“Cache-Control”:[“no-cache”],“Upgrade-Insecure-Requests”:[“1”],“User-Agent”:[“Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/76.0.3809.71 Safari/537.36”],“Accept-Language”:[“en-US,en;q=0.9”],“Accept”:[“text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,/;q=0.8,application/signed-exchange;v=b3”]},“tls”:{“resumed”:false,“version”:772,“ciphersuite”:4865,“proto”:“h2”,“proto_mutual”:true,“server_name”:“caddydash.someRegion.azurecontainer.io”}},“duration”:0.021471156,“status”:502,“err_id”:“y1h09z9tt”,“err_trace”:“reverseproxy.(*Handler).ServeHTTP (reverseproxy.go:411)”}

#3: One or more domains has an issue, not sure what it is:
{“level”:“info”,“ts”:1606295805.2672303,“msg”:“using provided configuration”,“config_file”:"/etc/caddy/Caddyfile",“config_adapter”:“caddyfile”}
{“level”:“info”,“ts”:1606295805.2685502,“logger”:“admin”,“msg”:“admin endpoint started”,“address”:“tcp/localhost:2019”,“enforce_origin”:false,“origins”:[“localhost:2019”,"[::1]:2019",“127.0.0.1:2019”]}
{“level”:“info”,“ts”:1606295805.2686732,“logger”:“http”,“msg”:“server is listening only on the HTTPS port but has no TLS connection policies; adding one to enable TLS”,“server_name”:“srv0”,“https_port”:443}
{“level”:“info”,“ts”:1606295805.2686977,“logger”:“http”,“msg”:“enabling automatic HTTP->HTTPS redirects”,“server_name”:“srv0”}
2020/11/25 09:16:45 [INFO][cache:0xc0001877a0] Started certificate maintenance routine
{“level”:“info”,“ts”:1606295805.2688997,“logger”:“http”,“msg”:“enabling automatic TLS certificate management”,“domains”:[“caddydash.westeurope.azurecontainer.io”]}
{“level”:“info”,“ts”:1606295805.268985,“logger”:“tls”,“msg”:“cleaned up storage units”}
{“level”:“info”,“ts”:1606295805.26906,“msg”:“autosaved config”,“file”:"/config/caddy/autosave.json"}
{“level”:“info”,“ts”:1606295805.2690742,“msg”:“serving initial configuration”}
2020/11/25 09:16:45 [INFO][caddydash.westeurope.azurecontainer.io] Obtain certificate; acquiring lock…
2020/11/25 09:16:45 [INFO][caddydash.westeurope.azurecontainer.io] Obtain: Lock acquired; proceeding…
2020/11/25 09:16:46 [INFO][caddydash.westeurope.azurecontainer.io] Waiting on rate limiter…
2020/11/25 09:16:46 [INFO][caddydash.westeurope.azurecontainer.io] Done waiting
2020/11/25 09:16:46 [INFO] [caddydash.westeurope.azurecontainer.io] acme: Obtaining bundled SAN certificate given a CSR
2020/11/25 09:16:46 [INFO] [caddydash.westeurope.azurecontainer.io] AuthURL: https://acme-v02.api.letsencrypt.org/acme/authz-v3/8851499004
2020/11/25 09:16:46 [INFO] [caddydash.westeurope.azurecontainer.io] acme: use tls-alpn-01 solver
2020/11/25 09:16:46 [INFO] [caddydash.westeurope.azurecontainer.io] acme: Trying to solve TLS-ALPN-01
2020/11/25 09:16:47 http: TLS handshake error from 127.0.0.1:33522: EOF
2020/11/25 09:17:01 [INFO] Deactivating auth: https://acme-v02.api.letsencrypt.org/acme/authz-v3/8851499004
2020/11/25 09:17:02 [INFO] Unable to deactivate the authorization: https://acme-v02.api.letsencrypt.org/acme/authz-v3/8851499004
2020/11/25 09:17:02 [ERROR] error: one or more domains had a problem:
[caddydash.westeurope.azurecontainer.io] acme: error: 400 :: urn:ietf:params:acme:error:connection :: Timeout during connect (likely firewall problem), url:
(challenge=tls-alpn-01 remaining=[http-01])
2020/11/25 09:17:04 [INFO] [caddydash.westeurope.azurecontainer.io] acme: Obtaining bundled SAN certificate given a CSR
2020/11/25 09:17:04 [INFO] [caddydash.westeurope.azurecontainer.io] AuthURL: https://acme-v02.api.letsencrypt.org/acme/authz-v3/8851503524
2020/11/25 09:17:04 [INFO] [caddydash.westeurope.azurecontainer.io] acme: Could not find solver for: tls-alpn-01
2020/11/25 09:17:04 [INFO] [caddydash.westeurope.azurecontainer.io] acme: use http-01 solver
2020/11/25 09:17:04 [INFO] [caddydash.westeurope.azurecontainer.io] acme: Trying to solve HTTP-01
2020/11/25 09:17:19 http: TLS handshake error from 10.240.255.56:55209: EOF
2020/11/25 09:17:19 http: TLS handshake error from 10.240.255.55:54013: EOF
2020/11/25 09:17:23 [INFO] Deactivating auth: https://acme-v02.api.letsencrypt.org/acme/authz-v3/8851503524
2020/11/25 09:17:23 [ERROR] error: one or more domains had a problem:
[caddydash.westeurope.azurecontainer.io] acme: error: 400 :: urn:ietf:params:acme:error:connection :: Fetching http://caddydash.westeurope.azurecontainer.io/.well-known/acme-challenge/vEA4EhdftIezymDOzAKhlkdoGa30EIuMetPnLc0O4Wo: Timeout during connect (likely firewall problem), url:
(challenge=http-01 remaining=[])

5. What I already tried:

I tried the 2 different Caddy Files listed above.

6. Links to relevant resources:

francislavoie · November 25, 2020, 9:53am

Latest version of Caddy is v2.2.1, first thing you could do is upgrade.

I know nothing about Azure’s container stack, so that’s all looks pretty foreign to me. How are you building the container? Looks like you’re not using the official docker image Docker Hub

Make sure to persist the /data and /config volumes, otherwise you’ll be losing your certs and keys any time you recreate the containers, and you risk hitting rate limits.

Does the Azure container stack use the standard docker bridge network adapter? Or is it doing some other funky stuff? Using dash as the hostname usually works okay with stock Docker, using its built-in DNS server. It looks like DNS resolution doesn’t happen correctly there though.

Are you sure ports 80 and 443 are open at the firewall level? That’s necessary for ACME challenges to succeed.

Those TLS handshake warnings are hidden as of v2.2.0 so if you upgrade, you’ll no longer see those. Basically it just means that you received a request without SNI, so Caddy doesn’t know how to complete the handshake. We’ve since moved them to DEBUG level because they’re noisy and usually just due to poorly configured clients or bots/crawlers not making requests with a valid hostname.

Kai_Amrhein · November 25, 2020, 1:29pm

First of all…huge THANK YOU for your replies, they are invaluable.
So I tried to fix some of the things you mentioned:

I upgraded the version to 2.2.1, by changing my Docker build file. I did not post that file in the original post, but just for reference, you find it below. That has worked as I can confirm with the caddy version command

FROM caddy/caddy:2.2.1-alpine
ENV CONFIG_DIR /etc/caddy
COPY Caddyfile $CONFIG_DIR/Caddyfile
I have (tried to) mount/ed the /data and /config volumes. I have both of them mapped to the same fileshare (and I also tried mapping them to two different ones, the result was not much different). I am not 100% comfortable with the outcome (Edit: by now, I am sure that it did not work: I am hitting the limit from letsencrypt, so it is using a new certificate every restart…arghhh): When uploading files (Azure Cloud Explorer), I can see them in the mounted directories /(sometimes), When recreating the containers, the uploaded files are sometimes there…more important however: certificate files - judging from their timestamp - seem to get recreated after the recreation of the container. Would you know a way to make sure that in a new container, I am using a certificate from an old one? Is it something you could se from the log? The yaml file that I used for the mount (with a single fileshare) is this:
apiVersion: 2019-12-01
location: someLocation
name: caddyContainerGroup
properties:
containers:
- name: caddy
  properties:
  image: caddy:latest
  resources:
  requests:
  cpu: 1
  memoryInGb: 0.5
  ports:
  - port: 80
  - port: 443
    volumeMounts:
  - mountPath: /data
    name: filesharevolume
    volumeMounts:
  - mountPath: /config
    name: filesharevolume
- name: dash
  properties:
  image: dashboard:latest
  resources:
  requests:
  cpu: 2
  memoryInGb: 2
  ports:
  - port: 8080
  - port: 8181
osType: Linux
ipAddress:
type: Public
ports:
- protocol: tcp
  port: 80
- protocol: tcp
  port: 8080
- protocol: tcp
  port: 443
- protocol: tcp
  port: 8181
  dnsNameLabel: someDNSName
  volumes:
- name: filesharevolume
  azureFile:
  sharename: caddy
  storageAccountName: mystorage
  storageAccountKey: key
  imageRegistryCredentials:
  - server: server
    username: user
    password: password
The fundamental thing I want to achiever (reverse-proxying from one container to the next) is still not successfull. Pretty much the same error like above, the target host/service/container cannot seem to be resolved by the docker dns. When using nslookup inside the container, I see that dash cannot be resolved, but the FQDN can be resolved…

“level”:“error”,“ts”:1606310794.2514346,“logger”:“http.log.error”,“msg”:“dial tcp: lookup dash on 168.63.129.16:53: no such host”,“request”:{“remote_addr”:“10.240.255.55:33670”,“proto”:“HTTP/2.0”,“method”:“GET”,“host”:“someUrl.someRegion.azurecontainer.io”,“uri”:“/login.xhtml”,“headers”:{“Sec-Fetch-User”:[“?1”],“Sec-Fetch-Dest”:[“document”],“Upgrade-Insecure-Requests”:[“1”],“Sec-Fetch-Site”:[“none”],“Sec-Fetch-Mode”:[“navigate”],“Accept-Encoding”:[“gzip, deflate, br”],“Accept-Language”:[“de-DE,de;q=0.9,en-US;q=0.8,en;q=0.7”],“User-Agent”:[“Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/87.0.4280.66 Safari/537.36”],“Accept”:[“text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,/;q=0.8,application/signed-exchange;v=b3;q=0.9”]},“tls”:{“resumed”:false,“version”:772,“cipher_suite”:4865,“proto”:“h2”,“proto_mutual”:true,“server_name”:“someUrl.someRegion.azurecontainer.io”}},“duration”:0.031115618,“status”:502,“err_id”:“i8pm13rsi”,“err_trace”:“reverseproxy.(*Handler).ServeHTTP (reverseproxy.go:441)”}

I am not sure about your question about the open ports (80,443). It is mentioned in the yaml file that these ports should be used for the containers. Is that not sufficient/what you are asking? Is there any way I can test (from inside the container) if these ports are open?
The log is still full of errors, like these below - would you have any further advice? Thank you so much!

{“level”:“error”,“ts”:1606309474.69734,“logger”:“tls.issuance.acme.acme_client”,“msg”:“challenge failed”,“identifier”:“someUrl.someRegion.azurecontainer.io”,“challenge_type”:“tls-alpn-01”,“status_code”:400,“problem_type”:“urn:ietf:params:acme:error:connection”,“error”:“Timeout during connect (likely firewall problem)”}
{“level”:“error”,“ts”:1606309474.6974242,“logger”:“tls.issuance.acme.acme_client”,“msg”:“validating authorization”,“identifier”:“someUrl.someRegion.azurecontainer.io”,“error”:“authorization failed: HTTP 400 urn:ietf:params:acme:error:connection - Timeout during connect (likely firewall problem)”,“order”:“https://acme-v02.api.letsencrypt.org/acme/order/103589905/6401726642",“attempt”:1,"max_attempts”:3}
{“level”:“info”,“ts”:1606309476.1250796,“logger”:“tls.issuance.acme.acme_client”,“msg”:“trying to solve challenge”,“identifier”:“someUrl.someRegion.azurecontainer.io”,“challenge_type”:“http-01”,“ca”:“https://acme-v02.api.letsencrypt.org/directory”}
{“level”:“error”,“ts”:1606309486.4869025,“logger”:“tls.issuance.acme.acme_client”,“msg”:“challenge failed”,“identifier”:“someUrl.westeurope.azurecontainer.io”,“challenge_type”:“http-01”,“status_code”:400,“problem_type”:“urn:ietf:params:acme:error:connection”,“error”:“Fetching http://mypumabiz.someRegion.azurecontainer.io/.well-known/acme-challenge/9UDnaRMQLRsAjrTfgxP2FnYxwUUDCVJCPXQgXrZ3Wj4: Timeout during connect (likely firewall problem)”}

francislavoie · November 25, 2020, 6:27pm

Please use ``` on the lines before and after your config/logs to use code formatting, it’s pretty difficult to read otherwise (especially because your yml is being parsed as markdown, looks real funky)

You should switch to the Let’s Encrypt staging endpoint while you’re testing your stack, to avoid hitting rate limits. Configure the acme_ca global option:

That would be a symptom of it not being persisted, yeah. No idea how Azure provides those options, but in docker-compose it’s pretty easy (see the example at the bottom of Docker)

Yeah, the logs will tell you when it’s doing certificate maintenance. If it already has certificates from a previous run it would be mostly quiet (cause it had nothing to do).

Azure probably has a firewall layer you need to configure to make sure those ports are open, maybe. But it probably is the case if you got far enough to see logs about the proxy. So nevermind on that point.

I think the error message is pretty self explanatory. Let’s Encrypt couldn’t reach your server to complete the ACME challenge. It’s hard to say why, it could be any number of reasons (especially because I don’t have a grasp of the whole Azure stack).

authorization failed: HTTP 400 urn:ietf:params:acme:error:connection - Timeout during connect (likely firewall problem)

Kai_Amrhein · November 29, 2020, 2:27pm

So the next round of questions, many thanks for your replies from the last round.

When looking at these lines from the logs, I assume that the certificate has been successfully obtained and should be ready? On the filesystem, i can find all kinds of cert and key , etc files…so it should be ok?

{"level":"info","ts":1606658539.1041746,"logger":"tls.issuance.acme","msg":"done waiting on internal rate limiter","identifiers":["mypumabiztest1.westeurope.azurecontainer.io"]}
{"level":"info","ts":1606658540.0557764,"logger":"tls.issuance.acme.acme_client","msg":"trying to solve challenge","identifier":"mypumabiztest1.westeurope.azurecontainer.io","challenge_type":"tls-alpn-01","ca":"https://acme-v02.api.letsencrypt.org/directory"}
{"level":"info","ts":1606658540.5226257,"logger":"tls","msg":"served key authentication certificate","server_name":"mypumabiztest1.westeurope.azurecontainer.io","challenge":"tls-alpn-01","remote":"10.240.255.56:28012"}
{"level":"info","ts":1606658540.8494112,"logger":"tls","msg":"served key authentication certificate","server_name":"mypumabiztest1.westeurope.azurecontainer.io","challenge":"tls-alpn-01","remote":"10.240.255.55:40810"}
{"level":"info","ts":1606658541.051945,"logger":"tls","msg":"served key authentication certificate","server_name":"mypumabiztest1.westeurope.azurecontainer.io","challenge":"tls-alpn-01","remote":"10.240.255.55:33390"}
{"level":"info","ts":1606658541.234594,"logger":"tls","msg":"served key authentication certificate","server_name":"mypumabiztest1.westeurope.azurecontainer.io","challenge":"tls-alpn-01","remote":"10.240.255.55:38294"}
{"level":"info","ts":1606658541.9419634,"logger":"tls.issuance.acme.acme_client","msg":"validations succeeded; finalizing order","order":"https://acme-v02.api.letsencrypt.org/acme/order/104228673/6472671387"}
{"level":"info","ts":1606658542.7704031,"logger":"tls.issuance.acme.acme_client","msg":"successfully downloaded available certificate chains","count":2,"first_url":"https://acme-v02.api.letsencrypt.org/acme/cert/0445003bc849dcd2d5ca064166bb8e75333"}

But still the forwarding to the backend does not work when using https and port 443
I am now wondering if I have a more fundamental issue. Could you pls verify that my general thinking is ok:

I use Caddy as a reverseproxy, it will have certificates and therefore be able to handle all the https/TLS topics.
My backend (payara server) is running without any certificates but that will be transparent for the request, as Caddy in the frontend is taking up https…
That is ho I understood this “automatic https” feature…is there anything wrong in that?

I have managed to set up the forwarding from Caddy to Payara on http ports (80 to 8080), but on 443 to 8181 i get nothing but 502s (would there be any debug options to get a bettter insight into what is wrong?), despite the fact that the Payara app can be reached directly on port 8181, without proper certificates then, however.

Any comments? Many thanks and best regards, Kai

francislavoie · November 29, 2020, 3:23pm

Yup, looks good to me!

Yeah that sounds correct.

What’s your full Caddyfile at this point? You can add this to the top of your config to get additional logging:

{
	debug
}

Kai_Amrhein · November 30, 2020, 7:09am

Thanks one more time over, you are marvelous.
So by now, thanks to your help, from my original three errirs I am down to this:

ACME challenge not working: Resolved, more or less by itself: while the error is still in the logs initially, it seems to resolve itself after a few minutes, after some certificate maintenance activity within Caddy.
Persisting /data and /config to avoid letsencrypet rate limit: Resolved, I am now properly mounting these two directories to an Azure file share. This is, just for everybodies reference, the relevant lines from the yaml file:

volumeMounts:
- mountPath: /data
  name: filesharevolumedata
- mountPath: /config
  name: filesharevolumeconfig
...and then...
volumes:
  - name: filesharevolumedata
    azureFile:
      sharename: data
      storageAccountName: mystorage
      storageAccountKey:key
  - name: filesharevolumeconfig
    azureFile:
      sharename: config
      storageAccountName: mystorage
      storageAccountKey: key

So finally, the “only” issue remaining is…that the forward to my backend des not work. Here is my latest Caddyfile:

    {
	debug
	}
	
	myXXX.YYY.azurecontainer.io:443 {
		reverse_proxy myXXX.YYY.azurecontainer.io:8181	
	}

And this is despite the fact that everything works fine when using a simpler http configuration:

    {
	debug
	}
	
	myXXX.YYY.azurecontainer.io:80 {
		reverse_proxy myXXX.YYY.azurecontainer.io:8080	
	}

The error I am getting for the https reverse-proxying is this (now with the “debug” option activated:

{"level":"debug","ts":1606718684.792248,"logger":"http.handlers.reverse_proxy","msg":"upstream roundtrip","upstream":"myXXX.YYY.azurecontainer.io:8181","request":{"remote_addr":"10.240.255.55:33551","proto":"HTTP/2.0","method":"GET","host":"myXXX.YYY.azurecontainer.io","uri":"/","headers":{"Accept":["text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.9"],"Sec-Fetch-Site":["none"],"Sec-Fetch-Mode":["navigate"],"Sec-Fetch-Dest":["document"],"Accept-Language":["de-DE,de;q=0.9,en-US;q=0.8,en;q=0.7"],"X-Forwarded-For":["10.240.255.55"],"X-Forwarded-Proto":["https"],"Upgrade-Insecure-Requests":["1"],"Sec-Fetch-User":["?1"],"Accept-Encoding":["gzip, deflate, br"],"User-Agent":["Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/87.0.4280.66 Safari/537.36"]},"tls":{"resumed":false,"version":772,"cipher_suite":4865,"proto":"h2","proto_mutual":true,"server_name":"myXXX.YYY.azurecontainer.io"}},"duration":0.058645815,"error":"EOF"}
{"level":"error","ts":1606718684.7923687,"logger":"http.log.error","msg":"EOF","request":{"remote_addr":"10.240.255.55:33551","proto":"HTTP/2.0","method":"GET","host":"myXXX.YYY.azurecontainer.io","uri":"/","headers":{"Upgrade-Insecure-Requests":["1"],"Accept":["text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.9"],"Sec-Fetch-Site":["none"],"Sec-Fetch-Mode":["navigate"],"Sec-Fetch-Dest":["document"],"Accept-Language":["de-DE,de;q=0.9,en-US;q=0.8,en;q=0.7"],"User-Agent":["Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/87.0.4280.66 Safari/537.36"],"Sec-Fetch-User":["?1"],"Accept-Encoding":["gzip, deflate, br"]},"tls":{"resumed":false,"version":772,"cipher_suite":4865,"proto":"h2","proto_mutual":true,"server_name":"myXXX.YYY.azurecontainer.io"}},"duration":0.058835516,"status":502,"err_id":"s3bcz8a5e","err_trace":"reverseproxy.(*Handler).ServeHTTP (reverseproxy.go:441)"}

Does the debug info help to get any idea why it is not working? Many thanks!

Kai_Amrhein · November 30, 2020, 8:39am

And one more little piece of information:

I have changed the Caddyfile to this:

myXXX.YYY.azurecontainer.io:443 {
		reverse_proxy https://myXXX.YYY.azurecontainer.io:8181	
	}

the difference to the revious setup is the https:// in the url that Caddy shall reverse-proxy to. The result is a change in the error message, it is now:

{"level":"debug","ts":1606725249.0880952,"logger":"http.handlers.reverse_proxy","msg":"upstream roundtrip","upstream":"myXXX.YYY.azurecontainer.io:8181","request":{"remote_addr":"10.240.255.56:33423","proto":"HTTP/2.0","method":"GET","host":"myXXX.YYY.azurecontainer.io","uri":"/login.xhtml","headers":{"Sec-Fetch-User":["?1"],"Sec-Fetch-Dest":["document"],"Accept-Language":["de-DE,de;q=0.9,en-US;q=0.8,en;q=0.7"],"X-Forwarded-For":["10.240.255.56"],"Upgrade-Insecure-Requests":["1"],"Accept":["text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.9"],"Sec-Fetch-Mode":["navigate"],"X-Forwarded-Proto":["https"],"User-Agent":["Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/87.0.4280.66 Safari/537.36"],"Sec-Fetch-Site":["none"],"Accept-Encoding":["gzip, deflate, br"]},"tls":{"resumed":false,"version":772,"cipher_suite":4865,"proto":"h2","proto_mutual":true,"server_name":"myXXX.YYY.azurecontainer.io"}},"duration":0.056558788,"error":"x509: certificate is not valid for any names, but wanted to match myXXX.YYY.azurecontainer.io"}
{"level":"error","ts":1606725249.0881643,"logger":"http.log.error","msg":"x509: certificate is not valid for any names, but wanted to match myXXX.YYY.azurecontainer.io","request":{"remote_addr":"10.240.255.56:33423","proto":"HTTP/2.0","method":"GET","host":"myXXX.YYY.azurecontainer.io","uri":"/login.xhtml","headers":{"Sec-Fetch-Dest":["document"],"Accept-Language":["de-DE,de;q=0.9,en-US;q=0.8,en;q=0.7"],"Upgrade-Insecure-Requests":["1"],"Accept":["text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.9"],"Sec-Fetch-Mode":["navigate"],"Sec-Fetch-User":["?1"],"User-Agent":["Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/87.0.4280.66 Safari/537.36"],"Sec-Fetch-Site":["none"],"Accept-Encoding":["gzip, deflate, br"]},"tls":{"resumed":false,"version":772,"cipher_suite":4865,"proto":"h2","proto_mutual":true,"server_name":"myXXX.YYY.azurecontainer.io"}},"duration":0.056698989,"status":502,"err_id":"b41tyy46z","err_trace":"reverseproxy.(*Handler).ServeHTTP (reverseproxy.go:441)"}

“x509: certificate is not valid for any names, but wanted to match myXXX.YYY.azurecontainer.io”

Does that give any further clues?
Many thanks,
Best Regards,
Kai

system · December 25, 2020, 9:25am

This topic was automatically closed after 30 days. New replies are no longer allowed.