[DNS challenge] write: operation not permitted

Spike · June 22, 2021, 2:11pm

1. Caddy version (`caddy version`):

v2.4.3 h1:Y1FaV2N4WO3rBqxSYA8UZsZTQdN+PwcoOcAiZTM8C0I=
custom build w/ GitHub - caddy-dns/gandi: Caddy module: dns.providers.gandi

2. How I run Caddy:

a. System environment:

using Fedora 34 w/ systemd through a VPN

b. Command:

sudo systemctl start caddy

c. Service/unit/compose file:

same as dist/caddy.service at master · caddyserver/dist · GitHub

d. My complete Caddyfile or JSON config:

{
http_port 58090
https_port 57294
}


example.com:58090 {
	reverse_proxy 127.0.0.1:8096
}

example.com:57294 {
	tls myemail@example.com {
		dns gandi my_key
	}
	reverse_proxy 127.0.0.1:8920
}

3. The problem I’m having:

I am trying to have a Jellyfin server with https. I cannot obtain a tls certificate from issuer.

4. Error messages and/or full log output:

Jun 22 08:24:26 fedora systemd[1]: Starting Caddy...
Jun 22 08:24:26 fedora caddy[40939]: caddy.HomeDir=/var/lib/caddy
Jun 22 08:24:26 fedora caddy[40939]: caddy.AppDataDir=/var/lib/caddy/.local/share/caddy
Jun 22 08:24:26 fedora caddy[40939]: caddy.AppConfigDir=/var/lib/caddy/.config/caddy
Jun 22 08:24:26 fedora caddy[40939]: caddy.ConfigAutosavePath=/var/lib/caddy/.config/caddy/autosave.json
Jun 22 08:24:26 fedora caddy[40939]: caddy.Version=v2.4.3 h1:Y1FaV2N4WO3rBqxSYA8UZsZTQdN+PwcoOcAiZTM8C0I=
Jun 22 08:24:26 fedora caddy[40939]: runtime.GOOS=linux
Jun 22 08:24:26 fedora caddy[40939]: runtime.GOARCH=amd64
Jun 22 08:24:26 fedora caddy[40939]: runtime.Compiler=gc
Jun 22 08:24:26 fedora caddy[40939]: runtime.NumCPU=4
Jun 22 08:24:26 fedora caddy[40939]: runtime.GOMAXPROCS=4
Jun 22 08:24:26 fedora caddy[40939]: runtime.Version=go1.16.5
Jun 22 08:24:26 fedora caddy[40939]: os.Getwd=/
Jun 22 08:24:26 fedora caddy[40939]: LANG=en_US.UTF-8
Jun 22 08:24:26 fedora caddy[40939]: PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin
Jun 22 08:24:26 fedora caddy[40939]: NOTIFY_SOCKET=/run/systemd/notify
Jun 22 08:24:26 fedora caddy[40939]: HOME=/var/lib/caddy
Jun 22 08:24:26 fedora caddy[40939]: LOGNAME=caddy
Jun 22 08:24:26 fedora caddy[40939]: USER=caddy
Jun 22 08:24:26 fedora caddy[40939]: INVOCATION_ID=dd811ab180304f0d87711167c2d49f7b
Jun 22 08:24:26 fedora caddy[40939]: JOURNAL_STREAM=8:180500
Jun 22 08:24:26 fedora caddy[40939]: SYSTEMD_EXEC_PID=40939
Jun 22 08:24:26 fedora caddy[40939]: {"level":"info","ts":1624368266.640844,"msg":"using provided configuration","config_file":"/etc/caddy/Caddyfile","config_adapter":""}
Jun 22 08:24:26 fedora caddy[40939]: {"level":"warn","ts":1624368266.6423728,"msg":"input is not formatted with 'caddy fmt'","adapter":"caddyfile","file":"/etc/caddy/Caddyfile","line":2}
Jun 22 08:24:26 fedora caddy[40939]: {"level":"info","ts":1624368266.6449437,"logger":"admin","msg":"admin endpoint started","address":"tcp/localhost:2019","enforce_origin":false,"origins":["localhost:2019","[::1]:2019","127.0.0.1:2019"]}
Jun 22 08:24:26 fedora caddy[40939]: {"level":"info","ts":1624368266.6452925,"logger":"tls.cache.maintenance","msg":"started background certificate maintenance","cache":"0xc0000a2d90"}
Jun 22 08:24:26 fedora caddy[40939]: {"level":"info","ts":1624368266.6452947,"logger":"http","msg":"server is listening only on the HTTP port, so no automatic HTTPS will be applied to this server","server_name":"srv1","http_port":58090}
Jun 22 08:24:26 fedora caddy[40939]: {"level":"info","ts":1624368266.6453555,"logger":"http","msg":"server is listening only on the HTTPS port but has no TLS connection policies; adding one to enable TLS","server_name":"srv0","https_port":57294}
Jun 22 08:24:26 fedora caddy[40939]: {"level":"info","ts":1624368266.6453764,"logger":"http","msg":"enabling automatic HTTP->HTTPS redirects","server_name":"srv0"}
Jun 22 08:24:26 fedora caddy[40939]: {"level":"info","ts":1624368266.6458557,"logger":"tls","msg":"cleaning storage unit","description":"FileStorage:/var/lib/caddy/.local/share/caddy"}
Jun 22 08:24:26 fedora caddy[40939]: {"level":"info","ts":1624368266.6458905,"logger":"tls","msg":"finished cleaning storage units"}
Jun 22 08:24:26 fedora caddy[40939]: {"level":"info","ts":1624368266.6459138,"logger":"http","msg":"enabling automatic TLS certificate management","domains":["example.com"]}
Jun 22 08:24:26 fedora caddy[40939]: {"level":"info","ts":1624368266.6461692,"msg":"autosaved config (load with --resume flag)","file":"/var/lib/caddy/.config/caddy/autosave.json"}
Jun 22 08:24:26 fedora caddy[40939]: {"level":"info","ts":1624368266.6462371,"msg":"serving initial configuration"}
Jun 22 08:24:26 fedora systemd[1]: Started Caddy.
Jun 22 08:24:26 fedora caddy[40939]: {"level":"info","ts":1624368266.64647,"logger":"tls.obtain","msg":"acquiring lock","identifier":"example.com"}
Jun 22 08:24:26 fedora caddy[40939]: {"level":"info","ts":1624368266.7050285,"logger":"tls.obtain","msg":"lock acquired","identifier":"example.com"}
Jun 22 08:24:26 fedora caddy[40939]: {"level":"info","ts":1624368266.7370594,"logger":"tls.issuance.acme","msg":"waiting on internal rate limiter","identifiers":["example.com"],"ca":"https://acme-v02.api.letsencrypt.org/directory","account":"myemail@example.com"}
Jun 22 08:24:26 fedora caddy[40939]: {"level":"info","ts":1624368266.7370849,"logger":"tls.issuance.acme","msg":"done waiting on internal rate limiter","identifiers":["example.com"],"ca":"https://acme-v02.api.letsencrypt.org/directory","account":"myemail@example.com"}
Jun 22 08:24:27 fedora caddy[40939]: {"level":"info","ts":1624368267.460404,"logger":"tls.issuance.acme.acme_client","msg":"trying to solve challenge","identifier":"example.com","challenge_type":"dns-01","ca":"https://acme-v02.api.letsencrypt.org/directory"}
Jun 22 08:24:32 fedora caddy[40939]: {"level":"error","ts":1624368272.124267,"logger":"tls.obtain","msg":"could not get certificate from issuer","identifier":"example.com","issuer":"acme-v02.api.letsencrypt.org-directory","error":"[example.com] solving challenges: waiting for solver certmagic.solverWrapper to be ready: checking DNS propagation of _acme-challenge.example.com: write udp 10.104.244.39:50157->213.167.230.182:53: write: operation not permitted (order=https://acme-v02.api.letsencrypt.org/acme/order/127864586/10569721066) (ca=https://acme-v02.api.letsencrypt.org/directory)"}
Jun 22 08:24:32 fedora caddy[40939]: {"level":"info","ts":1624368272.1247644,"logger":"tls.issuance.acme","msg":"waiting on internal rate limiter","identifiers":["example.com"],"ca":"https://acme.zerossl.com/v2/DV90","account":"myemail@example.com"}
Jun 22 08:24:32 fedora caddy[40939]: {"level":"info","ts":1624368272.1247885,"logger":"tls.issuance.acme","msg":"done waiting on internal rate limiter","identifiers":["example.com"],"ca":"https://acme.zerossl.com/v2/DV90","account":"myemail@example.com"}
Jun 22 08:24:32 fedora caddy[40939]: {"level":"info","ts":1624368272.5420308,"logger":"tls.issuance.acme.acme_client","msg":"trying to solve challenge","identifier":"example.com","challenge_type":"dns-01","ca":"https://acme.zerossl.com/v2/DV90"}
Jun 22 08:24:36 fedora caddy[40939]: {"level":"error","ts":1624368276.7212105,"logger":"tls.obtain","msg":"could not get certificate from issuer","identifier":"example.com","issuer":"acme.zerossl.com-v2-DV90","error":"[example.com] solving challenges: waiting for solver certmagic.solverWrapper to be ready: checking DNS propagation of _acme-challenge.example.com: write udp 10.104.244.39:38002->217.70.187.72:53: write: operation not permitted (order=https://acme.zerossl.com/v2/DV90/order/HHSsanH_pq3JaTmbSWptKA) (ca=https://acme.zerossl.com/v2/DV90)"}
Jun 22 08:24:36 fedora caddy[40939]: {"level":"error","ts":1624368276.721273,"logger":"tls.obtain","msg":"will retry","error":"[example.com] Obtain: [example.com] solving challenges: waiting for solver certmagic.solverWrapper to be ready: checking DNS propagation of _acme-challenge.example.com: write udp 10.104.244.39:38002->217.70.187.72:53: write: operation not permitted (order=https://acme.zerossl.com/v2/DV90/order/HHSsanH_pq3JaTmbSWptKA) (ca=https://acme.zerossl.com/v2/DV90)","attempt":1,"retrying_in":60,"elapsed":10.016209449,"max_duration":2592000}
Jun 22 08:25:38 fedora caddy[40939]: {"level":"info","ts":1624368338.1015477,"logger":"tls.issuance.acme.acme_client","msg":"trying to solve challenge","identifier":"example.com","challenge_type":"dns-01","ca":"https://acme-staging-v02.api.letsencrypt.org/directory"}
Jun 22 08:25:43 fedora caddy[40939]: {"level":"error","ts":1624368343.1068118,"logger":"tls.obtain","msg":"could not get certificate from issuer","identifier":"example.com","issuer":"acme-v02.api.letsencrypt.org-directory","error":"[example.com] solving challenges: waiting for solver certmagic.solverWrapper to be ready: checking DNS propagation of _acme-challenge.example.com: write udp 10.104.244.39:45446->173.246.100.49:53: write: operation not permitted (order=https://acme-staging-v02.api.letsencrypt.org/acme/order/19996898/82268718) (ca=https://acme-staging-v02.api.letsencrypt.org/directory)"}
Jun 22 08:25:44 fedora caddy[40939]: {"level":"info","ts":1624368344.427845,"logger":"tls.issuance.acme.acme_client","msg":"trying to solve challenge","identifier":"example.com","challenge_type":"dns-01","ca":"https://acme.zerossl.com/v2/DV90"}
Jun 22 08:25:48 fedora caddy[40939]: {"level":"error","ts":1624368348.5337574,"logger":"tls.obtain","msg":"could not get certificate from issuer","identifier":"example.com","issuer":"acme.zerossl.com-v2-DV90","error":"[example.com] solving challenges: waiting for solver certmagic.solverWrapper to be ready: checking DNS propagation of _acme-challenge.example.com: write udp 10.104.244.39:46655->173.246.100.49:53: write: operation not permitted (order=https://acme.zerossl.com/v2/DV90/order/qjNgMym-D90bfZefqRXnhQ) (ca=https://acme.zerossl.com/v2/DV90)"}
Jun 22 08:25:48 fedora caddy[40939]: {"level":"error","ts":1624368348.5338202,"logger":"tls.obtain","msg":"will retry","error":"[example.com] Obtain: [example.com] solving challenges: waiting for solver certmagic.solverWrapper to be ready: checking DNS propagation of _acme-challenge.example.com: write udp 10.104.244.39:46655->173.246.100.49:53: write: operation not permitted (order=https://acme.zerossl.com/v2/DV90/order/qjNgMym-D90bfZefqRXnhQ) (ca=https://acme.zerossl.com/v2/DV90)","attempt":2,"retrying_in":120,"elapsed":81.828756739,"max_duration":2592000}

5. What I already tried:

I am hosting this off a personal computer that is using a VPN with limited accessible external ports. I understand that, for the TLS-ALPN challenge, it must have port 443 accessible, so I have opted into using the DNS challenge. I have double checked folder ownerships and groups. everything seems to be setup correctly.

6. Links to relevant resources:

none

francislavoie · June 22, 2021, 3:53pm

Looks like when Caddy is trying to perform a DNS to make sure the DNS has properly been set, it’s getting blocked by something between it and the DNS server. Do you have some too-aggressive firewalls preventing outgoing UDP traffic on port 53?

This doesn’t look like an issue with Caddy, but rather a misconfiguration of your environment.

Spike · June 23, 2021, 7:14am

Ok, after some investigation, it seems that my vpn provider is blocking outgoing DNS both UDP and TCP on port 53. I am no rust programmer, but that is what it seems like from here.

github.com

mullvad/mullvadvpn-app/blob/96235b29e25ac3c73ee1d0b3ed89674dcc9eede7/talpid-core/src/firewall/linux.rs#L784

    
      
                  allow_rule.add_expr(&addr);
                  allow_rule.add_expr(&nft_expr!(cmp == host));
                  add_verdict(&mut allow_rule, &Verdict::Accept);
          
          
        self.batch.add(&allow_rule, nftnl::MsgType::Add);
              }
          
          
    Ok(())
          }
          
          
/// Blocks all outgoing DNS (port 53) on both TCP and UDP
          fn add_drop_dns_rule(&mut self) {
              for chain in &[&self.out_chain, &self.forward_chain] {
                  let mut block_udp_rule = Rule::new(chain);
                  check_port(&mut block_udp_rule, TransportProtocol::Udp, End::Dst, 53);
                  add_verdict(
                      &mut block_udp_rule,
                      &Verdict::Reject(RejectionType::Icmp(IcmpCode::PortUnreach)),
                  );
                  self.batch.add(&block_udp_rule, nftnl::MsgType::Add);

I think the only option I have right now is to reach out to their support and see what can be done. Thank you for the assistance so far. At the end of the day, I would have expected both Mullvad and Caddy to work cohesively, so I am unsure of who’s ownest this issue is. I will reply back with what I can gather.

Spike · June 23, 2021, 1:06pm

After contacting Mullvad support. They stated that they hijack DNS queries and recommended to use OpenVPN to connect on UDP port 1400 or 1401, where they do not hijack DNS queries. After reconfigure my environment, everything worked as expected.

system · July 22, 2021, 2:11pm

This topic was automatically closed after 30 days. New replies are no longer allowed.