Caddy on QNAP - set up reverse proxy

mupet0000 · July 24, 2019, 9:36pm

After running caddy -port 80 “status 200 /” I get this result:

$ curl -I http://myurl HTTP/1.1 504 Gateway Time-out Server: WebProxy/1.0 Pre-Alpha Date: Wed, 24 Jul 2019 21:22:14 GMT Content-Length: 0 Connection: keep-alive

It looks like Caddy is unable to get out onto the internet, maybe it’s unable to forward ports since the the NAS update, I am unsure. Is it possible to get a debug log output from Caddy to see if everything appears to be working as it should, or would I get an error in the normal log if it was unsuccessful at using the port it needs?

Nothing else has changed. My DDNS is serving pages on other ports that are unrelated to Caddy with no issues just as it was previously.

Nothing else is using port 80 or 443 so there should be no reason why Caddy can’t take those ports.

Whitestrake · July 24, 2019, 10:59pm

Server: WebProxy/1.0 Pre-Alpha

This part is concerning.

It means your port forwards aren’t reaching Caddy. They’re forwarding to another proxy server, and this proxy server can’t find Caddy.

mupet0000 · July 25, 2019, 7:16am

Ah I see. So it’s either going to be malware or that new qnap update has done something stupid. I think I’ll contact qnap and ask them as I’m relatively confident my NAS isn’t infected but I will double check.

There must be some network tools I can use to find out what program is running this proxy. I will do some further investigating. Once I’m able to eliminate whatever it is, I assume Caddy will begin working again.

I’ll post again once I have figured it out. Thanks for the help, it’s very much appreciated!

mupet0000 · August 12, 2019, 5:40pm

So after a long back and forth with QNAP on this issue, it appears to be resolved and Caddy is now working again. QNAP were not able to identify the cause or really help me at all, but I noticed some things which led me to resolving the issue.

First of all, I eliminated some programs that were running on my NAS that I did not need running, one of them was Qapache. It was not configured to use port 80 nor did it serve anything (on port 80), other than possibly that weird 503 error, but once Qapache stopped running, port 80 went back to being like any other closed port and refused connections.

Note that Qapache was able to reach the internet on port 80 in order to give a 503 error, without me forwarding any ports for it. Just like Caddy was working on port 80 and 443 without me forwarding any ports for it.

Once port 80 was back in that state I thought that Caddy would begin working again as it could now pick up the unused port. Nope, I still had the exact same output from Caddy. After some thinking I manually port forwarded 80 and 443 using the QNAP router config program and Caddy began working. Remember that I specifically had to disable port 80 and 443 from being forwarded previously as this caused Caddy to fail on the previous QTS update.

Some kind of behaviour has changed since this QTS update that’s for sure, QNAP have no idea of course but luckily with a bit of trial and error it’s sorted.

I hope my trial and error can help anyone facing similar issues with Caddy on QNAP.
I’ll see you at the next QTS update

giopas · August 14, 2019, 2:23pm

Hi,

I am a QNAP and Caddy user. Not sure I can really help you here as it seems you already went far beyond what I did on my Nas (i.e. use caddy for a lot of apps, but in http only).

However I write you to tell that, each time the Caddy package is updatesd, you have to reconfigure the Caddy start script to make it point to the right config file (and use the right ports, if you set them therein).

Finally, I am curious to know how you set up caddy in https, including QTS access (i.e. the Qnap administration GUI), as I always failed to do so.

As it may be OT here, I suggest you to open a post on Qnap community forum.

Thanks!

mupet0000 · August 14, 2019, 3:59pm

Hi giopas,

At the moment my QNAP+Caddy setup is working 100% correctly due to the reasons that I have specified in my previous post. I am aware that with the Caddy update on QTS that the config file was set back to default, but my issue began before the Caddy update and remained after the update. My issue was caused by a QTS update, seperate from the Caddy update. I appreciate you trying to help though.

I pretty much learned how to setup Caddy in https via this thread with much thanks going to Whitestrake for answering all of my questions. I was a complete noob when I started this thread and he helped me from not having any understanding at all to making it work 100%.

The great thing about Caddy is that it does all the HTTPS stuff by itself and you just need to setup the caddyfile pretty much. As long as your caddyfile is correct and your ports are forwarded, caddy does the rest.

My caddyfile is as follows:


	mynashere.myqnapcloud.com
	root /home/Qhttpd/
	gzip	


	proxy /radarr http://localhost:7878 {				# https://radarr.video/
        	transparent
        	header_upstream X-Forwarded-Host {host}
        	}
	
	proxy /sonarr http://localhost:8989 {			# https://sonarr.tv/
			transparent
			header_upstream X-Forwarded-Host {host}
			}

	proxy /nzbget http://localhost:6789 {			# http://nzbget.net/
			transparent
			header_upstream X-Forwarded-Host {host}
			}

	proxy / localhost:8080 {		# blank
			transparent
			header_upstream X-Forwarded-Host {host}
			}




}

The last part is the part that forwards the main QTS admin home page. Then inside my Caddy.sh file I have this line which runs Caddy with my caddyfile at every boot:

caddy -conf /share/CACHEDEV1_DATA/.qpkg/Caddy/Caddyfile -agree -log $QPKG_ROOT/var/logs/caddy.log -pidfile /var/run/caddy.pid &

Using this configuration along with the QTS settings of:

myQNAPCloud, Auto Router Configuration - Port 80 and Port 443 TCP forwarded

I also have NAS Web & Web Server, Multimedia Station forwarded to 8080 and 86. 8080 is the Nas Web that QTS forwards to, with caddy it will become HTTPS. 86 is my HTTP backup that can work without caddy.
You don’t need to use port 86, but you will need to go into Control Panel, Applications, Web Server & change the port from 80, otherwise caddy cannot use port 80. Also disable HTTPS there and also in System, General Settings. Caddy will manage HTTPS so you need to turn off QNAPs.

DDNS enabled using myqnapcloud url
SSL Certificate disabled and OFF (Caddy will manage this)

These are the settings I am using to successfully use QNAP+Caddy. Note that for other apps like Radarr, Sonarr etc, you need to configure them to have an URL base like /radarr so that it’s 192.0.0:1234/radarr rather than just 192.0.0:1234

I hope this helps.

giopas · August 26, 2019, 11:00am

Hi @mupet0000,

Thanks for your explanations!

If you have time, I would ask you advise then to move from HTTP to HTTPS reverse proxy in my configuration.

I try to be as exhaustive as possible:

Current router set up

External port 80 forwarded to internal port 8963 # this is used for HTTP reverse proxy with Caddy
External port 443 forwarded to internal port 443 # this is currently used to access Qnap QTS in HTTPS
IP static
myqnapcloud DDNS enabled

QTS configuration

QTS System port (HTTPS only): 443
Web Server: not enabled
Virtual Host: not enabled
WebDAV: enabled on HTTP port 5000, not enabled on HTTPS port
SSL purchased certificate on qnap.mydomain.com loaded # but I would happily use the one managed by Caddy

Caddy file

(gzipconf) {
	gzip {
		ext *
		level 7
		min_length 1
	}
}

(addheader) {
	header / {
		Strict-Transport-Security "max-age=31536000;"
		X-XSS-Protection "1; mode=block"
		X-Content-Type-Options "nosniff"
		X-Frame-Options "DENY"
		Referrer-Policy "strict-origin-when-cross-origin"
		-Server
	}
}

http://webdav.mydomain.com:8963 {
	import gzipconf
	import addheader
	proxy / http://10.0.0.1:5000 {
		keepalive 32
		transparent
		websocket
	}
}

http://sonarr.mydomain.com:8963 {
	import gzipconf
	import addheader
	proxy / http://10.0.0.30:8989 {
		keepalive 32
		transparent
		websocket
	}
}

http://transmission.mydomain.com:8963 {
	import gzipconf
	import addheader
	proxy / http://10.0.0.30:49091 {
		keepalive 32
		transparent
		websocket
	}
}

Caddy conf
caddy -conf /share/homes/user/script/caddy.conf -agree -port 20015 -root /home/Qthttpd -log $QPKG_root/var/logs/caddy.log -pidfile /var/run/caddy.pid &

As you can see, I can access to the different services simply using a subdomain, and I can access to QTS with https://qnap.mydomain.com (as any https calls are now taken by QTS).

Starting from this configuration, how could I bring everything to HTTPS?

Thanks for your hints!

Best regards,

giopas

giopas · September 25, 2019, 7:55pm

Hi,

I am finally trying to do the jump to HTTP but I have a problem.

To summarize: I can now access to QTS via HTTP only on port 8080 and disabled the Webserver. So port 443 and 80 should be free (those are also been forwarded on the router to port 8964 and 8963 for Caddy).

I have then configured Caddy to point to the config file, which looks as follows:

sub.domain.com {
proxy / https://10.1.1.1:8080 {
keepalive 32
transparent
websocket
}
}

But the problem, when I start Caddy is the following:

Activating privacy features… 2019/09/25 21:38:22 [INFO] [sub.domain.com] acme: Obtaining bundled SAN certificate

2019/09/25 21:38:23 [INFO] [sub.domain.com] acme: Obtaining bundled SAN certificate

2019/09/25 21:38:24 [INFO] [sub.domain.com] acme: Obtaining bundled SAN certificate

2019/09/25 21:38:26 [INFO] [sub.domain.com] acme: Obtaining bundled SAN certificate

2019/09/25 21:38:27 [INFO] [sub.domain.com] acme: Obtaining bundled SAN certificate

2019/09/25 21:38:28 [INFO] [sub.domain.com] acme: Obtaining bundled SAN certificate

2019/09/25 21:38:30 failed to obtain certificate: acme: error: 429 :: POST :: https://acme-v02.api.letsencrypt.org/acme/new-order :: urn:ietf:params:acme:error:rateLimited :: Error creating new order :: too many failed authorizations recently: see Rate Limits - Let's Encrypt, url:

What am I doing wrong/missing?

Whitestrake · September 25, 2019, 11:08pm

You’ve been rate limited.

The link has more info on rate limits.

giopas · September 26, 2019, 5:26am

Thank you for your answer Whitestrake.

Let’s add some information and question, so that I can more easily - with your support - idea tidy the problem.

Additional info:

In the caddy config file I am creating around 15 subdomains (sub.domain.com, sub2.domain.com, etc.). → would be this the reason for the problem?
One subdomain already has one purchased certificate (which however I did not specified in the config). → shall I modify the config file to point to the actual valid certificate? How?
Caddy (when I only use HTTP and use 443 to access directly QTS) currently uses port 8964 (HTTPS) and 8963 (HTTP). When trying to use HTTP, beside trying to free up port 443, I simply changed the router port forwarding of 443 and 80 to the two mentioned ports. → is that the correct way to do?
→ shall I understand from the error that the config is correct in terms of port forwarding/settings?
I do not remember how to check, verify and/or change the two ports used by Caddy (8964 and 8963). → How can I see it?
Beside waiting for a week to test again, is there a flag I can use to at least test that the config file is correct and that next week, once discovered what I am not doing well, the certificates can be issued?

Thanks!

Whitestrake · September 26, 2019, 5:38am

Not likely - you’ve got up to 50 per week.

The main limit is Certificates per Registered Domain (50 per week).
—Rate Limits - Let's Encrypt

Taking a closer look at the error you received (too many failed authorizations recently), I would wager that you’ve actually hit this one:

There is a Failed Validation limit of 5 failures per account, per hostname, per hour.
—Rate Limits - Let's Encrypt

Have a look at the tls directive:

To use Caddy with your own certificate and key:

tls cert key

—https://caddyserver.com/docs/tls#syntax

There’s multiple ways to go about it. This method should work, but forwarding 80/443 straight to 80/443 on the Caddy host is the simplest method. The error implies that something isn’t working, and doing port detouring might be the cause.

Those are not the default ports Caddy uses, so you must have changed them. If you didn’t do it in your Caddyfile, you probably did it by flag.

https://caddyserver.com/docs/cli#http-port

Remember, too, that Caddy outputs what sites it is serving, and on what scheme/port, when it starts up. You can double check that output to know for sure what Caddy’s listening on locally.

We recommend, when testing, troubleshooting, and configuring new or problematic HTTPS setups, that you use the LetsEncrypt ACME staging environment. They have relaxed rate limits there, and a rate limit on the production environment won’t stop you from testing on the staging environment. Use the -ca flag when running Caddy to specify which ACME endpoint to use.

giopas · October 2, 2019, 9:37pm

So, I tried again. I have completely shut down Qhttpd which was taking port 80 and ensured that ports 80 and 443 were free and forwarded by the router to the Nas.

Then I run Caddy with the following command line:

caddy -conf /share/Public/caddy/caddy.conf -agree -port 20015 -root /home/Qhttpd -log $QPKG_ROOT/var/logs/caddy.log -pidfile /var/run/caddy.pid &

The file is the following:

(gzipconf) {
        gzip {
                ext *
                level 7
                min_length 1
        }
}

(addheader) {
        header / {
                Strict-Transport-Security "max-age=31536000;"
                X-XSS-Protection "1; mode=block"
                X-Content-Type-Options "nosniff"
                X-Frame-Options "DENY"
                Referrer-Policy "strict-origin-when-cross-origin"
                -Server
        }
}

test.domain.com {
tls /share/Public/SSLcertificate.crt /share/Public/SSLprivatekey.key
        import gzipconf
        import addheader
        proxy / https://192.168.1.3:8080 {
                keepalive 32
                transparent
                websocket
        }
}

nas.domain.com {
tls sub@domain.com
        import gzipconf
        import addheader
        proxy / https://192.168.1.10:8080 {
                keepalive 32
                transparent
                websocket
        }
}

sub.domain.com {
tls sub@domain.com
        import gzipconf
        import addheader
        proxy / https://192.168.1.10:8080 {
                keepalive 32
                transparent
                websocket
        }
}

The errors I got are the following:

[/share/CACHEDEV1_DATA/.qpkg/Caddy] # Activating privacy features... 2019/10/01 14:58:38 [INFO] [nas.domain.com] acme: Obtaining bundled SAN certificate
2019/10/01 14:58:39 [INFO] [nas.domain.com] AuthURL: https://acme-v02.api.letsencrypt.org/acme/authz-v3/5851733xx
2019/10/01 14:58:39 [INFO] [nas.domain.com] acme: use tls-alpn-01 solver
2019/10/01 14:58:39 [INFO] [nas.domain.com] acme: Trying to solve TLS-ALPN-01
2019/10/01 14:58:46 [INFO] [nas.domain.com] The server validated our request
2019/10/01 14:58:46 [INFO] [nas.domain.com] acme: Validations succeeded; requesting certificates
2019/10/01 14:58:48 [INFO] [nas.domain.com] Server responded with a certificate.2019/10/01 14:58:48 [INFO] [sub.domain.com] acme: Obtaining bundled SAN certificate
2019/10/01 14:58:49 [INFO] [sub.domain.com] AuthURL: https://acme-v02.api.letsencrypt.org/acme/authz-v3/5851752xx
2019/10/01 14:58:49 [INFO] [sub.domain.com] acme: Could not find solver for: tls-alpn-01
2019/10/01 14:58:49 [INFO] [sub.domain.com] acme: use http-01 solver
2019/10/01 14:58:49 [INFO] [sub.domain.com] acme: Trying to solve HTTP-01
2019/10/01 14:58:57 [INFO] Unable to deactivated authorizations: https://acme-v02.api.letsencrypt.org/acme/authz-v3/5851752xx
2019/10/01 14:58:58 [INFO] [sub.domain.com] acme: Obtaining bundled SAN certificate
2019/10/01 14:58:59 [INFO] [sub.domain.com] AuthURL: https://acme-v02.api.letsencrypt.org/acme/authz-v3/5851772xx
2019/10/01 14:58:59 [INFO] [sub.domain.com] acme: Could not find solver for: tls-alpn-01
2019/10/01 14:58:59 [INFO] [sub.domain.com] acme: use http-01 solver
2019/10/01 14:58:59 [INFO] [sub.domain.com] acme: Trying to solve HTTP-01
2019/10/01 14:58:59 [INFO] Unable to deactivated authorizations: https://acme-v02.api.letsencrypt.org/acme/authz-v3/5851772xx
2019/10/01 14:59:00 [INFO] [sub.domain.com] acme: Obtaining bundled SAN certificate
2019/10/01 14:59:01 [INFO] [sub.domain.com] AuthURL: https://acme-v02.api.letsencrypt.org/acme/authz-v3/5851776xx
2019/10/01 14:59:01 [INFO] [sub.domain.com] acme: Could not find solver for: tls-alpn-01
2019/10/01 14:59:01 [INFO] [sub.domain.com] acme: use http-01 solver
2019/10/01 14:59:01 [INFO] [sub.domain.com] acme: Trying to solve HTTP-01
2019/10/01 14:59:01 [INFO] Unable to deactivated authorizations: https://acme-v02.api.letsencrypt.org/acme/authz-v3/5851776xx
2019/10/01 14:59:02 [INFO] [sub.domain.com] acme: Obtaining bundled SAN certificate
2019/10/01 14:59:03 [INFO] [sub.domain.com] AuthURL: https://acme-v02.api.letsencrypt.org/acme/authz-v3/5851780xx
2019/10/01 14:59:03 [INFO] [sub.domain.com] acme: use tls-alpn-01 solver
2019/10/01 14:59:03 [INFO] [sub.domain.com] acme: Trying to solve TLS-ALPN-01
2019/10/01 14:59:06 [INFO] Unable to deactivated authorizations: https://acme-v02.api.letsencrypt.org/acme/authz-v3/5851780xx
2019/10/01 14:59:07 [INFO] [sub.domain.com] acme: Obtaining bundled SAN certificate
[/share/CACHEDEV1_DATA/.qpkg/Caddy] # 2019/10/01 14:59:08 [INFO] [sub.domain.com] AuthURL: https://acme-v02.api.letsencrypt.org/acme/authz-v3/5851794xx
2019/10/01 14:59:08 [INFO] [sub.domain.com] acme: use tls-alpn-01 solver
2019/10/01 14:59:08 [INFO] [sub.domain.com] acme: Trying to solve TLS-ALPN-01
2019/10/01 14:59:09 [INFO] Unable to deactivated authorizations: https://acme-v02.api.letsencrypt.org/acme/authz-v3/5851794xx
2019/10/01 14:59:10 [INFO] [sub.domain.com] acme: Obtaining bundled SAN certificate
[/share/CACHEDEV1_DATA/.qpkg/Caddy] # 2019/10/01 14:59:11 failed to obtain certificate: acme: error: 429 :: POST :: https://acme-v02.api.letsencrypt.org/acme/new-order :: urn:ietf:params:acme:error:rateLimited :: Error creating new order :: too many failed authorizations recently: see https://letsencrypt.org/docs/rate-limits/, url:

Where am I doing wrong?

Whitestrake · October 2, 2019, 10:24pm

Not sure. Is the subdomain pointed at the correct IP address in DNS?

There might be more info in the (redacted) AuthURLs, and I can’t do any diagnostics (because I assume you don’t own domain.com and don’t know your real domain).

What I can see is that you got rate limited again at the end there. Did you swap to the staging server while testing?

giopas · October 3, 2019, 5:38am

I did not wanted to publicly write the domain name. I can send you the unredacted in private (if it is possible).

Ok, looking at the link, in one case I found that I did not point the DNS of one subdomain to the right IP (redacted output below):

detail": "DNS problem: NXDOMAIN looking up A for sub.domain.com",
        "status": 400

I have fixed it now.

For the other, I see that the certificate is created, the DNS points to the right address and to the right port (redacted output below):

{
      "type": "tls-alpn-01",
      "status": "valid",
      "url": "https://acme-v02.api.letsencrypt.org/acme/chall-v3/5851733xx/Lm2cbA",
      "token": "NKXBWbp1CCeEYwLokpv02gvRfQ_gh696W-Z7cx6Vu_t",
      "validationRecord": [
        {
          "hostname": "test.domain.com",
          "port": "443",
          "addressesResolved": [
            "187.92.110.9"
          ],
          "addressUsed": "187.92.110.9"
        }

I will make a test again to see if it works now.

I have two questions:

Why Caddy, even if there are many other other reverse proxies in the config file, if it finds an error (like the above) in one, it stops and does not try to also make the other working? It could could be useful also to try different set-up in one run.
Is the config and port settings correct in the configuration above or I am still misconfiguring something? Is it useful or is it a mistake to use the variable “-port 20015” to launch Caddy (see post above)? Does it make any difference for caddy to listen ports 80 and 443?

Thanks a lot for your help!

Whitestrake · October 3, 2019, 5:49am

This is a conscious decision the developers made when designing Caddy’s core behaviour. When the user configures the server to grab certs for two different sites, but it can only grab one, the question must be raised:

Do we start up with behaviour that is, strictly speaking, not what the user asked for? (i.e. proceed serving one site)
Or, do we tell the user that we’re not able to do what they’ve configured us to do? (i.e. fail out, produce an error for the user to troubleshoot and fix)

The decision was made to have the server strictly only do what the user asks, and be loud and obvious with problems. It’s all too common for simple log entries to be ignored, or worse, never logged at all (we see this constantly). During initial startup and configuration is the best time to ensure a user is aware that the server cannot support the configuration they’ve asked for. This decision was supported by the fact that it is possible, if absolutely necessary, for the user to manually configure Caddy such as to provide a best-effort attempt to serve the sites (for example, via On-Demand TLS).

But, ultimately, if you say “Serve these two sites with managed HTTPS!”, and Caddy can’t serve those two sites with managed HTTPS, it’s going to tell you as much.

The configuration looks serviceable. -port only governs which port Caddy will try to use for non-HTTPS-managed sites, so for all the valid sites with Automatic HTTPS, this flag is irrelevant. (To change the main ports, you’d need -http-port and -https-port.)

You might consider the -email flag (e.g. -email sub@domain.com) to set the default email address instead of specifying it in all the individual sites.

giopas · October 3, 2019, 7:46am

Thank you, this is helpful to understand the Caddy logic!

In relation to the ports, shall then I forward the port 80 of the router to port 20015? or should I change the lunch command as follows?

caddy -conf /share/Public/caddy/caddy.conf -agree -http-port 80 -https-port 443 -email sub@domain.com -root /home/Qhttpd -log $QPKG_ROOT/var/logs/caddy.log -pidfile /var/run/caddy.pid &

Again, a BIG thanks!

Whitestrake · October 3, 2019, 7:59am

Neither.

Remove all port-related flags. You don’t need to change the default -port (its default is :2015, but I don’t even think you’ve got any sites unmanaged). You don’t need to set the -http-port to 80, because that’s its default too, as is -https-port defaulted to 443.

giopas · October 3, 2019, 4:00pm

Ok, I tried again (using above settings) and all certificates are created correctly: “server responded with a certificate”.

At the end of the process, have around 15 reverse proxies, I have the following output:

Serving HTTPS on port 20015
https://sub.domain.com:20015
https://test.domain.com:20015

Serving HTTP on port 80
http://sub.domain.com
http://test.domain.com

I can confirm that ports 80 and 443 are correctly forwarded to the NAS. Moreover I can see that Caddy has taken port 80, but not port 443 (which not used).

If I type now sub.domain.com on the browser I am now redirected to https://sub.domain.com:20015.

Am I still missing something?

EDIT: It seems it is solved now taking out -port 20015 from the launching instruction, but I still have one problem for the (sole) domain for which I have a purchased certificate:

sub.domain.com {
tls /share/Public/SSLcertificate.crt /share/Public/SSLprivatekey.key
        import gzipconf
        import addheader
        proxy / http://192.168.1910:8080 {
                keepalive 32
                transparent
                websocket
        }
}

Indeed the output that I have from caddy is the following:

Serving HTTPS on port 2015
https://sub.domain.com:2015

I do not understand why it uses port 2015 even if there is no port set on caddy at launch…

Thanks (I am almost there!)

Whitestrake · October 3, 2019, 11:10pm

This is expected behaviour. Automatic HTTPS is what configures the ports, as well as the redirect listener.

Caddy automatically enables HTTPS for all your sites, given that some reasonable criteria are met:

Certificates and keys are not provided by you

—Automatic HTTPS — Caddy Documentation

Given that you’ve provided your own certificate, you’ll need to configure this manually.

Like so:

https://sub.example.com {
  # [site config here]
}

http://sub.example.com {
  redir https://sub.example.com{uri}
}

giopas · October 4, 2019, 12:04pm

Ok, so if my understanding is correct, I have to “split” the instruction to ensure it works. I will test it!

By the way, and I am continuing here as it relates to Caddy and QNAP, I have discovered that using HTTPS will create a problem for many QNAP applications used such as Notification Center, Help desk, Malware Remover, Photo Station, Download Station, etc… which are actually standalone applications developed by QNAP but linked to QTS through a proxy (for which the url is for example user.myqnapcloud.com/…).

For those I receive the error “connection refused”.

Should I specify something in the config file of Caddy to tell that it should also take into consideration all other applications?

Thanks again