Caddy 2 reverse proxying an existing nginx setup

Help
1

Hi all, i’ve been having some trouble setting up a reverse proxy of a nested nginx server, and am 100% out of ideas at this point. When researching i noticed that caddy seems to have a really friendly community on here, so figured you guys might be as helpful for me as you seem to be with most posts on here. Apologies in advance though: i’m not a webdev by trade, im more of a data scientist so sorry if i come across as a bit of a novice at times in this thread.

With that preface out of the way:

1. The problem I’m having:

I am attempting to deploy a site for a research project on AWS EC2 via docker containers. The site is comprised of a landing page, and a backend for data management. The frontend is just a simple sveltekit site: nothing fancy and i have got that working perfectly fine. The issue is arising with the data management backend.
This backend needs to be a deployment of Invenio. So, i started by building an instance of the invenio website using their invenio-cli tool, adding the containers produced by this CLI to my own docker compose file, and simply configuring my caddy container to reverse proxy the subdomain invenio.am-d-model.eu to the nginx server that acts as the frontend for invenio. And this is where the problem arises: the nginx configuration for invenio forces a HTTP to HTTPS upgrade, and that appears to be causing an error 502. I do not want to tamper too much with the invenio nginx configuration files because:

  • I want to be able to regenerate/upgrade this subdomain’s site without having to do a bunch of manual patching.
  • The underlying invenio server connects to nginx via uwsgi and i don’t want to have to get lost in the weeds delving into invenio’s source code to figure out how that works.
  • I am not extremely familiar with nginx and don’t want to break or misconfigure something.

So, i’ve tried a number of workarounds to solve this issue. Here’s a brief summary of what i’ve tried so far:

  • Having caddy ignore the certs from the nginx server
  • Replacing the nginx server’s certs with those fetched by caddy
  • Injecting extra config files into the directory that invenio’s nginx config wildcard imports (the one with default.conf) via docker compose’s volumes attribute.
  • Switching the underlying uwsgi server into http mode and directly proxying it via caddy
  • Using the caddy-uwsgi plugin to directly interface with invenio similar to what nginx is doing

After all of these attempts (and probably more i can’t remember) i have had to accept that i’m out of ideas and come here to hopefully get help from people with more expertise in web infrastructure.

2. Error messages and/or full log output:

Below is a full output of a curl request to the subdomain and the full docker compose logs since the timestamp just before that request.

$ curl -iv https://invenio.am-d-model.eu
* Host invenio.am-d-model.eu:443 was resolved.
* IPv6: (none)
* IPv4: 54.154.34.26
*   Trying 54.154.34.26:443...
* Connected to invenio.am-d-model.eu (54.154.34.26) port 443
* ALPN: curl offers h2,http/1.1
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
*  CAfile: /etc/pki/tls/certs/ca-bundle.crt
*  CApath: none
* TLSv1.3 (IN), TLS handshake, Server hello (2):
* TLSv1.3 (IN), TLS handshake, Encrypted Extensions (8):
* TLSv1.3 (IN), TLS handshake, Certificate (11):
* TLSv1.3 (IN), TLS handshake, CERT verify (15):
* TLSv1.3 (IN), TLS handshake, Finished (20):
* TLSv1.3 (OUT), TLS change cipher, Change cipher spec (1):
* TLSv1.3 (OUT), TLS handshake, Finished (20):
* SSL connection using TLSv1.3 / TLS_AES_128_GCM_SHA256 / X25519 / id-ecPublicKey
* ALPN: server accepted h2
* Server certificate:
*  subject: CN=invenio.am-d-model.eu
*  start date: Feb  4 11:14:00 2025 GMT
*  expire date: May  5 11:13:59 2025 GMT
*  subjectAltName: host "invenio.am-d-model.eu" matched cert's "invenio.am-d-model.eu"
*  issuer: C=US; O=Let's Encrypt; CN=E5
*  SSL certificate verify ok.
*   Certificate level 0: Public key type EC/prime256v1 (256/128 Bits/secBits), signed using ecdsa-with-SHA384
*   Certificate level 1: Public key type EC/secp384r1 (384/192 Bits/secBits), signed using sha256WithRSAEncryption
*   Certificate level 2: Public key type RSA (4096/152 Bits/secBits), signed using sha256WithRSAEncryption
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
* using HTTP/2
* [HTTP/2] [1] OPENED stream for https://invenio.am-d-model.eu/
* [HTTP/2] [1] [:method: GET]
* [HTTP/2] [1] [:scheme: https]
* [HTTP/2] [1] [:authority: invenio.am-d-model.eu]
* [HTTP/2] [1] [:path: /]
* [HTTP/2] [1] [user-agent: curl/8.5.0]
* [HTTP/2] [1] [accept: */*]
> GET / HTTP/2
> Host: invenio.am-d-model.eu
> User-Agent: curl/8.5.0
> Accept: */*
>
< HTTP/2 502
HTTP/2 502
< alt-svc: h3=":443"; ma=2592000
alt-svc: h3=":443"; ma=2592000
< server: Caddy
server: Caddy
< content-length: 0
content-length: 0
< date: Fri, 07 Feb 2025 11:37:00 GMT
date: Fri, 07 Feb 2025 11:37:00 GMT

<
* Connection #0 to host invenio.am-d-model.eu left intact

$ docker compose logs -t --since 2025-02-07T11:36:49.649063177Z
WARN[0000] The "Kow2" variable is not set. Defaulting to a blank string.
WARN[0000] The "uHLDTqL88lYv" variable is not set. Defaulting to a blank string.
WARN[0000] The "WXTgD3ieMtEpYCVkh7Z93B" variable is not set. Defaulting to a blank string.
WARN[0000] The "Kow2" variable is not set. Defaulting to a blank string.
WARN[0000] The "uHLDTqL88lYv" variable is not set. Defaulting to a blank string.
WARN[0000] The "WXTgD3ieMtEpYCVkh7Z93B" variable is not set. Defaulting to a blank string.
WARN[0000] The "Kow2" variable is not set. Defaulting to a blank string.
WARN[0000] The "uHLDTqL88lYv" variable is not set. Defaulting to a blank string.
WARN[0000] The "WXTgD3ieMtEpYCVkh7Z93B" variable is not set. Defaulting to a blank string.
WARN[0000] The "Kow2" variable is not set. Defaulting to a blank string.
WARN[0000] The "uHLDTqL88lYv" variable is not set. Defaulting to a blank string.
WARN[0000] The "WXTgD3ieMtEpYCVkh7Z93B" variable is not set. Defaulting to a blank string.
WARN[0000] The "Kow2" variable is not set. Defaulting to a blank string.
WARN[0000] The "uHLDTqL88lYv" variable is not set. Defaulting to a blank string.
WARN[0000] The "WXTgD3ieMtEpYCVkh7Z93B" variable is not set. Defaulting to a blank string.
WARN[0000] The "Kow2" variable is not set. Defaulting to a blank string.
WARN[0000] The "uHLDTqL88lYv" variable is not set. Defaulting to a blank string.
WARN[0000] The "WXTgD3ieMtEpYCVkh7Z93B" variable is not set. Defaulting to a blank string.
WARN[0000] The "Kow2" variable is not set. Defaulting to a blank string.
WARN[0000] The "uHLDTqL88lYv" variable is not set. Defaulting to a blank string.
WARN[0000] The "WXTgD3ieMtEpYCVkh7Z93B" variable is not set. Defaulting to a blank string.
WARN[0000] The "Kow2" variable is not set. Defaulting to a blank string.
WARN[0000] The "uHLDTqL88lYv" variable is not set. Defaulting to a blank string.
WARN[0000] The "WXTgD3ieMtEpYCVkh7Z93B" variable is not set. Defaulting to a blank string.
WARN[0000] The "Kow2" variable is not set. Defaulting to a blank string.
WARN[0000] The "uHLDTqL88lYv" variable is not set. Defaulting to a blank string.
WARN[0000] The "WXTgD3ieMtEpYCVkh7Z93B" variable is not set. Defaulting to a blank string.
WARN[0000] The "Kow2" variable is not set. Defaulting to a blank string.
WARN[0000] The "uHLDTqL88lYv" variable is not set. Defaulting to a blank string.
WARN[0000] The "WXTgD3ieMtEpYCVkh7Z93B" variable is not set. Defaulting to a blank string.
WARN[0000] The "Kow2" variable is not set. Defaulting to a blank string.
WARN[0000] The "uHLDTqL88lYv" variable is not set. Defaulting to a blank string.
WARN[0000] The "WXTgD3ieMtEpYCVkh7Z93B" variable is not set. Defaulting to a blank string.
WARN[0000] The "Kow2" variable is not set. Defaulting to a blank string.
WARN[0000] The "uHLDTqL88lYv" variable is not set. Defaulting to a blank string.
WARN[0000] The "WXTgD3ieMtEpYCVkh7Z93B" variable is not set. Defaulting to a blank string.
WARN[0000] The "Kow2" variable is not set. Defaulting to a blank string.
WARN[0000] The "uHLDTqL88lYv" variable is not set. Defaulting to a blank string.
WARN[0000] The "WXTgD3ieMtEpYCVkh7Z93B" variable is not set. Defaulting to a blank string.
WARN[0000] The "Kow2" variable is not set. Defaulting to a blank string.
WARN[0000] The "uHLDTqL88lYv" variable is not set. Defaulting to a blank string.
WARN[0000] The "WXTgD3ieMtEpYCVkh7Z93B" variable is not set. Defaulting to a blank string.
caddy-1  | 2025-02-07T11:37:00.829109627Z {"level":"debug","ts":1738928220.8011413,"logger":"events","msg":"event","name":"tls_get_certificate","id":"c325847e-82a3-4d4d-85ef-a8efb905022d","origin":"tls","data":{"client_hello":{"CipherSuites":[4866,4867,4865,4868,49196,49200,52393,52392,49325,49195,49199,49324,49187,49191,49162,49172,49161,49171,157,49309,156,49308,61,60,53,47,159,52394,49311,158,49310,107,103,57,51,255],"ServerName":"invenio.am-d-model.eu","SupportedCurves":[29,23,30,25,24,256,257,258,259,260],"SupportedPoints":"AAEC","SignatureSchemes":[1027,1283,1539,2055,2056,2057,2058,2059,2052,2053,2054,1025,1281,1537,771,769],"SupportedProtos":["h2","http/1.1"],"SupportedVersions":[772,771],"RemoteAddr":{"IP":"54.154.34.26","Port":37106,"Zone":""},"LocalAddr":{"IP":"172.29.0.3","Port":443,"Zone":""}}}}
caddy-1  | 2025-02-07T11:37:00.829840443Z {"level":"debug","ts":1738928220.8290317,"logger":"tls.handshake","msg":"choosing certificate","identifier":"invenio.am-d-model.eu","num_choices":1}
caddy-1  | 2025-02-07T11:37:00.829854426Z {"level":"debug","ts":1738928220.8296063,"logger":"tls.handshake","msg":"default certificate selection results","identifier":"invenio.am-d-model.eu","subjects":["invenio.am-d-model.eu"],"managed":true,"issuer_key":"acme-v02.api.letsencrypt.org-directory","hash":"1bb95de7535d0b9de13ec6ca46b18fc2ef805456f76c7461bd77129a612298fb"}
caddy-1  | 2025-02-07T11:37:00.830498420Z {"level":"debug","ts":1738928220.8296707,"logger":"tls.handshake","msg":"matched certificate in cache","remote_ip":"54.154.34.26","remote_port":"37106","subjects":["invenio.am-d-model.eu"],"managed":true,"expiration":1746443640,"hash":"1bb95de7535d0b9de13ec6ca46b18fc2ef805456f76c7461bd77129a612298fb"}
caddy-1  | 2025-02-07T11:37:00.886776252Z {"level":"debug","ts":1738928220.8801515,"logger":"http.handlers.reverse_proxy","msg":"selected upstream","dial":"invenio.am-d-model.eu:8080","total_upstreams":1}
caddy-1  | 2025-02-07T11:37:00.898132068Z {"level":"debug","ts":1738928220.896446,"logger":"http.handlers.reverse_proxy","msg":"upstream roundtrip","upstream":"invenio.am-d-model.eu:8080","duration":0.007683084,"request":{"remote_ip":"54.154.34.26","remote_port":"37106","client_ip":"54.154.34.26","proto":"HTTP/2.0","method":"GET","host":"invenio.am-d-model.eu","uri":"/","headers":{"X-Real-Ip":["54.154.34.26"],"Accept":["*/*"],"X-Forwarded-For":["54.154.34.26"],"X-Forwarded-Proto":["https"],"X-Forwarded-Host":["invenio.am-d-model.eu"],"User-Agent":["curl/8.5.0"]},"tls":{"resumed":false,"version":772,"cipher_suite":4865,"proto":"h2","server_name":"invenio.am-d-model.eu"}},"error":"dial tcp 172.29.0.2:8080: connect: connection refused"}
caddy-1  | 2025-02-07T11:37:00.899288283Z {"level":"error","ts":1738928220.899165,"logger":"http.log.error","msg":"dial tcp 172.29.0.2:8080: connect: connection refused","request":{"remote_ip":"54.154.34.26","remote_port":"37106","client_ip":"54.154.34.26","proto":"HTTP/2.0","method":"GET","host":"invenio.am-d-model.eu","uri":"/","headers":{"User-Agent":["curl/8.5.0"],"Accept":["*/*"]},"tls":{"resumed":false,"version":772,"cipher_suite":4865,"proto":"h2","server_name":"invenio.am-d-model.eu"}},"duration":0.022681359,"status":502,"err_id":"ybpg0269q","err_trace":"reverseproxy.statusError (reverseproxy.go:1373)"}
mq-1     | 2025-02-07T11:36:49.894101774Z 2025-02-07 11:36:49.859024+00:00 [info] <0.1210.0> accepting AMQP connection <0.1210.0> (172.18.0.2:37066 -> 172.18.0.8:5672)
mq-1     | 2025-02-07T11:36:49.935151286Z 2025-02-07 11:36:49.864382+00:00 [info] <0.1210.0> connection <0.1210.0> (172.18.0.2:37066 -> 172.18.0.8:5672): user 'guest' authenticated and granted access to vhost '/'
mq-1     | 2025-02-07T11:36:59.792083760Z 2025-02-07 11:36:59.790088+00:00 [info] <0.1221.0> accepting AMQP connection <0.1221.0> (172.18.0.2:43606 -> 172.18.0.8:5672)
mq-1     | 2025-02-07T11:36:59.794345347Z 2025-02-07 11:36:59.794017+00:00 [info] <0.1221.0> connection <0.1221.0> (172.18.0.2:43606 -> 172.18.0.8:5672): user 'guest' authenticated and granted access to vhost '/'
worker-1  | 2025-02-07T11:36:49.815508285Z [2025-02-07 11:36:49,771: INFO/Beat] Scheduler: Sending due task indexer (invenio_records_resources.tasks.manage_indexer_queues)
worker-1  | 2025-02-07T11:36:49.816567852Z [2025-02-07 11:36:49,779: INFO/MainProcess] Task invenio_records_resources.tasks.manage_indexer_queues[600df5e2-4a7a-457a-a621-e1def501f0c2] received
worker-1  | 2025-02-07T11:36:49.894515835Z [2025-02-07 11:36:49,877: INFO/ForkPoolWorker-3] Task invenio_records_resources.tasks.manage_indexer_queues[600df5e2-4a7a-457a-a621-e1def501f0c2] succeeded in 0.08726743690203875s: None
worker-1  | 2025-02-07T11:36:59.771790879Z [2025-02-07 11:36:59,771: INFO/Beat] Scheduler: Sending due task indexer (invenio_records_resources.tasks.manage_indexer_queues)
worker-1  | 2025-02-07T11:36:59.776751896Z [2025-02-07 11:36:59,775: INFO/MainProcess] Task invenio_records_resources.tasks.manage_indexer_queues[f7af8198-71bf-49b8-993d-ebf1a86fdd5f] received
worker-1  | 2025-02-07T11:36:59.804618313Z [2025-02-07 11:36:59,804: INFO/ForkPoolWorker-3] Task invenio_records_resources.tasks.manage_indexer_queues[f7af8198-71bf-49b8-993d-ebf1a86fdd5f] succeeded in 0.02520745398942381s: None
search-1  | 2025-02-07T11:36:49.649063177Z [2025-02-07T11:36:49,648][INFO ][o.r.Reflections          ] [33663bb1a797] Reflections took 239 ms to scan 1 urls, producing 26 keys and 67 values
search-1  | 2025-02-07T11:36:49.815463273Z [2025-02-07T11:36:49,781][INFO ][o.r.Reflections          ] [33663bb1a797] Reflections took 25 ms to scan 1 urls, producing 3 keys and 5 values
search-1  | 2025-02-07T11:36:50.794111409Z [2025-02-07T11:36:50,793][WARN ][o.o.s.p.SQLPlugin        ] [33663bb1a797] Master key is a required config for using create and update datasource APIs. Please set plugins.query.datasources.encryption.masterkey config in opensearch.yml in all the cluster nodes. More details can be found here: https://github.com/opensearch-project/sql/blob/main/docs/user/ppl/admin/datasources.rst#master-key-config-for-encrypting-credential-information
search-1  | 2025-02-07T11:36:53.045475522Z [2025-02-07T11:36:53,045][INFO ][o.o.t.NettyAllocator     ] [33663bb1a797] creating NettyAllocator with the following configs: [name=unpooled, suggested_max_allocation_size=256kb, factors={opensearch.unsafe.use_unpooled_allocator=null, g1gc_enabled=true, g1gc_region_size=1mb, heap_size=1gb}]
search-1  | 2025-02-07T11:36:53.066215960Z [2025-02-07T11:36:53,065][INFO ][o.o.s.s.t.SSLConfig      ] [33663bb1a797] SSL dual mode is disabled
search-1  | 2025-02-07T11:36:53.317961863Z [2025-02-07T11:36:53,317][INFO ][o.o.d.DiscoveryModule    ] [33663bb1a797] using discovery type [zen] and seed hosts providers [settings]
search-1  | 2025-02-07T11:36:54.706833307Z [2025-02-07T11:36:54,706][WARN ][o.o.g.DanglingIndicesState] [33663bb1a797] gateway.auto_import_dangling_indices is disabled, dangling indices will not be automatically detected or imported and must be managed manually
search-1  | 2025-02-07T11:36:56.276731485Z [2025-02-07T11:36:56,276][INFO ][o.o.p.h.c.PerformanceAnalyzerConfigAction] [33663bb1a797] PerformanceAnalyzer Enabled: false
search-1  | 2025-02-07T11:36:56.489165861Z [2025-02-07T11:36:56,485][INFO ][o.o.n.Node               ] [33663bb1a797] initialized
search-1  | 2025-02-07T11:36:56.489239995Z [2025-02-07T11:36:56,486][INFO ][o.o.n.Node               ] [33663bb1a797] starting ...
search-1  | 2025-02-07T11:36:56.529600395Z [2025-02-07T11:36:56,513][ERROR][o.o.s.l.BuiltinLogTypeLoader] [33663bb1a797] Failed loading builtin log types from disk!
search-1  | 2025-02-07T11:36:56.529633732Z java.nio.file.FileSystemNotFoundException: null
search-1  | 2025-02-07T11:36:56.529638546Z      at jdk.zipfs@21.0.4/jdk.nio.zipfs.ZipFileSystemProvider.getFileSystem(ZipFileSystemProvider.java:156) ~[?:?]
search-1  | 2025-02-07T11:36:56.529642833Z      at jdk.zipfs@21.0.4/jdk.nio.zipfs.ZipFileSystemProvider.getPath(ZipFileSystemProvider.java:142) ~[?:?]
search-1  | 2025-02-07T11:36:56.529646277Z      at java.base/java.nio.file.Path.of(Path.java:209) ~[?:?]
search-1  | 2025-02-07T11:36:56.529649908Z      at java.base/java.nio.file.Paths.get(Paths.java:98) ~[?:?]
search-1  | 2025-02-07T11:36:56.529653355Z      at org.opensearch.securityanalytics.logtype.BuiltinLogTypeLoader.loadBuiltinLogTypes(BuiltinLogTypeLoader.java:73) ~[opensearch-security-analytics-2.17.1.0.jar:2.17.1.0]
search-1  | 2025-02-07T11:36:56.529657206Z      at org.opensearch.securityanalytics.logtype.BuiltinLogTypeLoader.ensureLogTypesLoaded(BuiltinLogTypeLoader.java:62) [opensearch-security-analytics-2.17.1.0.jar:2.17.1.0]
search-1  | 2025-02-07T11:36:56.529661102Z      at org.opensearch.securityanalytics.logtype.BuiltinLogTypeLoader.doStart(BuiltinLogTypeLoader.java:146) [opensearch-security-analytics-2.17.1.0.jar:2.17.1.0]
search-1  | 2025-02-07T11:36:56.529664877Z      at org.opensearch.common.lifecycle.AbstractLifecycleComponent.start(AbstractLifecycleComponent.java:77) [opensearch-common-2.17.1.jar:2.17.1]
search-1  | 2025-02-07T11:36:56.529668377Z      at java.base/java.util.ArrayList.forEach(ArrayList.java:1596) [?:?]
search-1  | 2025-02-07T11:36:56.529672014Z      at java.base/java.util.Collections$UnmodifiableCollection.forEach(Collections.java:1116) [?:?]
search-1  | 2025-02-07T11:36:56.529675735Z      at org.opensearch.node.Node.start(Node.java:1564) [opensearch-2.17.1.jar:2.17.1]
search-1  | 2025-02-07T11:36:56.529678070Z      at org.opensearch.bootstrap.Bootstrap.start(Bootstrap.java:339) [opensearch-2.17.1.jar:2.17.1]
search-1  | 2025-02-07T11:36:56.529680272Z      at org.opensearch.bootstrap.Bootstrap.init(Bootstrap.java:413) [opensearch-2.17.1.jar:2.17.1]
search-1  | 2025-02-07T11:36:56.529682431Z      at org.opensearch.bootstrap.OpenSearch.init(OpenSearch.java:181) [opensearch-2.17.1.jar:2.17.1]
search-1  | 2025-02-07T11:36:56.529684559Z      at org.opensearch.bootstrap.OpenSearch.execute(OpenSearch.java:172) [opensearch-2.17.1.jar:2.17.1]
search-1  | 2025-02-07T11:36:56.529686794Z      at org.opensearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:104) [opensearch-2.17.1.jar:2.17.1]
search-1  | 2025-02-07T11:36:56.529688972Z      at org.opensearch.cli.Command.mainWithoutErrorHandling(Command.java:138) [opensearch-cli-2.17.1.jar:2.17.1]
search-1  | 2025-02-07T11:36:56.529692089Z      at org.opensearch.cli.Command.main(Command.java:101) [opensearch-cli-2.17.1.jar:2.17.1]
search-1  | 2025-02-07T11:36:56.529694358Z      at org.opensearch.bootstrap.OpenSearch.main(OpenSearch.java:138) [opensearch-2.17.1.jar:2.17.1]
search-1  | 2025-02-07T11:36:56.529696546Z      at org.opensearch.bootstrap.OpenSearch.main(OpenSearch.java:104) [opensearch-2.17.1.jar:2.17.1]
search-1  | 2025-02-07T11:36:56.889449758Z [2025-02-07T11:36:56,889][INFO ][o.o.t.TransportService   ] [33663bb1a797] publish_address {172.18.0.4:9300}, bound_addresses {0.0.0.0:9300}
search-1  | 2025-02-07T11:36:56.895237486Z [2025-02-07T11:36:56,894][INFO ][o.o.t.TransportService   ] [33663bb1a797] Remote clusters initialized successfully.
search-1  | 2025-02-07T11:36:57.249123594Z [2025-02-07T11:36:57,248][INFO ][o.o.b.BootstrapChecks    ] [33663bb1a797] bound or publishing to a non-loopback address, enforcing bootstrap checks
search-1  | 2025-02-07T11:36:57.267027775Z ERROR: [2] bootstrap checks failed
search-1  | 2025-02-07T11:36:57.267061023Z [1]: max virtual memory areas vm.max_map_count [65530] is too low, increase to at least [262144]
search-1  | 2025-02-07T11:36:57.267065966Z [2]: the default discovery settings are unsuitable for production use; at least one of [discovery.seed_hosts, discovery.seed_providers, cluster.initial_cluster_manager_nodes / cluster.initial_master_nodes] must be configured
search-1  | 2025-02-07T11:36:57.267875179Z ERROR: OpenSearch did not exit normally - check the logs at /usr/share/opensearch/logs/docker-cluster.log
search-1  | 2025-02-07T11:36:57.289218018Z [2025-02-07T11:36:57,288][INFO ][o.o.n.Node               ] [33663bb1a797] stopping ...
search-1  | 2025-02-07T11:36:57.292721052Z [2025-02-07T11:36:57,291][INFO ][o.o.s.a.r.AuditMessageRouter] [33663bb1a797] Closing AuditMessageRouter
search-1  | 2025-02-07T11:36:57.294358217Z [2025-02-07T11:36:57,294][INFO ][o.o.s.a.s.SinkProvider   ] [33663bb1a797] Closing InternalOpenSearchSink
search-1  | 2025-02-07T11:36:57.294604949Z [2025-02-07T11:36:57,294][INFO ][o.o.s.a.s.SinkProvider   ] [33663bb1a797] Closing DebugSink
search-1  | 2025-02-07T11:36:57.318289593Z [2025-02-07T11:36:57,317][INFO ][o.o.n.Node               ] [33663bb1a797] stopped
search-1  | 2025-02-07T11:36:57.318393071Z [2025-02-07T11:36:57,318][INFO ][o.o.n.Node               ] [33663bb1a797] closing ...
search-1  | 2025-02-07T11:36:57.334551514Z [2025-02-07T11:36:57,333][INFO ][o.o.s.a.i.AuditLogImpl   ] [33663bb1a797] Closing AuditLogImpl
search-1  | 2025-02-07T11:36:57.342507613Z [2025-02-07T11:36:57,342][INFO ][o.o.n.Node               ] [33663bb1a797] closed
search-1  | 2025-02-07T11:36:59.832852142Z Enabling OpenSearch Security Plugin
search-1  | 2025-02-07T11:36:59.832938115Z Enabling execution of install_demo_configuration.sh for OpenSearch Security Plugin
search-1  | 2025-02-07T11:36:59.832942968Z OpenSearch 2.12.0 onwards, the OpenSearch Security Plugin a change that requires an initial password for 'admin' user.
search-1  | 2025-02-07T11:36:59.832946729Z Please define an environment variable 'OPENSEARCH_INITIAL_ADMIN_PASSWORD' with a strong password string.
search-1  | 2025-02-07T11:36:59.832950399Z If a password is not provided, the setup will quit.
search-1  | 2025-02-07T11:36:59.832953442Z  For more details, please visit: https://opensearch.org/docs/latest/install-and-configure/install-opensearch/docker/
search-1  | 2025-02-07T11:37:00.627452673Z ### OpenSearch Security Demo Installer
search-1  | 2025-02-07T11:37:00.627538944Z ### ** Warning: Do not use on production or public reachable systems **
search-1  | 2025-02-07T11:37:00.645474327Z OpenSearch install type: rpm/deb on Linux 6.1.124-134.200.amzn2023.x86_64 amd64
search-1  | 2025-02-07T11:37:00.646393281Z OpenSearch config dir: /usr/share/opensearch/config/
search-1  | 2025-02-07T11:37:00.646561442Z OpenSearch config file: /usr/share/opensearch/config/opensearch.yml
search-1  | 2025-02-07T11:37:00.646727085Z OpenSearch bin dir: /usr/share/opensearch/bin/
search-1  | 2025-02-07T11:37:00.646937657Z OpenSearch plugins dir: /usr/share/opensearch/plugins/
search-1  | 2025-02-07T11:37:00.647092615Z OpenSearch lib dir: /usr/share/opensearch/lib/
search-1  | 2025-02-07T11:37:00.647252017Z Detected OpenSearch Version: 2.17.1
search-1  | 2025-02-07T11:37:00.647407849Z Detected OpenSearch Security Version: 2.17.1.0
search-1  | 2025-02-07T11:37:00.652554675Z /usr/share/opensearch/config/opensearch.yml seems to be already configured for Security. Quit.
search-1  | 2025-02-07T11:37:00.706960406Z Enabling execution of OPENSEARCH_HOME/bin/opensearch-performance-analyzer/performance-analyzer-agent-cli for OpenSearch Performance Analyzer Plugin
search-1  | 2025-02-07T11:37:06.994744775Z WARNING: Using incubator modules: jdk.incubator.vector
search-1  | 2025-02-07T11:37:07.253587073Z WARNING: A terminally deprecated method in java.lang.System has been called
search-1  | 2025-02-07T11:37:07.255766298Z WARNING: System::setSecurityManager has been called by org.opensearch.bootstrap.OpenSearch (file:/usr/share/opensearch/lib/opensearch-2.17.1.jar)
search-1  | 2025-02-07T11:37:07.256992732Z WARNING: Please consider reporting this to the maintainers of org.opensearch.bootstrap.OpenSearch
search-1  | 2025-02-07T11:37:07.257017329Z WARNING: System::setSecurityManager will be removed in a future release

3. Caddy version:

v2.9.1 h1:OEYiZ7DbCzAWVb6TNEkjRcSCRGHVoZsJinoDR/n9oaY=

4. How I installed and ran Caddy:

A simple direct docker compose pull of the image docker.io/library/caddy:latest

a. System environment:

Running on an AWS EC2 t3.small instance.

$ cat /etc/system-release
Amazon Linux release 2023.6.20250123 (Amazon Linux)

$ docker version
Client:
 Version:           25.0.5
 API version:       1.44
 Go version:        go1.22.5
 Git commit:        5dc9bcc
 Built:             Wed Aug 21 00:00:00 2024
 OS/Arch:           linux/amd64
 Context:           default

Server:
 Engine:
  Version:          25.0.6
  API version:      1.44 (minimum version 1.24)
  Go version:       go1.22.5
  Git commit:       b08a51f
  Built:            Wed Aug 21 00:00:00 2024
  OS/Arch:          linux/amd64
  Experimental:     false
 containerd:
  Version:          1.7.25
  GitCommit:        bcc810d6b9066471b0b6fa75f557a15a1cbf31bb
 runc:
  Version:          1.2.4
  GitCommit:        6c52b3fc541fb26fe8c374d5f58112a0a5dbda66
 docker-init:
  Version:          0.19.0
  GitCommit:        de40ad0

$ docker compose  version
Docker Compose version 083f676

b. Command:

docker compose up -d

c. Service/unit/compose file:

# yaml-language-server: $schema=https://raw.githubusercontent.com/compose-spec/compose-spec/master/schema/compose-spec.json
services:
    site:
        build: .
        networks:
            - am-d-model-site-network
        expose:
            - "3000"
        restart: unless-stopped
        labels:
            - "io.containers.autoupdate=local"

    caddy:
        image: docker.io/library/caddy:latest
        # entrypoint: ["tail", "-f", "/dev/null"] # Debug
        # command: ["tail", "-f", "/dev/null"] # Debug
        depends_on:
            - site
            - invenio.am-d-model.eu
        networks:
            - am-d-model-proxy-network
            - am-d-model-site-network
        ports:
            - "80:80"
            - "443:443"
        volumes:
            - ./data/caddy:/data/caddy
            - ./caddy:/etc/caddy
        restart: unless-stopped
        labels:
            - "io.containers.autoupdate=registry"
    # Repo containers start here
    # Repo frontend:
    invenio.am-d-model.eu:
        extends:
            file: repo/docker-services.yml
            service: frontend
        env_file: ./repo/.env
        depends_on:
            - web-ui
            - web-api
        volumes:
            - static_data:/opt/invenio/var/instance/static
            # Overwrite test certs with certs fetched by caddy
            - ./data/caddy/certificates/acme-v02.api.letsencrypt.org-directory/invenio.am-d-model.eu/invenio.am-d-model.eu.crt:/etc/ssl/certs/test.crt
            - ./data/caddy/certificates/acme-v02.api.letsencrypt.org-directory/invenio.am-d-model.eu/invenio.am-d-model.eu.key:/etc/ssl/private/test.key
            # Also, lets inject some configuration for the outer proxy
            # - ./caddy/proxy_settings.conf:/etc/nginx/conf.d/proxy_settings.conf
            # - ./caddy/internal_server.conf:/etc/nginx/conf.d/internal_server.conf
        ports:
            - "8080:80"
            - "8443:443"
        networks:
            - am-d-model-proxy-network
            - am-d-model-repo-network
    cache:
        extends:
            file: repo/docker-compose.full.yml
            service: cache
        env_file: ./repo/.env
        networks:
            - am-d-model-repo-network
    db:
        extends:
            file: repo/docker-compose.full.yml
            service: db
        env_file: ./repo/.env
        networks:
            - am-d-model-repo-network
    mq:
        extends:
            file: repo/docker-compose.full.yml
            service: mq
        env_file: ./repo/.env
        networks:
            - am-d-model-repo-network
    search:
        extends:
            file: repo/docker-compose.full.yml
            service: search
        env_file: ./repo/.env
        networks:
            - am-d-model-repo-network
    s3:
        extends:
            file: repo/docker-compose.full.yml
            service: s3
        env_file: ./repo/.env
        networks:
            - am-d-model-repo-network
    # UI Application
    web-ui:
        extends:
            file: repo/docker-compose.full.yml
            service: web-ui
        env_file: ./repo/.env
        build:
            context: ./repo/
        networks:
            - am-d-model-repo-network
        environment:
            - UWSGI_LOG_DEBUG=true
            - UWSGI_LOG_REQ=true     # Log requests
            - UWSGI_LOG_SLOW=true    # Log slow requests
            - UWSGI_LOG_ZERO=true    # Log empty requests
            - UWSGI_LOG_4XX=true     # Log 4xx errors
            - UWSGI_LOG_5XX=true     # Log 5xx errors
            - UWSGI_LOG_MICROS=true  # Add microseconds to logs
    # API Rest Application
    web-api:
        extends:
            file: repo/docker-compose.full.yml
            service: web-api
        env_file: ./repo/.env
        build:
            context: ./repo/
        networks:
            - am-d-model-repo-network
        environment:
            - UWSGI_LOG_DEBUG=true
            - UWSGI_LOG_REQ=true     # Log requests
            - UWSGI_LOG_SLOW=true    # Log slow requests
            - UWSGI_LOG_ZERO=true    # Log empty requests
            - UWSGI_LOG_4XX=true     # Log 4xx errors
            - UWSGI_LOG_5XX=true     # Log 5xx errors
            - UWSGI_LOG_MICROS=true  # Add microseconds to logs
    # Worker
    worker:
        extends:
            file: repo/docker-compose.full.yml
            service: worker
        env_file: ./repo/.env
        build:
            context: ./repo/
        networks:
            - am-d-model-repo-network

networks:
    am-d-model-site-network:
        name: am-d-model-site-network
    am-d-model-repo-network:
        name: am-d-model-repo-network
    am-d-model-proxy-network:
        name: am-d-model-proxy-network

volumes:
    caddy_data:
    static_data:
    redis_data:

d. My complete Caddy config:

{
        admin off
        debug
        log {
                level DEBUG
        }
}

am-d-model.eu {
        reverse_proxy site:3000 {
                health_uri /health
                health_interval 30s
                health_timeout 10s
                health_status 200
        }
}

invenio.am-d-model.eu {
        reverse_proxy invenio.am-d-model.eu:8080 {
                header_up Host {host}
                header_up X-Real-IP {remote_host}
                header_up X-Forwarded-Proto "https"
        }
}

5. Links to relevant resources:

https://github.com/Cian-H/am-d-model.eu/tree/invenio_backend
https://inveniordm.docs.cern.ch/

edit: changed github link to point to specific branch on which my current (not working) config is

2

Hey, man. You know way more than me. It sounds like Invenio is configured to require HTTPS, but your Caddyfile is trying to reverse_proxy over HTTP to port 80. So I think Nginx expects HTTPS, gets HTTP, and then returns a 502 (Bad Gateway) because it can’t handle the plain HTTP request.

Try changing your Caddyfile to:

invenio.am-d-model.eu {
    reverse_proxy invenio.am-d-model.eu:8443 {
        header_up Host {host}
        header_up X-Real-IP {remote_host}
        header_up X-Forwarded-Proto "https"
        insecure_skip_verify
    }
}

insecure_skip_verify I think is necessary since Invenio is managing the certificate internally. I would make sure you trust Invenio/Nginx before doing this.

3

Hey, thanks a million for trying to help with this. Your suggestion was helpful for getting past my nginx blockage, but then this whole attempt devolved back into a nightmare of uwsgi interfacing! Decided to try approaching it from the other side and went to ask the invenio guys for help on their discord.

Turns out i wasn’t the first person to discover that the wsgi -> uwsgi -> nginx -> caddy structure was an absolute nightmare to get working, and a very active community member over there was kind enough to share his own implementation that is structured as wsgi -> gunicorn -> caddy. Much cleaner, much more maintainable, and avoids the uwsgi binary interface nightmare i was having. If anyone with the same problem comes across this in future, i’d advise having a look at that setup instead of the traditional invenio stack. At least until caddy gets better support for uwsgi, which does appear to be on the radar of the maintainers and there is a plugin for it but it’s not quite there yet as a plug-and-play alternative for setups as complex as invenio (if my admittedly clumsy attempts to use it were anything to go off, at least).

This post can be marked as closed: i have managed to get a nice clean invenio deployment working using caddy, which should be far more approachable for me and the rest of my non webdev colleagues than nginx was. Thanks again to the community here for trying to help, and thanks for the efforts on making such an awesome, user friendly (when not using uwsgi, at least!) webserver. :grin:

2 Likes
4

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.