How can I use fastCGI with caddy in k8s ingress?

1. The problem I’m having:

Im running caddy ingress in k8s to serve a php-fpm container. I only get 502 responses. You can see it here: https://homolog.main-api.fretegestao.com.br/

I have a golang image running in this cluster, and it is working fine as it doesnt need fastcgi.

I didnt find how to use a custom caddy configuration in the ingress or how to use the fastcgi directive in the yaml in the docs. GitHub - caddyserver/ingress: WIP Caddy 2 ingress controller for Kubernetes

I need to use fastCGI for this container, how can i do that?

2. Error messages and/or full log output:

Deployment and service yaml:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: main-api-deployment
  labels:
    app: main-api
spec:
  replicas: 1
  strategy:
    type: RollingUpdate
    rollingUpdate:
      maxUnavailable: 0
      maxSurge: 2
  selector:
    matchLabels:
      app: main-api
  template:
    metadata:
      labels:
        app: main-api
    spec:
      containers:
        - name: main-api
          image: <image>
          ports:
            - containerPort: 9000
          envFrom:
            - configMapRef:
                name: main-api
---
apiVersion: v1
kind: Service
metadata:
  name: main-api-service
spec:
  selector:
    app: main-api
  ports:
    - name: fpm
      port: 9000
      targetPort: 9000

Ingress yaml

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: ingress
  annotations:
    kubernetes.io/ingress.class: caddy
spec:
  rules:
  - host: homolog.main-api.fretegestao.com.br
    http:
      paths:
      - path: /
        pathType: Prefix
        backend:
          service:
            name: main-api-service
            port:
              number: 9000

Curl from inside the container pod: (This one does not pass by caddy, its using http instead of fastcgi, thats the explation for the error)

* Trying 127.0.0.1:9000...
* Connected to localhost (127.0.0.1) port 9000 (#0)
> GET / HTTP/1.1
> Host: localhost:9000
> User-Agent: curl/8.0.1-DEV
> Accept: */*
>
* Recv failure: Connection reset by peer
* Closing connection 0
curl: (56) Recv failure: Connection reset by peer

pod description:

Name:             main-api-deployment-7cd9d47886-c5g45
Namespace:        default
Priority:         0
Service Account:  default
Node:             pool-q5an7/10.116.0.6
Start Time:       Thu, 23 Mar 2023 16:59:37 -0300
Labels:           app=main-api
                  pod-template-hash=7cd9d47886
Annotations:      kubectl.kubernetes.io/restartedAt: 2023-03-23T16:59:37-03:00
Status:           Running
IP:               10.244.0.126
IPs:
  IP:           10.244.0.126
Controlled By:  ReplicaSet/main-api-deployment-7cd9d47886
Containers:
  main-api:
    Container ID:   containerd://04b6d8f62295e174ed196b0ad0b3002fe0b37c64faafe9cfd623abb4e98a30c7
    Image:          ...
    Image ID:       ...
    Port:           9000/TCP
    Host Port:      0/TCP
    State:          Running
      Started:      Thu, 23 Mar 2023 16:59:40 -0300
    Ready:          True
    Restart Count:  0
    Environment Variables from:
      main-api    ConfigMap  Optional: false
    Environment:  <none>
    Mounts:
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-54c4q (ro)
Conditions:
  Type              Status
  Initialized       True 
  Ready             True 
  ContainersReady   True 
  PodScheduled      True 
Volumes:
  kube-api-access-54c4q:
    Type:                    Projected (a volume that contains injected data from multiple sources)
    TokenExpirationSeconds:  3607
    ConfigMapName:           kube-root-ca.crt
    ConfigMapOptional:       <nil>
    DownwardAPI:             true
QoS Class:                   BestEffort
Node-Selectors:              <none>
Tolerations:                 node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
                             node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
Events:                      <none>

service description:

Name:              main-api-service
Namespace:         default
Labels:            <none>
Annotations:       <none>
Selector:          app=main-api
Type:              ClusterIP
IP Family Policy:  SingleStack
IP Families:       IPv4
IP:                10.245.139.92
IPs:               10.245.139.92
Port:              fpm  9000/TCP
TargetPort:        9000/TCP
Endpoints:         10.244.0.126:9000
Session Affinity:  None
Events:            <none>

pod logs: (Im using laravel)

  ...
  - Installing ezyang/htmlpurifier (v4.16.0): Extracting archive
  - Installing phpoffice/phpspreadsheet (1.28.0): Extracting archive
  - Installing spatie/laravel-package-tools (1.12.1): Extracting archive
  - Installing spatie/laravel-activitylog (4.7.3): Extracting archive
84 package suggestions were added by new dependencies, use `composer suggest` to see details.
Generating optimized autoload files
58 packages you are using are looking for funding.
Use the `composer fund` command to find out more!
Nothing to migrate.
Configuration cache cleared!
Configuration cached successfully!
Route cache cleared!
Routes cached successfully!
Files cached successfully!
Configuration cache cleared!
Configuration cached successfully!
Configuration cache cleared!
Application cache cleared!
Generating optimized autoload files
> Illuminate\Foundation\ComposerScripts::postAutoloadDump
> @php artisan package:discover --ansi
Discovered Package: fruitcake/laravel-cors
Discovered Package: laravel/sanctum
Discovered Package: laravel/tinker
Discovered Package: nesbot/carbon
Discovered Package: nunomaduro/collision
Discovered Package: spatie/laravel-activitylog
Package manifest generated successfully.
Generated optimized autoload files containing 5701 classes
Compiled views cleared!
Route cache cleared!
Compiled services and packages files removed!
[23-Mar-2023 20:01:20] NOTICE: [pool www] 'user' directive is ignored when FPM is not running as root
[23-Mar-2023 20:01:20] NOTICE: [pool www] 'user' directive is ignored when FPM is not running as root
[23-Mar-2023 20:01:20] NOTICE: [pool www] 'group' directive is ignored when FPM is not running as root
[23-Mar-2023 20:01:20] NOTICE: [pool www] 'group' directive is ignored when FPM is not running as root
[23-Mar-2023 20:01:20] NOTICE: fpm is running, pid 289
[23-Mar-2023 20:01:20] NOTICE: ready to handle connections

Ingress description:

Name:             ingress
Labels:           <none>
Namespace:        default
Address:          ...  
Ingress Class:    <none>
Default backend:  <default>
Rules:
  Host                                 Path  Backends
  ----                                 ----  --------
  homolog.go.fretegestao.com.br
                                       /   go-service-service:80 (10.244.0.76:80)
  homolog.main-api.fretegestao.com.br
                                       /   main-api-service:9000 (10.244.0.126:9000)
Annotations:                           kubernetes.io/ingress.class: caddy
Events:                                <none>

Error in the caddy pod:

{"level":"error","ts":1679639705.9354064,"logger":"http.log.error","msg":"read tcp 10.244.0.86:35710->10.245.139.92:9000: read: connection reset by peer","request":{"remote_ip":"10.244.0.11","remote_port":"1363","proto":"HTTP/1.1","method":"GET","host":"homolog.main-api.fretegestao.com.br","uri":"/Public/home/js/check.js","headers":{"User-Agent":["Mozilla/5.0 (Linux; Android 11; M2004J15SC) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/103.0.5060.114 Mobile Safari/537.36"],"Accept":["*/*"],"Accept-Encoding":["gzip"]},"tls":{"resumed":false,"version":771,"cipher_suite":49195,"proto":"","server_name":"homolog.main-api.fretegestao.com.br"}},"duration":0.008618425,"status":502,"err_id":"xg8825qc7","err_trace":"reverseproxy.statusError (reverseproxy.go:1196)"}

3. Caddy version:

4. How I installed and ran Caddy:

a. System environment:

K8S

b. Command:

I just followed the guide on the ghub repo to install it in my k8s cluster. I used helm.

helm install \                                           
  --namespace=caddy-system \
  --repo https://caddyserver.github.io/ingress/ \
  --atomic \
  --set ingressController.config.email=redacted@email \
  caddy \
  caddy-ingress-controller

c. Service/unit/compose file:

d. My complete Caddy config:

I dont have any caddyfile or custom config as I dont know how to provide it to the ingress. Probably thats how I should be setting up fastCGI to my container. Could not find how to do it in the ingress docs.

5. Links to relevant resources:

I see you opened an issue:

I’ll have to leave it to the ingress maintainers to answer that. I think ingress is set up with a custom config layer, so it would need to explicitly support fastcgi, I think. But I’m not involved in the ingress project, since I don’t use k8s at all.

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.