1. The problem I’m having:
Im running caddy ingress in k8s to serve a php-fpm container. I only get 502 responses. You can see it here: https://homolog.main-api.fretegestao.com.br/
I have a golang image running in this cluster, and it is working fine as it doesnt need fastcgi.
I didnt find how to use a custom caddy configuration in the ingress or how to use the fastcgi directive in the yaml in the docs. GitHub - caddyserver/ingress: WIP Caddy 2 ingress controller for Kubernetes
I need to use fastCGI for this container, how can i do that?
2. Error messages and/or full log output:
Deployment and service yaml:
apiVersion: apps/v1
kind: Deployment
metadata:
name: main-api-deployment
labels:
app: main-api
spec:
replicas: 1
strategy:
type: RollingUpdate
rollingUpdate:
maxUnavailable: 0
maxSurge: 2
selector:
matchLabels:
app: main-api
template:
metadata:
labels:
app: main-api
spec:
containers:
- name: main-api
image: <image>
ports:
- containerPort: 9000
envFrom:
- configMapRef:
name: main-api
---
apiVersion: v1
kind: Service
metadata:
name: main-api-service
spec:
selector:
app: main-api
ports:
- name: fpm
port: 9000
targetPort: 9000
Ingress yaml
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: ingress
annotations:
kubernetes.io/ingress.class: caddy
spec:
rules:
- host: homolog.main-api.fretegestao.com.br
http:
paths:
- path: /
pathType: Prefix
backend:
service:
name: main-api-service
port:
number: 9000
Curl from inside the container pod: (This one does not pass by caddy, its using http instead of fastcgi, thats the explation for the error)
* Trying 127.0.0.1:9000...
* Connected to localhost (127.0.0.1) port 9000 (#0)
> GET / HTTP/1.1
> Host: localhost:9000
> User-Agent: curl/8.0.1-DEV
> Accept: */*
>
* Recv failure: Connection reset by peer
* Closing connection 0
curl: (56) Recv failure: Connection reset by peer
pod description:
Name: main-api-deployment-7cd9d47886-c5g45
Namespace: default
Priority: 0
Service Account: default
Node: pool-q5an7/10.116.0.6
Start Time: Thu, 23 Mar 2023 16:59:37 -0300
Labels: app=main-api
pod-template-hash=7cd9d47886
Annotations: kubectl.kubernetes.io/restartedAt: 2023-03-23T16:59:37-03:00
Status: Running
IP: 10.244.0.126
IPs:
IP: 10.244.0.126
Controlled By: ReplicaSet/main-api-deployment-7cd9d47886
Containers:
main-api:
Container ID: containerd://04b6d8f62295e174ed196b0ad0b3002fe0b37c64faafe9cfd623abb4e98a30c7
Image: ...
Image ID: ...
Port: 9000/TCP
Host Port: 0/TCP
State: Running
Started: Thu, 23 Mar 2023 16:59:40 -0300
Ready: True
Restart Count: 0
Environment Variables from:
main-api ConfigMap Optional: false
Environment: <none>
Mounts:
/var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-54c4q (ro)
Conditions:
Type Status
Initialized True
Ready True
ContainersReady True
PodScheduled True
Volumes:
kube-api-access-54c4q:
Type: Projected (a volume that contains injected data from multiple sources)
TokenExpirationSeconds: 3607
ConfigMapName: kube-root-ca.crt
ConfigMapOptional: <nil>
DownwardAPI: true
QoS Class: BestEffort
Node-Selectors: <none>
Tolerations: node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
Events: <none>
service description:
Name: main-api-service
Namespace: default
Labels: <none>
Annotations: <none>
Selector: app=main-api
Type: ClusterIP
IP Family Policy: SingleStack
IP Families: IPv4
IP: 10.245.139.92
IPs: 10.245.139.92
Port: fpm 9000/TCP
TargetPort: 9000/TCP
Endpoints: 10.244.0.126:9000
Session Affinity: None
Events: <none>
pod logs: (Im using laravel)
...
- Installing ezyang/htmlpurifier (v4.16.0): Extracting archive
- Installing phpoffice/phpspreadsheet (1.28.0): Extracting archive
- Installing spatie/laravel-package-tools (1.12.1): Extracting archive
- Installing spatie/laravel-activitylog (4.7.3): Extracting archive
84 package suggestions were added by new dependencies, use `composer suggest` to see details.
Generating optimized autoload files
58 packages you are using are looking for funding.
Use the `composer fund` command to find out more!
Nothing to migrate.
Configuration cache cleared!
Configuration cached successfully!
Route cache cleared!
Routes cached successfully!
Files cached successfully!
Configuration cache cleared!
Configuration cached successfully!
Configuration cache cleared!
Application cache cleared!
Generating optimized autoload files
> Illuminate\Foundation\ComposerScripts::postAutoloadDump
> @php artisan package:discover --ansi
Discovered Package: fruitcake/laravel-cors
Discovered Package: laravel/sanctum
Discovered Package: laravel/tinker
Discovered Package: nesbot/carbon
Discovered Package: nunomaduro/collision
Discovered Package: spatie/laravel-activitylog
Package manifest generated successfully.
Generated optimized autoload files containing 5701 classes
Compiled views cleared!
Route cache cleared!
Compiled services and packages files removed!
[23-Mar-2023 20:01:20] NOTICE: [pool www] 'user' directive is ignored when FPM is not running as root
[23-Mar-2023 20:01:20] NOTICE: [pool www] 'user' directive is ignored when FPM is not running as root
[23-Mar-2023 20:01:20] NOTICE: [pool www] 'group' directive is ignored when FPM is not running as root
[23-Mar-2023 20:01:20] NOTICE: [pool www] 'group' directive is ignored when FPM is not running as root
[23-Mar-2023 20:01:20] NOTICE: fpm is running, pid 289
[23-Mar-2023 20:01:20] NOTICE: ready to handle connections
Ingress description:
Name: ingress
Labels: <none>
Namespace: default
Address: ...
Ingress Class: <none>
Default backend: <default>
Rules:
Host Path Backends
---- ---- --------
homolog.go.fretegestao.com.br
/ go-service-service:80 (10.244.0.76:80)
homolog.main-api.fretegestao.com.br
/ main-api-service:9000 (10.244.0.126:9000)
Annotations: kubernetes.io/ingress.class: caddy
Events: <none>
Error in the caddy pod:
{"level":"error","ts":1679639705.9354064,"logger":"http.log.error","msg":"read tcp 10.244.0.86:35710->10.245.139.92:9000: read: connection reset by peer","request":{"remote_ip":"10.244.0.11","remote_port":"1363","proto":"HTTP/1.1","method":"GET","host":"homolog.main-api.fretegestao.com.br","uri":"/Public/home/js/check.js","headers":{"User-Agent":["Mozilla/5.0 (Linux; Android 11; M2004J15SC) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/103.0.5060.114 Mobile Safari/537.36"],"Accept":["*/*"],"Accept-Encoding":["gzip"]},"tls":{"resumed":false,"version":771,"cipher_suite":49195,"proto":"","server_name":"homolog.main-api.fretegestao.com.br"}},"duration":0.008618425,"status":502,"err_id":"xg8825qc7","err_trace":"reverseproxy.statusError (reverseproxy.go:1196)"}
3. Caddy version:
4. How I installed and ran Caddy:
a. System environment:
K8S
b. Command:
I just followed the guide on the ghub repo to install it in my k8s cluster. I used helm.
helm install \
--namespace=caddy-system \
--repo https://caddyserver.github.io/ingress/ \
--atomic \
--set ingressController.config.email=redacted@email \
caddy \
caddy-ingress-controller
c. Service/unit/compose file:
d. My complete Caddy config:
I dont have any caddyfile or custom config as I dont know how to provide it to the ingress. Probably thats how I should be setting up fastCGI to my container. Could not find how to do it in the ingress docs.