What is the maximum number of pods that KEDA can handle on one node? #6867
-
The problem is that I run KEDA on a vcluster that runs on one node. I run an application there that can scale up to 100 pods (1CPU request + 2Gi RAM Request) and I noticed that after exceeding the limit of 70/80 pods started by KEDA, the next pods do not start and their containers end with exit code -1. Interestingly, if I set in the scaling object that I should start 100 pods as minReplicaCount: 100, all my pods start and are in a healthy state. I use the CRI-O runtime. When debugging CRI-O itself, you can see that the container starts and then immediately dies. Therefore, I wonder how KEDA is responsible for newly created pods and whether there is a moment when KEDA itself does not kill these containers in some way. |
Beta Was this translation helpful? Give feedback.
Replies: 1 comment 1 reply
-
KEDA does not manage individual pod/container lifecycles after requesting a scale-out. KEDA monitors external metrics, and when a trigger is hit, KEDA updates the replicaCount on the ScaledObject’s target resource. Kubernetes itself schedules new pods and the container runtime (CRI-O in your case) starts them. KEDA doesn’t intervene after requesting the scale-out. |
Beta Was this translation helpful? Give feedback.
I just discovered what my problem was.
When I was running 80 pods on one worker, I didn't notice that the disk was very busy (for the scale, a server with 128 CPUs showed throttling at 500). The reason was that applications under heavy load wrote a lot of logs, which in addition to being displayed on stdout, also wrote to a file.
After fixing this writing to a file (removing this functionality), the problem does not occur.
Thanks a lot for confirming my belief that KEDA does not affect the application runtime in any way :)