Why does TargetQueueSize have to be a number greater than 0 for Temporal trigger in scaled object

[LarsAlmgren]

We are trying to scale out as soon as there is an item in the queue, but this is not possible right now since TargetQueueSize need to be at least 1. Is there some reason TargetQueueSize need to be greater than 0 in Temporal trigger?

[LarsAlmgren]

So, I was able to come around the problem by defining a scaling modifier for the metric with target 0.1 (I guess I could have used anything in the interval (0,1)). I assume the target queue size need to be greater than 0 for Temporal trigger as the underlying framework have checks to ensure > 0.0.

[LarsAlmgren]

If someone want to see the full template, ours looks something like this:

apiVersion: keda.sh/v1alpha1
kind: ScaledObject
metadata:
  name: {{ .Release.Name }}
  namespace: {{ .Release.Namespace }}
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: {{ .Release.Name }}
  minReplicaCount: 1
  cooldownPeriod: 300
  advanced:
    scalingModifiers:
      formula: "seriousserialstuff"
      target: "0.1"
      activationTarget: "0"
      metricType: "Value"
  triggers:
    - type: temporal
      name: seriousserialstuff
      metadata:
        namespace: "mynamespace.cool"
        taskQueue: "work-work-work/mytaskqueue"
        endpoint: "go.figure.it"
        queueTypes: activity
      authenticationRef:
        kind: ClusterTriggerAuthentication
        name: temporal-cloud

[rickbrouwer]

Hi!

I think it is so that if you have TargetQueueSize set to 5, for example, you get 1 pod for everything between 1 and 5.
As soon as there are 6 messages on it, for example, you get a 2nd pod (because it is then above those 5).
The activationTargetQueueSize is 0 by default, so everything above that will be acted on.
Now I don't know the Temporal scaler very well, but I assume that if there is 1 item on the queue, you get the value '1' back.
If you were to set the TargetQueueSize to 1, that would mean that you would get 1 pod per message on a queue.

[LarsAlmgren]

If you were to set the TargetQueueSize to 1, that would mean that you would get 1 pod per message on a queue.

It kind of makes sense to me, but I see some weird behavior now which I cannot really explain.

We have parallelization = 1 on the task workers, so we will only handle one task per worker at the time.
New tasks will be put on the task queue.

Using:

        minReplicaCount: "1"
        targetQueueSize: "1"
        activationTargetQueueSize: "0"

The sequence of events is the following:

1. One pod is running (minReplicaCount)
2. Create task which start on that pod
3. Create another task which will end up in the task queue
4. Metric goes from 0 to 1
5. A second pod is started
6. One of the two pods is killed after 5 minutes
7. Metric goes from 0 to 1
8. A second pod is started
9. One of the two pods is killed after 5 minutes
10. Metric from 0 to 1
11. No scaling happens

NAME                                        READY   STATUS    RESTARTS   AGE
my-worker-pod-5997c6469d-gptd9   1/1     Running   0          2m53s
my-worker-pod-5997c6469d-hdrz6   1/1     Running   0          7m26s
my-worker-pod-5997c6469d-gptd9   1/1     Terminating   0          5m33s
my-worker-pod-5997c6469d-gptd9   0/1     Completed     0          5m34s
my-worker-pod-5997c6469d-gptd9   0/1     Completed     0          5m34s
my-worker-pod-5997c6469d-gptd9   0/1     Completed     0          5m34s
my-worker-pod-5997c6469d-x7q6l   0/1     Pending       0          0s
my-worker-pod-5997c6469d-x7q6l   0/1     Pending       0          0s
my-worker-pod-5997c6469d-x7q6l   0/1     ContainerCreating   0          0s
my-worker-pod-5997c6469d-x7q6l   1/1     Running             0          1s
my-worker-pod-5997c6469d-x7q6l   1/1     Terminating         0          5m3s

NAME                                        READY   STATUS    RESTARTS   AGE
data-pipelines-workflows-5997c6469d-hdrz6   1/1     Running   0          20m

data-pipelines-workflows-5997c6469d-hdrz6   1/1     Running   0          46m

Why would it scale up the first two times, but not later? Will I get minReplicaCount + metric scaling pods or will I get MAX(minReplicaCount, metric scaling pods)?

[rickbrouwer]

Can you try this one:

apiVersion: keda.sh/v1alpha1
kind: ScaledObject
metadata:
  name: {{ .Release.Name }}
  namespace: {{ .Release.Namespace }}
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: {{ .Release.Name }}
  minReplicaCount: 1
  cooldownPeriod: 300
  triggers:
    - type: temporal
      name: seriousserialstuff
      metadata:
        namespace: "mynamespace.cool"
        taskQueue: "work-work-work/mytaskqueue"
        targetQueueSize: "1"
        endpoint: "go.figure.it"
        queueTypes: activity
      authenticationRef:
        kind: ClusterTriggerAuthentication
        name: temporal-cloud

It will then run with at least 1 pod continuously.
If there is 1 message on the queue it will get a metric 1 and therefore want 1 pod. That is already there, so it must remain at 1.
Only if there are 2 messages on a queue at the time of checking you will get an extra pod (2 pods).

[LarsAlmgren]

Can you try this one:

I'm more or less using that one, only difference it that I have set activationTargetQueueSize: "0" explicitly, but that is the default value anyway.