Kafka Trigger: "error parsing kafka metadata: no username given"

[christian-ber]

Report

I'd like to report an issue that was already reported once by another user but not investigated, the issue was then closed due to inactivity.

#5287

Expected Behavior

The operator should read username and password from the scecret and provide the values via TriggerAuthentication.

Actual Behavior

Operator throws error reporting that the username is empty.

Steps to Reproduce the Problem

1. Kafka Secret

apiVersion: v1                                                                                                                                                                                                
 data:                                                                                                                                                                                                         
   password: <REDACTED>                                                                                                          
   username: <REDACTED>                                                                                                                                                                          
 kind: Secret                                                                                                                                                                                                  
 metadata:                                                                                                                                                                                                     
   creationTimestamp: "2025-08-11T07:37:29Z"                                                                                                                                                                   
   name: keda-kafka-secrets                                                                                                                                                                                    
   namespace: <REDACTED>                                                                                                                                                                                     
   resourceVersion: "1023943695"                                                                                                                                                                               
   uid: 3752a1cd-a62f-4df0-8837-dda7ab98c4c2                                                                                                                                                                   
 type: Opaque

2. TriggerAuthentication

apiVersion: keda.sh/v1alpha1
kind: TriggerAuthentication
metadata:
  name: keda-kafka-credentials
spec:
  secretTargetRef:
    - parameter: username
      name: keda-kafka-secrets
      key: username
    - parameter: password
      name: keda-kafka-secrets
      key: password

3. Scaled Object

apiVersion: keda.sh/v1alpha1
kind: ScaledObject
metadata:
  annotations:
    meta.helm.sh/release-name: <REDACTED>
    meta.helm.sh/release-namespace: <REDACTED>
  creationTimestamp: "2025-08-11T07:37:41Z"
  finalizers:
  - finalizer.keda.sh
  generation: 6
  labels:
    app.kubernetes.io/managed-by: Helm
    deploymentName: <REDACTED>-chart
    scaledobject.keda.sh/name: <REDACTED>-chart-scaledobject
  name: <REDACTED>-chart-scaledobject
  namespace: <REDACTED>
  resourceVersion: "1023969094"
  uid: <REDACTED>
spec:
  maxReplicaCount: 2
  minReplicaCount: 1
  pollingInterval: 5
  scaleTargetRef:
    apiVersion: apps/v1
    kind: deployment
    name: <REDACTED>-chart
  triggers:
  - authenticationRef:
      name: keda-kafka-credentials
    metadata:
      bootstrapServers: <REDACTED>
      consumerGroup: <REDACTED>
      lagThreshold: "100"
      limitToPartitionsWithLag: "true"
      offsetRestPolicy: earliest
      sasl: plaintext
      topic: <REDACTED>
    type: kafka

Logs from KEDA operator

ERROR    Reconciler error   {"controller": "scaledobject", "controllerGroup": "keda.sh", "controllerKind": "ScaledObject", 
"ScaledObject": {"name":"<REDACTED>","namespace":"<REDACTED>"}, "namespace": "<REDACTED>", "name": "<REDACTED>", 
"reconcileID": "b455bef3-10a6-4721-875b-816c16a7aef2" 
 , "error": "error parsing kafka metadata: no username given"}                                                                                                                                                 
 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler                                                                                                                    
     /workspace/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:316                                                                                                                
 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem                                                                                                                 
     /workspace/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:263                                                                                                                
 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func2.2                                                                                                                       
     /workspace/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:224  

KEDA Version

2.17.1

Kubernetes Version

1.31

Platform

Microsoft Azure

Scaler Details

Kafka

Anything else?

Any input on how to resolve this would be highly appreciated. We currently cannot use the kafka trigger at all because of this issue.

[soma-kurisu]

We're encountering the same error described by @christian-ber. We're currently using the Helm chart from patch release 2.17.1, and haven't yet tested the latest 2.17.2 release.

We also experimented with the AKS-managed KEDA integration, but unfortunately, we don't recall which KEDA version was deployed by AKS.

Additionally, we tried extending RBAC permissions for the KEDA operator's service account by adding a Role and RoleBinding to allow list and watch access to secrets in the application's namespace, alongside the default get permission. However, this did not resolve the issue.

apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  name: keda-list-secrets
  namespace: my-keda-scaled-application-ns
rules:
  - apiGroups: [""]
    resources: ["secrets"]
    verbs: ["get", "list", "watch"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: keda-list-secrets-binding
  namespace: my-keda-scaled-application-ns
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: Role
  name: keda-list-secrets
subjects:
  - kind: ServiceAccount
    name: keda-operator
    namespace: keda

[soma-kurisu]

As a follow-up to my previous comment, we also experimented with providing Kafka authentication credentials directly in the ScaledObject's trigger metadata section, using username and password, in addition to the standard approach via secretTargetRef in the TriggerAuthentication manifest.

Based on our review of the codebase, both the official Kafka scaler and the experimental Apache Kafka scaler appear to support SASL authentication using username and password parameters.

Although we haven’t built a custom version to debug this behavior directly, the test suite for the official Kafka scaler also demonstrates usage of these parameters:

var validWithAuthParams = map[string]string{
    "sasl":     "plaintext",
    "username": "admin",
    "password": "admin",
}

This suggests that the scaler should be able to handle SASL authentication correctly when these values are provided directly in the metadata.

[rickbrouwer]

Can you test by also specifying sasl via a secret?

Like:

apiVersion: v1
kind: Secret
metadata:
  name: keda-kafka-secrets
  namespace: default
data:
  sasl: "plaintext"
  username: "admin"
  password: "admin"
---
apiVersion: keda.sh/v1alpha1
kind: TriggerAuthentication
metadata:
  name: keda-trigger-auth-kafka-credential
  namespace: default
spec:
  secretTargetRef:
  - parameter: sasl
    name: keda-kafka-secrets
    key: sasl
  - parameter: username
    name: keda-kafka-secrets
    key: username
  - parameter: password
    name: keda-kafka-secrets
    key: password

[christian-ber]

@rickbrouwer Moving the sasl property to the secret fixed the issue, thanks a lot. Is this unexpected behavior from the operator? If not, updating the examples in the README might be good.

[rickbrouwer]

@rickbrouwer Moving the sasl property to the secret fixed the issue, thanks a lot. Is this unexpected behavior from the operator? If not, updating the examples in the README might be good.

Great! If you're interested, you can see if you can make a suggestion in keda-docs to update the docs.
If you don't get around to it, I'll check the docs later, to see if I can clarify something

[christian-ber]

@rickbrouwer We're still struggling to connect to our kafka cluster. Our setup requires SASL_SSL/PLAIN authentication. I haven't seen this mentioned in the documentation. Any idea how to make this work?

We use these settings:

tls: enable
sasl: plaintext (+ username / password)

Or does sasl:plaintext translate to plaintext authentication (without TLS).

Error message from operator is:

error creating kafka client: kafka: client has run out of available brokers to talk to: EOF

[soma-kurisu]

Adding on @christian-ber's observation i tried following configurations with the kcat kafka connection test tool:

/ # kcat -b REDACTED.westeurope.azure.glb.confluent.cloud:9092 -t REDACTED -X security.protocol=SASL_PLAINTEXT -X sasl.mechanisms=PLAIN -X sasl.username='REDACTED
SJBH2Q' -X sasl.password='REDACTED' -L
%6|1755081960.561|FAIL|rdkafka#producer-1| [thrd:sasl_plaintext://REDACTEDwesteurope.azure.glb.confluent]: sasl_plaintext://REDACTEDwesteurope.azure.glb.confluent.cloud:9092/bootstrap: Disconnected while requesting ApiVersion: might be caused by incorrect security.protocol configuration (connecting to a SSL listener?) or broker version is < 0.10 (see api.version.request) (after 7ms in state APIVERSION_QUERY)
%6|1755081960.802|FAIL|rdkafka#producer-1| [thrd:sasl_plaintext://REDACTED.westeurope.azure.glb.confluent]: sasl_plaintext://REDACTED.westeurope.azure.glb.confluent.cloud:9092/bootstrap: Disconnected while requesting ApiVersion: might be caused by incorrect security.protocol configuration (connecting to a SSL listener?) or broker version is < 0.10 (see api.version.request) (after 7ms in state APIVERSION_QUERY, 1 identical error(s) suppressed)
% ERROR: Failed to acquire metadata: Local: Broker transport failure (Are the brokers reachable? Also try increasing the metadata timeout with -m <timeout>?)

/ # kcat -b REDACTED.westeurope.azure.glb.confluent.cloud:9092 -t REDACTED -X security.protocol=SASL_SSL -X sasl.mechanisms=PLAIN -X sasl.username='REDACTED
' -X sasl.password='REDACTED' -L
Metadata for REDACTED (from broker -1: sasl_ssl://REDACTED.westeurope.azure.glb.confluent.cloud:9092/bootstrap):
 3 brokers:
  broker 0 at REDACTED.westeurope.azure.glb.confluent.cloud:9092 (controller)
  broker 1 at REDACTED.westeurope.azure.glb.confluent.cloud:9092
  broker 2 at REDACTED.westeurope.azure.glb.confluent.cloud:9092
 1 topics:
  topic "REDACTED" with 6 partitions:
    partition 0, leader 1, replicas: 1,0,2, isrs: 0,1,2
    partition 1, leader 0, replicas: 0,2,1, isrs: 0,1,2
    partition 2, leader 2, replicas: 2,1,0, isrs: 0,1,2
    partition 3, leader 1, replicas: 1,2,0, isrs: 0,1,2
    partition 4, leader 0, replicas: 0,1,2, isrs: 0,1,2
    partition 5, leader 2, replicas: 2,0,1, isrs: 0,1,2
/ #

So setting security.protocol=SASL_SSL fixes the broker conntection errors. Since we weren't able to find an approprite configuration option in the codebase for the kafka scaler the question remains how to advise the scaler or the backing sarama library to use securityProtocol: SASL_SSL instead of securityProtocol: SASL_PLAINTEXT?

[rickbrouwer]

@dttung2905 , is that something you know from your experience?

[dttung2905]

Hi there,
usually, a typical config for java producer consumer when using with Confluent API KEY would be

security.protocol=SASL_SSL
sasl.mechanism=PLAIN
sasl.jaas.config=org.apache.kafka.common.security.plain.PlainLoginModule required \
        username="xxxxx" \
        password="yyyyy";

This translate to the following KEDA config (cmiiw)

tls: enable
sasl: plaintext

The code logic should be here

keda/pkg/scalers/kafka_scaler.go

Lines 633 to 650 in 794fe80

	func getKafkaClientConfig(ctx context.Context, metadata kafkaMetadata) (*sarama.Config, error) {
	config := sarama.NewConfig()
	config.Version = metadata.version
	if metadata.saslType != KafkaSASLTypeNone && metadata.saslType != KafkaSASLTypeGSSAPI {
	config.Net.SASL.Enable = true
	config.Net.SASL.User = metadata.username
	config.Net.SASL.Password = metadata.password
	}
	if metadata.enableTLS {
	config.Net.TLS.Enable = true
	tlsConfig, err := kedautil.NewTLSConfigWithPassword(metadata.cert, metadata.key, metadata.keyPassword, metadata.ca, metadata.unsafeSsl)
	if err != nil {
	return nil, err
	}
	config.Net.TLS.Config = tlsConfig
	}

My understanding iabout the different combination is

No TLS + No SASL -> PLAINTEXT
TLS + No SASL -> SSL
No TLS + SASL -> SASL_PLAINTEXT
TLS + SASL -> SASL_SSL # Your case