Azure EventHubs KEDA Scaler with Azure Container Apps - FailedGetExternalMetric

[ALS-Adebeer]

Report

I'm having difficulty with configuring the azure-eventhub for our Event hub consumers. Checkpoints is successfully saved via Azure.Messaging.EventHubs.Primitives.BlobCheckpointStore but the KEDA scaler reports FailedGetExternalMetric error

Expected Behavior

KEDA is able to fetch required metrics so that consumers replicas can be scaled down to 0 when no messages to process

Actual Behavior

There are no container application errors - only errors in the container app system logs for FailedGetExternalMetric

And for completeness - I get successful log entries for KEDAScalersStarted and SyncingSecretFromAzureKeyVaultForContainerAppSucceeded (we use User assigned managed identity).

Steps to Reproduce the Problem

Scaler configuration (bicep file):

scale: {
        minReplicas: 0
        maxReplicas: 1 // we don't support more than 1 as we run 1 container app per consumer partition
        rules: [
          {
            name: 'ev-keda-scale'
            custom: {
              type: 'azure-eventhub'
              metadata: {
                //connectionFromEnv: 'ConnectionStrings__ConsumerEventHub' - not needed as we use auth below
                eventHubName: eventhub.name
                //storageConnectionFromEnv: 'ConnectionStrings__EventHubBlobStorage' - not needed as we use auth below
                blobContainer: 'consumer-checkpoints'
                storageAccountName: blobStorageAccount.name 
                checkpointStrategy: 'blobMetadata'
                consumerGroup: 'workorderlogin'
                activationUnprocessedEventThreshold: '0'
                eventHubNamespace: toLower(eventHubNamespace.name) 
              }
              identity: appMi.id
              auth: [
                {
                  secretRef: kvEventConsumer.name
                  triggerParameter: 'connection'
                }
                {
                  secretRef: kvEventHubBlobStorage.name
                  triggerParameter: 'storageConnection'
                }
              ]
            }
          }
        ]
      }

We're not using legacy implementation as blob checkpoints are saved via Azure.Messaging.EventHubs.Primitives.BlobCheckpointStore - nuget package v5.12.1

Note too that when I didn't use toLower on eventHubNamespace.name (ours have some uppercase letters) - the KEDAScalerFailed error message is slightly different saying unable to get unprocessedEventCount for metrics: unable to get checkpoint from storage: Get "https://./consumer-checkpoints/<EventHubNamespaceWithSomeUpperCase>.servicebus.windows.net%2F<eventhubname>%2Fworkorderlogin%2Fcheckpoint%2F0": dial tcp: lookup . on 10.0.0.2.53: server misbehaving. In blob storage all the container folder names are lowercase, so hence it makes sense to change to lowercase (bug?)

We also don't use Azure.Messaging.EventHubs.EventProcessorClient, I did validate that blob storage checkpoint information (folder structure and meta info) matches exactly the checkpoint info written to blob storage and the blob meta data from a native EventProcessorClient implementation. We also don't use the dynamic consumer ownership as we run 1 container app per event hub partition - as such we only write checkpoints to /.servicebus.windows.net///checkpoint' and there is no paired 'ownership' folder and I presume ownership info is irrelevant to KEDA scaler.

Logs from KEDA operator

These are from ContainerAppSystemLogs_CL

Reason_s	ContainerAppName_s	TimeStamp_s	Type_s	Log_s	EventSource_s
FailedGetExternalMetric	dwol-workorderconsumer2	2025-07-08 01:24:41 +0000 UTC	Warning	unable to get external metric azure-eventhub-workorderlogin for dwol-workorderconsumer2: unable to fetch metrics from external metrics api: rpc error: code = unknown desc = error when getting metric values metric:s0-azure-eventhub-workorderlogin encountered error	Scaling
KEDAScalersStarted	dwol-workorderconsumer2	2025-07-08 01:23:40 +0000 UTC	Normal	Scaler azure-eventhub is built.	KEDA
FailedGetExternalMetric	dwol-workorderconsumer2	2025-07-08 01:19:35 +0000 UTC	Warning	unable to get external metric azure-eventhub-workorderlogin for dwol-workorderconsumer2: unable to fetch metrics from external metrics api: rpc error: code = unknown desc = error when getting metric values metric:s0-azure-eventhub-workorderlogin encountered error	Scaling
SyncingSecretFromAzureKeyVaultForContainerAppSucceeded	dwol-workorderconsumer2	2025-07-08 01:18:57 +0000 UTC	Normal	Sync with secrets from Azure Key Vault was successful for container app dwol-workorderconsumer2	ContainerAppController
KEDAScalersStarted	dwol-workorderconsumer2	2025-07-08 01:18:40 +0000 UTC	Normal	Scaler azure-eventhub is built.	KEDA

KEDA Version

2.16.1

Kubernetes Version

Unknown - we use Azure Container Apps with private VNET

Platform

Azure (public cloud)

Scaler Details

azure-eventhub