Where does KEDA store credentials fetched from AWS Secrets Manager using podIdenity of aws and identityOwner as workload?

[rajaghan]

The following is in the context of KEDA Kafka Scaler and AWS Secrets Manager

• KEDA Kafka scaler uses AWS Secrets Manager as the Auth Provider to source credentials needed to authenticate to kakfa to fetch lag related metrics to take scaling decisions on the workload to be scaled.
• KEDA uses podIdentity.identityOwner as workload (IRSA is setup on the workload to allow KEDA Service Account to assume the IAM role that grants permissions to GetSecretValue for a secret from AWS Secrets Manager)
• It was observed from the AWS Cloud Trail that KEDA calls AWS Secrets Manager to GetSecretValue only once when a ScaledObject with kafka trigger is created. (TriggerAuthentication uses podIdentity.provider as aws and podIdentity.identityOwner as workload)
• KEDA polls kafka every 30 seconds to get the lag related metrics, but it was observed from AWS Cloud Trail that calls to AWS Secrets Manager are not being made during this polling
• Question being, where are the secrets fetched from AWS Secrets Manager being stored, to allow successful authentication with Kafka after the initial call?

@JorTurFer greatly appreciate your quick response please.

[JorTurFer]

Hello
Secrets are stored in memory inside the scaler. The scaler is created on SO/SJ creation/update and during the creation of scaler, all the secrets are fetched from the sources and passed to the scaler for the building process. After the creation, the scaler is cached in memory until:

• There is an error inside the scaler
• The SO/SJ changes

In both cases, the old cached entry is deleted from memory and another new one is created from scratch again, pulling the secrets once again. This means that if the Kafka scaler doesn't fail, the scaler won't be recreated and the fetched value is still valid.

In KEDA code, we don't store anything anywhere, and everything is stored in memory just for operations. There is only a single exception to this and it's Kafka using Kerberos auth, as Kerberos needs to get the credentials from a file, KEDA stores the secret in a temporal file in /tmp -> https://keda.sh/docs/2.17/scalers/apache-kafka/#your-kafka-cluster-turns-on-saslgssapi-auth-without-tls

If you are afraid about KEDA storing your secrets somewhere (filesystem, external db, etc), it doesn't happen except for this Kerberos case, and in general the whole filesystem is in read-only mode.
You can check that on SO/SJ changes, KEDA fetches again the secrets from secrets manager.

[rajaghan]

Thanks a lot @JorTurFer for taking out time to provide such a detailed explanation. Greatly appreciated.
Yes, you're right on the premise where I was coming from, can secrets stored in memory or otherwise in the operator pod be a security concern; can they be extracted through ephemeral debug containers?
PS: I have noticed that KEDA pods run distroless images that do not pack a shell and hence one cannot kubectl exec into these.

[JorTurFer]

The secrets are stored in memory as part of the process, so they can't be recovered just connecting to the pod via ephemeral container., To recover them from memory, you should use forensic methods, dumping the memory and scanning the dump (like any other process in software). As you said, no shell is available, so the only option to have a shell is to use an ephemeral container, but as KEDA runs using non root users, the options are limited too.

In any case, if you are using Kafka via AWS MKS, you can use your role assumption directly to access Kafka, not needing to provide secrets and relying on short-lived tokens. Latest KEDA versions support this auth mode

[rajaghan]

Thanks a ton @JorTurFer for your clarifications. Much appreciated.