Memory leak and stuck pod creation when using EFS PVC on Bottlerocket (v1.39.1)

[Kuzmenko-Pavel]

/kind bug

What happened?
We're experiencing memory leaks in efs-csi-node and stuck pod creation when using EFS volumes.

What you expected to happen?
Please advise if this is a known issue with Bottlerocket or the current driver version and if any workarounds exist.

Observed issues:

• Pods mounting EFS volumes hang indefinitely.
• efs-csi-node container shows increasing memory usage over time.
• Zombie stunnel processes accumulate under the driver pod.
• dmesg is full of NFS timeout errors:

Impact:

• PVC mounting is unreliable
• Cluster nodes get saturated with zombie processes
• Memory usage of efs-csi-node grows continuously, potentially causing node pressure

Environment

• OS: Bottlerocket OS 1.39.1 (aws-k8s-1.32)
• Kernel: 6.1.134
• Container runtime: containerd://1.7.27+bottlerocket
• Kubelet: v1.32.2-eks-677bac1
• Helm chart: aws-efs-csi-driver-3.1.9

processes in efs-csi-node pod

[root@admin]# ps -fp $(grep -l 'pod5c77b4db_18c6_47cf_a2b5_3d5feae978cb' /proc/*/cgroup 2>/dev/null | sed 's#/proc/\(.*\)/cgroup#\1#')
UID          PID    PPID  C STIME TTY      STAT   TIME CMD
root     3079105 3079083  0 Jun01 ?        Ss     0:00 /pause
root     3079138 3079083  0 Jun01 ?        Dsl    1:58 /bin/aws-efs-csi-driver --endpoint=unix:/csi/csi.sock --logtostderr --v=2 --vol-metrics-opt-in=false --vol-metrics-refresh-period=240 --vol-metrics-fs-rate-limit=5
root     3079162 3079138  0 Jun01 ?        S      0:53 /usr/bin/python3 /usr/bin/amazon-efs-mount-watchdog
root     3079306 3079083  0 Jun01 ?        Ssl    0:01 /csi-node-driver-registrar --csi-address=/csi/csi.sock --kubelet-registration-path=/var/lib/kubelet/plugins/efs.csi.aws.com/csi.sock --v=2
root     3079500 3079083  0 Jun01 ?        Ssl    0:37 /livenessprobe --csi-address=/csi/csi.sock --health-port=9809 --v=2
root     3089054 3079138  0 Jun01 ?        Ssl    0:01 /sbin/efs-proxy /var/run/efs/stunnel-config.fs-00cb6e82cf3e8beb4.var.lib.kubelet.pods.9fd5c826-3ccf-433c-82aa-52ff83491d0e.volumes.kubernetes.io~csi.pvc-9c3bb10e-c3b8-468f-9f44-169b13f45c47.mount.205
root     3092086 3079138  0 Jun01 ?        Ssl    0:01 /sbin/efs-proxy /var/run/efs/stunnel-config.fs-00cb6e82cf3e8beb4.var.lib.kubelet.pods.7f34f5dc-7d18-4813-802e-1f54f6a28177.volumes.kubernetes.io~csi.pvc-b3e000e9-2eae-4d8a-9afa-a717a8b2b26f.mount.205
root     3092333 3079138  0 Jun01 ?        Ssl    0:01 /sbin/efs-proxy /var/run/efs/stunnel-config.fs-00cb6e82cf3e8beb4.var.lib.kubelet.pods.7c6df583-9e4f-4d5e-9730-e2e8f38c4ff9.volumes.kubernetes.io~csi.pvc-b3e000e9-2eae-4d8a-9afa-a717a8b2b26f.mount.206
root     3093098 3079138  0 Jun01 ?        Ssl    0:01 /sbin/efs-proxy /var/run/efs/stunnel-config.fs-00cb6e82cf3e8beb4.var.lib.kubelet.pods.a3a23cc9-1c48-41b7-a76c-307f9e71dd43.volumes.kubernetes.io~csi.pvc-b3e000e9-2eae-4d8a-9afa-a717a8b2b26f.mount.206
root     3094698 3079138  0 Jun01 ?        Ssl    0:01 /sbin/efs-proxy /var/run/efs/stunnel-config.fs-00cb6e82cf3e8beb4.var.lib.kubelet.pods.0311ad9e-0e80-4b9e-9457-2526e9571c19.volumes.kubernetes.io~csi.pvc-e2daa308-7a3c-4e5c-a0c0-d7a8327f0bc3.mount.207
root     3094717 3079138  0 Jun01 ?        Ssl    0:11 /sbin/efs-proxy /var/run/efs/stunnel-config.fs-00cb6e82cf3e8beb4.var.lib.kubelet.pods.0311ad9e-0e80-4b9e-9457-2526e9571c19.volumes.kubernetes.io~csi.pvc-e6cce698-52e7-4d62-8de9-0a8977fa308b.mount.206
root     3095180 3079138  0 Jun01 ?        Ssl    0:01 /sbin/efs-proxy /var/run/efs/stunnel-config.fs-00cb6e82cf3e8beb4.var.lib.kubelet.pods.fa58e62d-50ed-4f78-93dd-c0d47b3dd472.volumes.kubernetes.io~csi.pvc-9d4ef8a3-1804-4ff8-aed0-47fdbf5d6004.mount.207
root     3098527 3079138  0 Jun01 ?        Ssl    0:01 /sbin/efs-proxy /var/run/efs/stunnel-config.fs-00cb6e82cf3e8beb4.var.lib.kubelet.pods.e3864605-1cc4-4974-a1fc-4a688359e066.volumes.kubernetes.io~csi.pvc-8d417dde-af0c-42e7-926d-a66a4604ca40.mount.201
root     3098866 3079138  0 Jun01 ?        Ssl    0:01 /sbin/efs-proxy /var/run/efs/stunnel-config.fs-00cb6e82cf3e8beb4.var.lib.kubelet.pods.124604d4-d41a-4cd3-b96d-4ed4e49d27a0.volumes.kubernetes.io~csi.pvc-9c59f7da-a56a-4fca-9caa-c8934eb98a75.mount.200
root     3103083 3079138  0 Jun01 ?        Ssl    0:01 /sbin/efs-proxy /var/run/efs/stunnel-config.fs-00cb6e82cf3e8beb4.var.lib.kubelet.pods.5cf67876-e5e9-4f4b-92c9-e40a85c67e9c.volumes.kubernetes.io~csi.pvc-9d4ef8a3-1804-4ff8-aed0-47fdbf5d6004.mount.205
root     3105607 3079138  0 Jun01 ?        Ssl    0:01 /sbin/efs-proxy /var/run/efs/stunnel-config.fs-00cb6e82cf3e8beb4.var.lib.kubelet.pods.7dad2514-2286-4c30-b2c6-4386465ac6b6.volumes.kubernetes.io~csi.pvc-9d4ef8a3-1804-4ff8-aed0-47fdbf5d6004.mount.202
root     3659271 3079083  0 06:24 pts/0    Ss     0:00 sh -c clear; (bash || ash || sh)
root     3659277 3659271  0 06:24 pts/0    S      0:00 sh -c clear; (bash || ash || sh)
root     3659278 3659277  0 06:24 pts/0    S+     0:00 bash
root     3665340 3079162  0 06:30 ?        Zs     0:00 [stunnel] <defunct>
root     3670793 3079162  0 06:35 ?        Zs     0:00 [stunnel] <defunct>
root     3673767 3079162  0 06:38 ?        Zs     0:00 [stunnel] <defunct>
root     3675730 3079162  0 06:40 ?        Zs     0:00 [stunnel] <defunct>
root     3677768 3079162  0 06:42 ?        Zs     0:00 [stunnel] <defunct>
root     3678764 3079162  0 06:43 ?        Zs     0:00 [stunnel] <defunct>
root     3679759 3079162  0 06:44 ?        Zs     0:00 [stunnel] <defunct>
root     3680732 3079162  0 06:45 ?        Zs     0:00 [stunnel] <defunct>
root     3681733 3079162  0 06:46 ?        Zs     0:00 [stunnel] <defunct>
root     3682791 3079162  0 06:48 ?        Zs     0:00 [stunnel] <defunct>
root     3683284 3079162  0 06:48 ?        Zs     0:00 [stunnel] <defunct>
root     3683763 3079162  0 06:49 ?        Zs     0:00 [stunnel] <defunct>
root     3684266 3079162  0 06:49 ?        Zs     0:00 [stunnel] <defunct>
root     3684752 3079162  0 06:50 ?        Zs     0:00 [stunnel] <defunct>
root     3685231 3079162  0 06:50 ?        Ssl    0:00 /usr/bin/stunnel /var/run/efs/stunnel-config.fs-00cb6e82cf3e8beb4.var.lib.kubelet.pods.6549df91-9a7a-4d88-8df4-a235094292fe.volumes.kubernetes.io~csi.pvc-325a0f9b-1f22-49cb-8b0f-13a90c382b77.mount.20
root     3685742 3079162  0 06:51 ?        Ssl    0:00 /usr/bin/stunnel /var/run/efs/stunnel-config.fs-00cb6e82cf3e8beb4.var.lib.kubelet.pods.ffc930cd-1ab4-4f62-9f69-dcfac0a8ea5c.volumes.kubernetes.io~csi.pvc-8d417dde-af0c-42e7-926d-a66a4604ca40.mount.20
root     3686223 3079162  0 06:51 ?        Ssl    0:00 /usr/bin/stunnel /var/run/efs/stunnel-config.fs-00cb6e82cf3e8beb4.var.lib.kubelet.pods.5471e847-c8f7-4515-b67a-901905a1f33f.volumes.kubernetes.io~csi.pvc-9c59f7da-a56a-4fca-9caa-c8934eb98a75.mount.20
root     3686712 3079162  0 06:52 ?        Ssl    0:00 /usr/bin/stunnel5 /var/run/efs/stunnel-config.fs-00cb6e82cf3e8beb4.var.lib.kubelet.pods.ab289b51-1b02-428c-8ed3-73dec1c17359.volumes.kubernetes.io~csi.pvc-9d4ef8a3-1804-4ff8-aed0-47fdbf5d6004.mount.2
root     3687228 3079162  0 06:52 ?        Ssl    0:00 /usr/bin/stunnel5 /var/run/efs/stunnel-config.fs-00cb6e82cf3e8beb4.var.lib.kubelet.pods.e0723447-ea7e-4341-ac30-1e6c4808bb1a.volumes.kubernetes.io~csi.pvc-8d417dde-af0c-42e7-926d-a66a4604ca40.mount.2
root     3687731 3079162  0 06:53 ?        Zs     0:00 [stunnel] <defunct>
root     3688215 3079162  0 06:53 ?        Ssl    0:00 /usr/bin/stunnel /var/run/efs/stunnel-config.fs-00cb6e82cf3e8beb4.var.lib.kubelet.pods.de619eb8-d2a4-4274-bb9c-c00720dd2031.volumes.kubernetes.io~csi.pvc-e6cce698-52e7-4d62-8de9-0a8977fa308b.mount.20
root     3688732 3079162  0 06:54 ?        Ssl    0:00 /usr/bin/stunnel /var/run/efs/stunnel-config.fs-00cb6e82cf3e8beb4.var.lib.kubelet.pods.b5f61114-ef9c-43ea-9b75-1f0cb9ed680b.volumes.kubernetes.io~csi.pvc-3b549f7f-aba7-457e-b2c7-ee9d24174ec5.mount.20
root     3689212 3079162  0 06:54 ?        Ssl    0:00 /usr/bin/stunnel /var/run/efs/stunnel-config.fs-00cb6e82cf3e8beb4.var.lib.kubelet.pods.04b267a3-ce67-4c0c-9832-80fbeb983c27.volumes.kubernetes.io~csi.pvc-e2daa308-7a3c-4e5c-a0c0-d7a8327f0bc3.mount.20
root     3689698 3079162  0 06:55 ?        Ssl    0:00 /usr/bin/stunnel /var/run/efs/stunnel-config.fs-00cb6e82cf3e8beb4.var.lib.kubelet.pods.6549df91-9a7a-4d88-8df4-a235094292fe.volumes.kubernetes.io~csi.pvc-029f0195-b05f-412a-a9e5-e4710d4a7fb5.mount.20
root     3690200 3079162  0 06:55 ?        Ssl    0:00 /usr/bin/stunnel /var/run/efs/stunnel-config.fs-00cb6e82cf3e8beb4.var.lib.kubelet.pods.20250af5-3038-47e5-b98e-f21a0a2af167.volumes.kubernetes.io~csi.pvc-3b549f7f-aba7-457e-b2c7-ee9d24174ec5.mount.20
root     3690678 3079162  0 06:56 ?        Ssl    0:00 /usr/bin/stunnel /var/run/efs/stunnel-config.fs-00cb6e82cf3e8beb4.var.lib.kubelet.pods.cc59ad5e-77f2-4442-9929-c69b64155e11.volumes.kubernetes.io~csi.pvc-e0d02abf-200a-4fa2-8a80-d0084d5b7ce8.mount.20
root     3691154 3079162  0 06:56 ?        Ssl    0:00 /usr/bin/stunnel /var/run/efs/stunnel-config.fs-00cb6e82cf3e8beb4.var.lib.kubelet.pods.ed813042-0752-435f-a551-f40f23442f99.volumes.kubernetes.io~csi.pvc-8d417dde-af0c-42e7-926d-a66a4604ca40.mount.20
root     3691654 3079162  0 06:57 ?        Ssl    0:00 /usr/bin/stunnel /var/run/efs/stunnel-config.fs-00cb6e82cf3e8beb4.var.lib.kubelet.pods.6549df91-9a7a-4d88-8df4-a235094292fe.volumes.kubernetes.io~csi.pvc-93938c16-a9ff-48e0-8d88-78996939997b.mount.20
root     3692134 3079162  0 06:57 ?        Ssl    0:00 /usr/bin/stunnel /var/run/efs/stunnel-config.fs-00cb6e82cf3e8beb4.var.lib.kubelet.pods.2faa8fe9-79cd-4769-b307-7f9373baea38.volumes.kubernetes.io~csi.pvc-8d417dde-af0c-42e7-926d-a66a4604ca40.mount.20
root     3692659 3079162  0 06:58 ?        Ssl    0:00 /usr/bin/stunnel /var/run/efs/stunnel-config.fs-00cb6e82cf3e8beb4.var.lib.kubelet.pods.04b267a3-ce67-4c0c-9832-80fbeb983c27.volumes.kubernetes.io~csi.pvc-e6cce698-52e7-4d62-8de9-0a8977fa308b.mount.20
root     3693140 3079162  0 06:58 ?        Ssl    0:00 /usr/bin/stunnel /var/run/efs/stunnel-config.fs-00cb6e82cf3e8beb4.var.lib.kubelet.pods.56db3b11-9140-4027-a383-2fd9a6599e20.volumes.kubernetes.io~csi.pvc-8d417dde-af0c-42e7-926d-a66a4604ca40.mount.20
root     3693618 3079162  0 06:59 ?        Ssl    0:00 /usr/bin/stunnel /var/run/efs/stunnel-config.fs-00cb6e82cf3e8beb4.var.lib.kubelet.pods.86eb7a0b-031a-4598-b845-a8991b854d2b.volumes.kubernetes.io~csi.pvc-e0d02abf-200a-4fa2-8a80-d0084d5b7ce8.mount.20
root     3694127 3079162  0 06:59 ?        Ssl    0:00 /usr/bin/stunnel /var/run/efs/stunnel-config.fs-00cb6e82cf3e8beb4.var.lib.kubelet.pods.bf1dbd30-b67d-406c-98bd-2499939bf8f6.volumes.kubernetes.io~csi.pvc-8d417dde-af0c-42e7-926d-a66a4604ca40.mount.20
root     3694613 3079162  0 07:00 ?        Ssl    0:00 /usr/bin/stunnel /var/run/efs/stunnel-config.fs-00cb6e82cf3e8beb4.var.lib.kubelet.pods.6549df91-9a7a-4d88-8df4-a235094292fe.volumes.kubernetes.io~csi.pvc-44f4362b-c17d-436e-afb2-483ae101eacb.mount.20
root     3694616 3079162  0 07:00 ?        D      0:00 df /var/lib/kubelet/pods/6549df91-9a7a-4d88-8df4-a235094292fe/volumes/kubernetes.io~csi/pvc-325a0f9b-1f22-49cb-8b0f-13a90c382b77/mount
[root@admin]# 

The following errors are in dmesg

[167343.953900] nfs: server 127.0.0.1 not responding, timed out
[167343.953915] nfs: server 127.0.0.1 not responding, timed out
[167433.100011] nfs: server 127.0.0.1 not responding, timed out
[167583.233983] nfs: server 127.0.0.1 not responding, timed out
[167855.223822] nfs: server 127.0.0.1 not responding, timed out
[167904.463810] nfs: server 127.0.0.1 not responding, timed out
[168367.393427] nfs: server 127.0.0.1 not responding, timed out
[168447.447073] nfs: server 127.0.0.1 not responding, timed out
[168648.523654] nfs: server 127.0.0.1 not responding, timed out
[168648.523679] nfs: server 127.0.0.1 not responding, timed out

EFSkubelet logs

root@ip-10-100-26-198:/# chroot /host journalctl -u kubelet -l | grep efs
Jun 02 07:02:07 ip-10-100-26-198.eu-west-1.compute.internal kubelet[2251]: E0602 07:02:07.027807    2251 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/csi/efs.csi.aws.com^fs-00cb6e82cf3e8beb4::fsap-0163f2557f011b17c podName:ed813042-0752-435f-a551-f40f23442f99 nodeName:}" failed. No retries permitted until 2025-06-02 07:04:09.027781637 +0000 UTC m=+170652.518432783 (durationBeforeRetry 2m2s). Error: UnmountVolume.TearDown failed for volume "config-files" (UniqueName: "kubernetes.io/csi/efs.csi.aws.com^fs-00cb6e82cf3e8beb4::fsap-0163f2557f011b17c") pod "ed813042-0752-435f-a551-f40f23442f99" (UID: "ed813042-0752-435f-a551-f40f23442f99") : kubernetes.io/csi: Unmounter.TearDownAt failed: rpc error: code = DeadlineExceeded desc = context deadline exceeded
Jun 02 07:02:09 ip-10-100-26-198.eu-west-1.compute.internal kubelet[2251]: E0602 07:02:09.055334    2251 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/csi/efs.csi.aws.com^fs-00cb6e82cf3e8beb4::fsap-0163f2557f011b17c podName:e0723447-ea7e-4341-ac30-1e6c4808bb1a nodeName:}" failed. No retries permitted until 2025-06-02 07:04:11.055306857 +0000 UTC m=+170654.545958003 (durationBeforeRetry 2m2s). Error: UnmountVolume.TearDown failed for volume "config-files" (UniqueName: "kubernetes.io/csi/efs.csi.aws.com^fs-00cb6e82cf3e8beb4::fsap-0163f2557f011b17c") pod "e0723447-ea7e-4341-ac30-1e6c4808bb1a" (UID: "e0723447-ea7e-4341-ac30-1e6c4808bb1a") : kubernetes.io/csi: Unmounter.TearDownAt failed: rpc error: code = DeadlineExceeded desc = context deadline exceeded
Jun 02 07:02:13 ip-10-100-26-198.eu-west-1.compute.internal kubelet[2251]: E0602 07:02:13.116797    2251 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/csi/efs.csi.aws.com^fs-00cb6e82cf3e8beb4::fsap-0a5d6da1a2b883c96 podName:5471e847-c8f7-4515-b67a-901905a1f33f nodeName:}" failed. No retries permitted until 2025-06-02 07:04:15.116769629 +0000 UTC m=+170658.607420776 (durationBeforeRetry 2m2s). Error: UnmountVolume.TearDown failed for volume "config-files" (UniqueName: "kubernetes.io/csi/efs.csi.aws.com^fs-00cb6e82cf3e8beb4::fsap-0a5d6da1a2b883c96") pod "5471e847-c8f7-4515-b67a-901905a1f33f" (UID: "5471e847-c8f7-4515-b67a-901905a1f33f") : kubernetes.io/csi: Unmounter.TearDownAt failed: rpc error: code = DeadlineExceeded desc = context deadline exceeded
Jun 02 07:02:17 ip-10-100-26-198.eu-west-1.compute.internal kubelet[2251]: I0602 07:02:17.942536    2251 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"media\" (UniqueName: \"kubernetes.io/csi/efs.csi.aws.com^fs-00cb6e82cf3e8beb4::fsap-0eccf76f606e2107d\") pod \"de619eb8-d2a4-4274-bb9c-c00720dd2031\" (UID: \"de619eb8-d2a4-4274-bb9c-c00720dd2031\") "
Jun 02 07:02:17 ip-10-100-26-198.eu-west-1.compute.internal kubelet[2251]: I0602 07:02:17.942695    2251 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"config-files\" (UniqueName: \"kubernetes.io/csi/efs.csi.aws.com^fs-00cb6e82cf3e8beb4::fsap-0d2a7ad88595fae55\") pod \"20250af5-3038-47e5-b98e-f21a0a2af167\" (UID: \"20250af5-3038-47e5-b98e-f21a0a2af167\") "
Jun 02 07:02:19 ip-10-100-26-198.eu-west-1.compute.internal kubelet[2251]: E0602 07:02:19.306219    2251 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/csi/efs.csi.aws.com^fs-00cb6e82cf3e8beb4::fsap-0d2a7ad88595fae55 podName:b5f61114-ef9c-43ea-9b75-1f0cb9ed680b nodeName:}" failed. No retries permitted until 2025-06-02 07:04:21.306195383 +0000 UTC m=+170664.796846529 (durationBeforeRetry 2m2s). Error: UnmountVolume.TearDown failed for volume "config-files" (UniqueName: "kubernetes.io/csi/efs.csi.aws.com^fs-00cb6e82cf3e8beb4::fsap-0d2a7ad88595fae55") pod "b5f61114-ef9c-43ea-9b75-1f0cb9ed680b" (UID: "b5f61114-ef9c-43ea-9b75-1f0cb9ed680b") : kubernetes.io/csi: Unmounter.TearDownAt failed: rpc error: code = DeadlineExceeded desc = context deadline exceeded

Pod logs: efs-csi-node-n29vn

I0601 20:53:12.961149       1 config_dir.go:88] Creating symlink from '/etc/amazon/efs' to '/var/amazon/efs'
I0601 20:53:12.961692       1 metadata.go:67] getting MetadataService...
I0601 20:53:12.964000       1 metadata.go:72] retrieving metadata from EC2 metadata service
I0601 20:53:12.967734       1 driver.go:154] Did not find any input tags.
I0601 20:53:12.967940       1 driver.go:121] Registering Node Server
I0601 20:53:12.967947       1 driver.go:123] Registering Controller Server
I0601 20:53:12.967954       1 driver.go:126] Starting efs-utils watchdog
I0601 20:53:12.968033       1 efs_watch_dog.go:262] Skip copying /etc/amazon/efs/efs-utils.conf since it exists already
I0601 20:53:12.968045       1 efs_watch_dog.go:257] Copying /etc/amazon/efs/efs-utils.crt
I0601 20:53:12.970713       1 driver.go:132] Starting reaper
I0601 20:53:12.970739       1 driver.go:141] Listening for connections on address: &net.UnixAddr{Name:"/csi/csi.sock", Net:"unix"}
I0601 20:53:13.138460       1 main.go:150] "Version" version="v2.13.0"
I0601 20:53:13.138502       1 main.go:151] "Running node-driver-registrar" mode=""
I0601 20:53:13.138506       1 main.go:172] "Attempting to open a gRPC connection" csiAddress="/csi/csi.sock"
I0601 20:53:13.139246       1 main.go:180] "Calling CSI driver to discover driver name"
I0601 20:53:13.140700       1 main.go:189] "CSI driver name" csiDriverName="efs.csi.aws.com"
I0601 20:53:13.140737       1 node_register.go:56] "Starting Registration Server" socketPath="/registration/efs.csi.aws.com-reg.sock"
I0601 20:53:13.140972       1 node_register.go:66] "Registration Server started" socketPath="/registration/efs.csi.aws.com-reg.sock"
I0601 20:53:13.141038       1 node_register.go:96] "Skipping HTTP server"
I0601 20:53:13.248348       1 main.go:135] "Calling CSI driver to discover driver name"
I0601 20:53:13.250826       1 main.go:143] "CSI driver name" driver="efs.csi.aws.com"
I0601 20:53:13.250906       1 main.go:172] "ServeMux listening" address="0.0.0.0:9809"
I0601 20:53:13.954623       1 main.go:96] "Received GetInfo call" request="&InfoRequest{}"
I0601 20:53:13.979296       1 main.go:108] "Received NotifyRegistrationStatus call" status="&RegistrationStatus{PluginRegistered:true,Error:,}"
W0601 20:59:01.830424       1 reaper.go:105] reaper: failed to wait for process &{88 17 90 88 88 stunnel}: no child processes
W0601 20:59:18.869573       1 reaper.go:105] reaper: failed to wait for process &{79 17 90 79 79 stunnel}: no child processes
W0601 20:59:18.890732       1 reaper.go:105] reaper: failed to wait for process &{79 17 90 79 79 stunnel}: no child processes
W0601 20:59:18.891560       1 reaper.go:105] reaper: failed to wait for process &{79 17 90 79 79 stunnel}: no child processes
I0601 20:59:20.039199       1 mount_linux.go:243] Detected OS without systemd
W0601 20:59:20.399307       1 reaper.go:105] reaper: failed to wait for process &{79 17 90 79 79 stunnel}: no child processes
W0601 21:00:04.555079       1 reaper.go:105] reaper: failed to wait for process &{76 17 90 76 76 stunnel5}: no child processes
W0601 21:00:05.799231       1 reaper.go:105] reaper: failed to wait for process &{76 17 90 76 76 stunnel5}: no child processes
W0601 21:00:09.065799       1 reaper.go:105] reaper: failed to wait for process &{76 17 90 76 76 stunnel5}: no child processes
W0601 21:00:21.950128       1 reaper.go:105] reaper: failed to wait for process &{76 17 90 76 76 stunnel5}: no child processes
W0601 21:00:22.099659       1 reaper.go:105] reaper: failed to wait for process &{76 17 90 76 76 stunnel5}: no child processes
W0601 21:00:22.118006       1 reaper.go:105] reaper: failed to wait for process &{76 17 90 76 76 stunnel5}: no child processes
W0601 21:00:26.120773       1 reaper.go:105] reaper: failed to wait for process &{76 17 90 76 76 stunnel5}: no child processes
W0601 21:00:31.599743       1 reaper.go:105] reaper: failed to wait for process &{76 17 90 76 76 stunnel5}: no child processes
W0601 21:00:33.440934       1 reaper.go:105] reaper: failed to wait for process &{76 17 90 76 76 stunnel5}: no child processes
W0601 21:00:33.441827       1 reaper.go:105] reaper: failed to wait for process &{76 17 90 76 76 stunnel5}: no child processes
W0601 21:00:42.489910       1 reaper.go:105] reaper: failed to wait for process &{76 17 90 76 76 stunnel5}: no child processes

kubectl describe pod efs-csi-node-n29vn -n kube-system

Name:                 efs-csi-node-n29vn
Namespace:            kube-system
Priority:             2000001000
Priority Class Name:  system-node-critical
Service Account:      efs-csi-node-sa
Node:                 ip-10-100-26-198.eu-west-1.compute.internal/10.100.26.198
Start Time:           Sun, 01 Jun 2025 23:53:12 +0300
Labels:               app=efs-csi-node
                      app.kubernetes.io/instance=aws-efs-csi
                      app.kubernetes.io/name=aws-efs
                      controller-revision-hash=55f8597bc6
                      pod-template-generation=16
Annotations:          kubectl.kubernetes.io/restartedAt: 2025-06-01T20:50:06.394Z
Status:               Running
IP:                   10.100.26.198
IPs:
  IP:           10.100.26.198
Controlled By:  DaemonSet/efs-csi-node
Containers:
  efs-plugin:
    Container ID:  containerd://05c7653f09574de3d11b8b4072e8103de6ba0772c12832465e0a419034cfd3d6
    Image:         public.ecr.aws/efs-csi-driver/amazon/aws-efs-csi-driver:v2.1.8
    Image ID:      public.ecr.aws/efs-csi-driver/amazon/aws-efs-csi-driver@sha256:691ae9947da35be49bdbf4d89133527eb58c5ca1d0d18056950592e2a92973eb
    Port:          9809/TCP
    Host Port:     9809/TCP
    Args:
      --endpoint=$(CSI_ENDPOINT)
      --logtostderr
      --v=2
      --vol-metrics-opt-in=false
      --vol-metrics-refresh-period=240
      --vol-metrics-fs-rate-limit=5
    State:          Running
      Started:      Sun, 01 Jun 2025 23:53:12 +0300
    Ready:          True
    Restart Count:  0
    Limits:
      memory:  2Gi
    Requests:
      cpu:     100m
      memory:  512Mi
    Liveness:  http-get http://:healthz/healthz delay=10s timeout=3s period=2s #success=1 #failure=5
    Environment:
      CSI_ENDPOINT:                 unix:/csi/csi.sock
      CSI_NODE_NAME:                 (v1:spec.nodeName)
      PORT_RANGE_UPPER_BOUND:       21049
      AWS_STS_REGIONAL_ENDPOINTS:   regional
      AWS_DEFAULT_REGION:           eu-west-1
      AWS_REGION:                   eu-west-1
      AWS_ROLE_ARN:                 arn:aws:iam::XXXXXXXXXX:role/AWSEfsCsiIAMRole
      AWS_WEB_IDENTITY_TOKEN_FILE:  /var/run/secrets/eks.amazonaws.com/serviceaccount/token
    Mounts:
      /csi from plugin-dir (rw)
      /etc/amazon/efs-legacy from efs-utils-config-legacy (rw)
      /var/amazon/efs from efs-utils-config (rw)
      /var/lib/kubelet from kubelet-dir (rw)
      /var/run/efs from efs-state-dir (rw)
      /var/run/secrets/eks.amazonaws.com/serviceaccount from aws-iam-token (ro)
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-whtn6 (ro)
  csi-driver-registrar:
    Container ID:  containerd://a6764aff6916f1bbcbf0b9fc05dd1642a9abac9a0ab7ec29f77af83b00015fe9
    Image:         public.ecr.aws/eks-distro/kubernetes-csi/node-driver-registrar:v2.13.0-eks-1-32-6
    Image ID:      public.ecr.aws/eks-distro/kubernetes-csi/node-driver-registrar@sha256:714d918d4b3510eb0e66c38f11c61095ba25980b121c192dace47bf9e003acc3
    Port:          <none>
    Host Port:     <none>
    Args:
      --csi-address=$(ADDRESS)
      --kubelet-registration-path=$(DRIVER_REG_SOCK_PATH)
      --v=2
    State:          Running
      Started:      Sun, 01 Jun 2025 23:53:13 +0300
    Ready:          True
    Restart Count:  0
    Limits:
      memory:  64Mi
    Requests:
      cpu:     100m
      memory:  32Mi
    Environment:
      ADDRESS:                      /csi/csi.sock
      DRIVER_REG_SOCK_PATH:         /var/lib/kubelet/plugins/efs.csi.aws.com/csi.sock
      KUBE_NODE_NAME:                (v1:spec.nodeName)
      AWS_STS_REGIONAL_ENDPOINTS:   regional
      AWS_DEFAULT_REGION:           eu-west-1
      AWS_REGION:                   eu-west-1
      AWS_ROLE_ARN:                 arn:aws:iam::XXXXXXXXXX:role/AWSEfsCsiIAMRole
      AWS_WEB_IDENTITY_TOKEN_FILE:  /var/run/secrets/eks.amazonaws.com/serviceaccount/token
    Mounts:
      /csi from plugin-dir (rw)
      /registration from registration-dir (rw)
      /var/run/secrets/eks.amazonaws.com/serviceaccount from aws-iam-token (ro)
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-whtn6 (ro)
  liveness-probe:
    Container ID:  containerd://5eb81355b2b561af3a9b6590ab08bd22fa8f081fa2f8b5c6a63e4f94687ef34f
    Image:         public.ecr.aws/eks-distro/kubernetes-csi/livenessprobe:v2.14.0-eks-1-32-6
    Image ID:      public.ecr.aws/eks-distro/kubernetes-csi/livenessprobe@sha256:d92f5afbdf30cbca79ad1e584dae4ee6d0d4ff7541d021e28a52cb2f3720bcfe
    Port:          <none>
    Host Port:     <none>
    Args:
      --csi-address=/csi/csi.sock
      --health-port=9809
      --v=2
    State:          Running
      Started:      Sun, 01 Jun 2025 23:53:13 +0300
    Ready:          True
    Restart Count:  0
    Limits:
      memory:  64Mi
    Requests:
      cpu:     100m
      memory:  32Mi
    Environment:
      AWS_STS_REGIONAL_ENDPOINTS:   regional
      AWS_DEFAULT_REGION:           eu-west-1
      AWS_REGION:                   eu-west-1
      AWS_ROLE_ARN:                 arn:aws:iam::XXXXXXXXXX:role/AWSEfsCsiIAMRole
      AWS_WEB_IDENTITY_TOKEN_FILE:  /var/run/secrets/eks.amazonaws.com/serviceaccount/token
    Mounts:
      /csi from plugin-dir (rw)
      /var/run/secrets/eks.amazonaws.com/serviceaccount from aws-iam-token (ro)
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-whtn6 (ro)
Conditions:
  Type                        Status
  PodReadyToStartContainers   True 
  Initialized                 True 
  Ready                       True 
  ContainersReady             True 
  PodScheduled                True 
Volumes:
  aws-iam-token:
    Type:                    Projected (a volume that contains injected data from multiple sources)
    TokenExpirationSeconds:  86400
  kubelet-dir:
    Type:          HostPath (bare host directory volume)
    Path:          /var/lib/kubelet
    HostPathType:  Directory
  plugin-dir:
    Type:          HostPath (bare host directory volume)
    Path:          /var/lib/kubelet/plugins/efs.csi.aws.com/
    HostPathType:  DirectoryOrCreate
  registration-dir:
    Type:          HostPath (bare host directory volume)
    Path:          /var/lib/kubelet/plugins_registry/
    HostPathType:  Directory
  efs-state-dir:
    Type:          HostPath (bare host directory volume)
    Path:          /var/run/efs
    HostPathType:  DirectoryOrCreate
  efs-utils-config:
    Type:          HostPath (bare host directory volume)
    Path:          /var/amazon/efs
    HostPathType:  DirectoryOrCreate
  efs-utils-config-legacy:
    Type:          HostPath (bare host directory volume)
    Path:          /etc/amazon/efs
    HostPathType:  DirectoryOrCreate
  kube-api-access-whtn6:
    Type:                    Projected (a volume that contains injected data from multiple sources)
    TokenExpirationSeconds:  3607
    ConfigMapName:           kube-root-ca.crt
    ConfigMapOptional:       <nil>
    DownwardAPI:             true
QoS Class:                   Burstable
Node-Selectors:              kubernetes.io/os=linux
Tolerations:                 op=Exists
                             node.kubernetes.io/disk-pressure:NoSchedule op=Exists
                             node.kubernetes.io/memory-pressure:NoSchedule op=Exists
                             node.kubernetes.io/network-unavailable:NoSchedule op=Exists
                             node.kubernetes.io/not-ready:NoExecute op=Exists
                             node.kubernetes.io/pid-pressure:NoSchedule op=Exists
                             node.kubernetes.io/unreachable:NoExecute op=Exists
                             node.kubernetes.io/unschedulable:NoSchedule op=Exists
Events:                      <none>

Please also attach debug logs to help us better diagnose

• Instructions to gather debug logs can be found here

[Kuzmenko-Pavel]

Memory Usage efs-csi-node-n29vn

[Kuzmenko-Pavel]

Detailed output ps with memory

[root@admin]# ps -eo pid,ppid,uid,%mem,vsz,rss,cmd --sort=-rss | grep -Ff <(grep -l 'pod5c77b4db_18c6_47cf_a2b5_3d5feae978cb' /proc/*/cgroup 2>/dev/null | sed 's#/proc/\(.*\)/cgroup#\1#')
3079138 3079083     0  1.0 1930388 354136 /bin/aws-efs-csi-driver --endpoint=unix:/csi/csi.sock --logtostderr --v=2 --vol-metrics-opt-in=false --vol-metrics-refresh-period=240 --vol-metrics-fs-rate-limit=5
3079162 3079138     0  0.0  30544 24976 /usr/bin/python3 /usr/bin/amazon-efs-mount-watchdog
3079500 3079083     0  0.0 1246640 24368 /livenessprobe --csi-address=/csi/csi.sock --health-port=9809 --v=2
3079306 3079083     0  0.0 1246328 19732 /csi-node-driver-registrar --csi-address=/csi/csi.sock --kubelet-registration-path=/var/lib/kubelet/plugins/efs.csi.aws.com/csi.sock --v=2
3094717 3079138     0  0.0 758636 13096 /sbin/efs-proxy /var/run/efs/stunnel-config.fs-00cb6e82cf3e8beb4.var.lib.kubelet.pods.0311ad9e-0e80-4b9e-9457-2526e9571c19.volumes.kubernetes.io~csi.pvc-e6cce698-52e7-4d62-8de9-0a8977fa308b.mount.20688 --tls
3092333 3079138     0  0.0 759668 12692 /sbin/efs-proxy /var/run/efs/stunnel-config.fs-00cb6e82cf3e8beb4.var.lib.kubelet.pods.7c6df583-9e4f-4d5e-9730-e2e8f38c4ff9.volumes.kubernetes.io~csi.pvc-b3e000e9-2eae-4d8a-9afa-a717a8b2b26f.mount.20600 --tls
3098527 3079138     0  0.0 624492 12352 /sbin/efs-proxy /var/run/efs/stunnel-config.fs-00cb6e82cf3e8beb4.var.lib.kubelet.pods.e3864605-1cc4-4974-a1fc-4a688359e066.volumes.kubernetes.io~csi.pvc-8d417dde-af0c-42e7-926d-a66a4604ca40.mount.20145 --tls
3092086 3079138     0  0.0 624492 12072 /sbin/efs-proxy /var/run/efs/stunnel-config.fs-00cb6e82cf3e8beb4.var.lib.kubelet.pods.7f34f5dc-7d18-4813-802e-1f54f6a28177.volumes.kubernetes.io~csi.pvc-b3e000e9-2eae-4d8a-9afa-a717a8b2b26f.mount.20590 --tls
3093098 3079138     0  0.0 759668 11992 /sbin/efs-proxy /var/run/efs/stunnel-config.fs-00cb6e82cf3e8beb4.var.lib.kubelet.pods.a3a23cc9-1c48-41b7-a76c-307f9e71dd43.volumes.kubernetes.io~csi.pvc-b3e000e9-2eae-4d8a-9afa-a717a8b2b26f.mount.20698 --tls
3103083 3079138     0  0.0 624492 11948 /sbin/efs-proxy /var/run/efs/stunnel-config.fs-00cb6e82cf3e8beb4.var.lib.kubelet.pods.5cf67876-e5e9-4f4b-92c9-e40a85c67e9c.volumes.kubernetes.io~csi.pvc-9d4ef8a3-1804-4ff8-aed0-47fdbf5d6004.mount.20504 --tls
3105607 3079138     0  0.0 624492 11944 /sbin/efs-proxy /var/run/efs/stunnel-config.fs-00cb6e82cf3e8beb4.var.lib.kubelet.pods.7dad2514-2286-4c30-b2c6-4386465ac6b6.volumes.kubernetes.io~csi.pvc-9d4ef8a3-1804-4ff8-aed0-47fdbf5d6004.mount.20295 --tls
3094698 3079138     0  0.0 624492 11596 /sbin/efs-proxy /var/run/efs/stunnel-config.fs-00cb6e82cf3e8beb4.var.lib.kubelet.pods.0311ad9e-0e80-4b9e-9457-2526e9571c19.volumes.kubernetes.io~csi.pvc-e2daa308-7a3c-4e5c-a0c0-d7a8327f0bc3.mount.20714 --tls
3098866 3079138     0  0.0 624492 11480 /sbin/efs-proxy /var/run/efs/stunnel-config.fs-00cb6e82cf3e8beb4.var.lib.kubelet.pods.124604d4-d41a-4cd3-b96d-4ed4e49d27a0.volumes.kubernetes.io~csi.pvc-9c59f7da-a56a-4fca-9caa-c8934eb98a75.mount.20076 --tls
3095180 3079138     0  0.0 624492 11352 /sbin/efs-proxy /var/run/efs/stunnel-config.fs-00cb6e82cf3e8beb4.var.lib.kubelet.pods.fa58e62d-50ed-4f78-93dd-c0d47b3dd472.volumes.kubernetes.io~csi.pvc-9d4ef8a3-1804-4ff8-aed0-47fdbf5d6004.mount.20773 --tls
3089054 3079138     0  0.0 624492 11208 /sbin/efs-proxy /var/run/efs/stunnel-config.fs-00cb6e82cf3e8beb4.var.lib.kubelet.pods.9fd5c826-3ccf-433c-82aa-52ff83491d0e.volumes.kubernetes.io~csi.pvc-9c3bb10e-c3b8-468f-9f44-169b13f45c47.mount.20522 --tls
3808378 3079162     0  0.0 148424  7760 /usr/bin/stunnel /var/run/efs/stunnel-config.fs-00cb6e82cf3e8beb4.var.lib.kubelet.pods.04b267a3-ce67-4c0c-9832-80fbeb983c27.volumes.kubernetes.io~csi.pvc-e2daa308-7a3c-4e5c-a0c0-d7a8327f0bc3.mount.20389
3806405 3079162     0  0.0 148424  7748 /usr/bin/stunnel5 /var/run/efs/stunnel-config.fs-00cb6e82cf3e8beb4.var.lib.kubelet.pods.e0723447-ea7e-4341-ac30-1e6c4808bb1a.volumes.kubernetes.io~csi.pvc-8d417dde-af0c-42e7-926d-a66a4604ca40.mount.20434
3808880 3079162     0  0.0 148424  7748 /usr/bin/stunnel /var/run/efs/stunnel-config.fs-00cb6e82cf3e8beb4.var.lib.kubelet.pods.6549df91-9a7a-4d88-8df4-a235094292fe.volumes.kubernetes.io~csi.pvc-029f0195-b05f-412a-a9e5-e4710d4a7fb5.mount.20266
3809362 3079162     0  0.0 148424  7748 /usr/bin/stunnel /var/run/efs/stunnel-config.fs-00cb6e82cf3e8beb4.var.lib.kubelet.pods.20250af5-3038-47e5-b98e-f21a0a2af167.volumes.kubernetes.io~csi.pvc-3b549f7f-aba7-457e-b2c7-ee9d24174ec5.mount.20430
3811324 3079162     0  0.0 148424  7748 /usr/bin/stunnel /var/run/efs/stunnel-config.fs-00cb6e82cf3e8beb4.var.lib.kubelet.pods.2faa8fe9-79cd-4769-b307-7f9373baea38.volumes.kubernetes.io~csi.pvc-8d417dde-af0c-42e7-926d-a66a4604ca40.mount.20152
3803908 3079162     0  0.0 148424  7720 /usr/bin/stunnel /var/run/efs/stunnel-config.fs-00cb6e82cf3e8beb4.var.lib.kubelet.pods.6549df91-9a7a-4d88-8df4-a235094292fe.volumes.kubernetes.io~csi.pvc-44f4362b-c17d-436e-afb2-483ae101eacb.mount.20433
3804893 3079162     0  0.0 148424  7720 /usr/bin/stunnel /var/run/efs/stunnel-config.fs-00cb6e82cf3e8beb4.var.lib.kubelet.pods.ffc930cd-1ab4-4f62-9f69-dcfac0a8ea5c.volumes.kubernetes.io~csi.pvc-8d417dde-af0c-42e7-926d-a66a4604ca40.mount.20322
3804411 3079162     0  0.0 148424  7680 /usr/bin/stunnel /var/run/efs/stunnel-config.fs-00cb6e82cf3e8beb4.var.lib.kubelet.pods.6549df91-9a7a-4d88-8df4-a235094292fe.volumes.kubernetes.io~csi.pvc-325a0f9b-1f22-49cb-8b0f-13a90c382b77.mount.20244
3809847 3079162     0  0.0 148424  7668 /usr/bin/stunnel /var/run/efs/stunnel-config.fs-00cb6e82cf3e8beb4.var.lib.kubelet.pods.cc59ad5e-77f2-4442-9929-c69b64155e11.volumes.kubernetes.io~csi.pvc-e0d02abf-200a-4fa2-8a80-d0084d5b7ce8.mount.20189
3810839 3079162     0  0.0 148424  7668 /usr/bin/stunnel /var/run/efs/stunnel-config.fs-00cb6e82cf3e8beb4.var.lib.kubelet.pods.6549df91-9a7a-4d88-8df4-a235094292fe.volumes.kubernetes.io~csi.pvc-93938c16-a9ff-48e0-8d88-78996939997b.mount.20099
3805886 3079162     0  0.0 148424  7624 /usr/bin/stunnel5 /var/run/efs/stunnel-config.fs-00cb6e82cf3e8beb4.var.lib.kubelet.pods.ab289b51-1b02-428c-8ed3-73dec1c17359.volumes.kubernetes.io~csi.pvc-9d4ef8a3-1804-4ff8-aed0-47fdbf5d6004.mount.20084
3807894 3079162     0  0.0 148424  7600 /usr/bin/stunnel /var/run/efs/stunnel-config.fs-00cb6e82cf3e8beb4.var.lib.kubelet.pods.b5f61114-ef9c-43ea-9b75-1f0cb9ed680b.volumes.kubernetes.io~csi.pvc-3b549f7f-aba7-457e-b2c7-ee9d24174ec5.mount.20346
3811895 3079162     0  0.0 148424  7584 /usr/bin/stunnel /var/run/efs/stunnel-config.fs-00cb6e82cf3e8beb4.var.lib.kubelet.pods.04b267a3-ce67-4c0c-9832-80fbeb983c27.volumes.kubernetes.io~csi.pvc-e6cce698-52e7-4d62-8de9-0a8977fa308b.mount.20106
3805371 3079162     0  0.0 148424  7544 /usr/bin/stunnel /var/run/efs/stunnel-config.fs-00cb6e82cf3e8beb4.var.lib.kubelet.pods.5471e847-c8f7-4515-b67a-901905a1f33f.volumes.kubernetes.io~csi.pvc-9c59f7da-a56a-4fca-9caa-c8934eb98a75.mount.20396
3812876 3079162     0  0.0 148424  7524 /usr/bin/stunnel /var/run/efs/stunnel-config.fs-00cb6e82cf3e8beb4.var.lib.kubelet.pods.86eb7a0b-031a-4598-b845-a8991b854d2b.volumes.kubernetes.io~csi.pvc-e0d02abf-200a-4fa2-8a80-d0084d5b7ce8.mount.20222
3807419 3079162     0  0.0 148424  7520 /usr/bin/stunnel /var/run/efs/stunnel-config.fs-00cb6e82cf3e8beb4.var.lib.kubelet.pods.de619eb8-d2a4-4274-bb9c-c00720dd2031.volumes.kubernetes.io~csi.pvc-e6cce698-52e7-4d62-8de9-0a8977fa308b.mount.20268
3813379 3079162     0  0.0 148424  7488 /usr/bin/stunnel /var/run/efs/stunnel-config.fs-00cb6e82cf3e8beb4.var.lib.kubelet.pods.bf1dbd30-b67d-406c-98bd-2499939bf8f6.volumes.kubernetes.io~csi.pvc-8d417dde-af0c-42e7-926d-a66a4604ca40.mount.20245
3810353 3079162     0  0.0 148424  7332 /usr/bin/stunnel /var/run/efs/stunnel-config.fs-00cb6e82cf3e8beb4.var.lib.kubelet.pods.ed813042-0752-435f-a551-f40f23442f99.volumes.kubernetes.io~csi.pvc-8d417dde-af0c-42e7-926d-a66a4604ca40.mount.20348
3812389 3079162     0  0.0 148424  7096 /usr/bin/stunnel /var/run/efs/stunnel-config.fs-00cb6e82cf3e8beb4.var.lib.kubelet.pods.56db3b11-9140-4027-a383-2fd9a6599e20.volumes.kubernetes.io~csi.pvc-8d417dde-af0c-42e7-926d-a66a4604ca40.mount.20100
3659278 3659277     0  0.0   4544  3928 bash
3713923 3713922     0  0.0   4544  3928 bash
3713915 3079083     0  0.0   4280  3304 sh -c clear; (bash || ash || sh)
3659271 3079083     0  0.0   4280  3108 sh -c clear; (bash || ash || sh)
3813382 3079162     0  0.0   3040  1052 df /var/lib/kubelet/pods/6549df91-9a7a-4d88-8df4-a235094292fe/volumes/kubernetes.io~csi/pvc-44f4362b-c17d-436e-afb2-483ae101eacb/mount
3659277 3659271     0  0.0   4280   280 sh -c clear; (bash || ash || sh)
3713922 3713915     0  0.0   4280   280 sh -c clear; (bash || ash || sh)
3079105 3079083     0  0.0    196     4 /pause
3785070 3079162     0  0.0      0     0 [stunnel] <defunct>
3790029 3079162     0  0.0      0     0 [stunnel] <defunct>
3792988 3079162     0  0.0      0     0 [stunnel] <defunct>
3794950 3079162     0  0.0      0     0 [stunnel] <defunct>
3796983 3079162     0  0.0      0     0 [stunnel] <defunct>
3797957 3079162     0  0.0      0     0 [stunnel] <defunct>
3798950 3079162     0  0.0      0     0 [stunnel] <defunct>
3799945 3079162     0  0.0      0     0 [stunnel] <defunct>
3800914 3079162     0  0.0      0     0 [stunnel] <defunct>
3801949 3079162     0  0.0      0     0 [stunnel] <defunct>
3802437 3079162     0  0.0      0     0 [stunnel] <defunct>
3802946 3079162     0  0.0      0     0 [stunnel] <defunct>
3803430 3079162     0  0.0      0     0 [stunnel] <defunct>
3806911 3079162     0  0.0      0     0 [stunnel] <defunct>
[root@admin]# 

Parameter --tls is present but missing in StorageClass, PersistentVolume, PersistentVolumeClaim why?

3094717 3079138     0  0.0 758636 13096 /sbin/efs-proxy /var/run/efs/stunnel-config.fs-00cb6e82cf3e8beb4.var.lib.kubelet.pods.0311ad9e-0e80-4b9e-9457-2526e9571c19.volumes.kubernetes.io~csi.pvc-e6cce698-52e7-4d62-8de9-0a8977fa308b.mount.20688 --tls

3811895 3079162     0  0.0 148424  7584 /usr/bin/stunnel /var/run/efs/stunnel-config.fs-00cb6e82cf3e8beb4.var.lib.kubelet.pods.04b267a3-ce67-4c0c-9832-80fbeb983c27.volumes.kubernetes.io~csi.pvc-e6cce698-52e7-4d62-8de9-0a8977fa308b.mount.20106

3807419 3079162     0  0.0 148424  7520 /usr/bin/stunnel /var/run/efs/stunnel-config.fs-00cb6e82cf3e8beb4.var.lib.kubelet.pods.de619eb8-d2a4-4274-bb9c-c00720dd2031.volumes.kubernetes.io~csi.pvc-e6cce698-52e7-4d62-8de9-0a8977fa308b.mount.20268

StorageClass, PersistentVolume, PersistentVolumeClaim

apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: efs-sc
  uid: c2003199-034f-4770-a664-ceaba4b1170e
  resourceVersion: '515478933'
  creationTimestamp: '2023-01-12T18:46:16Z'
  labels:
    app.kubernetes.io/managed-by: Helm
    k8slens-edit-resource-version: v1
  annotations:
    meta.helm.sh/release-name: aws-efs-csi
    meta.helm.sh/release-namespace: kube-system
  selfLink: /apis/storage.k8s.io/v1/storageclasses/efs-sc
spec: {}
provisioner: efs.csi.aws.com
parameters:
  basePath: /dynamic_provisioning
  directoryPerms: '700'
  fileSystemId: fs-00cb6e82cf3e8beb4
  gidRangeEnd: '2000'
  gidRangeStart: '1000'
  provisioningMode: efs-ap
reclaimPolicy: Delete
volumeBindingMode: Immediate



apiVersion: v1
kind: PersistentVolume
metadata:
  name: pvc-e6cce698-52e7-4d62-8de9-0a8977fa308b
  uid: 2545240a-fefe-4233-b8c0-f44e36562f79
  resourceVersion: '515414500'
  creationTimestamp: '2024-03-19T15:23:25Z'
  labels:
    k8slens-edit-resource-version: v1
  annotations:
    pv.kubernetes.io/provisioned-by: efs.csi.aws.com
  finalizers:
    - kubernetes.io/pv-protection
    - external-provisioner.volume.kubernetes.io/finalizer
  selfLink: /api/v1/persistentvolumes/pvc-e6cce698-52e7-4d62-8de9-0a8977fa308b
status:
  phase: Bound
spec:
  capacity:
    storage: 3Gi
  csi:
    driver: efs.csi.aws.com
    volumeHandle: fs-00cb6e82cf3e8beb4::fsap-0eccf76f606e2107d
    volumeAttributes:
      storage.kubernetes.io/csiProvisionerIdentity: 1694244631176-8081-efs.csi.aws.com
  accessModes:
    - ReadWriteMany
  claimRef:
    kind: PersistentVolumeClaim
    namespace: marketplace-portal-demo
    name: marketplace-portal-demo-media
    uid: e6cce698-52e7-4d62-8de9-0a8977fa308b
    apiVersion: v1
    resourceVersion: '218610270'
  persistentVolumeReclaimPolicy: Delete
  storageClassName: efs-sc
  volumeMode: Filesystem

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: marketplace-portal-demo-media
  namespace: marketplace-portal-demo
  uid: e6cce698-52e7-4d62-8de9-0a8977fa308b
  resourceVersion: '218610417'
  creationTimestamp: '2024-03-19T15:23:25Z'
  labels:
    app.kubernetes.io/managed-by: Helm
  annotations:
    meta.helm.sh/release-name: marketplace-portal-demo
    meta.helm.sh/release-namespace: marketplace-portal-demo
    pv.kubernetes.io/bind-completed: 'yes'
    pv.kubernetes.io/bound-by-controller: 'yes'
    volume.beta.kubernetes.io/storage-provisioner: efs.csi.aws.com
    volume.kubernetes.io/storage-provisioner: efs.csi.aws.com
  finalizers:
    - kubernetes.io/pvc-protection
  selfLink: >-
    /api/v1/namespaces/marketplace-portal-demo/persistentvolumeclaims/marketplace-portal-demo-media
status:
  phase: Bound
  accessModes:
    - ReadWriteMany
  capacity:
    storage: 3Gi
spec:
  accessModes:
    - ReadWriteMany
  resources:
    requests:
      storage: 3Gi
  volumeName: pvc-e6cce698-52e7-4d62-8de9-0a8977fa308b
  storageClassName: efs-sc
  volumeMode: Filesystem

[Kuzmenko-Pavel]

Parameter --tls is present but missing in StorageClass, PersistentVolume, PersistentVolumeClaim why?

I think I got it. Where there are options --tls, this is a connection to efs.eu-west-1.amazonaws.com:2049. And where there is no option, this is a connection to the pod

[Kuzmenko-Pavel]

@tdachille-dev
You wrote in this post #1523 (comment) that efs-proxy consumes memory.
But in the helm version aws-efs-csi-driver-3.1.9, it is not efs-proxy that consumes memory and leaks, but bin/aws-efs-csi-driver.
This is clearly visible on the screen. What in aws-efs-csi-driver makes it consume so much memory? And zombie processes seem to appear because of amazon-efs-mount-watchdog, which does not handle process closures quite correctly.
aws/efs-utils#175 (comment)

[Kuzmenko-Pavel]

@tdachille-dev You wrote in this post #1523 (comment) that efs-proxy consumes memory. But in the helm version aws-efs-csi-driver-3.1.9, it is not efs-proxy that consumes memory and leaks, but bin/aws-efs-csi-driver. This is clearly visible on the screen. What in aws-efs-csi-driver makes it consume so much memory? And zombie processes seem to appear because of amazon-efs-mount-watchdog, which does not handle process closures quite correctly. aws/efs-utils#175 (comment)

After 5 hours, the same pod. You can see the same pid for the efs proxy and that bin/aws-efs-csi-driver is leaking.

Do you need some additional information to investigate the problem?

[Kuzmenko-Pavel]

python3 log_collector.py --driver-pod-name efs-csi-node-jbjrv
results_log_efs-csi-node-jbjrv.zip

[Kuzmenko-Pavel]

I may have localized the problem and found a solution.
The driver was not installed from scratch, it was an update to 3.1.9
After analyzing the logs and code of aws-efs-csi-driver, the configuration files located in /var/run/efs/ seemed quite strange to me.
Since /var/run/efs/ is a temporary directory that disappears when the node reboots, I decided to try rebooting the node.

I executed the following sequence of commands

• kubectl cordon ip-10-100-16-14.eu-west-1.compute.internal
• kubectl drain ip-10-100-16-14.eu-west-1.compute.internal --ignore-daemonsets --delete-emptydir-data
• kubectl debug node/ip-10-100-16-14.eu-west-1.compute.internal -it --image=ubuntu --profile=sysadmin # For Bottlerocket
• chroot /host apiclient reboot # Bottlerocket node reboot
• python3 log_collector.py --driver-pod-name efs-csi-node-jbjrv # Collect the log after start without pods
• kubectl uncordon ip-10-100-16-14.eu-west-1.compute.internal
• python3 log_collector.py --driver-pod-name efs-csi-node-jbjrv # Collect the log with pods

And it helped. The directory /var/run/efs/ was empty after startup and the memory was fine.
Zombie processes disappeared and the structure of running processes itself changed.
Probably there is some drift of configuration files, which leads to problems after the update. And simply remounting pvc by rebooting all pods does not help. Only rebooting the node helps.

All logs and work described in the ticket were performed on one node. I am attaching all the logs, perhaps this will help the project team to understand the problem in more detail.

My solution is a complete reboot of the nodes after updating the driver.

results_after_start.zip

results_after_uncordon.zip

[zmaguire]

Hi, thanks for raising the issue, we are actively looking into it. Despite our attempts, we have been unable to reproduce the issue so far.

Do you have any more information you could share? How many pods? What kind of workload? Also, I see you mentioned that this issue was seen after an upgrade. Which specific version did you upgrade from?

[vchan2002]

So, we have reproduced this ourselves when we did upgrade from driver 1.6.0 to 2.1.3. Here's what we noticed.

a. The EFS CSI driver pod complained about not able to initialize stunnel for a specific EFS mount. Pods that tries to talk to the EFS mount does not work. Just restarting the daemonset and/or restarting the pods with the EFS mount does not fix it.

b. Looking into the /var/efs/run of that node, the EFS mount's timestamp that the watchdog complained about is BEFORE the EFS upgrade

c. After bouncing the node (just bouncing is on the AWS console), the /var/run/efs directory is indeed cleaned up, and no more complaints.

So, This condition seems to only occur when on nodes with pods that has existing EFS mounts that pre-dated the upgrade from v1 to v2. Hopefully that makes sense.