Describe the bug (🐛 if you encounter this issue)

The memory usage of the CSI plugins has been growing steadily for the past few weeks after upgrading to version 1.4.0

To Reproduce

No specific steps. It has been like that since upgrading to version Longhorn 1.4.0.

Expected behavior

The CSI plugin memory usage should stay relatively small without growing steadily.

Log or Support bundle

supportbundle_413f9f8b-328c-47d2-a25d-4f7c903bf06f_2023-01-16T08-27-12Z.zip

Environment

Longhorn version: 1.4.0
Installation method (e.g. Rancher Catalog App/Helm/Kubectl): Helm
Kubernetes distro (e.g. RKE/K3s/EKS/OpenShift) and version: v1.25.4+k3s1
Number of management node in the cluster: 1
Number of worker node in the cluster: 4
Node config
OS type and version: Raspbian (5.15.76-v8) and Ubuntu server (22.04.1)
CPU per node: 2-4
Memory per node: 4-8G
Disk type(e.g. SSD/NVMe): SSD
Network bandwidth between the nodes: 1G
Underlying Infrastructure (e.g. on AWS/GCE, EKS/GKE, VMWare/KVM, Baremetal): Baremetal
Number of Longhorn volumes in the cluster: 15

Workaround

Force umount the stuck umount process by umoun -f.

Additional context

Initially discussed as part of another issue #5195 (comment)