Is your feature request related to a problem? Please describe (👍 if you like this request)

The longhorn instance manager component is a mission-critical component responsible for the volume lifecycle including volume frontend, engine, and replica, so it should continue working with the right resource allocation w/o any downtime to prevent any potential resource starvation to impact its running state. (here, we are not handling involuntary issues to cause node down and then impact instance manager pods to stop working accidentally)

Currently, a user is able to set a CPU request to it as per https://longhorn.io/docs/1.5.0/references/settings/#guaranteed-instance-manager-cpu, but it's not quite clear for users to understand why they need to have pre-allocated resources to it, even though there is no much volume usage on that node, and it turns out to a resource usage concern)

It's better to consider different strategies for users and clearly tell users how to do and what options able to choose like no resource request (best effort), and pre-requested resource (strict mode) by considering if the node role like the node is a pure storage node w/o user workloads or only frontend w/o replicas, etc.

Should leverage priority class for this component to ensure it's liveness, https://kubernetes.io/docs/concepts/scheduling-eviction/pod-priority-preemption/#priorityclass by default.

All the above should be considered with other unclarified factors, and make a strategy global setting for it. Also, need to review the resource CPU calculation for the engine & replica when it's used by a data-intensive application, so we can use the extreme case for the estimation formula instead of a unclear theory assumption)

Describe the solution you'd like

A clear and concise description of what you want to happen

Describe alternatives you've considered

A clear and concise description of any alternative solutions or features you've considered.

Additional context

cc @longhorn/dev