Is your feature request related to a problem? Please describe (👍 if you like this request)

Currently, when creating a longhorn snapshot (in-cluster) of a volume will not respect any provision factors except the fixed upper limit of the number of volume snapshots, so it would cause unexpected disk space exhaustion which means it's unable to prevent this beforehand by having a better space plan.

The goal is to introduce a mechanism to manage snapshot space usage when creating snapshots.

Describe the solution you'd like

When creating a snapshot, should respect the below factors. All factors should be considered together.

• A customized max number of snapshots allowed globally or volume-specific setting.
• A quota for snapshot usage global, node, or disk-specific. This will be another factor to impact replica scheduling.
• Introduce snapshot scheduling like replica scheduling to determine if the disk space is enough for creating a snapshot next to the replica. (cross disks allowed). This should be similar to creating a volume by respecting disk usage first.

Describe alternatives you've considered

Additional context

cc @longhorn/dev