Describe the bug

can't upgrade engine if a volume was created in Longhorn v1.0 and its volume.spec.dataLocality ""

v1.0 does not have the dataLocality parameter, and the value is not handled in the upgrade path (v1.0->v1.1 and v1.1->v1.2) and always keeps an empty sting, so users might hit this issue while upgrading to v1.3.x if the volume was created in v1.0.

To Reproduce

Steps to reproduce the behavior:

1. Deploy Longhorn v1.0
2. Create a volume
3. Upgrade Longhorn to v1.2.x and then upgrade the volume's engine
4. Upgrade Longhorn to v1.3.x. Then, you might hit

Error: UPGRADE FAILED: cannot patch "engineimages.longhorn.io" with kind CustomResourceDefinition: CustomResourceDefinition.apiextensions.k8s.io "engineimages.longhorn.io" is invalid: spec.conversion.strategy: Invalid value: "Webhook": must be None if spec.preserveUnknownFields is true && cannot patch "nodes.longhorn.io" with kind CustomResourceDefinition: CustomResourceDefinition.apiextensions.k8s.io "nodes.longhorn.io" is invalid: spec.conversion.strategy: Invalid value: "Webhook": must be None if spec.preserveUnknownFields is true && cannot patch "volumes.longhorn.io" with kind CustomResourceDefinition: CustomResourceDefinition.apiextensions.k8s.io "volumes.longhorn.io" is invalid: spec.conversion.strategy: Invalid value: "Webhook": must be None if spec.preserveUnknownFields is true

Workaround: #4340 (comment)

5. After upgrade to v1.3.x, upgrade the volume's engine. The issue will be hit while upgrading the engine.

Expected behavior

Can upgrade engine successfully

Log or Support bundle

If applicable, add the Longhorn managers' log or support bundle when the issue happens.
You can generate a Support Bundle using the link at the footer of the Longhorn UI.

Environment

• Longhorn version:
• Installation method (e.g. Rancher Catalog App/Helm/Kubectl):
• Kubernetes distro (e.g. RKE/K3s/EKS/OpenShift) and version:
  • Number of management node in the cluster:
  • Number of worker node in the cluster:
• Node config
  • OS type and version:
  • CPU per node:
  • Memory per node:
  • Disk type(e.g. SSD/NVMe):
  • Network bandwidth between the nodes:
• Underlying Infrastructure (e.g. on AWS/GCE, EKS/GKE, VMWare/KVM, Baremetal):
• Number of Longhorn volumes in the cluster:

Additional context

#4387

Workaround

Set volume.spec.dataLocality from "" to disabled manually