Add `preserveUnknownFields: false` to crd specs #4237

tristanlins · 2022-07-15T04:35:01Z

I recently tried to upgrade from 1.2 to 1.3 and run into this error:

$ helm upgrade longhorn longhorn/longhorn --namespace longhorn-system --values longhorn.yaml --debug
upgrade.go:139: [debug] preparing upgrade for longhorn
upgrade.go:147: [debug] performing update for longhorn
upgrade.go:319: [debug] creating upgraded release for longhorn
client.go:218: [debug] checking 37 resources for changes
client.go:501: [debug] Looks like there are no changes for ServiceAccount "longhorn-service-account"
client.go:501: [debug] Looks like there are no changes for ConfigMap "longhorn-default-setting"
client.go:501: [debug] Looks like there are no changes for ConfigMap "longhorn-storageclass"
client.go:250: [debug] error updating the resource "engineimages.longhorn.io":
         cannot patch "engineimages.longhorn.io" with kind CustomResourceDefinition: CustomResourceDefinition.apiextensions.k8s.io "engineimages.longhorn.io" is invalid: spec.conversion.strategy: Invalid value: "Webhook": must be None if spec.preserveUnknownFields is true
client.go:250: [debug] error updating the resource "nodes.longhorn.io":
         cannot patch "nodes.longhorn.io" with kind CustomResourceDefinition: CustomResourceDefinition.apiextensions.k8s.io "nodes.longhorn.io" is invalid: spec.conversion.strategy: Invalid value: "Webhook": must be None if spec.preserveUnknownFields is true
client.go:250: [debug] error updating the resource "volumes.longhorn.io":
         cannot patch "volumes.longhorn.io" with kind CustomResourceDefinition: CustomResourceDefinition.apiextensions.k8s.io "volumes.longhorn.io" is invalid: spec.conversion.strategy: Invalid value: "Webhook": must be None if spec.preserveUnknownFields is true
client.go:501: [debug] Looks like there are no changes for ClusterRole "longhorn-role"
client.go:501: [debug] Looks like there are no changes for ClusterRoleBinding "longhorn-bind"
client.go:501: [debug] Looks like there are no changes for Role "longhorn-psp-role"
client.go:501: [debug] Looks like there are no changes for RoleBinding "longhorn-psp-binding"
client.go:501: [debug] Looks like there are no changes for Service "longhorn-backend"
client.go:501: [debug] Looks like there are no changes for Service "longhorn-conversion-webhook"
client.go:501: [debug] Looks like there are no changes for Service "longhorn-admission-webhook"
client.go:501: [debug] Looks like there are no changes for Service "longhorn-engine-manager"
client.go:501: [debug] Looks like there are no changes for Service "longhorn-replica-manager"
client.go:501: [debug] Looks like there are no changes for Deployment "longhorn-driver-deployer"
client.go:501: [debug] Looks like there are no changes for Deployment "longhorn-conversion-webhook"
client.go:501: [debug] Looks like there are no changes for Deployment "longhorn-admission-webhook"
client.go:501: [debug] Looks like there are no changes for Ingress "longhorn-ingress"
upgrade.go:420: [debug] warning: Upgrade "longhorn" failed: cannot patch "engineimages.longhorn.io" with kind CustomResourceDefinition: CustomResourceDefinition.apiextensions.k8s.io "engineimages.longhorn.io" is invalid: spec.conversion.strategy: Invalid value: "Webhook": must be None if spec.preserveUnknownFields is true && cannot patch "nodes.longhorn.io" with kind CustomResourceDefinition: CustomResourceDefinition.apiextensions.k8s.io "nodes.longhorn.io" is invalid: spec.conversion.strategy: Invalid value: "Webhook": must be None if spec.preserveUnknownFields is true && cannot patch "volumes.longhorn.io" with kind CustomResourceDefinition: CustomResourceDefinition.apiextensions.k8s.io "volumes.longhorn.io" is invalid: spec.conversion.strategy: Invalid value: "Webhook": must be None if spec.preserveUnknownFields is true
Error: UPGRADE FAILED: cannot patch "engineimages.longhorn.io" with kind CustomResourceDefinition: CustomResourceDefinition.apiextensions.k8s.io "engineimages.longhorn.io" is invalid: spec.conversion.strategy: Invalid value: "Webhook": must be None if spec.preserveUnknownFields is true && cannot patch "nodes.longhorn.io" with kind CustomResourceDefinition: CustomResourceDefinition.apiextensions.k8s.io "nodes.longhorn.io" is invalid: spec.conversion.strategy: Invalid value: "Webhook": must be None if spec.preserveUnknownFields is true && cannot patch "volumes.longhorn.io" with kind CustomResourceDefinition: CustomResourceDefinition.apiextensions.k8s.io "volumes.longhorn.io" is invalid: spec.conversion.strategy: Invalid value: "Webhook": must be None if spec.preserveUnknownFields is true
helm.go:88: [debug] cannot patch "engineimages.longhorn.io" with kind CustomResourceDefinition: CustomResourceDefinition.apiextensions.k8s.io "engineimages.longhorn.io" is invalid: spec.conversion.strategy: Invalid value: "Webhook": must be None if spec.preserveUnknownFields is true && cannot patch "nodes.longhorn.io" with kind CustomResourceDefinition: CustomResourceDefinition.apiextensions.k8s.io "nodes.longhorn.io" is invalid: spec.conversion.strategy: Invalid value: "Webhook": must be None if spec.preserveUnknownFields is true && cannot patch "volumes.longhorn.io" with kind CustomResourceDefinition: CustomResourceDefinition.apiextensions.k8s.io "volumes.longhorn.io" is invalid: spec.conversion.strategy: Invalid value: "Webhook": must be None if spec.preserveUnknownFields is true
helm.sh/helm/v3/pkg/kube.(*Client).Update
        helm.sh/helm/v3/pkg/kube/client.go:263
helm.sh/helm/v3/pkg/action.(*Upgrade).releasingUpgrade
        helm.sh/helm/v3/pkg/action/upgrade.go:365
runtime.goexit
        runtime/asm_amd64.s:1371
UPGRADE FAILED
main.newUpgradeCmd.func2
        helm.sh/helm/v3/cmd/helm/upgrade.go:200
github.com/spf13/cobra.(*Command).execute
        github.com/spf13/cobra@v1.2.1/command.go:856
github.com/spf13/cobra.(*Command).ExecuteC
        github.com/spf13/cobra@v1.2.1/command.go:974
github.com/spf13/cobra.(*Command).Execute
        github.com/spf13/cobra@v1.2.1/command.go:902
main.main
        helm.sh/helm/v3/cmd/helm/helm.go:87
runtime.main
        runtime/proc.go:225
runtime.goexit
        runtime/asm_amd64.s:1371

I found a solution here: projectcalico/calico#4237 (comment)
Adding preserveUnknownFields: false to the CRD spec fixes the problem.

Environment

Longhorn version: 1.2.3
Installation method (e.g. Rancher Catalog App/Helm/Kubectl): Helm
Kubernetes distro (e.g. RKE/K3s/EKS/OpenShift) and version: k3s v1.19.3+k3s3
- Number of management node in the cluster: 1
- Number of worker node in the cluster: 1
Node config
- OS type and version: Ubuntu 18.04.6 LTS
- CPU per node: 8
- Memory per node: 40 GB
- Disk type(e.g. SSD/NVMe): Virtual
- Network bandwidth between the nodes:
Underlying Infrastructure (e.g. on AWS/GCE, EKS/GKE, VMWare/KVM, Baremetal): Proxmox/KVM
Number of Longhorn volumes in the cluster: ~40

Workaround

#4198 (comment)

Signed-off-by: Tristan Lins <tristan.lins@chamaeleon.de>

derekbit

LGTM.

It happens if the upgrade is from the LH version (v1.0.x) using apiextensions.k8s.io/v1beta1.

Related issue: #4198

shuo-wu

Should we set this field for other CRD specs?

derekbit · 2022-07-25T14:16:43Z

Should we set this field for other CRD specs?

No. Only the three CRDs are upgraded from apiextensions.k8s.io/v1beta1 (Longhorn v1.0.x) and have the conversion webhooks in v1.3.0 as well.

innobead · 2022-07-26T09:43:42Z

@tristanlins Did you encounter this issue when upgrading from 1.2 or 1.0 to 1.3? because we would assume users only hit this when upgrading from 1.0.

The official upgrade path should be at most one minor version less than the target version you want to upgrade to. For 1.3.0, you should upgrade from 1.2.x.

innobead · 2022-07-26T11:06:06Z

@tristanlins Did you encounter this issue when upgrading from1.2 or 1.0 to 1.3? because we would assume users only hit this when upgrading from 1.0.

The official upgrade path should be at most one minor version less than the target version you want to upgrade to. For 1.3.0, you should upgrade from 1.2.x.

Ok, it's clarified now. @tristanlins actually upgraded from 1.2 to 1.3, but started using Longhorn from 1.0 version which is based on v1beta CustomResourceDefinition.apiextensions.k8s.io, so this will keep spec.preserveUnknownFields true (this value is true by default in v1beta1, but false for v1) along the time until upgrading to 1.3, the validation error was encountered when the helm was trying to patch the update from the new version of CustomResourceDefinition.apiextensions.k8s.io CRD w/ webhook strategy introduction.

validation error: https://github.com/kubernetes/kubernetes/blob/f0406c8d2d18a00622dd7bdc5a51f0dbf593c20b/staging/src/k8s.io/apiextensions-apiserver/pkg/apis/apiextensions/validation/validation.go#L340-L342

The solution is just to update spec.preserveUnknownFields to false` to bypass the validation error.
https://kubernetes.io/docs/tasks/extend-kubernetes/custom-resources/custom-resource-definitions/#field-pruning

note: backingimages & backuptargets have no this issues, because they are introduced from 1.2 which uses v1 CustomResourceDefinition.apiextensions.k8s.io and the default value of spec.preserveUnknownFields is false.

innobead · 2022-07-26T11:15:18Z

@derekbit Could you help backport this to 1.3.1? Also, add this to https://github.com/longhorn/longhorn/wiki/Known-Issues-of-Feature-Releases. Thanks

@tristanlins Thanks for the contribution!

Add preserveUnknownFields: false to crd specs

3e20b86

Signed-off-by: Tristan Lins <tristan.lins@chamaeleon.de>

tristanlins force-pushed the hotfix-invalid-value-webhook branch from 503f4c8 to 3e20b86 Compare July 15, 2022 04:36

Merge branch 'master' into hotfix-invalid-value-webhook

2beca35

innobead requested review from c3y1huang and derekbit July 15, 2022 06:13

derekbit approved these changes Jul 15, 2022

View reviewed changes

innobead requested a review from shuo-wu July 15, 2022 10:08

shuo-wu reviewed Jul 20, 2022

View reviewed changes

Merge branch 'master' into hotfix-invalid-value-webhook

0f66efa

innobead approved these changes Jul 26, 2022

View reviewed changes

innobead merged commit d3e4d6e into longhorn:master Jul 26, 2022

derekbit mentioned this pull request Aug 1, 2022

[BUG] Update to 1.3.0 failed after chart migration #4340

Closed

ejweber mentioned this pull request Feb 12, 2024

[BUG] Volumes stuck upgrading after 1.5.3 -> 1.6.0 upgrade. #7887

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Add `preserveUnknownFields: false` to crd specs #4237

Add `preserveUnknownFields: false` to crd specs #4237

Uh oh!

tristanlins commented Jul 15, 2022 •

edited by derekbit

Loading

Uh oh!

derekbit left a comment •

edited

Loading

Uh oh!

shuo-wu left a comment

Uh oh!

derekbit commented Jul 25, 2022

Uh oh!

innobead commented Jul 26, 2022 •

edited

Loading

Uh oh!

innobead commented Jul 26, 2022 •

edited

Loading

Uh oh!

innobead commented Jul 26, 2022 •

edited

Loading

Uh oh!

Uh oh!

Add preserveUnknownFields: false to crd specs #4237

Add preserveUnknownFields: false to crd specs #4237

Uh oh!

Conversation

tristanlins commented Jul 15, 2022 • edited by derekbit Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Environment

Workaround

Uh oh!

derekbit left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

shuo-wu left a comment

Choose a reason for hiding this comment

Uh oh!

derekbit commented Jul 25, 2022

Uh oh!

innobead commented Jul 26, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

innobead commented Jul 26, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

innobead commented Jul 26, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

Add `preserveUnknownFields: false` to crd specs #4237

Add `preserveUnknownFields: false` to crd specs #4237

tristanlins commented Jul 15, 2022 •

edited by derekbit

Loading

derekbit left a comment •

edited

Loading

innobead commented Jul 26, 2022 •

edited

Loading

innobead commented Jul 26, 2022 •

edited

Loading

innobead commented Jul 26, 2022 •

edited

Loading