Trident snapshotter pod log spam after upgrading from older pre-CSI snapshot version
Applies to
- Trident 20.07 and higher
- Kubernetes 1.20 and higher
Issue
When upgrading from a release of Trident or Kubernetes before the CSI Snapshot support, the snapshotter will attempt to access two separate v1beta1 classes
(VolumeSnapshotClass and VolumeSnapshotContent), and fail access:I[MMdd] [TIME] 1 round_trippers.go:420] GET https://[TRIDENT_CONTROLLER_IP]:443/apis/snapshot.storage.k8s.io/v1beta1/volumesnapshotclasses?limit=500&resourceVersion=0
I[MMdd] [TIME] 1 round_trippers.go:427] Request Headers:
I[MMdd] [TIME] 1 round_trippers.go:431] User-Agent: csi-snapshotter/v0.0.0 (linux/amd64) kubernetes/$Format
I[MMdd] [TIME] 1 round_trippers.go:431] Accept: application/json, */*
I[MMdd] [TIME] 1 round_trippers.go:431] Authorization: Bearer <masked>
I[MMdd] [TIME] 1 round_trippers.go:420] GET https://[TRIDENT_CONTROLLER_IP]/apis/snapshot.storage.k8s.io/v1beta1/volumesnapshotcontents?limit=500&resourceVersion=0
I[MMdd] [TIME] 1 round_trippers.go:427] Request Headers:
I[MMdd] [TIME] 1 round_trippers.go:431] Accept: application/json, */*
I[MMdd] [TIME] 1 round_trippers.go:431] User-Agent: csi-snapshotter/v0.0.0 (linux/amd64) kubernetes/$Format
I[MMdd] [TIME] 1 round_trippers.go:431] Authorization: Bearer <masked>
I[MMdd] [TIME] 1 round_trippers.go:446] Response Status: 404 Not Found in 6 milliseconds
I[MMdd] [TIME] 1 round_trippers.go:449] Response Headers:
I[MMdd] [TIME] 1 round_trippers.go:446] Response Status: 404 Not Found in 6 milliseconds
I[MMdd] [TIME] 1 round_trippers.go:452] Cache-Control: no-cache, private
I[MMdd] [TIME] 1 round_trippers.go:449] Response Headers:
I[MMdd] [TIME] 1 round_trippers.go:452] Content-Type: text/plain; charset=utf-8
I[MMdd] [TIME] 1 round_trippers.go:452] X-Kubernetes-Pf-Flowschema-Uid: f978b7f4-c3f3-4736-aedc-36f873e2e698
I[MMdd] [TIME] 1 round_trippers.go:452] X-Content-Type-Options: nosniff
I[MMdd] [TIME] 1 round_trippers.go:452] X-Kubernetes-Pf-Prioritylevel-Uid: a26a1e9c-85a4-4742-8844-c44bd8d00267
I[MMdd] [TIME] 1 round_trippers.go:452] Content-Length: 19
I[MMdd] [TIME] 1 round_trippers.go:452] X-Kubernetes-Pf-Flowschema-Uid: f978b7f4-c3f3-4736-aedc-36f873e2e698
I[MMdd] [TIME] 1 round_trippers.go:452] X-Kubernetes-Pf-Prioritylevel-Uid: a26a1e9c-85a4-4742-8844-c44bd8d00267
I[MMdd] [TIME] 1 round_trippers.go:452] Date: Thu, 02 Jun 2022 07:28:17 GMT
I[MMdd] [TIME] 1 round_trippers.go:452] Cache-Control: no-cache, private
I[MMdd] [TIME] 1 round_trippers.go:452] Content-Type: text/plain; charset=utf-8
I[MMdd] [TIME] 1 round_trippers.go:452] X-Content-Type-Options: nosniff
I[MMdd] [TIME] 1 request.go:1097] Response Body: 404 page not found
I[MMdd] [TIME] 1 round_trippers.go:452] Content-Length: 19
I[MMdd] [TIME] 1 round_trippers.go:452] Date: Thu, 02 Jun 2022 07:28:17 GMT
I[MMdd] [TIME] 1 request.go:1097] Response Body: 404 page not found
E[MMdd] [TIME] 1 reflector.go:127] github.com/kubernetes-csi/external-snapshotter/client/v3/informers/externalversions/factory.go:117: Failed to watch *v1beta1.VolumeSnapshotContent: failed to list *v1beta1.VolumeSnapshotContent: the server could not find the requested resource (get volumesnapshotcontents.snapshot.storage.k8s.io)
E[MMdd] [TIME] 1 reflector.go:127] github.com/kubernetes-csi/external-snapshotter/client/v3/informers/externalversions/factory.go:117: Failed to watch *v1beta1.VolumeSnapshotClass: failed to list *v1beta1.VolumeSnapshotClass: the server could not find the requested resource (get volumesnapshotclasses.snapshot.storage.k8s.io)This repeats for each of these checks, on average 30 top 40 seconds, bloating the log for trident-controller-sidecar-csi-snapshotter to tens of Megabytes.
Note: the actual line numbers with
round_trippers.go depend on the version of Trident, and the version of the classes on whether v1beta1 or v1 is used (v1 snapshots use v5 informer classes).