Add VolumeDevices to PodSpec (#2572) #2573

jirislav · 2024-08-16T00:39:11Z

Description

Resolves #2572

Acceptance tests

Have you added an acceptance test for the functionality being added?
Have you run the acceptance tests on this branch?

Output from acceptance testing follows, filtered only to the new EKS tests, since it's hard to test locally:

/usr/lib/go-1.21/bin/go tool test2json -t ~/.cache/JetBrains/IntelliJIdea2024.2/tmp/GoLand/___2resource_kubernetes_stateful_set_v1_eks_test_go.test -test.v=test2json -test.paniconexit0 -test.run ^\QTestAccKubernetesStatefulSetV1_EKS_basic\E|\QTestAccKubernetesStatefulSetV1_EKS_basic_idempotency\E|\QTestAccKubernetesStatefulSetV1_EKS_Update\E|\QTestAccKubernetesStatefulSetV1_EKS_waitForRollout\E$
=== RUN   TestAccKubernetesStatefulSetV1_EKS_basic
=== PAUSE TestAccKubernetesStatefulSetV1_EKS_basic
=== CONT  TestAccKubernetesStatefulSetV1_EKS_basic
--- PASS: TestAccKubernetesStatefulSetV1_EKS_basic (25.41s)
=== RUN   TestAccKubernetesStatefulSetV1_EKS_basic_idempotency
=== PAUSE TestAccKubernetesStatefulSetV1_EKS_basic_idempotency
=== CONT  TestAccKubernetesStatefulSetV1_EKS_basic_idempotency
--- PASS: TestAccKubernetesStatefulSetV1_EKS_basic_idempotency (36.40s)
=== RUN   TestAccKubernetesStatefulSetV1_EKS_Update
=== PAUSE TestAccKubernetesStatefulSetV1_EKS_Update
=== CONT  TestAccKubernetesStatefulSetV1_EKS_Update
=== RUN   TestAccKubernetesStatefulSetV1_EKS_waitForRollout
=== PAUSE TestAccKubernetesStatefulSetV1_EKS_waitForRollout
=== CONT  TestAccKubernetesStatefulSetV1_EKS_waitForRollout
--- PASS: TestAccKubernetesStatefulSetV1_EKS_waitForRollout (19.02s)
--- PASS: TestAccKubernetesStatefulSetV1_EKS_Update (145.19s)
PASS

Release Note

Release note for CHANGELOG:

Add VolumeDevices to PodSpec (#2572)

References

Community Note

Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
If you are interested in working on this issue or have submitted a pull request, please leave a comment

hashicorp-cla-app · 2024-08-16T00:39:25Z

All committers have signed the CLA.

jirislav · 2024-08-16T01:39:12Z

I have published this version to the registry for early access and easier docs review.

To get the early access, use:

terraform {
  required_providers {
    kubernetes = {
      source = "jirislav/kubernetes"
      version = "2.32.0-issue2572"
    }
  }
}

provider "kubernetes" {
  # Configuration options
}

BBBmau · 2024-08-16T18:01:37Z

kubernetes/resource_kubernetes_stateful_set_v1_eks_test.go

+	appsv1 "k8s.io/api/apps/v1"
+)
+
+func TestAccKubernetesStatefulSetV1_EKS_basic(t *testing.T) {


From your description, you added these tests because it's difficult to test locally. Is their no simple way to include volume_device into the other already existing stateful tests? Would like to know what makes the local testing difficult.

By default, Kind provides StorageClass with rancher.io/local-path provisioner using the docker.io/kindest/local-path-provisioner image.

Now, if you create PersistentVolumeClaim with VolumeMode: Block, the default StorageClass kicks in, instructing rancher.io/local-path provisioner to obtain the block storage volume.

Then you will get the rancher.io/local-path does not support block volume provisioning message saying you need to install whole new provisioner and StorageClass, none of which is provided by Kind out of the box.

Describe StatefulSet (ss-test)

$ k describe sts tf-acc-test-103te79aze Name: tf-acc-test-103te79aze Namespace: default CreationTimestamp: Sat, 17 Aug 2024 00:25:20 +0200 Selector: app=ss-test Labels: TestLabelOne=one TestLabelThree=three TestLabelTwo=two Annotations: TestAnnotationOne: one TestAnnotationTwo: two Replicas: 1 desired | 1 total Update Strategy: RollingUpdate Partition: 1 Pods Status: 0 Running / 1 Waiting / 0 Succeeded / 0 Failed Pod Template: Labels: app=ss-test Containers: ss-test: Image: registry.k8s.io/e2e-test-images/agnhost:2.43 Port: 80/TCP Host Port: 0/TCP Args: pause Environment: <none> Mounts: /work-dir from ss-test (rw) Devices: /dev/xvda from ss-device-test Volumes: <none> Node-Selectors: <none> Tolerations: <none> Volume Claims: Name: ss-test StorageClass: Labels: <none> Annotations: <none> Capacity: 1Gi Access Modes: [ReadWriteOnce] Name: ss-device-test StorageClass: Labels: <none> Annotations: <none> Capacity: 1Gi Access Modes: [ReadWriteOnce] Events: Type Reason Age From Message ---- ------ ---- ---- ------- Normal SuccessfulCreate 80s statefulset-controller create Claim ss-device-test-tf-acc-test-103te79aze-0 Pod tf-acc-test-103te79aze-0 in StatefulSet tf-acc-test-103te79aze success Normal SuccessfulCreate 80s statefulset-controller create Claim ss-test-tf-acc-test-103te79aze-0 Pod tf-acc-test-103te79aze-0 in StatefulSet tf-acc-test-103te79aze success Normal SuccessfulCreate 75s statefulset-controller create Pod tf-acc-test-103te79aze-0 in StatefulSet tf-acc-test-103te79aze successful

Describe Pod (ss-test)

$ k describe pod tf-acc-test-103te79aze-0 Name: tf-acc-test-103te79aze-0 Namespace: default Priority: 0 Service Account: default Node: <none> Labels: app=ss-test apps.kubernetes.io/pod-index=0 controller-revision-hash=tf-acc-test-103te79aze-5ddf5fb74d statefulset.kubernetes.io/pod-name=tf-acc-test-103te79aze-0 Annotations: <none> Status: Pending IP: IPs: <none> Controlled By: StatefulSet/tf-acc-test-103te79aze Containers: ss-test: Image: registry.k8s.io/e2e-test-images/agnhost:2.43 Port: 80/TCP Host Port: 0/TCP Args: pause Environment: <none> Mounts: /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-zptvp (ro) /work-dir from ss-test (rw) Devices: /dev/xvda from ss-device-test Volumes: ss-test: Type: PersistentVolumeClaim (a reference to a PersistentVolumeClaim in the same namespace) ClaimName: ss-test-tf-acc-test-103te79aze-0 ReadOnly: false ss-device-test: Type: PersistentVolumeClaim (a reference to a PersistentVolumeClaim in the same namespace) ClaimName: ss-device-test-tf-acc-test-103te79aze-0 ReadOnly: false kube-api-access-zptvp: Type: Projected (a volume that contains injected data from multiple sources) TokenExpirationSeconds: 3607 ConfigMapName: kube-root-ca.crt ConfigMapOptional: <nil> DownwardAPI: true QoS Class: BestEffort Node-Selectors: <none> Tolerations: node.kubernetes.io/not-ready:NoExecute op=Exists for 300s node.kubernetes.io/unreachable:NoExecute op=Exists for 300s Events: <none>

Describe PersistentVolumeClaim (ss-device-test)

$ k describe pvc ss-device-test-tf-acc-test-103te79aze-0 Name: ss-device-test-tf-acc-test-103te79aze-0 Namespace: default StorageClass: standard Status: Pending Volume: Labels: app=ss-test Annotations: volume.beta.kubernetes.io/storage-provisioner: rancher.io/local-path volume.kubernetes.io/selected-node: kind-control-plane volume.kubernetes.io/storage-provisioner: rancher.io/local-path Finalizers: [kubernetes.io/pvc-protection] Capacity: Access Modes: VolumeMode: Block Used By: tf-acc-test-103te79aze-0 Events: Type Reason Age From Message ---- ------ ---- ---- ------- Normal WaitForFirstConsumer 106s persistentvolume-controller waiting for first consumer to be created before binding Warning ProvisioningFailed 106s (x2 over 106s) rancher.io/local-path_local-path-provisioner-57c5987fd4-5fprk_7ec3ce96-a1c6-4a3b-94c8-078cce4831cb rancher.io/local-path does not support block volume provisioning Normal ExternalProvisioning 12s (x9 over 106s) persistentvolume-controller Waiting for a volume to be created either by the external provisioner 'rancher.io/local-path' or manually by the system administrator. If volume creation is delayed, please verify that the provisioner is running and correctly registered.

Describe local-path-provisioner Pod

$ k describe pod local-path-provisioner-57c5987fd4-5fprk -n local-path-storage Name: local-path-provisioner-57c5987fd4-5fprk Namespace: local-path-storage Priority: 0 Service Account: local-path-provisioner-service-account Node: kind-control-plane/10.89.0.2 Start Time: Sat, 17 Aug 2024 00:14:00 +0200 Labels: app=local-path-provisioner pod-template-hash=57c5987fd4 Annotations: <none> Status: Running IP: 10.244.0.4 IPs: IP: 10.244.0.4 Controlled By: ReplicaSet/local-path-provisioner-57c5987fd4 Containers: local-path-provisioner: Container ID: containerd://e8df5db803bf7d6464c95986bc9574d44650b88ea746924255b2e4a634c4325d Image: docker.io/kindest/local-path-provisioner:v20240813-c6f155d6 Image ID: sha256:3a195b56ff1540463285c7f6287ef0d4ec18754013f68a3666b363b7893a62ad Port: <none> Host Port: <none> Command: local-path-provisioner --debug start --helper-image docker.io/kindest/local-path-helper:v20230510-486859a6 --config /etc/config/config.json State: Running Started: Sat, 17 Aug 2024 00:14:01 +0200 Ready: True Restart Count: 0 Environment: POD_NAMESPACE: local-path-storage (v1:metadata.namespace) Mounts: /etc/config/ from config-volume (rw) /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-6fpn9 (ro) Conditions: Type Status PodReadyToStartContainers True Initialized True Ready True ContainersReady True PodScheduled True Volumes: config-volume: Type: ConfigMap (a volume populated by a ConfigMap) Name: local-path-config Optional: false kube-api-access-6fpn9: Type: Projected (a volume that contains injected data from multiple sources) TokenExpirationSeconds: 3607 ConfigMapName: kube-root-ca.crt ConfigMapOptional: <nil> DownwardAPI: true QoS Class: BestEffort Node-Selectors: kubernetes.io/os=linux Tolerations: node-role.kubernetes.io/control-plane:NoSchedule node-role.kubernetes.io/master:NoSchedule node.kubernetes.io/not-ready:NoExecute op=Exists for 300s node.kubernetes.io/unreachable:NoExecute op=Exists for 300s Events: Type Reason Age From Message ---- ------ ---- ---- ------- Warning FailedScheduling 14m default-scheduler 0/1 nodes are available: 1 node(s) had untolerated taint {node.kubernetes.io/not-ready: }. preemption: 0/1 nodes are available: 1 Preemption is not helpful for scheduling. Normal Scheduled 14m default-scheduler Successfully assigned local-path-storage/local-path-provisioner-57c5987fd4-5fprk to kind-control-plane Normal Pulled 14m kubelet Container image "docker.io/kindest/local-path-provisioner:v20240813-c6f155d6" already present on machine Normal Created 14m kubelet Created container local-path-provisioner Normal Started 14m kubelet Started container local-path-provisioner

Filtered logs:

$ k logs local-path-provisioner-57c5987fd4-5fprk -n local-path-storage | grep 'ss-device-test' I0816 22:25:12.656609 1 controller.go:1337] provision "default/ss-device-test-tf-acc-test-oww82yagz2-0" class "standard": started E0816 22:25:12.656641 1 controller.go:1361] provision "default/ss-device-test-tf-acc-test-oww82yagz2-0" class "standard": failed to provision volume: rancher.io/local-path does not support block volume provisioning I0816 22:25:12.656705 1 event.go:282] Event(v1.ObjectReference{Kind:"PersistentVolumeClaim", Namespace:"default", Name:"ss-device-test-tf-acc-test-oww82yagz2-0", UID:"b9413651-c1db-4095-8926-63edf3bc132a", APIVersion:"v1", ResourceVersion:"1385", FieldPath:""}): type: 'Warning' reason: 'ProvisioningFailed' rancher.io/local-path does not support block volume provisioning I0816 22:25:12.663412 1 controller.go:1337] provision "default/ss-device-test-tf-acc-test-oww82yagz2-0" class "standard": started E0816 22:25:12.663449 1 controller.go:1361] provision "default/ss-device-test-tf-acc-test-oww82yagz2-0" class "standard": failed to provision volume: rancher.io/local-path does not support block volume provisioning I0816 22:25:12.663511 1 event.go:282] Event(v1.ObjectReference{Kind:"PersistentVolumeClaim", Namespace:"default", Name:"ss-device-test-tf-acc-test-oww82yagz2-0", UID:"b9413651-c1db-4095-8926-63edf3bc132a", APIVersion:"v1", ResourceVersion:"1392", FieldPath:""}): type: 'Warning' reason: 'ProvisioningFailed' rancher.io/local-path does not support block volume provisioning I0816 22:25:13.004019 1 controller.go:1337] provision "default/ss-device-test-tf-acc-test-ai47eoiufr-0" class "standard": started I0816 22:25:13.004156 1 event.go:282] Event(v1.ObjectReference{Kind:"PersistentVolumeClaim", Namespace:"default", Name:"ss-device-test-tf-acc-test-ai47eoiufr-0", UID:"186a28a0-b637-4c29-922b-a523a26d4c23", APIVersion:"v1", ResourceVersion:"1402", FieldPath:""}): type: 'Warning' reason: 'ProvisioningFailed' rancher.io/local-path does not support block volume provisioning E0816 22:25:13.004241 1 controller.go:1361] provision "default/ss-device-test-tf-acc-test-ai47eoiufr-0" class "standard": failed to provision volume: rancher.io/local-path does not support block volume provisioning I0816 22:25:25.813517 1 controller.go:1337] provision "default/ss-device-test-tf-acc-test-103te79aze-0" class "standard": started E0816 22:25:25.813589 1 controller.go:1361] provision "default/ss-device-test-tf-acc-test-103te79aze-0" class "standard": failed to provision volume: rancher.io/local-path does not support block volume provisioning I0816 22:25:25.813716 1 event.go:282] Event(v1.ObjectReference{Kind:"PersistentVolumeClaim", Namespace:"default", Name:"ss-device-test-tf-acc-test-103te79aze-0", UID:"cb8a9b89-e6b2-4fb7-b755-ba2bea175956", APIVersion:"v1", ResourceVersion:"1507", FieldPath:""}): type: 'Warning' reason: 'ProvisioningFailed' rancher.io/local-path does not support block volume provisioning I0816 22:29:01.613933 1 controller.go:1337] provision "default/ss-device-test-tf-acc-test-ai47eoiufr-0" class "standard": started E0816 22:29:01.613996 1 controller.go:1361] provision "default/ss-device-test-tf-acc-test-ai47eoiufr-0" class "standard": failed to provision volume: rancher.io/local-path does not support block volume provisioning I0816 22:29:01.614008 1 controller.go:1337] provision "default/ss-device-test-tf-acc-test-oww82yagz2-0" class "standard": started I0816 22:29:01.614028 1 controller.go:1337] provision "default/ss-device-test-tf-acc-test-103te79aze-0" class "standard": started E0816 22:29:01.614088 1 controller.go:1361] provision "default/ss-device-test-tf-acc-test-oww82yagz2-0" class "standard": failed to provision volume: rancher.io/local-path does not support block volume provisioning E0816 22:29:01.614127 1 controller.go:1361] provision "default/ss-device-test-tf-acc-test-103te79aze-0" class "standard": failed to provision volume: rancher.io/local-path does not support block volume provisioning I0816 22:29:01.614135 1 event.go:282] Event(v1.ObjectReference{Kind:"PersistentVolumeClaim", Namespace:"default", Name:"ss-device-test-tf-acc-test-ai47eoiufr-0", UID:"186a28a0-b637-4c29-922b-a523a26d4c23", APIVersion:"v1", ResourceVersion:"1402", FieldPath:""}): type: 'Warning' reason: 'ProvisioningFailed' rancher.io/local-path does not support block volume provisioning I0816 22:29:01.614169 1 event.go:282] Event(v1.ObjectReference{Kind:"PersistentVolumeClaim", Namespace:"default", Name:"ss-device-test-tf-acc-test-oww82yagz2-0", UID:"b9413651-c1db-4095-8926-63edf3bc132a", APIVersion:"v1", ResourceVersion:"1392", FieldPath:""}): type: 'Warning' reason: 'ProvisioningFailed' rancher.io/local-path does not support block volume provisioning I0816 22:29:01.614186 1 event.go:282] Event(v1.ObjectReference{Kind:"PersistentVolumeClaim", Namespace:"default", Name:"ss-device-test-tf-acc-test-103te79aze-0", UID:"cb8a9b89-e6b2-4fb7-b755-ba2bea175956", APIVersion:"v1", ResourceVersion:"1507", FieldPath:""}): type: 'Warning' reason: 'ProvisioningFailed' rancher.io/local-path does not support block volume provisioning I0816 22:35:12.095561 1 controller.go:1337] provision "default/ss-device-test-tf-acc-test-ai47eoiufr-0" class "standard": started E0816 22:35:12.095758 1 controller.go:1361] provision "default/ss-device-test-tf-acc-test-ai47eoiufr-0" class "standard": failed to provision volume: rancher.io/local-path does not support block volume provisioning I0816 22:35:12.095830 1 event.go:282] Event(v1.ObjectReference{Kind:"PersistentVolumeClaim", Namespace:"default", Name:"ss-device-test-tf-acc-test-ai47eoiufr-0", UID:"186a28a0-b637-4c29-922b-a523a26d4c23", APIVersion:"v1", ResourceVersion:"2379", FieldPath:""}): type: 'Warning' reason: 'ProvisioningFailed' rancher.io/local-path does not support block volume provisioning I0816 22:35:12.887555 1 controller.go:1337] provision "default/ss-device-test-tf-acc-test-oww82yagz2-0" class "standard": started E0816 22:35:12.887684 1 controller.go:1361] provision "default/ss-device-test-tf-acc-test-oww82yagz2-0" class "standard": failed to provision volume: rancher.io/local-path does not support block volume provisioning I0816 22:35:12.888008 1 event.go:282] Event(v1.ObjectReference{Kind:"PersistentVolumeClaim", Namespace:"default", Name:"ss-device-test-tf-acc-test-oww82yagz2-0", UID:"b9413651-c1db-4095-8926-63edf3bc132a", APIVersion:"v1", ResourceVersion:"2397", FieldPath:""}): type: 'Warning' reason: 'ProvisioningFailed' rancher.io/local-path does not support block volume provisioning I0816 22:35:20.885571 1 controller.go:1337] provision "default/ss-device-test-tf-acc-test-103te79aze-0" class "standard": started E0816 22:35:20.885753 1 controller.go:1361] provision "default/ss-device-test-tf-acc-test-103te79aze-0" class "standard": failed to provision volume: rancher.io/local-path does not support block volume provisioning I0816 22:35:20.885840 1 event.go:282] Event(v1.ObjectReference{Kind:"PersistentVolumeClaim", Namespace:"default", Name:"ss-device-test-tf-acc-test-103te79aze-0", UID:"cb8a9b89-e6b2-4fb7-b755-ba2bea175956", APIVersion:"v1", ResourceVersion:"2436", FieldPath:""}): type: 'Warning' reason: 'ProvisioningFailed' rancher.io/local-path does not support block volume provisioning

Thanks for the explanation! My only request then is to remove the mention of EKS in the tests since volumeDevices is dependent on EKS and can be done on any platform. Be sure to include a changelog as well!

Good point, thanks! I have removed the mention of EKS and included the changelog.

kubernetes/provider_test.go

BBBmau

thanks for adding this in! I've updated the changelog to include the specific location of where volume_devices was added.

jirislav requested a review from a team as a code owner August 16, 2024 00:39

github-actions bot added the size/XL label Aug 16, 2024

jirislav mentioned this pull request Aug 16, 2024

StatefulSet's PodSpec should allow specifying volumeDevices #2572

Closed

BBBmau reviewed Aug 16, 2024

View reviewed changes

alexsomesan reviewed Aug 22, 2024

View reviewed changes

kubernetes/provider_test.go Outdated Show resolved Hide resolved

jirislav force-pushed the 2572-add-volume-devices-to-pod-spec branch from 760080d to 38a6c23 Compare August 22, 2024 11:34

BBBmau added this to the v2.23.0 milestone Aug 23, 2024

BBBmau self-assigned this Sep 6, 2024

Add VolumeDevices to PodSpec (hashicorp#2572)

6ec1c85

jirislav force-pushed the 2572-add-volume-devices-to-pod-spec branch from 38a6c23 to 6ec1c85 Compare September 16, 2024 05:27

Merge branch 'main' into 2572-add-volume-devices-to-pod-spec

80decc8

BBBmau added the no-documentation This issue or pull request does not have documentation changes. label Sep 20, 2024

Update changelog to include schema_container

483cc3d

BBBmau approved these changes Sep 20, 2024

View reviewed changes

BBBmau merged commit 507da66 into hashicorp:main Sep 20, 2024
22 of 23 checks passed

jirislav deleted the 2572-add-volume-devices-to-pod-spec branch September 20, 2024 19:00

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add VolumeDevices to PodSpec (#2572) #2573

Add VolumeDevices to PodSpec (#2572) #2573

jirislav commented Aug 16, 2024 •

edited

Loading

hashicorp-cla-app bot commented Aug 16, 2024 •

edited

Loading

jirislav commented Aug 16, 2024

BBBmau Aug 16, 2024

jirislav Aug 16, 2024

BBBmau Sep 10, 2024

jirislav Sep 16, 2024

BBBmau left a comment

Add VolumeDevices to PodSpec (#2572) #2573

Add VolumeDevices to PodSpec (#2572) #2573

Conversation

jirislav commented Aug 16, 2024 • edited Loading

Description

Acceptance tests

Release Note

References

Community Note

hashicorp-cla-app bot commented Aug 16, 2024 • edited Loading

jirislav commented Aug 16, 2024

BBBmau Aug 16, 2024

Choose a reason for hiding this comment

jirislav Aug 16, 2024

Choose a reason for hiding this comment

BBBmau Sep 10, 2024

Choose a reason for hiding this comment

jirislav Sep 16, 2024

Choose a reason for hiding this comment

BBBmau left a comment

Choose a reason for hiding this comment

jirislav commented Aug 16, 2024 •

edited

Loading

hashicorp-cla-app bot commented Aug 16, 2024 •

edited

Loading