Required time.sleep(...) to get expected result when calling get_namespaced_custom_object after create

[yannickperrenet]

What happened (please include outputs or screenshots):

    my_resource = {
        "apiVersion": "orchest.io/v1alpha1",
        "kind": "OrchestCluster",
        "metadata": {
            "name": "cluster-1",
            "namespace": "orchest",
        },
        "spec": {
            "singleNode": True,
            "orchest": {
                "nodeAgent": {"image": "orchest/node-agent"},
            },
        },
    }

    # create the resource
    API.create_namespaced_custom_object(
        group="orchest.io",
        version="v1alpha1",
        namespace=namespace,
        plural="orchestclusters",
        body=my_resource,
    )
    print("Resource created")

    # !!!!
    # time.sleep(1)
    resource = API.get_namespaced_custom_object(
        group="orchest.io",
        version="v1alpha1",
        name="cluster-1",
        namespace=namespace,
        plural="orchestclusters",
    )

Would output:

Resource created
Resource details:
{'apiVersion': 'orchest.io/v1alpha1',
 'kind': 'OrchestCluster',
 'metadata': {'creationTimestamp': '2022-04-08T12:37:18Z',
              'generation': 1,
              'managedFields': [{'apiVersion': 'orchest.io/v1alpha1',
                                 'fieldsType': 'FieldsV1',
                                 'fieldsV1': {'f:spec': {'.': {},
                                                         'f:orchest': {'.': {},
                                                                       'f:nodeAgent': {'.': {},
                                                                                       'f:image': {}}},
                                                         'f:singleNode': {}}},
                                 'manager': 'OpenAPI-Generator',
                                 'operation': 'Update',
                                 'time': '2022-04-08T12:37:18Z'}],
              'name': 'cluster-1',
              'namespace': 'orchest',
              'resourceVersion': '10877',
              'uid': 'c28a43ba-593b-4d05-894a-46e516f823e8'},
 'spec': {'orchest': {'nodeAgent': {'image': 'orchest/node-agent'}},
          'singleNode': True}}

What you expected to happen:
Output of:

Resource created
Resource details:
{'apiVersion': 'orchest.io/v1alpha1',
 'kind': 'OrchestCluster',
 'metadata': {'creationTimestamp': '2022-04-08T12:38:41Z',
              'finalizers': ['controller.orchest.io'],
              'generation': 4,
              'managedFields': [{'apiVersion': 'orchest.io/v1alpha1',
                                 'fieldsType': 'FieldsV1',
                                 'fieldsV1': {'f:spec': {'.': {},
                                                         'f:orchest': {'.': {},
                                                                       'f:nodeAgent': {'.': {},
                                                                                       'f:image': {}}},
                                                         'f:singleNode': {}}},
                                 'manager': 'OpenAPI-Generator',
                                 'operation': 'Update',
                                 'time': '2022-04-08T12:38:41Z'},
                                {'apiVersion': 'orchest.io/v1alpha1',
                                 'fieldsType': 'FieldsV1',
                                 'fieldsV1': {'f:metadata': {'f:finalizers': {'.': {},
                                                                              'v:"controller.orchest.io"': {}}},
                                              'f:spec': {'f:orchest': {'f:authServer': {},
                                                                       'f:celeryWorker': {},
                                                                       'f:orchestApi': {},
                                                                       'f:orchestWebServer': {}},
                                                         'f:postgres': {},
                                                         'f:registry': {}},
                                              'f:status': {'.': {},
                                                           'f:message': {},
                                                           'f:state': {}}},
                                 'manager': 'controller',
                                 'operation': 'Update',
                                 'time': '2022-04-08T12:38:41Z'}],
              'name': 'cluster-1',
              'namespace': 'orchest',
              'resourceVersion': '10948',
              'uid': 'ae5e390c-b82b-4f6e-96d1-8f3f3ea7ace5'},
 'spec': {'orchest': {'authServer': {},
                      'celeryWorker': {},
                      'nodeAgent': {'image': 'orchest/node-agent'},
                      'orchestApi': {},
                      'orchestWebServer': {}},
          'postgres': {},
          'registry': {},
          'singleNode': True},
 'status': {'message': 'Deploying Argo', 'state': 'DeployingArgo'}}

which can be achieved by commenting the time.sleep(1) in the code sample.

How to reproduce it (as minimally and precisely as possible):
See above.

Anything else we need to know?:
Ran similar code using the Go SDK which did output the expected result.

Happy to hear your thoughts on the required time.sleep(1).

Environment:

• Kubernetes version (kubectl version): v1.22.2 (client) and v1.22.3 (server)
• OS (e.g., MacOS 10.13.6): Ubuntu 21.04
• Python version (python --version): 3.9
• Python client version (pip list | grep kubernetes): v23.3.0

[roycaihw]

/assign

[roycaihw]

Could you clarify what difference you're looking for between the expected and what you got?

You can run a watch in parallel to see who updated the object in that 1s period.

[yannickperrenet]

Could you clarify what difference you're looking for between the expected and what you got?

Of course, my apologies for not being clear at first. What you can see from both responses (the current response and expected response) is that the current response does not include some information such as the top-level status entry ('status': {'message': 'Deploying Argo', 'state': 'DeployingArgo'}). Probably because I instantly get the object instead of waiting (probably why the time.sleep(1) fixing it) for the changes to be propagated.

When first coding this I was confused because the returned object did not include any status information, especially since the go-client did not seem to "suffer" from this (and includes the status entry).

You can run a watch in parallel to see who updated the object in that 1s period.

I would indeed like to watch the object instead, but that doesn't seem to be supported according to: #1679

The reason for opening this issue was the question whether the get_namespaced_custom_object should be allowed to return an "incomplete" object. Although that could of course depend on how you define "incomplete".

[roycaihw]

It's working as intended for the server to return the object with no status. What you want is to wait for the status to get populated (likely by some other controller running in the cluster). The go client didn't have this problem probably due to a race condition.

What you can do is to:

• sleep and poll the object (what you did)
• run a watch to check the status-- it's still "wait and check", but it's more efficient compared to the sleep, depending on how long you may have to wait

Thanks for bringing the watch issue to my attention. I will take a look.
/close

[k8s-ci-robot]

@roycaihw: Closing this issue.

In response to this:

It's working as intended for the server to return the object with no status. What you want is to wait for the status to get populated (likely by some other controller running in the cluster). The go client didn't have this problem probably due to a race condition.

What you can do is to:

• sleep and poll the object (what you did)
• run a watch to check the status-- it's still "wait and check", but it's more efficient compared to the sleep, depending on how long you may have to wait

Thanks for bringing the watch issue to my attention. I will take a look.
/close

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.