'KubeCluster' object has no attribute '_lock'

[rajasgs]

I am just testing dask-kubernetes.

Here is my code

from dask_kubernetes import KubeCluster, make_pod_spec
pod_spec = make_pod_spec(image='daskdev/dask:latest',
                              memory_limit='4G', memory_request='4G',
                              cpu_limit=1, cpu_request=1,
                              env={'EXTRA_PIP_PACKAGES': 'fastparquet git+https://github.com/dask/distributed'})

cluster = KubeCluster(pod_spec)

When I the the above code on Jupyter lab, I get the error as bleow:

tornado.application - ERROR - Exception in callback functools.partial(<bound method IOLoop._discard_future_result of <zmq.eventloop.ioloop.ZMQIOLoop object at 0x10e9bec88>>, <Task finished coro=<SpecCluster._correct_state_internal() done, defined at /Users/xyz/anaconda3/envs/py36/lib/python3.6/site-packages/distributed/deploy/spec.py:300> exception=AttributeError("'KubeCluster' object has no attribute '_lock'",)>)

Not sure what I am doing here. Some help would be great.

[jsignell]

Hi @rajasgs, thanks for raising an issue. It'd be helpful to know what versions of dask, distributed, and dask-kubernetes you have installed. I see you are pip installing distributed master in the pod. That might result in a mismatch between the versions of distributed on the worker and in your work environment (where you run cluster = KubeCluster)

[arrjaytee]

Hi - we saw the same that was caused by a ~/.kube/config file which was zero-length, giving 'None' as the config dict. A similar problem occurs if it is missing.

It looks like KubeCluster will just carry on regardless with an invalid object if the config fails, so tornodo reports errors afterwards with _lock missing on the partially setup object:

 File "/local/data/scratch/thatro/conda/envs/dask-k8s/lib/python3.7/site-packages/kubernetes/config/kube_config.py", line 659, in load_config
    with open(path) as f:
TypeError: 'NoneType' object does not support item assignment
tornado.application - ERROR - Exception in callback functools.partial(<bound method IOLoop._discard_future_result of <tornado.platform.asyncio.AsyncIOLoop object at 0x7f2a37dccd50>>, <Task finished coro=<SpecCluster._correct_state_internal() done, defined at /local/data/scratch/thatro/conda/envs/dask-k8s/lib/python3.7/site-packages/distributed/deploy/spec.py:300> exception=AttributeError("'KubeCluster' object has no attribute '_lock'")>)
Traceback (most recent call last):
  File "/local/data/scratch/thatro/conda/envs/dask-k8s/lib/python3.7/site-packages/tornado/ioloop.py", line 743, in _run_callback
    ret = callback()
  File "/local/data/scratch/thatro/conda/envs/dask-k8s/lib/python3.7/site-packages/tornado/ioloop.py", line 767, in _discard_future_result
    future.result()
  File "/local/data/scratch/thatro/conda/envs/dask-k8s/lib/python3.7/site-packages/distributed/deploy/spec.py", line 381, in _close
    await self._correct_state()
  File "/local/data/scratch/thatro/conda/envs/dask-k8s/lib/python3.7/site-packages/distributed/deploy/spec.py", line 301, in _correct_state_internal
    async with self._lock:

[jsignell]

Ah thanks for that @thatcr! I guess this issue can just be a case for better error handling in general.

[MridulS]

Ran into a similar issue and kubernetes-client/python-base#223 just got merged which should take care of the error.

Now this will fail a bit more gracefully :)

---------------------------------------------------------------------------
ConfigException                           Traceback (most recent call last)
<ipython-input-1-a6aab33ba3f3> in <module>
      6 
----> 7 cluster = KubeCluster(pod_spec)

~/miniconda3/envs/dask/lib/python3.8/site-packages/dask_kubernetes/core.py in __init__(self, pod_template, name, namespace, n_workers, host, port, env, auth, idle_timeout, deploy_mode, interface, protocol, dashboard_address, security, scheduler_service_wait_timeout, scheduler_pod_template, **kwargs)
    412         self.auth = auth
    413         self.kwargs = kwargs
--> 414         super().__init__(**self.kwargs)
    415 
    416     def _get_pod_template(self, pod_template, pod_type):

.....
.....
.....

~/dev/k8s/python/kubernetes/config/kube_config.py in load_config(self, path)
    704 
    705         if config is None:
--> 706             raise ConfigException(
    707                 'Invalid kube-config. '
    708                 '%s file is empty' % path)

ConfigException: Invalid kube-config. /Users/ms/.kube/config file is empty

[jsignell]

Great! Thanks for mentioning that @MridulS

[jacobtomlinson]

Great thanks! Closing as fixed upstream.