Bump qhub-dask version to 0.4.3

[peytondmurray]

Closes #1296

Changes introduced in this PR:

• Bumped qhub-dask to version 0.4.3, which includes distributed==2022.04.2 and dask-gateway==2022.4.0

Types of changes

• Bug fix (non-breaking change which fixes an issue)
• New feature (non-breaking change which adds a feature)
• Breaking change (fix or feature that would cause existing features to not work as expected)
• Documentation Update
• Code style update (formatting, renaming)
• Refactoring (no functional changes, no API changes)
• Build related changes
• Other (please describe): Enhancement

Testing

Requires testing

• Yes
• No

In case you checked yes, did you write tests?

• Yes
• No

Github workflows for testing new versions of qhub-dask before they get released to conda-forge will be addressing in #1339. Until then, testing was carried out on an existing qhub deployment, see the discussion at conda-forge/qhub-dask-feedstock#14 for more information.

Documentation

Does your contribution include breaking changes or deprecations?
If so have you updated the documentation?

• Yes, docstrings
• Yes, main documentation
• Yes, deprecation notices

[viniciusdc]

OK. Some steps here:

• We need to wait until the new version of qhub-dask is available for downloading (its already on conda, but CF needs some time 1~3h to update the indexes). Until there, the envs will install the previous version
• Once that's done, re-trigger the docker build in this above PR to create a new quansight/qhub-dask-worker:bump-qhub-dask-version-0.4.3 that should then content the new package version -- which might be the reason the Minikube tests are failing now:

 > [dask-worker  9/11] RUN /opt/scripts/install-conda-environment.sh /opt/dask-worker/environment.yaml 'false':
#30 25.90 
#30 25.90 Encountered problems while solving:
#30 25.90   - nothing provides requested qhub-dask 0.4.3

• If the above passes and the image is downloadable, you can pass that in the qhub-config.yaml fo the deployment you want to update (Quansight/Quansight-beta):

profiles:
  dask_worker:
    "Small Worker":
      worker_cores_limit: 1
      worker_cores: 1
      worker_memory_limit: 1G
      worker_memory: 1G
      image: "quansight/qhub-dask-worker:bump..."

you can create a copy of the above profile like "dask-worker-updated" if you don't want to affect the global environment;

[peytondmurray]

Merging is delayed because qhub-dask 0.4.3 must either choose to

1. Drop support for Python 3.7, allowing us to use distributed==2022.4.2 and dask-gateway==2022.4.0, or
2. Keep support for Python 3.7, and be restricted to distributed==2022.2.0 and dask-gateway==0.9.0

See conda-forge/qhub-dask-feedstock#15

[trallard]

Action plan - drop 3.7 -> push to QHub beta and see how that works out

[viniciusdc]

I've tested this, and everything seems to be working -- will check again on Monday to be sure.

[viniciusdc]

/home/conda/filesystem/hash-dask/lib/python3.9/site-packages/distributed/client.py:1288:
 VersionMismatchWarning: Mismatched versions found
+-------------+-----------+-----------+---------+
| Package     | client    | scheduler | workers |
+-------------+-----------+-----------+---------+
| blosc       | MISSING   | None      | None    |
| dask        | 2022.04.2 | 2021.07.2 | None    |
| distributed | 2022.4.2  | 2021.07.2 | None    |
| lz4         | 4.0.0     | None      | None    |
+-------------+-----------+-----------+---------+
  warnings.warn(version_module.VersionMismatchWarning(msg[0]["warning"]))

It seems that the current scheduler is still using an outdated version, @costrouc how do we pass over the `cluster-image var to the gateway deployments? (I was able to find for worker/gateway, but not sure about the above), is it in here:

c.KubeClusterConfig.image = (
    f"{config['cluster-image']['name']}:{config['cluster-image']['tag']}"

Also, I found config for dask that we can use as well, is this worth it?

  # backend nested configuration relates to the scheduler and worker resources
  # created for DaskCluster k8s resources by the controller.
  backend:
    # The image to use for both schedulers and workers.
    image:
      name: ghcr.io/dask/dask-gateway
      tag: "set-by-chartpress"
      pullPolicy: IfNotPresent

[trallard]

@costrouc can you please follow up on this issue?

[viniciusdc]

Oh, silly me! I was using the wrong environment for testing. Thanks, @costrouc and @peytondmurray, all seem okay now 😄

Just to keep a record of the test here:

the version outputs:

client = cluster.get_client()
client.get_versions()

{'scheduler': {'host': {'python': '3.9.13.final.0',
   'python-bits': 64,
   'OS': 'Linux',
   'OS-release': '5.10.90+',
   'machine': 'x86_64',
   'processor': 'x86_64',
   'byteorder': 'little',
   'LC_ALL': 'None',
   'LANG': 'None'},
  'packages': {'python': '3.9.13.final.0',
   'dask': '2022.04.2',
   'distributed': '2022.4.2',
   'msgpack': '1.0.4',
   'cloudpickle': '2.1.0',
   'tornado': '6.1',
   'toolz': '0.11.2',
   'numpy': '1.23.0',
   'pandas': '1.4.3',
   'lz4': '4.0.0'}},
 'workers': {'tls://10.48.1.24:40695': {'host': {'python': '3.9.13.final.0',
    'python-bits': 64,
    'OS': 'Linux',
    'OS-release': '5.10.90+',
    'machine': 'x86_64',
    'processor': 'x86_64',
    'byteorder': 'little',
    'LC_ALL': 'None',
    'LANG': 'None'},
   'packages': {'python': '3.9.13.final.0',
    'dask': '2022.04.2',
    'distributed': '2022.4.2',
    'msgpack': '1.0.4',
    'cloudpickle': '2.1.0',
    'tornado': '6.1',
    'toolz': '0.11.2',
    'numpy': '1.23.0',
    'pandas': '1.4.3',
    'lz4': '4.0.0'}}},
 'client': {'host': {'python': '3.9.13.final.0',
   'python-bits': 64,
   'OS': 'Linux',
   'OS-release': '5.10.90+',
   'machine': 'x86_64',
   'processor': 'x86_64',
   'byteorder': 'little',
   'LC_ALL': 'C.UTF-8',
   'LANG': 'C.UTF-8'},
  'packages': {'python': '3.9.13.final.0',
   'dask': '2022.04.2',
   'distributed': '2022.4.2',
   'msgpack': '1.0.4',
   'cloudpickle': '2.1.0',
   'tornado': '6.1',
   'toolz': '0.11.2',
   'numpy': '1.23.0',
   'pandas': '1.4.3',
   'lz4': '4.0.0'}}}

[viniciusdc]

LGTM, next is updating the docs and closing the PR open for qhub-dask to undo the latest changes.