Overloaded dask gateway fix

[Adam-D-Lewis]

Reference Issues or PRs

Closes #1750.

What does this implement/fix?

Put a x in the boxes that apply

• Bug fix (non-breaking change which fixes an issue)
• New feature (non-breaking change which adds a feature)
• Breaking change (fix or feature that would cause existing features not to work as expected)
• Documentation Update
• Code style update (formatting, renaming)
• Refactoring (no functional changes, no API changes)
• Build related changes
• Other (please describe):

Testing

• Did you test the pull request locally?
• Did you add new tests?

Any other comments?

[Adam-D-Lewis]

I spun up 50 clusters in 30 seconds with the following code (after nodes were already spun up):

%%time
import concurrent.futures

# define the function you want to run
def my_function(num):
    from dask_gateway import Gateway
    gateway = Gateway()
    return gateway.new_cluster()

# create a thread pool with 10 threads
with concurrent.futures.ThreadPoolExecutor(max_workers=50) as executor:
    # submit the function to the thread pool 50 times
    results = [executor.submit(my_function, i) for i in range(50)]
    
    # retrieve the results of the function calls
    for i, future in enumerate(concurrent.futures.as_completed(results)):
        print(i+1, future.result())

[dharhas]

@Adam-D-Lewis

can you test getting options from dask gateway as well. Also run those scaling tests on the existing nebari cluster and see if they fail.

[Adam-D-Lewis]

@dharhas I did both of those things. The test fails in the nebari.quansight.dev, and options worked as expected.

[Adam-D-Lewis]

I can probably fix it so it works with self signed certs by using the k8s service endpoint (e.g. curl -kL http://nebari-conda-store-server.dev.svc:5000/conda-store/api/v1/environment -H "Authorization: Bearer <token>)

[iameskild]

I tested this and everything looks good to me! I had to use ProcessPoolManager instead of:

with concurrent.futures.ThreadPoolExecutor(max_workers=50) as executor:

Either way, I was able to connect to 50 dask schedulers within a matter of seconds 🎉

[rsignell-usgs]

This is awesome! @kcpevey let's test this out on the ESIP nebari training deployment at https://nebari-workshop.esipfed.org as soon as it's ready: I've added you as an admin on the private repo: https://github.com/ESIPFed/nebari-workshop

It looks like we need to get @amsnyder to push the deployment as she was the one who configured and deployed this! Let's continue this discussion on ESIP slack...

[amsnyder]

We would like to push these updates to our deployment so that we can use them in a workshop we are running on May 10th.

Would you be able to create a release with these updates so that we can pull them?

[iameskild]

Hi @amsnyder, we working on getting a new release out either today or early next week :)

[amsnyder]

Thanks @iameskild! If the release is delayed, is it possible for us to just update our images from:

default_images:
  jupyterhub: quay.io/nebari/nebari-jupyterhub:2023.4.1
  jupyterlab: quay.io/nebari/nebari-jupyterlab:2023.4.1
  dask_worker: quay.io/nebari/nebari-dask-worker:2023.4.1

to:

default_images:
  jupyterhub: quay.io/nebari/nebari-jupyterhub:main
  jupyterlab: quay.io/nebari/nebari-jupyterlab:main
  dask_worker: quay.io/nebari/nebari-dask-worker:main

[Adam-D-Lewis]

Thanks @iameskild! If the release is delayed, is it possible for us to just update our images from:

Unfortunately, I don't think that will work since conda store needs to add a token for dask gateway which is only done during the nebari deploy step with the latest version of nebari. You may be able to pip install the latest version of the develop branch of nebari as shown (https://stackoverflow.com/questions/20101834/pip-install-from-git-repo-branch) if needed, and deploy with this fix. I think that would work, but you may encounter other unforeseen issues doing that.

[dharhas]

Is there anything holding back the release at this point?

[kcpevey]

No blockers. We're hoping to get it out today 🤞