[core] Release function actor lock while waiting for actor class to be loaded by import thread

[ckw017]

Why are these changes needed?

Actor manager now drops the lock introduced in #16278 which can cause deadlock described in #17980 while it's waiting for actor classes to be loaded.

Related issue number

Closes #16278

Checks

• I've run scripts/format.sh to lint the changes in this PR.
• I've included any doc changes needed for https://docs.ray.io/en/master/.
• I've made sure the tests are passing. Note that there might be a few flaky tests, see the recent failures at https://flakey-tests.ray.io/
• Testing Strategy
  • Unit tests
  • Release tests
  • This PR is not tested :(

[ericl]

This seems like a classic case where you should use a condition variable: https://docs.python.org/3/library/threading.html

Would the following fix the issue?

1. Change the time.sleep in function_manager.py:load_actor_class_from_gcs to cv.wait() for some condition variable of the function manager.
2. After each new class is registered in function_manager, call cv.notify().

This release the lock during load_actor_class_from_gcs().

[ericl]

Another workaround that I can see, if the issue is that you want to be holding the lock across the entire deserialization--- is to raise an exception if the class is missing instead of sleeping.

Then, we can catch that exception, sleep, and retry the entire deserialization at a higher level.

[ericl]

Seems that strategy is failing all tests, did you test locally? Otherwise probably should try the high level exception/retry.

[ckw017]

Sorry, lots of silly mistakes in that run. Retrying right now with fixes

[ckw017]

@ekl Windows failure looks unrelated, but lmk if you want me to rerun/investigate