Add hiredis dependency on supported platforms

[wuisawesome]

Why are these changes needed?

This PR adds the hiredis dependency for non M1 machines.

This removes the redis < 4.0 pin.

Since hiredis doesn't have M1 mac wheels yet, so users there will have extra warning messages in their outputs if they use redis 4.0.

Related issue number

Checks

• I've run scripts/format.sh to lint the changes in this PR.
• I've included any doc changes needed for https://docs.ray.io/en/master/.
• I've made sure the tests are passing. Note that there might be a few flaky tests, see the recent failures at https://flakey-tests.ray.io/
• Testing Strategy
  • Unit tests
  • Release tests
  • This PR is not tested :(

[ericl]

Don't you need to remove this?

[ericl]

Add comment as to why?

[ericl]

LGTM, let's wait for tests.

[pcmoritz]

Windows failure! Can you fix @wuisawesome

+ cd /d/a/ray/ray/python/../python
+ /d/a/ray/ray/python/../ci/keep_alive pip install -v -e .
Obtaining file:///D:/a/ray/ray/python
  Preparing metadata (setup.py): started
  Preparing metadata (setup.py): finished with status 'error'
  ERROR: Command errored out with exit status 1:
   command: 'c:\hostedtoolcache\windows\python\3.6.8\x64\python.exe' -c 'import io, os, sys, setuptools, tokenize; sys.argv[0] = '"'"'D:\\a\\ray\\ray\\python\\setup.py'"'"'; __file__='"'"'D:\\a\\ray\\ray\\python\\setup.py'"'"';f = getattr(tokenize, '"'"'open'"'"', open)(__file__) if os.path.exists(__file__) else io.StringIO('"'"'from setuptools import setup; setup()'"'"');code = f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' egg_info --egg-base 'C:\Users\RUNNER~1\AppData\Local\Temp\pip-pip-egg-info-su7manth'
       cwd: D:\a\ray\ray\python\
  Complete output (1 lines):
  error in ray setup command: 'install_requires' must be a string or list of strings containing valid project/version requirement specifiers; Invalid requirement, parse error at "', redis '"
  ----------------------------------------
WARNING: Discarding file:///D:/a/ray/ray/python. Command errored out with exit status 1: python setup.py egg_info Check the logs for full command output.
ERROR: Command errored out with exit status 1: python setup.py egg_info Check the logs for full command output.

[wuisawesome]

Tune workflow and dashboard errors all look unrelated. merging.

cc @zhe-thoughts @iycheng we need to cherry-pick this one

[zhe-thoughts]

@wuisawesome @ericl @pcmoritz Verifying: this PR fixes #20416, is that right? (It's not linked to the issue)

[wuisawesome]

Actually @ericl, Waleed, and I talked an I think we may not want to cherry-pick this one (though we still want it in master).

[pcmoritz]

@wuisawesome and @ericl can you say what the rationale is?

[wuisawesome]

The release branch currently has redis < 4.0 on all platforms.

The main drawback is that in the future, if people won't be able to use ray 1.9 with redis 4.0. We’ll probably be able to support it by 1.10 though.

The main drawback of cherry picking this PR is that there's some additional risk in adding a new dependency and major version upgrade this late in the release cycle. Note that this is somewhat mitigated by the fact that we use hiredis as the primary redis client in the ray core already, and redis 3.9 already did feature detection and used hiredis (and chances are you already have hiredis installed and have been using it, even though release testing didn’t).

[pcmoritz]

Thanks for the summary!

I think we should try hard to be as compatible as possible with external packages, it is very reasonable to assume that somebody wants to use their own redis client in a Ray application or mix Ray with other packages that might have their own requirements on the redis client and therefore we should strive to support as many versions for that as possible. Given that we have pretty exhaustive tests for the codepaths involving redis in the CI and the release tests and they are passing with redis-py 4 on most platforms, supporting redis-py 4 on these platforms should be low risk and if that is the case we should support it.

I'd love to hear other opinions on this if somebody disagrees :)

Ideally of course we can also be compatible with redis-py 4 on Windows and M1 Macs, hopefully we can still make that happen together with upstream, but we certainly shouldn't be blocked on it.

[zhe-thoughts]

And just linking that the redis < 4.0 setting was introduced in #20430

[wuisawesome]

Looks like we're pretty close to re-simplifying all these PRs. redis/redis-py#1726 has been approved, so once it makes its way into a pypi release, we can just go back to the redis >= 3.5.0 rule.