Detect cgroups resource limits

[jcrist]

This adds support for detecting resource (CPU and memory) limits set
using cgroups. This makes dask resource detection play nicer with
container systems (e.g. docker), rather than detecting the host memory
and cpus avaialable.

This also centralizes all queries about the host platform to a single
module (distributed.platform), with top-level constants defined for
common usage.

Fixes #1712

[mrocklin]

This sounds like a great idea to me!

I don't know enough about cgroups to verify that the choices here are sufficiently generic, but I suspect that @jacobtomlinson or @pentschev may know more.

[pentschev]

On a very high-level overview, this looks good to me and I also like this idea, so thanks for working on that @jcrist.

One alternative to reading /sys directly is to use https://pypi.org/project/cgroup-utils/, have you checked/thought about that? Not trying to make a case in favor of it, just pointing to its existence, since adding that dependency may also be undesirable.

[pentschev]

Forgot to mention, but I can take a more detailed look at this PR tomorrow.

[jcrist]

One alternative to reading /sys directly is to use https://pypi.org/project/cgroup-utils/, have you checked/thought about that?

There's a few different libraries for doing this sort of thing, but all of them seemed to be overkill for the simple queries we need, and there didn't seem to be a strong community standard lib to use. Opted for the simplest thing that works.

[jacobtomlinson]

This all looks good to me. I don't feel I can add more to the existing reviews right now.

[mrocklin]

If you're good to change PLATFORM to .system then I'd be happy to merge

…

On Tue, Sep 10, 2019 at 7:16 AM Jim Crist ***@***.***> wrote: ***@***.**** commented on this pull request. ------------------------------ In distributed/platform.py <#3039 (comment)>: > @@ -0,0 +1,70 @@ +import os No, I think this should be just another internal module. Kept it out of utils.py, as that module is huge. — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#3039?email_source=notifications&email_token=AACKZTF5MZUODFQST7N54ALQI6T3DA5CNFSM4IU67QYKYY3PNVWWK3TUL52HS4DFWFIHK3DMKJSXC5LFON2FEZLWNFSXPKTDN5WW2ZLOORPWSZGOCEHNUVQ#discussion_r322769010>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AACKZTF646RH557INVE36R3QI6T3DANCNFSM4IU67QYA> .

[jcrist]

I believe all comments have been addressed. Should be good-to-go on tests pass.

[mrocklin]

Thanks @jcrist . This is in. I think that this will make folks happy.