Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Windows] Windows init() fails, Video Controller #38388

Closed
jaxnb opened this issue Aug 13, 2023 · 5 comments
Closed

[Windows] Windows init() fails, Video Controller #38388

jaxnb opened this issue Aug 13, 2023 · 5 comments
Assignees
Labels
bug Something that is supposed to be working; but isn't P2 Important issue, but not time-critical QS Quantsight triage label windows

Comments

@jaxnb
Copy link

jaxnb commented Aug 13, 2023

What happened + What you expected to happen

The _autodetect_num_gpus() function in resource_spec.py on Windows calls the command: WMICPATH Win32_VideoController GET AdapterCompatibility. However, this command (at least on my virtualized qemu windows) returns an error.
My system has 1x NVIDIA GTX 970 using passthrough, although I'm not attempting to use it so I personally am not too concerned with that usecase being supported. However, Ray should initialize without error even if a GPU is not detected.

Python 3.11.4 (tags/v3.11.4:d2340ef, Jun  7 2023, 05:45:37) [MSC v.1934 64 bit (AMD64)] on win32
Type "help", "copyright", "credits" or "license" for more information.
>>> import ray
>>> ray.init()
ERROR:
Description = Not found
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "...\Python311\Lib\site-packages\ray\_private\client_mode_hook.py", line 103, in wrapper
    return func(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^
  File "...\Python311\Lib\site-packages\ray\_private\worker.py", line 1514, in init
    _global_node = ray._private.node.Node(
                   ^^^^^^^^^^^^^^^^^^^^^^^
  File "...\Python311\Lib\site-packages\ray\_private\node.py", line 290, in __init__
    self.start_ray_processes()
  File "...\Python311\Lib\site-packages\ray\_private\node.py", line 1214, in start_ray_processes
    resource_spec = self.get_resource_spec()
  File "...\Python311\Lib\site-packages\ray\_private\node.py", line 468, in get_resource_spec
    ).resolve(is_head=self.head, node_ip_address=self.node_ip_address)
      ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "...\Python311\Lib\site-packages\ray\_private\resource_spec.py", line 186, in resolve
    num_gpus = _autodetect_num_gpus()
               ^^^^^^^^^^^^^^^^^^^^^^
  File "...\Python311\Lib\site-packages\ray\_private\resource_spec.py", line 298, in _autodetect_num_gpus
    lines = subprocess.check_output(cmdargs).splitlines()[1:]
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "...\Python311\Lib\subprocess.py", line 466, in check_output
    return run(*popenargs, stdout=PIPE, timeout=timeout, check=True,
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "...\Python311\Lib\subprocess.py", line 571, in run
    raise CalledProcessError(retcode, process.args,
subprocess.CalledProcessError: Command '['WMIC', 'PATH', 'Win32_VideoController', 'GET', 'AdapterCompatibility']' returned non-zero exit status 2147749890.
PS> WIMC PATH Win32_VideoController GET AdapterCompatibility
ERROR:
Description = Not found
PS> $LASTEXITCODE
-2147217406

Versions / Dependencies

Python 3.11
Ray 2.6.2
Windows 10 (virtualized)

Reproduction script

import ray
ray.init()

Note: likely system dependent

Issue Severity

High: It blocks me from completing my task.

@jaxnb jaxnb added bug Something that is supposed to be working; but isn't triage Needs triage (eg: priority, bug/not-bug, and owning component) labels Aug 13, 2023
@jjyao jjyao added the windows label Aug 14, 2023
@rkooo567 rkooo567 added the QS Quantsight triage label label Aug 21, 2023
@rkooo567 rkooo567 changed the title [Core] Windows init() fails, Video Controller [Windows] Windows init() fails, Video Controller Aug 21, 2023
@anyscalesam
Copy link
Contributor

anyscalesam commented Sep 26, 2023

@mattip can you please take a look?

@mattip
Copy link
Contributor

mattip commented Oct 5, 2023

I think this would be solved by using gpustat everywhere ray wants to find gpu resources. See PR #35581.

@mattip
Copy link
Contributor

mattip commented Oct 5, 2023

For now, I think you can pip install GPUtil

@mattip
Copy link
Contributor

mattip commented Oct 29, 2023

@jaxnb did installing GPUtil help?

@peytondmurray peytondmurray added the @external-author-action-required Alternate tag for PRs where the author doesn't have labeling permission. label Nov 1, 2023
@anyscalesam anyscalesam assigned jaxnb and mattip and unassigned mattip Nov 8, 2023
@anyscalesam anyscalesam added P2 Important issue, but not time-critical and removed @external-author-action-required Alternate tag for PRs where the author doesn't have labeling permission. triage Needs triage (eg: priority, bug/not-bug, and owning component) labels Nov 8, 2023
@mattip
Copy link
Contributor

mattip commented Nov 30, 2023

I will close this, the need for GPutil has been removed and the detection code refactored. Please open a new issue if this reappears.

@mattip mattip closed this as completed Nov 30, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something that is supposed to be working; but isn't P2 Important issue, but not time-critical QS Quantsight triage label windows
Projects
None yet
Development

No branches or pull requests

6 participants