Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[Ray release infra] RLLib dependency on Atari ROMs causing multiple release test failures #31880

Closed
cadedaniel opened this issue Jan 24, 2023 · 4 comments · Fixed by #31933
Closed

[Ray release infra] RLLib dependency on Atari ROMs causing multiple release test failures #31880

cadedaniel opened this issue Jan 24, 2023 · 4 comments · Fixed by #31933
Assignees
@kouroshHakha
Labels
P0 Issues that should be fixed in short order release-blocker P0 Issue that blocks the release

Comments

@cadedaniel
Copy link
Member

cadedaniel commented Jan 24, 2023 •
edited
Loading

Some release tests are failing because we're not able to build the cluster environment. The cluster environment logs fail when attempting to download an Atari ROM for RL simulations.

Failing tests caused by this:

there may be others.

[ERROR] 1/21/2023, 10:28:56 PM:   error: subprocess-exited-with-error
  
  × Building wheel for AutoROM.accept-rom-license (pyproject.toml) did not run successfully.
  │ exit code: 1
  ╰─> [241 lines of output]
      running bdist_wheel
      running build
      running build_py
      creating build
      creating build/lib
      copying AutoROM.py -> build/lib
      installing to build/bdist.linux-x86_64/wheel
      running install
      running install_lib
      creating build/bdist.linux-x86_64
      creating build/bdist.linux-x86_64/wheel
      copying build/lib/AutoROM.py -> build/bdist.linux-x86_64/wheel
      running install_egg_info
      running egg_info
      writing AutoROM.accept_rom_license.egg-info/PKG-INFO
      writing dependency_links to AutoROM.accept_rom_license.egg-info/dependency_links.txt
      writing requirements to AutoROM.accept_rom_license.egg-info/requires.txt
      writing top-level names to AutoROM.accept_rom_license.egg-info/top_level.txt
      reading manifest file 'AutoROM.accept_rom_license.egg-info/SOURCES.txt'
      reading manifest template 'MANIFEST.in'
      adding license file 'LICENSE.txt'
      writing manifest file 'AutoROM.accept_rom_license.egg-info/SOURCES.txt'
      Copying AutoROM.accept_rom_license.egg-info to build/bdist.linux-x86_64/wheel/AutoROM.accept_rom_license-0.5.4-py3.7.egg-info
      running install_scripts
      /tmp/pip-build-env-pco1a5lu/overlay/lib/python3.7/site-packages/setuptools/config/setupcfg.py:515: SetuptoolsDeprecationWarning: The license_file parameter is deprecated, use license_files instead.
        warnings.warn(msg, warning_class)
      time=1.0/60 seconds - Trying to download atari roms
          current status=downloading metadata (downloading_metadata)
          total downloaded bytes=0
          total payload download=0
          total failed bytes=0
<snip />
      time=11.0/60 seconds - Trying to download atari roms
          current status=download (downloading)
          total downloaded bytes=383148
          total payload download=381584
          total failed bytes=0
      Terminating attempt to download ROMs after 60 seconds, trying again
      AutoROM will download the Atari 2600 ROMs.
      They will be installed to:
          /tmp/pip-install-xrwywji6/autorom-accept-rom-license/build/bdist.linux-x86_64/wheel/AutoROM/roms
      
      Existing ROMs will be overwritten.
      Traceback (most recent call last):
        File "/home/ray/anaconda3/lib/python3.7/site-packages/pip/_vendor/pep517/in_process/_in_process.py", line 351, in <module>
          main()
        File "/home/ray/anaconda3/lib/python3.7/site-packages/pip/_vendor/pep517/in_process/_in_process.py", line 333, in main
          json_out['return_val'] = hook(**hook_input['kwargs'])
        File "/home/ray/anaconda3/lib/python3.7/site-packages/pip/_vendor/pep517/in_process/_in_process.py", line 250, in build_wheel
          metadata_directory)
        File "/tmp/pip-build-env-pco1a5lu/overlay/lib/python3.7/site-packages/setuptools/build_meta.py", line 414, in build_wheel
          wheel_directory, config_settings)
        File "/tmp/pip-build-env-pco1a5lu/overlay/lib/python3.7/site-packages/setuptools/build_meta.py", line 398, in _build_with_temp_dir
          self.run_setup()
        File "/tmp/pip-build-env-pco1a5lu/overlay/lib/python3.7/site-packages/setuptools/build_meta.py", line 485, in run_setup
          self).run_setup(setup_script=setup_script)
        File "/tmp/pip-build-env-pco1a5lu/overlay/lib/python3.7/site-packages/setuptools/build_meta.py", line 335, in run_setup
          exec(code, locals())
        File "<string>", line 20, in <module>
        File "/tmp/pip-build-env-pco1a5lu/overlay/lib/python3.7/site-packages/setuptools/__init__.py", line 108, in setup
          return distutils.core.setup(**attrs)
        File "/tmp/pip-build-env-pco1a5lu/overlay/lib/python3.7/site-packages/setuptools/_distutils/core.py", line 185, in setup
          return run_commands(dist)
        File "/tmp/pip-build-env-pco1a5lu/overlay/lib/python3.7/site-packages/setuptools/_distutils/core.py", line 201, in run_commands
          dist.run_commands()
        File "/tmp/pip-build-env-pco1a5lu/overlay/lib/python3.7/site-packages/setuptools/_distutils/dist.py", line 969, in run_commands
          self.run_command(cmd)
        File "/tmp/pip-build-env-pco1a5lu/overlay/lib/python3.7/site-packages/setuptools/dist.py", line 1213, in run_command
          super().run_command(command)
        File "/tmp/pip-build-env-pco1a5lu/overlay/lib/python3.7/site-packages/setuptools/_distutils/dist.py", line 988, in run_command
          cmd_obj.run()
        File "/tmp/pip-build-env-pco1a5lu/overlay/lib/python3.7/site-packages/wheel/bdist_wheel.py", line 360, in run
          self.run_command("install")
        File "/tmp/pip-build-env-pco1a5lu/overlay/lib/python3.7/site-packages/setuptools/_distutils/cmd.py", line 318, in run_command
          self.distribution.run_command(command)
        File "/tmp/pip-build-env-pco1a5lu/overlay/lib/python3.7/site-packages/setuptools/dist.py", line 1213, in run_command
          super().run_command(command)
        File "/tmp/pip-build-env-pco1a5lu/overlay/lib/python3.7/site-packages/setuptools/_distutils/dist.py", line 988, in run_command
          cmd_obj.run()
        File "<string>", line 15, in run
        File "/tmp/pip-install-xrwywji6/autorom-accept-rom-license/AutoROM.py", line 330, in main
          buffer = torrent_tar_to_buffer()
        File "/tmp/pip-install-xrwywji6/autorom-accept-rom-license/AutoROM.py", line 200, in torrent_tar_to_buffer
          raise RuntimeError("Tried to download ROMs 3 times, which have all failed, please try again or report this issue.")
      RuntimeError: Tried to download ROMs 3 times, which have all failed, please try again or report this issue.
      [end of output]
  
  note: This error originates from a subprocess, and is likely not a problem with pip.

[ERROR] 1/21/2023, 10:28:56 PM:   ERROR: Failed building wheel for AutoROM.accept-rom-license
@cadedaniel cadedaniel added release-blocker P0 Issue that blocks the release P0 Issues that should be fixed in short order labels Jan 24, 2023
@kouroshHakha
Copy link
Contributor

kouroshHakha commented Jan 24, 2023 •
edited
Loading

@cadedaniel For the long running APEX test we need that dependency. Do we know why the download fails on these3 particular tests and not on other ones? Please tag me going fwd so that I get notified.

@cadedaniel
Copy link
Member Author

Have we had issues in the past with this? I am worried that we'll be blocked from running release tests if this pip package cannot be downloaded.

@cadedaniel
Copy link
Member Author

@kouroshHakha can we repro this locally?

@cadedaniel
Copy link
Member Author

We're fixing by uploading built wheel to s3

@kouroshHakha kouroshHakha mentioned this issue Jan 25, 2023
Merged
7 tasks
richardliaw pushed a commit that referenced this issue Jan 27, 2023
@kouroshHakha
[RLlib] Fixed the autorom dependency issue (#31933)
76d7467 
Co-authored-by: Cade Daniel <edacih@gmail.com>
Closes #31880
edoakes pushed a commit to edoakes/ray that referenced this issue Mar 22, 2023
@kouroshHakha @edoakes
[RLlib] Fixed the autorom dependency issue (ray-project#31933)
f103522 
Co-authored-by: Cade Daniel <edacih@gmail.com>
Closes ray-project#31880

Signed-off-by: Edward Oakes <ed.nmi.oakes@gmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@kouroshHakha kouroshHakha

Labels
P0 Issues that should be fixed in short order release-blocker P0 Issue that blocks the release
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

[RLlib] Fixed the autorom dependency issue kouroshHakha/ray
4 participants
@cadedaniel @ArturNiederfahrenhorst @kouroshHakha @avnishn