Trouble Installing, sample virtual environment config?

[edfink234]

Hello.

I have tried with Python version 3.11, 3.9, 3.8, and now 3.7, with pip versions 23.2.1 and 20.3.4, and wheel versions 0.36.2 and 0.41.1, and numpy versions 1.16 - 1.19 and for the life of me no matter how many different clean-slate virtual environments I use I can't seem to get pip install -e . to work. For example, with pip-23.2.1 I get:

Obtaining file:///Users/edwardfinkelstein/deep-symbolic-optimization/dso
  Installing build dependencies ... done
  Checking if build backend supports build_editable ... done
  Getting requirements to build editable ... error
  error: subprocess-exited-with-error
  
  × Getting requirements to build editable did not run successfully.
  │ exit code: 1
  ╰─> [53 lines of output]
      /Users/edwardfinkelstein/deepSRvenv/bin/python3.7: No module named pip
      /Users/edwardfinkelstein/deep-symbolic-optimization/dso/setup.py:5: _DeprecatedInstaller: setuptools.installer and fetch_build_eggs are deprecated.
      !!
      
              ********************************************************************************
              Requirements should be satisfied by a PEP 517 installer.
              If you are using pip, you can try `pip install --use-pep517`.
              ********************************************************************************
      
      !!
        dist.Distribution().fetch_build_eggs(['Cython', 'numpy'])
      Traceback (most recent call last):
        File "/private/var/folders/n1/0_k13bw50977fy1vswx1ln4m0000gn/T/pip-build-env-184jyeef/overlay/lib/python3.7/site-packages/setuptools/installer.py", line 96, in _fetch_build_egg_no_warn
          subprocess.check_call(cmd)
        File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/subprocess.py", line 328, in check_call
          raise CalledProcessError(retcode, cmd)
      subprocess.CalledProcessError: Command '['/Users/edwardfinkelstein/deepSRvenv/bin/python3.7', '-m', 'pip', '--disable-pip-version-check', 'wheel', '--no-deps', '-w', '/var/folders/n1/0_k13bw50977fy1vswx1ln4m0000gn/T/tmp7sr8fk9_', '--quiet', 'numpy']' returned non-zero exit status 1.
      
      The above exception was the direct cause of the following exception:
      
      Traceback (most recent call last):
        File "/Users/edwardfinkelstein/deepSRvenv/lib/python3.7/site-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py", line 353, in <module>
          main()
        File "/Users/edwardfinkelstein/deepSRvenv/lib/python3.7/site-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py", line 335, in main
          json_out['return_val'] = hook(**hook_input['kwargs'])
        File "/Users/edwardfinkelstein/deepSRvenv/lib/python3.7/site-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py", line 132, in get_requires_for_build_editable
          return hook(config_settings)
        File "/private/var/folders/n1/0_k13bw50977fy1vswx1ln4m0000gn/T/pip-build-env-184jyeef/overlay/lib/python3.7/site-packages/setuptools/build_meta.py", line 450, in get_requires_for_build_editable
          return self.get_requires_for_build_wheel(config_settings)
        File "/private/var/folders/n1/0_k13bw50977fy1vswx1ln4m0000gn/T/pip-build-env-184jyeef/overlay/lib/python3.7/site-packages/setuptools/build_meta.py", line 341, in get_requires_for_build_wheel
          return self._get_build_requires(config_settings, requirements=['wheel'])
        File "/private/var/folders/n1/0_k13bw50977fy1vswx1ln4m0000gn/T/pip-build-env-184jyeef/overlay/lib/python3.7/site-packages/setuptools/build_meta.py", line 323, in _get_build_requires
          self.run_setup()
        File "/private/var/folders/n1/0_k13bw50977fy1vswx1ln4m0000gn/T/pip-build-env-184jyeef/overlay/lib/python3.7/site-packages/setuptools/build_meta.py", line 488, in run_setup
          self).run_setup(setup_script=setup_script)
        File "/private/var/folders/n1/0_k13bw50977fy1vswx1ln4m0000gn/T/pip-build-env-184jyeef/overlay/lib/python3.7/site-packages/setuptools/build_meta.py", line 338, in run_setup
          exec(code, locals())
        File "<string>", line 5, in <module>
        File "/private/var/folders/n1/0_k13bw50977fy1vswx1ln4m0000gn/T/pip-build-env-184jyeef/overlay/lib/python3.7/site-packages/setuptools/dist.py", line 907, in fetch_build_eggs
          return _fetch_build_eggs(self, requires)
        File "/private/var/folders/n1/0_k13bw50977fy1vswx1ln4m0000gn/T/pip-build-env-184jyeef/overlay/lib/python3.7/site-packages/setuptools/installer.py", line 41, in _fetch_build_eggs
          replace_conflicting=True,
        File "/private/var/folders/n1/0_k13bw50977fy1vswx1ln4m0000gn/T/pip-build-env-184jyeef/overlay/lib/python3.7/site-packages/pkg_resources/__init__.py", line 830, in resolve
          req, best, replace_conflicting, env, installer, required_by, to_activate
        File "/private/var/folders/n1/0_k13bw50977fy1vswx1ln4m0000gn/T/pip-build-env-184jyeef/overlay/lib/python3.7/site-packages/pkg_resources/__init__.py", line 866, in _resolve_dist
          req, ws, installer, replace_conflicting=replace_conflicting
        File "/private/var/folders/n1/0_k13bw50977fy1vswx1ln4m0000gn/T/pip-build-env-184jyeef/overlay/lib/python3.7/site-packages/pkg_resources/__init__.py", line 1135, in best_match
          return self.obtain(req, installer)
        File "/private/var/folders/n1/0_k13bw50977fy1vswx1ln4m0000gn/T/pip-build-env-184jyeef/overlay/lib/python3.7/site-packages/pkg_resources/__init__.py", line 1147, in obtain
          return installer(requirement)
        File "/private/var/folders/n1/0_k13bw50977fy1vswx1ln4m0000gn/T/pip-build-env-184jyeef/overlay/lib/python3.7/site-packages/setuptools/installer.py", line 98, in _fetch_build_egg_no_warn
          raise DistutilsError(str(e)) from e
      distutils.errors.DistutilsError: Command '['/Users/edwardfinkelstein/deepSRvenv/bin/python3.7', '-m', 'pip', '--disable-pip-version-check', 'wheel', '--no-deps', '-w', '/var/folders/n1/0_k13bw50977fy1vswx1ln4m0000gn/T/tmp7sr8fk9_', '--quiet', 'numpy']' returned non-zero exit status 1.
      [end of output]
  
  note: This error originates from a subprocess, and is likely not a problem with pip.
error: subprocess-exited-with-error

× Getting requirements to build editable did not run successfully.
│ exit code: 1
╰─> See above for output.

note: This error originates from a subprocess, and is likely not a problem with pip.

Doing pip install --use-pep517 -e . changes nothing. With pip version 20.3.4 I get:

  Installing build dependencies ... done
  Getting requirements to build wheel ... error
  ERROR: Command errored out with exit status 1:
   command: /Users/edwardfinkelstein/deepSRvenv/bin/python3.7 /Users/edwardfinkelstein/deepSRvenv/lib/python3.7/site-packages/pip/_vendor/pep517/_in_process.py get_requires_for_build_wheel /var/folders/n1/0_k13bw50977fy1vswx1ln4m0000gn/T/tmpdfdn1xir
       cwd: /Users/edwardfinkelstein/deep-symbolic-optimization/dso
  Complete output (51 lines):
  /Users/edwardfinkelstein/deepSRvenv/bin/python3.7: No module named pip
  /Users/edwardfinkelstein/deep-symbolic-optimization/dso/setup.py:5: _DeprecatedInstaller: setuptools.installer and fetch_build_eggs are deprecated.
  !!
  
          ********************************************************************************
          Requirements should be satisfied by a PEP 517 installer.
          If you are using pip, you can try `pip install --use-pep517`.
          ********************************************************************************
  
  !!
    dist.Distribution().fetch_build_eggs(['Cython', 'numpy'])
  Traceback (most recent call last):
    File "/private/var/folders/n1/0_k13bw50977fy1vswx1ln4m0000gn/T/pip-build-env-8nqb_t8o/overlay/lib/python3.7/site-packages/setuptools/installer.py", line 96, in _fetch_build_egg_no_warn
      subprocess.check_call(cmd)
    File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/subprocess.py", line 328, in check_call
      raise CalledProcessError(retcode, cmd)
  subprocess.CalledProcessError: Command '['/Users/edwardfinkelstein/deepSRvenv/bin/python3.7', '-m', 'pip', '--disable-pip-version-check', 'wheel', '--no-deps', '-w', '/var/folders/n1/0_k13bw50977fy1vswx1ln4m0000gn/T/tmpu1788i4d', '--quiet', 'numpy']' returned non-zero exit status 1.
  
  The above exception was the direct cause of the following exception:
  
  Traceback (most recent call last):
    File "/Users/edwardfinkelstein/deepSRvenv/lib/python3.7/site-packages/pip/_vendor/pep517/_in_process.py", line 280, in <module>
      main()
    File "/Users/edwardfinkelstein/deepSRvenv/lib/python3.7/site-packages/pip/_vendor/pep517/_in_process.py", line 263, in main
      json_out['return_val'] = hook(**hook_input['kwargs'])
    File "/Users/edwardfinkelstein/deepSRvenv/lib/python3.7/site-packages/pip/_vendor/pep517/_in_process.py", line 114, in get_requires_for_build_wheel
      return hook(config_settings)
    File "/private/var/folders/n1/0_k13bw50977fy1vswx1ln4m0000gn/T/pip-build-env-8nqb_t8o/overlay/lib/python3.7/site-packages/setuptools/build_meta.py", line 341, in get_requires_for_build_wheel
      return self._get_build_requires(config_settings, requirements=['wheel'])
    File "/private/var/folders/n1/0_k13bw50977fy1vswx1ln4m0000gn/T/pip-build-env-8nqb_t8o/overlay/lib/python3.7/site-packages/setuptools/build_meta.py", line 323, in _get_build_requires
      self.run_setup()
    File "/private/var/folders/n1/0_k13bw50977fy1vswx1ln4m0000gn/T/pip-build-env-8nqb_t8o/overlay/lib/python3.7/site-packages/setuptools/build_meta.py", line 488, in run_setup
      self).run_setup(setup_script=setup_script)
    File "/private/var/folders/n1/0_k13bw50977fy1vswx1ln4m0000gn/T/pip-build-env-8nqb_t8o/overlay/lib/python3.7/site-packages/setuptools/build_meta.py", line 338, in run_setup
      exec(code, locals())
    File "<string>", line 5, in <module>
    File "/private/var/folders/n1/0_k13bw50977fy1vswx1ln4m0000gn/T/pip-build-env-8nqb_t8o/overlay/lib/python3.7/site-packages/setuptools/dist.py", line 907, in fetch_build_eggs
      return _fetch_build_eggs(self, requires)
    File "/private/var/folders/n1/0_k13bw50977fy1vswx1ln4m0000gn/T/pip-build-env-8nqb_t8o/overlay/lib/python3.7/site-packages/setuptools/installer.py", line 41, in _fetch_build_eggs
      replace_conflicting=True,
    File "/private/var/folders/n1/0_k13bw50977fy1vswx1ln4m0000gn/T/pip-build-env-8nqb_t8o/overlay/lib/python3.7/site-packages/pkg_resources/__init__.py", line 830, in resolve
      req, best, replace_conflicting, env, installer, required_by, to_activate
    File "/private/var/folders/n1/0_k13bw50977fy1vswx1ln4m0000gn/T/pip-build-env-8nqb_t8o/overlay/lib/python3.7/site-packages/pkg_resources/__init__.py", line 866, in _resolve_dist
      req, ws, installer, replace_conflicting=replace_conflicting
    File "/private/var/folders/n1/0_k13bw50977fy1vswx1ln4m0000gn/T/pip-build-env-8nqb_t8o/overlay/lib/python3.7/site-packages/pkg_resources/__init__.py", line 1135, in best_match
      return self.obtain(req, installer)
    File "/private/var/folders/n1/0_k13bw50977fy1vswx1ln4m0000gn/T/pip-build-env-8nqb_t8o/overlay/lib/python3.7/site-packages/pkg_resources/__init__.py", line 1147, in obtain
      return installer(requirement)
    File "/private/var/folders/n1/0_k13bw50977fy1vswx1ln4m0000gn/T/pip-build-env-8nqb_t8o/overlay/lib/python3.7/site-packages/setuptools/installer.py", line 98, in _fetch_build_egg_no_warn
      raise DistutilsError(str(e)) from e
  distutils.errors.DistutilsError: Command '['/Users/edwardfinkelstein/deepSRvenv/bin/python3.7', '-m', 'pip', '--disable-pip-version-check', 'wheel', '--no-deps', '-w', '/var/folders/n1/0_k13bw50977fy1vswx1ln4m0000gn/T/tmpu1788i4d', '--quiet', 'numpy']' returned non-zero exit status 1.
  ----------------------------------------
WARNING: Discarding file:///Users/edwardfinkelstein/deep-symbolic-optimization/dso. Command errored out with exit status 1: /Users/edwardfinkelstein/deepSRvenv/bin/python3.7 /Users/edwardfinkelstein/deepSRvenv/lib/python3.7/site-packages/pip/_vendor/pep517/_in_process.py get_requires_for_build_wheel /var/folders/n1/0_k13bw50977fy1vswx1ln4m0000gn/T/tmpdfdn1xir Check the logs for full command output.
ERROR: Command errored out with exit status 1: /Users/edwardfinkelstein/deepSRvenv/bin/python3.7 /Users/edwardfinkelstein/deepSRvenv/lib/python3.7/site-packages/pip/_vendor/pep517/_in_process.py get_requires_for_build_wheel /var/folders/n1/0_k13bw50977fy1vswx1ln4m0000gn/T/tmpdfdn1xir Check the logs for full command output.
WARNING: You are using pip version 20.3.4; however, version 23.2.1 is available.
You should consider upgrading via the '/Users/edwardfinkelstein/deepSRvenv/bin/python3.7 -m pip install --upgrade pip' command.

I've followed the steps in the README every time to no avail. Computer info for my Apple M1 (uname -a):

Darwin Edwards-MBP.webspeed.dk 22.5.0 Darwin Kernel Version 22.5.0: Thu Jun  8 22:22:19 PDT 2023; root:xnu-8796.121.3~7/RELEASE_ARM64_T8103 arm64

Does anyone by chance have a python/pip/wheel/numpy/anything_else_relevant version combination that worked for them? Any help's greatly appreciated and would be happy to answer any questions that might help further diagnose the issue.

[mtrautner]

I had the same error following the README instructions. What worked for me was doing
/usr/bin/python3.6 -m venv venv-dso-3.6
(or whereever you have python3.6 located)
instead of just python. I think some of the packages depend on version 3.6. Then when actually running dso inside the venv, I also specify python3.6.

[jeroenwijnen98]

@edfink234, did you eventually manage? I'm having the same issues, and running on python3.6 as @mtrautner suggested also does not work for me.

Kind Regards,
Jeroen

[snapo]

I also had the same issue and gave up on linux (as my linux distro requires 3.10 and i dont wanna destroy it) because of the requirements and dependencys... so i created a docker file to run it in a container. Maybe this might help you too...

FROM ubuntu:18.04
RUN apt update
RUN apt install python3-pip software-properties-common -y

ENV LANG C.UTF-8
ENV LC_ALL C.UTF-8

RUN ln -s /usr/bin/pip3 /usr/bin/pip
RUN ln -s /usr/bin/python3.6 /usr/bin/python

RUN pip install --upgrade setuptools pip

WORKDIR /code

COPY ./deep-symbolic-optimization/dso ./dso
RUN pip install -e ./dso

# COPY ./mydata ./mydata

# python -m dso.run /code/mydata/Data1.json

CMD ["python3", "--version"]

after that you can either edit the dso and rebuild the container, or link the mydata folder to /code like i do....

docker run -it --name ia-dso -v ./mydata:/code/mydata dso:latest /bin/bash

after that you are in the interactive shell and can use

python -m dso.run /code/mydata2/Data1.json --runs 100

to make your first run...

[brendenpetersen]

@edfink234 What OS are you using?

@snapo Thank you very much for providing a Dockerfile. I think we could take that as a PR if you'd like to create one.

Similar for any improvements to the README! Internally, we have a slightly different setup, so it can be hard for us to keep the public README up to date.

[snapo]

@brendenpetersen i was thinking about creating a PR... but i have to do some more work on the docker file to make it much more easy to scale / run tasks in the future and make it able to orchestrate tasks with different seeds via docker compose. When its ready yes i will do it, this is just a quick and dirty hackup to test the framework. with docker compose/kubernetis you would then be able to distribute each search to the cloud where each seed runs in its own docker container. (at least what i am planning).

This would allow you then to have
datafile , json config

then lets say run it on 1k amazon cloud instances where each instance has its own seed for 1 run. (therefore i have to still think about a little to collect data after the run , maybe i am script something ....).
With that setup you would then be able to say 30 runs per instance, 10k instances and have a huge time reduction for the serach space.

[edfink234]

@jeroenwijnen98 I never managed. @brendenpetersen macOS Sonoma 14.2

[gpzhaome]

I got a similar error with python 3.9 on both windows and ubuntu os. The error disappeared if I installed it with python 3.6.

Error info:

Obtaining file:///home/gpzhao/Github/deep-symbolic-optimization-master/dso
Preparing metadata (setup.py) ... done
Collecting pytest (from dso==1.0.dev0)
Using cached pytest-7.4.3-py3-none-any.whl.metadata (7.9 kB)
Requirement already satisfied: cython in /home/gpzhao/.conda/envs/dsr39/lib/python3.9/site-packages (from dso==1.0.dev0) (3.0.7)
Collecting numpy<=1.19 (from dso==1.0.dev0)
Using cached numpy-1.19.0.zip (7.3 MB)
Installing build dependencies ... done
Getting requirements to build wheel ... done
Preparing metadata (pyproject.toml) ... error
error: subprocess-exited-with-error

× Preparing metadata (pyproject.toml) did not run successfully.
│ exit code: 1
╰─> [54 lines of output]
Running from numpy source directory.
:460: UserWarning: Unrecognized setuptools command, proceeding with generating Cython sources and expanding templates
/tmp/pip-install-s5u3gw4k/numpy_c70991b77c1648a99494c52834ba182a/tools/cythonize.py:73: DeprecationWarning: distutils Version classes are deprecated. Use packaging.version instead.
required_version = LooseVersion('0.29.14')
/tmp/pip-install-s5u3gw4k/numpy_c70991b77c1648a99494c52834ba182a/tools/cythonize.py:75: DeprecationWarning: distutils Version classes are deprecated. Use packaging.version instead.
if LooseVersion(cython_version) < required_version:

  Error compiling Cython file:
  ------------------------------------------------------------
  ...
              self.rng_state.ctr.v[i] = counter[i]

          self._reset_state_variables()

          self._bitgen.state = <void *>&self.rng_state
          self._bitgen.next_uint64 = &philox_uint64
                                     ^
  ------------------------------------------------------------

  _philox.pyx:195:35: Cannot assign type 'uint64_t (*)(void *) except? -1 nogil' to 'uint64_t (*)(void *) noexcept nogil'. Exception values are incompatible. Suggest adding 'noexcept' to type 'uint64_t (void *) except? -1 nogil'.
  Processing numpy/random/_bounded_integers.pxd.in
  Processing numpy/random/_philox.pyx
  Traceback (most recent call last):
    File "/tmp/pip-install-s5u3gw4k/numpy_c70991b77c1648a99494c52834ba182a/tools/cythonize.py", line 235, in <module>
      main()
    File "/tmp/pip-install-s5u3gw4k/numpy_c70991b77c1648a99494c52834ba182a/tools/cythonize.py", line 231, in main
      find_process_files(root_dir)
    File "/tmp/pip-install-s5u3gw4k/numpy_c70991b77c1648a99494c52834ba182a/tools/cythonize.py", line 222, in find_process_files
      process(root_dir, fromfile, tofile, function, hash_db)
    File "/tmp/pip-install-s5u3gw4k/numpy_c70991b77c1648a99494c52834ba182a/tools/cythonize.py", line 188, in process
      processor_function(fromfile, tofile)
    File "/tmp/pip-install-s5u3gw4k/numpy_c70991b77c1648a99494c52834ba182a/tools/cythonize.py", line 77, in process_pyx
      subprocess.check_call(
    File "/home/gpzhao/.conda/envs/dsr39/lib/python3.9/subprocess.py", line 373, in check_call
      raise CalledProcessError(retcode, cmd)
  subprocess.CalledProcessError: Command '['/home/gpzhao/.conda/envs/dsr39/bin/python', '-m', 'cython', '-3', '--fast-fail', '-o', '_philox.c', '_philox.pyx']' returned non-zero exit status 1.
  Cythonizing sources
  Traceback (most recent call last):
    File "/home/gpzhao/.conda/envs/dsr39/lib/python3.9/site-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py", line 353, in <module>
      main()
    File "/home/gpzhao/.conda/envs/dsr39/lib/python3.9/site-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py", line 335, in main
      json_out['return_val'] = hook(**hook_input['kwargs'])
    File "/home/gpzhao/.conda/envs/dsr39/lib/python3.9/site-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py", line 149, in prepare_metadata_for_build_wheel
      return hook(metadata_directory, config_settings)
    File "/tmp/pip-build-env-4tnhlnaz/overlay/lib/python3.9/site-packages/setuptools/build_meta.py", line 366, in prepare_metadata_for_build_wheel
      self.run_setup()
    File "/tmp/pip-build-env-4tnhlnaz/overlay/lib/python3.9/site-packages/setuptools/build_meta.py", line 480, in run_setup
      super(_BuildMetaLegacyBackend, self).run_setup(setup_script=setup_script)
    File "/tmp/pip-build-env-4tnhlnaz/overlay/lib/python3.9/site-packages/setuptools/build_meta.py", line 311, in run_setup
      exec(code, locals())
    File "<string>", line 489, in <module>
    File "<string>", line 469, in setup_package
    File "<string>", line 274, in generate_cython
  RuntimeError: Running cythonize failed!
  [end of output]

note: This error originates from a subprocess, and is likely not a problem with pip.
error: metadata-generation-failed

× Encountered error while generating package metadata.
╰─> See above for output.

note: This is an issue with the package mentioned above, not pip.
hint: See above for details.