Default source install instructions shows scary errors and seems to fail...

[iislucas]

(MacBook Pro M1 2021. MacOS-14.4 + Docker - 4.28.0 (139021))

Following: https://github.com/pair-code/lit?tab=readme-ov-file#install-from-source

(using venv)

$ python -m pip install -r requirements.txt
Collecting absl-py==1.4.0 (from -r requirements_core.txt (line 16))
  Obtaining dependency information for absl-py==1.4.0 from https://files.pythonhosted.org/packages/dd/87/de5c32fa1b1c6c3305d576e299801d8655c175ca9557019906247b994331/absl_py-1.4.0-py3-none-any.whl.metadata
  Downloading absl_py-1.4.0-py3-none-any.whl.metadata (2.3 kB)
Collecting annoy==1.17.3 (from -r requirements_core.txt (line 17))
  Downloading annoy-1.17.3.tar.gz (647 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 647.5/647.5 kB 4.9 MB/s eta 0:00:00
  Installing build dependencies ... done
  Getting requirements to build wheel ... done
  Installing backend dependencies ... done
  Preparing metadata (pyproject.toml) ... done
Collecting attrs==22.1.0 (from -r requirements_core.txt (line 18))
  Obtaining dependency information for attrs==22.1.0 from https://files.pythonhosted.org/packages/f2/bc/d817287d1aa01878af07c19505fafd1165cd6a119e9d0821ca1d1c20312d/attrs-22.1.0-py2.py3-none-any.whl.metadata
  Downloading attrs-22.1.0-py2.py3-none-any.whl.metadata (11 kB)
Collecting etils[epath]==1.3.0 (from -r requirements_core.txt (line 19))
  Obtaining dependency information for etils[epath]==1.3.0 from https://files.pythonhosted.org/packages/ca/db/47ffb866d7a1aa21132a72f67e84c4f03a4cad11ae9d069dd61c52f929de/etils-1.3.0-py3-none-any.whl.metadata
  Downloading etils-1.3.0-py3-none-any.whl.metadata (5.5 kB)
Collecting filelock==3.12.2 (from -r requirements_core.txt (line 20))
  Obtaining dependency information for filelock==3.12.2 from https://files.pythonhosted.org/packages/00/45/ec3407adf6f6b5bf867a4462b2b0af27597a26bd3cd6e2534cb6ab029938/filelock-3.12.2-py3-none-any.whl.metadata
  Downloading filelock-3.12.2-py3-none-any.whl.metadata (2.7 kB)
Collecting google-cloud-translate==3.11.1 (from -r requirements_core.txt (line 21))
  Obtaining dependency information for google-cloud-translate==3.11.1 from https://files.pythonhosted.org/packages/94/5a/7937935c66ea0360f81a1465f24e884ac6df3f9a9860940de10a06b13730/google_cloud_translate-3.11.1-py2.py3-none-any.whl.metadata
  Downloading google_cloud_translate-3.11.1-py2.py3-none-any.whl.metadata (5.1 kB)
Collecting ipython==8.14.0 (from -r requirements_core.txt (line 22))
  Obtaining dependency information for ipython==8.14.0 from https://files.pythonhosted.org/packages/52/d1/f70cdafba20030cbc1412d7a7d6a89c5035071835cc50e47fc5ed8da553c/ipython-8.14.0-py3-none-any.whl.metadata
  Downloading ipython-8.14.0-py3-none-any.whl.metadata (5.8 kB)
Collecting Levenshtein==0.21.1 (from -r requirements_core.txt (line 23))
  Downloading Levenshtein-0.21.1.tar.gz (139 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 139.2/139.2 kB 12.5 MB/s eta 0:00:00
  Installing build dependencies ... done
  Getting requirements to build wheel ... done
  Installing backend dependencies ... done
  Preparing metadata (pyproject.toml) ... done
Collecting matplotlib==3.6.1 (from -r requirements_core.txt (line 24))
  Downloading matplotlib-3.6.1.tar.gz (35.8 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 35.8/35.8 MB 9.8 MB/s eta 0:00:00
  Installing build dependencies ... done
  Getting requirements to build wheel ... done
  Installing backend dependencies ... done
  Preparing metadata (pyproject.toml) ... done
Collecting ml-collections==0.1.1 (from -r requirements_core.txt (line 25))
  Downloading ml_collections-0.1.1.tar.gz (77 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 77.9/77.9 kB 9.7 MB/s eta 0:00:00
  Installing build dependencies ... done
  Getting requirements to build wheel ... done
  Preparing metadata (pyproject.toml) ... done
Collecting numpy==1.24.1 (from -r requirements_core.txt (line 26))
  Downloading numpy-1.24.1.tar.gz (10.9 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 10.9/10.9 MB 17.3 MB/s eta 0:00:00
  Installing build dependencies ... done
  Getting requirements to build wheel ... done
ERROR: Exception:
Traceback (most recent call last):
  File "/Users/ldixon/code/lit/lit/.venv/lib/python3.12/site-packages/pip/_internal/cli/base_command.py", line 180, in exc_logging_wrapper
    status = run_func(*args)
             ^^^^^^^^^^^^^^^
  File "/Users/ldixon/code/lit/lit/.venv/lib/python3.12/site-packages/pip/_internal/cli/req_command.py", line 248, in wrapper
    return func(self, options, args)
           ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/ldixon/code/lit/lit/.venv/lib/python3.12/site-packages/pip/_internal/commands/install.py", line 377, in run
    requirement_set = resolver.resolve(
                      ^^^^^^^^^^^^^^^^^
  File "/Users/ldixon/code/lit/lit/.venv/lib/python3.12/site-packages/pip/_internal/resolution/resolvelib/resolver.py", line 92, in resolve
    result = self._result = resolver.resolve(
                            ^^^^^^^^^^^^^^^^^
  File "/Users/ldixon/code/lit/lit/.venv/lib/python3.12/site-packages/pip/_vendor/resolvelib/resolvers.py", line 546, in resolve
    state = resolution.resolve(requirements, max_rounds=max_rounds)
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/ldixon/code/lit/lit/.venv/lib/python3.12/site-packages/pip/_vendor/resolvelib/resolvers.py", line 397, in resolve
    self._add_to_criteria(self.state.criteria, r, parent=None)
  File "/Users/ldixon/code/lit/lit/.venv/lib/python3.12/site-packages/pip/_vendor/resolvelib/resolvers.py", line 173, in _add_to_criteria
    if not criterion.candidates:
           ^^^^^^^^^^^^^^^^^^^^
  File "/Users/ldixon/code/lit/lit/.venv/lib/python3.12/site-packages/pip/_vendor/resolvelib/structs.py", line 156, in __bool__
    return bool(self._sequence)
           ^^^^^^^^^^^^^^^^^^^^
  File "/Users/ldixon/code/lit/lit/.venv/lib/python3.12/site-packages/pip/_internal/resolution/resolvelib/found_candidates.py", line 155, in __bool__
    return any(self)
           ^^^^^^^^^
  File "/Users/ldixon/code/lit/lit/.venv/lib/python3.12/site-packages/pip/_internal/resolution/resolvelib/found_candidates.py", line 143, in <genexpr>
    return (c for c in iterator if id(c) not in self._incompatible_ids)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/ldixon/code/lit/lit/.venv/lib/python3.12/site-packages/pip/_internal/resolution/resolvelib/found_candidates.py", line 47, in _iter_built
    candidate = func()
                ^^^^^^
  File "/Users/ldixon/code/lit/lit/.venv/lib/python3.12/site-packages/pip/_internal/resolution/resolvelib/factory.py", line 206, in _make_candidate_from_link
    self._link_candidate_cache[link] = LinkCandidate(
                                       ^^^^^^^^^^^^^^
  File "/Users/ldixon/code/lit/lit/.venv/lib/python3.12/site-packages/pip/_internal/resolution/resolvelib/candidates.py", line 293, in __init__
    super().__init__(
  File "/Users/ldixon/code/lit/lit/.venv/lib/python3.12/site-packages/pip/_internal/resolution/resolvelib/candidates.py", line 156, in __init__
    self.dist = self._prepare()
                ^^^^^^^^^^^^^^^
  File "/Users/ldixon/code/lit/lit/.venv/lib/python3.12/site-packages/pip/_internal/resolution/resolvelib/candidates.py", line 225, in _prepare
    dist = self._prepare_distribution()
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/ldixon/code/lit/lit/.venv/lib/python3.12/site-packages/pip/_internal/resolution/resolvelib/candidates.py", line 304, in _prepare_distribution
    return preparer.prepare_linked_requirement(self._ireq, parallel_builds=True)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/ldixon/code/lit/lit/.venv/lib/python3.12/site-packages/pip/_internal/operations/prepare.py", line 538, in prepare_linked_requirement
    return self._prepare_linked_requirement(req, parallel_builds)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/ldixon/code/lit/lit/.venv/lib/python3.12/site-packages/pip/_internal/operations/prepare.py", line 653, in _prepare_linked_requirement
    dist = _get_prepared_distribution(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/ldixon/code/lit/lit/.venv/lib/python3.12/site-packages/pip/_internal/operations/prepare.py", line 69, in _get_prepared_distribution
    abstract_dist.prepare_distribution_metadata(
  File "/Users/ldixon/code/lit/lit/.venv/lib/python3.12/site-packages/pip/_internal/distributions/sdist.py", line 48, in prepare_distribution_metadata
    self._install_build_reqs(finder)
  File "/Users/ldixon/code/lit/lit/.venv/lib/python3.12/site-packages/pip/_internal/distributions/sdist.py", line 118, in _install_build_reqs
    build_reqs = self._get_build_requires_wheel()
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/ldixon/code/lit/lit/.venv/lib/python3.12/site-packages/pip/_internal/distributions/sdist.py", line 95, in _get_build_requires_wheel
    return backend.get_requires_for_build_wheel()
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/ldixon/code/lit/lit/.venv/lib/python3.12/site-packages/pip/_internal/utils/misc.py", line 697, in get_requires_for_build_wheel
    return super().get_requires_for_build_wheel(config_settings=cs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/ldixon/code/lit/lit/.venv/lib/python3.12/site-packages/pip/_vendor/pyproject_hooks/_impl.py", line 166, in get_requires_for_build_wheel
    return self._call_hook('get_requires_for_build_wheel', {
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/ldixon/code/lit/lit/.venv/lib/python3.12/site-packages/pip/_vendor/pyproject_hooks/_impl.py", line 321, in _call_hook
    raise BackendUnavailable(data.get('traceback', ''))
pip._vendor.pyproject_hooks._impl.BackendUnavailable: Traceback (most recent call last):
  File "/Users/ldixon/code/lit/lit/.venv/lib/python3.12/site-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py", line 77, in _build_backend
    obj = import_module(mod_path)
          ^^^^^^^^^^^^^^^^^^^^^^^
  File "/Library/Frameworks/Python.framework/Versions/3.12/lib/python3.12/importlib/__init__.py", line 90, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "<frozen importlib._bootstrap>", line 1387, in _gcd_import
  File "<frozen importlib._bootstrap>", line 1360, in _find_and_load
  File "<frozen importlib._bootstrap>", line 1310, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 488, in _call_with_frames_removed
  File "<frozen importlib._bootstrap>", line 1387, in _gcd_import
  File "<frozen importlib._bootstrap>", line 1360, in _find_and_load
  File "<frozen importlib._bootstrap>", line 1331, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 935, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 994, in exec_module
  File "<frozen importlib._bootstrap>", line 488, in _call_with_frames_removed
  File "/private/var/folders/jw/fn8nkzrd3zbf7qccf4kq90nr003zq5/T/pip-build-env-uug8d1_4/overlay/lib/python3.12/site-packages/setuptools/__init__.py", line 10, in <module>
    import distutils.core
ModuleNotFoundError: No module named 'distutils'

[RyanMullins]

@iislucas I think that the dependency changes in #1364 will address the Apple Silicon incompatibilities, but I still need to verify that.

[olympus-terminal]

I am also getting very strange errors that I cannot address (for example, below, the requested packages were already installed and verified to be installed and working, yet still errors):

21.78 ERROR: Ignored the following versions that require a different python version: 1.6.2 Requires-Python >=3.7,<3.10; 1.6.3 Requires-Python >=3.7,<3.10; 1.7.0 Requires-Python >=3.7,<3.10; 1.7.1 Requires-Python >=3.7,<3.10; 1.8.0 Requires-Python >=3.11; 1.9.0 Requires-Python >=3.11; 1.9.1 Requires-Python >=3.11; 1.9.2 Requires-Python >=3.11
21.78 ERROR: Could not find a version that satisfies the requirement tensorflow-text<2.16.0,>=2.10.0 (from versions: none)
21.78 ERROR: No matching distribution found for tensorflow-text<2.16.0,>=2.10.0
21.89
21.89 [notice] A new release of pip is available: 23.0.1 -> 24.1.1
21.89 [notice] To update, run: pip install --upgrade pip

Dockerfile:36

34 | # This step is slow as it installs many packages.
35 | COPY ./requirements*.txt ./
36 | >>> RUN python -m pip install -r requirements.txt
37 |
38 | # Copy the rest of the lit_nlp package

ERROR: failed to solve: process "/bin/sh -c python -m pip install -r requirements.txt" did not complete successfully: exit code: 1
(lit_env) drn2@ADUAED16547LPMX lit % pip install --upgrade pip
Requirement already satisfied: pip in /Users/drn2/miniconda3/envs/lit_env/lib/python3.10/site-packages (24.1.1)
(lit_env) drn2@ADUAED16547LPMX lit % pip install 'tensorflow-text<2.16.0,>=2.10.0'