make prophet, lightgbm and catboost optional #1589

hrzn · 2023-02-22T16:23:04Z

Now that I have access to an M1 Mac, I was able to run more tests.

This is up for discussion, but I propose the following:

We make prophet optional again, and not included anymore in the darts package. That's because it's just too painful to install and I'm not even sure it can always be done on Apple silicon without using x64 emulation, which is not satisfying. Having it as a hard dependency is too much troubles for many users I suspect...
Adapt the INSTALL guide to simplify things for LightGBM and Apple Silicon, and add a section explaining that now prophet has to be installed separately.

Edit (@dennisbader, 2023-08-03):
Removed Prophet, LightGBM, and CatBoost dependencies from PyPI packages (darts, u8darts, u8darts[torch]), and conda-forge packages (u8darts, u8darts-torch) to avoid installation issues that some users were facing (installation on Apple M1/M2 devices, ...).

The models are still supported by installing the required packages as described in our installation guide.
PyPi package u8darts[all] and conda-forge package u8darts-all are now equivalent to the old darts package (all dependencies).
Added new PyPI flavor u8darts[notorch], and conda-forge flavor u8darts-notorch which are equivalent to the old u8darts installation.

Eliotdoesprogramming · 2023-02-22T22:24:21Z

@hrzn

Just wanted to comment to add my own user experience on a M1 Macbook (2020 macbook pro) if its all helpful to you:

I have no issues installing prophet on my m1, even on arm64 architecture. I did however have issues with LightGBM installation through pip, but was easily fixed with an install through conda forge (all conda environments in this example, not virtualenv)

I did have issues with installing prophet on windows x86_64 architecture, but solved by using WSL.

hrzn · 2023-02-23T08:27:46Z

@hrzn

Just wanted to comment to add my own user experience on a M1 Macbook (2020 macbook pro) if its all helpful to you:

I have no issues installing prophet on my m1, even on arm64 architecture. I did however have issues with LightGBM installation through pip, but was easily fixed with an install through conda forge (all conda environments in this example, not virtualenv)

I did have issues with installing prophet on windows x86_64 architecture, but solved by using WSL.

Thanks for sharing. I'm now wondering if I did anything wrong when trying to install Prophet. I tried a couple of ways, including different versions of Python, installing pystan or cmdstan first etc. How are you installing it?

For the record, here's what I get when I attempt pip install prophet:

      --- CmdStan v2.26.1 built ---
      Test model compilation
      cmd: make examples/bernoulli/bernoulli
      cwd: None
      09:25:56 - cmdstanpy - WARNING - CmdStan installation failed.
      Command "make clean-all" failed
      Command ['make', 'examples/bernoulli/bernoulli']
          error during processing No such file or directory
      CmdStan installation failed.
      Command "make clean-all" failed
      Command ['make', 'examples/bernoulli/bernoulli']
          error during processing No such file or directory
      clang++ -std=c++1y -Wno-unknown-warning-option -Wno-tautological-compare -Wno-sign-compare -D_REENTRANT -Wno-ignored-attributes      -I stan/lib/stan_math/lib/tbb_2019_U8/include   -O3 -I src -I stan/src -I lib/rapidjson_1.1.0/ -I lib/CLI11-1.9.1/ -I stan/lib/stan_math/ -I stan/lib/stan_math/lib/eigen_3.3.9 -I stan/lib/stan_math/lib/boost_1.72.0 -I stan/lib/stan_math/lib/sundials_5.6.1/include    -DBOOST_DISABLE_ASSERTS         -c -MT stan/src/stan/model/model_header.hpp.gch -MT stan/src/stan/model/model_header.d -MM -E -MG -MP -MF stan/src/stan/model/model_header.d stan/src/stan/model/model_header.hpp
      clang++ -std=c++1y -Wno-unknown-warning-option -Wno-tautological-compare -Wno-sign-compare -D_REENTRANT -Wno-ignored-attributes      -I stan/lib/stan_math/lib/tbb_2019_U8/include   -O3 -I src -I stan/src -I lib/rapidjson_1.1.0/ -I lib/CLI11-1.9.1/ -I stan/lib/stan_math/ -I stan/lib/stan_math/lib/eigen_3.3.9 -I stan/lib/stan_math/lib/boost_1.72.0 -I stan/lib/stan_math/lib/sundials_5.6.1/include    -DBOOST_DISABLE_ASSERTS         -c -MT src/cmdstan/main.o -MM -E -MG -MP -MF src/cmdstan/main.d src/cmdstan/main.cpp

      --- Translating Stan model to C++ code ---
      bin/stanc  --o=examples/bernoulli/bernoulli.hpp examples/bernoulli/bernoulli.stan
      make: bin/stanc: Bad CPU type in executable
      make: *** [examples/bernoulli/bernoulli.hpp] Error 1
      Traceback (most recent call last):
        File "/Users/julien/miniconda3/envs/darts/lib/python3.10/site-packages/pip/_vendor/pep517/in_process/_in_process.py", line 351, in <module>
          main()
        File "/Users/julien/miniconda3/envs/darts/lib/python3.10/site-packages/pip/_vendor/pep517/in_process/_in_process.py", line 333, in main
          json_out['return_val'] = hook(**hook_input['kwargs'])
        File "/Users/julien/miniconda3/envs/darts/lib/python3.10/site-packages/pip/_vendor/pep517/in_process/_in_process.py", line 249, in build_wheel
          return _build_backend().build_wheel(wheel_directory, config_settings,
        File "/private/var/folders/c3/0hfh1r6j2t944n9kwd8mg5c00000gn/T/pip-build-env-mr7_c1ya/overlay/lib/python3.10/site-packages/setuptools/build_meta.py", line 413, in build_wheel
          return self._build_with_temp_dir(['bdist_wheel'], '.whl',
        File "/private/var/folders/c3/0hfh1r6j2t944n9kwd8mg5c00000gn/T/pip-build-env-mr7_c1ya/overlay/lib/python3.10/site-packages/setuptools/build_meta.py", line 398, in _build_with_temp_dir
          self.run_setup()
        File "/private/var/folders/c3/0hfh1r6j2t944n9kwd8mg5c00000gn/T/pip-build-env-mr7_c1ya/overlay/lib/python3.10/site-packages/setuptools/build_meta.py", line 335, in run_setup
          exec(code, locals())
        File "<string>", line 183, in <module>
        File "/private/var/folders/c3/0hfh1r6j2t944n9kwd8mg5c00000gn/T/pip-build-env-mr7_c1ya/overlay/lib/python3.10/site-packages/setuptools/__init__.py", line 108, in setup
          return distutils.core.setup(**attrs)
        File "/private/var/folders/c3/0hfh1r6j2t944n9kwd8mg5c00000gn/T/pip-build-env-mr7_c1ya/overlay/lib/python3.10/site-packages/setuptools/_distutils/core.py", line 185, in setup
          return run_commands(dist)
        File "/private/var/folders/c3/0hfh1r6j2t944n9kwd8mg5c00000gn/T/pip-build-env-mr7_c1ya/overlay/lib/python3.10/site-packages/setuptools/_distutils/core.py", line 201, in run_commands
          dist.run_commands()
        File "/private/var/folders/c3/0hfh1r6j2t944n9kwd8mg5c00000gn/T/pip-build-env-mr7_c1ya/overlay/lib/python3.10/site-packages/setuptools/_distutils/dist.py", line 969, in run_commands
          self.run_command(cmd)
        File "/private/var/folders/c3/0hfh1r6j2t944n9kwd8mg5c00000gn/T/pip-build-env-mr7_c1ya/overlay/lib/python3.10/site-packages/setuptools/dist.py", line 1221, in run_command
          super().run_command(command)
        File "/private/var/folders/c3/0hfh1r6j2t944n9kwd8mg5c00000gn/T/pip-build-env-mr7_c1ya/overlay/lib/python3.10/site-packages/setuptools/_distutils/dist.py", line 988, in run_command
          cmd_obj.run()
        File "/private/var/folders/c3/0hfh1r6j2t944n9kwd8mg5c00000gn/T/pip-build-env-mr7_c1ya/overlay/lib/python3.10/site-packages/wheel/bdist_wheel.py", line 325, in run
          self.run_command("build")
        File "/private/var/folders/c3/0hfh1r6j2t944n9kwd8mg5c00000gn/T/pip-build-env-mr7_c1ya/overlay/lib/python3.10/site-packages/setuptools/_distutils/cmd.py", line 318, in run_command
          self.distribution.run_command(command)
        File "/private/var/folders/c3/0hfh1r6j2t944n9kwd8mg5c00000gn/T/pip-build-env-mr7_c1ya/overlay/lib/python3.10/site-packages/setuptools/dist.py", line 1221, in run_command
          super().run_command(command)
        File "/private/var/folders/c3/0hfh1r6j2t944n9kwd8mg5c00000gn/T/pip-build-env-mr7_c1ya/overlay/lib/python3.10/site-packages/setuptools/_distutils/dist.py", line 988, in run_command
          cmd_obj.run()
        File "/private/var/folders/c3/0hfh1r6j2t944n9kwd8mg5c00000gn/T/pip-build-env-mr7_c1ya/overlay/lib/python3.10/site-packages/setuptools/_distutils/command/build.py", line 131, in run
          self.run_command(cmd_name)
        File "/private/var/folders/c3/0hfh1r6j2t944n9kwd8mg5c00000gn/T/pip-build-env-mr7_c1ya/overlay/lib/python3.10/site-packages/setuptools/_distutils/cmd.py", line 318, in run_command
          self.distribution.run_command(command)
        File "/private/var/folders/c3/0hfh1r6j2t944n9kwd8mg5c00000gn/T/pip-build-env-mr7_c1ya/overlay/lib/python3.10/site-packages/setuptools/dist.py", line 1221, in run_command
          super().run_command(command)
        File "/private/var/folders/c3/0hfh1r6j2t944n9kwd8mg5c00000gn/T/pip-build-env-mr7_c1ya/overlay/lib/python3.10/site-packages/setuptools/_distutils/dist.py", line 988, in run_command
          cmd_obj.run()
        File "<string>", line 144, in run
        File "<string>", line 131, in build_models
        File "<string>", line 110, in build_cmdstan_model
        File "<string>", line 94, in install_cmdstan_deps
      RuntimeError: CmdStan failed to install in repackaged directory
      [end of output]

  note: This error originates from a subprocess, and is likely not a problem with pip.
  ERROR: Failed building wheel for prophet
Failed to build prophet
ERROR: Could not build wheels for prophet, which is required to install pyproject.toml-based projects

Eliotdoesprogramming · 2023-02-24T19:09:40Z

honestly its just a normal pip install for me. here was my process:

conda create -n install_demo python=3.9 -c conda-forge

pip install prophet --no-cache-dir # no cache for clean install

brew list

the one thing I can think of off the top of my head is I've installed mamba to make some of the package management faster
hope this helps!

martinb-ai · 2023-02-28T00:08:30Z

@hrzn Could we make lightgbm optional when installing darts pytorch? It is a pain to deal with on Macs 👎

Seems it always bundles together even if I don't use the lib.

eshwen · 2023-03-07T11:30:29Z

I've been struggling with this recently. But I think I've found a way to install both LightGBM and prophet without resorting to emulation.

As @Eliotdoesprogramming suggested, install LightGBM with conda:

conda install -c conda-forge 'lightgbm>=3.3.3'

As for prophet, I installed CmdStan through conda, then installed the Python client and rebuilt it:

conda install -c conda-forge cmdstan
pip install cmdstanpy
python -c "from cmdstanpy import rebuild_cmdstan; rebuild_cmdstan()"
export PROPHET_REPACKAGE_CMDSTAN=False

Then I was able to just pip install darts. My full list of instructions can be found here, and I'm happy to open a PR to fold them into INSTALL.md

madtoinou · 2023-03-23T10:06:10Z

Hi @eshwen,

Sorry for the delay and thank you for sharing your solution!

Could you please check if your approach still works after the release of PyTorch 2.0? If, a PR adding these instructions to the documentation would be very welcome!

eshwen · 2023-04-03T09:48:08Z

Hi @eshwen,

Sorry for the delay and thank you for sharing your solution!

Could you please check if your approach still works after the release of PyTorch 2.0? If, a PR adding these instructions to the documentation would be very welcome!

No problem. I've briefly tested, and it does still work with PyTorch 2.0. In fact, it doesn't appear to need the cmdstanpy block anymore. When I have some free time, I'll open a PR to update the documentation

TueLindhart · 2023-05-01T12:25:02Z

Not sure if this is the correct forum, but I found a way to install prophet on MacOS 13.2.1 with M2 processor.

pip install cmdstanpy
Use cmdstanpy to install CMDStan either by python or terminal. (See here)
In terminal set export PROPHET_REPACKAGE_CMDSTAN=False. It is the repackaging which makes it fail.
pip install prophet

I do not know if prophet actually works properly as I am not interesting in using it. However, I am forced to install prophet=1.1.1 to be able to use the newest darts features.

ale-dg · 2023-05-15T16:31:15Z

Hi,

As a fellow M1 Mac user, I have found that most of the packages used in Darts fail with "used" or "old" environments (at least Conda environments). Therefore I'd suggest to create a new environment and install all the dependencies first before installing darts. It goes even smoother.

As for the specific case of Prophet, I have raised a couple of issues in their repo because the cmdstan shouldn't try to build the wheel locally since the pip command should be retrieving the built wheel. You can see it here facebook/prophet#2402. They have found the issue on Windows but still missing the solution for macOS.

On a second note, as for now, be sure to install prophet's version 1.1.1 since the newest one has an issue when trying to fit some data. Hence the suggestion to first install first Darts' external dependencies (i.e. PyTorch, Prophet, LightGBM, etc) and then Darts in whatever package you want.

Best

dennisbader · 2023-08-02T07:58:27Z

I will pick this up and additionally exclude lightgbm (there are issues for many users installing on M1) and catboost (catboost>=1.2.0 installation requires a lot of additional build tools (cmake, ninja, ...) which we should avoid forcing on the user)

codecov-commenter · 2023-08-02T15:15:20Z

Codecov Report

Patch coverage: 38.09% and project coverage change: +0.16% 🎉

Comparison is base (137cf8c) 93.72% compared to head (e0cd981) 93.88%.

❗ Current head e0cd981 differs from pull request most recent head eebd227. Consider uploading reports for the commit eebd227 to get more accurate results

❗ Your organization is not using the GitHub App Integration. As a result you may experience degraded service beginning May 15th. Please install the Github App Integration for your organization. Read more.

Additional details and impacted files

@@            Coverage Diff             @@
##           master    #1589      +/-   ##
==========================================
+ Coverage   93.72%   93.88%   +0.16%     
==========================================
  Files         131      132       +1     
  Lines       12662    12639      -23     
==========================================
- Hits        11867    11866       -1     
+ Misses        795      773      -22

Files Changed	Coverage Δ
darts/models/__init__.py	`73.01% <10.00%> (+15.32%)`	⬆️
darts/models/utils.py	`60.00% <60.00%> (ø)`
darts/models/forecasting/regression_model.py	`97.02% <100.00%> (+1.78%)`	⬆️

... and 7 files with indirect coverage changes

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

darts/models/utils.py

madtoinou

I really like the way the optional dependencies are handled, just spotted a small typo.

madtoinou

LGTM 🚀

With this, XGBoost becomes the most extensively tested model of the library if the optional dependencies are not installed...

madtoinou · 2023-08-03T09:40:05Z

INSTALL.md

-* Install darts with all available models (recommended): `conda install -c conda-forge -c pytorch u8darts-all`.
+* Install darts with all available models: `conda install -c conda-forge -c pytorch u8darts-all`
+* Install core only (without neural networks, Prophet, LightGBM and Catboost): `conda install -c conda-forge u8darts`
+* Install core + Prophet + LightGBM + CatBoost: `pip install "u8darts[notorch]"`


to be coherent with the others, should this flavor of the package be called udarts-notorch?

madtoinou · 2023-08-03T09:41:20Z

INSTALL.md

 softwareupdate --install-rosetta
 ```

 Below are the necessary instructions to create and configure the environment:
- Start by installing conda (e.g., with miniforge : `brew install miniforge`).
+- Install conda if you haven't done so (e.g., with miniforge : `brew install miniforge`).
 - Create the x_64 environment : `CONDA_SUBDIR=osx-64 conda create -n env_name python=3.9 pip`


update this to 3.10 to be coherent with the instruction above?

make prophet optional again, adapt install readme

3c2c40a

hrzn requested a review from dennisbader as a code owner February 22, 2023 16:23

Merge branch 'master' into feat/improve-install

871a2b6

adapt u8darts setup as well

cd717d7

dennisbader mentioned this pull request Apr 27, 2023

pip install u8darts tries to install prophet #1728

Closed

dennisbader added 4 commits August 2, 2023 10:03

Merge branch 'master' into feat/improve-install

66c4ec3

remove prophet, lightgbm, catboost for core requirements

8f2c1b4

simplify the module importing

6b38ce9

update tests with new optional dependencies

17904a9

dennisbader changed the title ~~make prophet optional again, adapt install readme~~ make prophet, lightgbm and catboost optional Aug 2, 2023

update gradle for including optional dependencies in unittest_all

51efc4f

madtoinou reviewed Aug 3, 2023

View reviewed changes

darts/models/utils.py Outdated Show resolved Hide resolved

madtoinou approved these changes Aug 3, 2023

View reviewed changes

dennisbader added 4 commits August 3, 2023 10:15

update checkExamples to show all installed dependencies before test

7cc0b19

apply suggestions from PR review

be35339

update install guide

1cafa79

update shap unit tests

e0cd981

madtoinou approved these changes Aug 3, 2023

View reviewed changes

dennisbader added 2 commits August 3, 2023 11:51

apply suggestions from PR review

eebd227

update changelog

a1ce78d

madtoinou approved these changes Aug 3, 2023

View reviewed changes

dennisbader merged commit ecacd93 into master Aug 3, 2023

dennisbader deleted the feat/improve-install branch August 3, 2023 11:33

dennisbader mentioned this pull request Aug 1, 2023

Relax dependency caps once fixed #1772

Closed

3 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

make prophet, lightgbm and catboost optional #1589

make prophet, lightgbm and catboost optional #1589

hrzn commented Feb 22, 2023 •

edited by dennisbader

Loading

Eliotdoesprogramming commented Feb 22, 2023

hrzn commented Feb 23, 2023

Eliotdoesprogramming commented Feb 24, 2023 •

edited

Loading

martinb-ai commented Feb 28, 2023 •

edited

Loading

eshwen commented Mar 7, 2023

madtoinou commented Mar 23, 2023

eshwen commented Apr 3, 2023

TueLindhart commented May 1, 2023

ale-dg commented May 15, 2023 •

edited

Loading

dennisbader commented Aug 2, 2023

codecov-commenter commented Aug 2, 2023 •

edited

Loading

madtoinou left a comment

madtoinou left a comment

madtoinou Aug 3, 2023

dennisbader Aug 3, 2023

madtoinou Aug 3, 2023

dennisbader Aug 3, 2023

make prophet, lightgbm and catboost optional #1589

make prophet, lightgbm and catboost optional #1589

Conversation

hrzn commented Feb 22, 2023 • edited by dennisbader Loading

Eliotdoesprogramming commented Feb 22, 2023

hrzn commented Feb 23, 2023

Eliotdoesprogramming commented Feb 24, 2023 • edited Loading

martinb-ai commented Feb 28, 2023 • edited Loading

eshwen commented Mar 7, 2023

madtoinou commented Mar 23, 2023

eshwen commented Apr 3, 2023

TueLindhart commented May 1, 2023

ale-dg commented May 15, 2023 • edited Loading

dennisbader commented Aug 2, 2023

codecov-commenter commented Aug 2, 2023 • edited Loading

Codecov Report

madtoinou left a comment

Choose a reason for hiding this comment

madtoinou left a comment

Choose a reason for hiding this comment

madtoinou Aug 3, 2023

Choose a reason for hiding this comment

dennisbader Aug 3, 2023

Choose a reason for hiding this comment

madtoinou Aug 3, 2023

Choose a reason for hiding this comment

dennisbader Aug 3, 2023

Choose a reason for hiding this comment

hrzn commented Feb 22, 2023 •

edited by dennisbader

Loading

Eliotdoesprogramming commented Feb 24, 2023 •

edited

Loading

martinb-ai commented Feb 28, 2023 •

edited

Loading

ale-dg commented May 15, 2023 •

edited

Loading

codecov-commenter commented Aug 2, 2023 •

edited

Loading