Another error with pip install

[mkleinbort]

Hello,

When trying to run pip install fbprophet (in a python3.8 docker container) it tells me the convertdate module is not installed.

When I install the convertdate module it tells me the lunarcalendar module is not installed, and if I install convertdate and lunarcalendar it then tells me holidays is not installed.

Thus I need to do

pip install convertdate 
pip install lunarcalendar
pip install holidays

and then pip install fbprophet

(still not working, figuring out pystan)

[bletham]

All of the dependencies for fbprophet are build dependencies, meaning they have to be installed before fbprophet can be installed. Usually if you are installing with pip and don't have the dependencies installed, it will first try to build a wheel, that will fail because the build dependencies are not in place, but then it will install the dependencies and install from source which succeeds. See #401 for lengthy discussion of this. It sounds like for you it is just exiting after the failure to build the wheel? What version of pip are you using?

You can get all the dependencies with

pip install -r requirements.txt

for this requirements.txt: https://github.com/facebook/prophet/blob/master/python/requirements.txt

but like I said I expect pip to install the dependencies itself, so am surprised to hear it isn't.

[sarusso]

@bletham it seems like there is a problem when using requirements to build the wheel itself (see the following Dockerfile for how to reproduce it).

@mkleinbort this Dockerfile works:

FROM ubuntu:18.04

# Set non-interactive
ENV DEBIAN_FRONTEND noninteractive

# Update apt cache
RUN apt-get update

# Curl
RUN apt-get install curl -y

# Install get-pip script
RUN curl -O https://bootstrap.pypa.io/get-pip.py

# Install Python3 and Pip3, force pip to 20.2 to improve reproducibility
RUN apt-get install python3 python3-distutils -y 
RUN python3 get-pip.py && pip install -I pip==20.2

# Install dev & build packages
RUN apt-get install python3-dev build-essential -y

# Install deps now to avoid misleading error messages which are actually non-errors at the next step
RUN pip3 install convertdate==2.1.2 lunarcalendar==0.0.9 holidays==0.10.3 cython==0.29.21 pystan==2.19.1.1 pandas==0.23.4 

# Install/build Prophet
RUN pip3 install fbprophet==0.6

[bletham]

It makes sense to me that it would fail when trying to build the wheel, what's surprising to me is that the install then exits with an error. When I do pip install fbprophet with a missing dependency (here I uninstalled lunarcalendar) I get this:

Collecting fbprophet
  Using cached fbprophet-0.6.tar.gz (54 kB)
Requirement already satisfied: Cython>=0.22 in /usr/local/lib64/python3.7/site-packages (from fbprophet) (0.29.7)
Collecting cmdstanpy==0.4
  Using cached cmdstanpy-0.4.0-py3-none-any.whl (22 kB)
Requirement already satisfied: pystan>=2.14 in /usr/local/lib64/python3.7/site-packages (from fbprophet) (2.19.1.1)
Requirement already satisfied: numpy>=1.10.0 in /usr/local/lib64/python3.7/site-packages (from fbprophet) (1.18.2)
Requirement already satisfied: pandas>=0.23.4 in /usr/local/lib64/python3.7/site-packages (from fbprophet) (0.25.3)
Requirement already satisfied: matplotlib>=2.0.0 in /usr/local/lib64/python3.7/site-packages (from fbprophet) (3.0.3)
Collecting LunarCalendar>=0.0.9
  Downloading LunarCalendar-0.0.9-py2.py3-none-any.whl (18 kB)
Requirement already satisfied: convertdate>=2.1.2 in /usr/local/lib/python3.7/site-packages (from fbprophet) (2.1.3)
Requirement already satisfied: holidays>=0.9.5 in /usr/local/lib/python3.7/site-packages (from fbprophet) (0.10.2)
Requirement already satisfied: setuptools-git>=1.2 in /usr/local/lib/python3.7/site-packages (from fbprophet) (1.2)
Requirement already satisfied: python-dateutil>=2.8.0 in /usr/lib/python3.7/site-packages (from fbprophet) (2.8.0)
Requirement already satisfied: pytz>=2017.2 in /usr/lib/python3.7/site-packages (from pandas>=0.23.4->fbprophet) (2019.2)
Requirement already satisfied: pyparsing!=2.0.4,!=2.1.2,!=2.1.6,>=2.0.1 in /usr/lib/python3.7/site-packages (from matplotlib>=2.0.0->fbprophet) (2.4.0)
Requirement already satisfied: kiwisolver>=1.0.1 in /usr/local/lib64/python3.7/site-packages (from matplotlib>=2.0.0->fbprophet) (1.0.1)
Requirement already satisfied: cycler>=0.10 in /usr/local/lib/python3.7/site-packages (from matplotlib>=2.0.0->fbprophet) (0.10.0)
Requirement already satisfied: ephem>=3.7.5.3 in /usr/local/lib64/python3.7/site-packages (from LunarCalendar>=0.0.9->fbprophet) (3.7.6.0)
Requirement already satisfied: six in /usr/lib/python3.7/site-packages (from holidays>=0.9.5->fbprophet) (1.12.0)
Requirement already satisfied: korean-lunar-calendar in /usr/local/lib/python3.7/site-packages (from holidays>=0.9.5->fbprophet) (0.2.1)
Requirement already satisfied: setuptools in /usr/lib/python3.7/site-packages (from kiwisolver>=1.0.1->matplotlib>=2.0.0->fbprophet) (41.6.0)
Building wheels for collected packages: fbprophet
  Building wheel for fbprophet (setup.py) ... error
  ERROR: Command errored out with exit status 1:
   command: /usr/bin/python3 -u -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'/tmp/pip-install-pddsa_fi/fbprophet/setup.py'"'"'; __file__='"'"'/tmp/pip-install-pddsa_fi/fbprophet/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' bdist_wheel -d /tmp/pip-wheel-z_vuywmz
       cwd: /tmp/pip-install-pddsa_fi/fbprophet/
  Complete output (44 lines):
  running bdist_wheel
  running build
  running build_py
  creating build
  creating build/lib
  creating build/lib/fbprophet
  creating build/lib/fbprophet/stan_model
  Traceback (most recent call last):
    File "<string>", line 1, in <module>
    File "/tmp/pip-install-pddsa_fi/fbprophet/setup.py", line 148, in <module>
      """
    File "/usr/lib/python3.7/site-packages/setuptools/__init__.py", line 145, in setup
      return distutils.core.setup(**attrs)
    File "/usr/lib64/python3.7/distutils/core.py", line 148, in setup
      dist.run_commands()
    File "/usr/lib64/python3.7/distutils/dist.py", line 966, in run_commands
      self.run_command(cmd)
    File "/usr/lib64/python3.7/distutils/dist.py", line 985, in run_command
      cmd_obj.run()
    File "/usr/local/lib/python3.7/site-packages/wheel/bdist_wheel.py", line 223, in run
      self.run_command('build')
    File "/usr/lib64/python3.7/distutils/cmd.py", line 313, in run_command
      self.distribution.run_command(command)
    File "/usr/lib64/python3.7/distutils/dist.py", line 985, in run_command
      cmd_obj.run()
    File "/usr/lib64/python3.7/distutils/command/build.py", line 135, in run
      self.run_command(cmd_name)
    File "/usr/lib64/python3.7/distutils/cmd.py", line 313, in run_command
      self.distribution.run_command(command)
    File "/usr/lib64/python3.7/distutils/dist.py", line 985, in run_command
      cmd_obj.run()
    File "/tmp/pip-install-pddsa_fi/fbprophet/setup.py", line 48, in run
      build_models(target_dir)
    File "/tmp/pip-install-pddsa_fi/fbprophet/setup.py", line 36, in build_models
      from fbprophet.models import StanBackendEnum
    File "/tmp/pip-install-pddsa_fi/fbprophet/fbprophet/__init__.py", line 8, in <module>
      from fbprophet.forecaster import Prophet
    File "/tmp/pip-install-pddsa_fi/fbprophet/fbprophet/forecaster.py", line 17, in <module>
      from fbprophet.make_holidays import get_holiday_names, make_holidays_df
    File "/tmp/pip-install-pddsa_fi/fbprophet/fbprophet/make_holidays.py", line 14, in <module>
      import fbprophet.hdays as hdays_part2
    File "/tmp/pip-install-pddsa_fi/fbprophet/fbprophet/hdays.py", line 14, in <module>
      from lunarcalendar import Lunar, Converter
  ModuleNotFoundError: No module named 'lunarcalendar'
  ----------------------------------------
  ERROR: Failed building wheel for fbprophet
  Running setup.py clean for fbprophet
Failed to build fbprophet
Installing collected packages: cmdstanpy, LunarCalendar, fbprophet
  Attempting uninstall: cmdstanpy
    Found existing installation: cmdstanpy 0.9.5
    Uninstalling cmdstanpy-0.9.5:
      Successfully uninstalled cmdstanpy-0.9.5
    Running setup.py install for fbprophet ... done
Successfully installed LunarCalendar-0.0.9 cmdstanpy-0.4.0 fbprophet-0.6

So you can see that it is failing to build the wheel, but then after it fails to build the wheel it installs the missing dependencies (lunarcalendar and cmdstanpy in this case) and then successfully installs. Is this different behavior something with different versions of pip I wonder? I used pip 20.1.1 above.

[sarusso]

You are right, it should not happen, ad indeed it is not (I edited my Dockerfile above accordingly).

As far as I understood here is what happens:

1. You do pip install of the dependencies and then you do another pip install of propfhet: it builds the wheel and there are no errors at all.

2. You do a pip install of the dependencies plus fbprophet altogether (or just of fbprophet, which will in any case trigger the install of all the dependencies within the same install session): it will give first an error about a missing dependency (with a lot of ugly red output on the terminal, as you reported in your previous comment), but then it will try again to build (?) and successfully build the wheel. Pretty confusing as a user.

3. you do 1) or 2) but without dev tools installed (i.e. on Ubuntu python3-dev and build-essential, in this case it will just not build at all.

Note that the missing dependency of point 2) can be anything, depending of what it is already installed or not, I personally saw errors about missing convertdate, lunarcalendar, pandas, numpy..

...so I guess that the OP experienced step 2), he saw one of the errors of a missing build deps (i.e. convertdate as he reported in first instance, then either:

A) he stopped the pip install as soon as he seen the first error about the missing dependency (as I did!), without letting the wheel to be correctly built in the "second try", or:

B) he got the error about the missing dependency, he did not stop the process, but then the missing dev tools error piled up and even the "second try" of building the wheel failed (command 'x86_64-linux-gnu-gcc' failed with exit status 1)

I have to say that this is a pretty messy situation to navigate, is there anything that you can do on your side of the install/build process to help the user understand what is going on...?

Best,
Stefano.

[bletham]

@sarusso thanks for the summary, that all seems correct to me. I agree it's really non-ideal and seeing error messages during install can be really confusing, even if it does ultimately install correctly. We've tried a few things to resolve this but it's pretty complicated. I don't see a clean way to have it build the wheel without pystan at the least. We did try to have it install the build dependencies before trying to build the wheel (with a pyproject.toml file), and #401 describes why that ended up not being an option. Hopefully we can work out a clean solution for this at some point.

[ttaahhaa]

use these commands to in docker to install fbprohet

USER root
RUN python -m pip install --upgrade pip
RUN pip install --upgrade setuptools

fbprophet

RUN apt-get update && apt-get install build-essential unixodbc-dev -y
RUN pip3 install convertdate==2.1.2 lunarcalendar==0.0.9 holidays==0.10.3 cython==0.29.21 pystan==2.19.1.1 pandas
RUN pip3 install fbprophet==0.6