[Draft] Python Wheels for PyPi #2010

tcuongd · 2021-08-28T16:59:18Z

Having a go at creating wheels for the Python package with the aim of publishing to PyPi. We're trying to create a wheel that has both pystan and cmdstanpy models compiled, so the end user just has to install the wheel and prophet should work out of the box.

Tasks

Install cmdstan as a part of setup.py, and include the core cmdstan files in the prophet distribution as well. This allows end users to fit models on the CMDSTANPY backend without an existing installation of cmdstan, and ensures that we the model is fit using the same version of cmdstan that compiled it.
Use pyproject.toml to specify build dependencies, with the aim of using pypa/build as the build engine. This seems to be the recommended approach to building wheels.
Remove imports of prophet.models from the setup.py script -- the build system was complaining that packages could not be found (because the build happens in an isolated environment). I think this is actually a bit cleaner.
Create a wheel.yml Github Actions workflow. The aim is to run this when we release the package, and publish to PyPi automatically. Currently using cibuildwheel to try and simplify this process.
Get all relevant wheels built successfully via wheel.yml (across different platforms and for Python versions 3.6, 3.7, 3.8).
- macOS. Test wheel here for Python 3.8: https://test.pypi.org/project/prophet/#description
- Linux -- having issues with running auditwheel to create the manylinux wheels.
Finalise the testing + automated publishing via wheel.yml
Spot check that building from source on the end user's machines is not broken (would be required for versions / platforms where we haven't published a wheel).

Open questions

Windows

Getting the necessary build tools on Windows will be tricky, but I think cmdstanpy might make it easier with the install_cxx_toolchain utility function. This should be its own PR though.

cmdstan

The PR as it currently stands removes the option for the user to use their own cmdstan installation. Should we still make this available for end-users who want to build their own wheel?

tcuongd · 2021-08-29T05:09:52Z

.github/workflows/wheel.yml

+ push:
+ branches: [ master ]
+ pull_request:
+ branches: [ master ]


Temporary, for testing.

tcuongd · 2021-08-29T05:11:32Z

python/setup.py


 def get_backends_from_env() -> List[str]:
- from prophet.models import StanBackendEnum


This was causing the build system to fail, since the build happens in an isolated environment with only the dependencies needed for setup.py. I think removing this makes things a bit cleaner, but keen to hear thoughts.

python/setup.py

tcuongd · 2021-08-29T05:14:05Z

python/setup.py

@@ -129,15 +181,15 @@ def with_project_on_sys_path(self, func):
 author_email='sjtz@pm.me',
 license='MIT',
 packages=find_packages(),
- setup_requires=[


Ensures that we read from pyproject.toml for build dependencies instead.

freddyaboulton · 2021-09-03T20:29:30Z

@tcuongd Thanks for starting this! I think this is really cool.

I tried some stuff out on my fork: freddyaboulton#1 to no avail as well.

I think the problem with auditwheel is that it's finding compiled code in the wheel when the package does not specify any extensions in the setup.py so it thinks it's "pure".

We can obviously skip the audit step but then the problem is that cmdstan dynamically links against some libraries that would not be included in the wheel so it's actually not exportable.

One thing that came to mind is that if cmdstan statically linked against its dependencies, then we could skip the audit step and I think it should "work". So maybe that's something worth trying?

I'm pretty new to this myself so I don't have any concrete answer (and it's possible what I'm saying here doesn't make sense at all!) but just want to let you know that this is cool and I'm willing to help if possible.

Another point to discuss is that I added testing to the cibuildwheel script and the macosx build passes but then the tests start failing in a really weird way?

https://github.com/freddyaboulton/prophet/runs/3498406868

tcuongd · 2021-09-04T08:45:35Z

Thanks heaps for looking into this as well @freddyaboulton !! Yeah definitely a new area for me too so still trying to piece everything together 😅

Linux wheel

Yeah the Linux one is weird. I did try to bundle the necessary cmdstan files into the wheel without running the repair command, but the wheel was rejected by PyPi when I tried to upload it. I've received some good feedback about this here: pypa/packaging-problems#542 - seems like we'll need to change our setup.py file.

MacOS tests

Oh I didn't realise cibuildwheel had wheel tests baked in as well, nice find! I think what's happening on your branch is that it's trying to run the tests against the source code ({project}/tests), but that's not where we built the binaries. 🤞 if we change that to {package}/tests it'll work.

Keep me updated! I'll keep trying to get Linux working as well.

…stan logic

freddyaboulton · 2021-09-05T18:46:37Z

@tcuongd great call adding the custom build_ext command to setup.py! I was able to fix the unit test issue and add a couple of cmdstanpy tests to verify the built wheels would pass the unit tests for both back ends as expected! I opened up those changes as a PR to this branch.

freddyaboulton · 2021-09-18T16:07:04Z

@tcuongd this looks ready to merge for me. What do you think?

tcuongd · 2021-10-02T11:30:09Z

@freddyaboulton Thanks for the reminder! Totally dropped the ball on this one 😅 Yeah it's probably a good idea to merge some of this into master, so it's easier to extend and test before we do a release.

Going to work on cleaning this up over the next 2 days.

…ls code

…/prophet into tcuongd-cmdstanpy-wheel

tcuongd · 2021-10-03T01:02:37Z

Alrighty I will merge this into master now since it's passing the regular build-and-test CI, and we can iterate on it. I've also set the wheel workflow runs to be triggered on release create and workflow_dispatch (i.e. manual runs). Since the manual run only becomes trigger-able after this is merged into master, I'll do another check that nothing's broken after the merge.

freddyaboulton · 2022-02-08T23:54:07Z

@tcuongd When is this being released?

octavd · 2022-02-09T16:54:37Z

@tcuongd looking forward for this also.

lmmentel · 2022-02-10T08:26:45Z

Awesome! Looking forward to it. It'll have a lot of impact reducing the install times 🙌

akosfurton · 2022-02-14T16:21:12Z

I believe this PR + a follow up PR to remove the PyStan backend should unblock #2041

Then, prophet 1.1 can be released to PyPi #2064

freddyaboulton · 2022-03-23T15:12:29Z

@tcuongd @akosfurton Any update on when prophet 1.1 will be released? Thank you!

akosfurton · 2022-03-29T14:21:09Z

@tcuongd @akosfurton Any update on when prophet 1.1 will be released? Thank you!

I don't have write access to merge the PR, only @tcuongd and @bletham do. Will try to contact them.

tcuongd added 10 commits August 28, 2021 22:57

init

983e08b

allow manual run:

042c912

temp triggers for testing

a156cfd

ensure pyproject.toml is used

50e8ca6

fixes

40e8f8f

prophet namespace

85f75c0

fix

a6beac7

fix

44e7eb3

expose tests

2f9b7ff

clarify setup

ab845cd

facebook-github-bot added the CLA Signed label Aug 28, 2021

tcuongd added 7 commits August 29, 2021 03:19

fix

765f71a

use cibuildwheel

f59468c

set cibw options

7b7f2b7

dont attempt windows wheel

523d9a2

add cmdstanpy to the wheel

37dd50d

remove unnecessary cmdstan files from wheel

292705c

fix linux repair

79c6f45

tcuongd changed the title ~~[Draft] Python Wheels~~ [Draft] Python Wheels for PyPi Aug 29, 2021

tcuongd commented Aug 29, 2021

View reviewed changes

python/setup.py Outdated Show resolved Hide resolved

tcuongd commented Aug 29, 2021

View reviewed changes

python/setup.py Outdated Show resolved Hide resolved

tcuongd commented Aug 29, 2021

View reviewed changes

tcuongd mentioned this pull request Aug 29, 2021

Upload wheel to PyPI #664

Closed

simplify setup.py

65fea9d

tcuongd added 2 commits September 5, 2021 02:32

cache cmdstan files, make linux aware of extension modules, prune cmd…

ee1abe7

…stan logic

cleaner prune function, use cache for pip as well

4d45c59

tcuongd added 2 commits September 5, 2021 17:54

build for py36 and py37 as well

e4fae99

consistent usage of platform variable

e20d88c

tcuongd linked an issue Sep 5, 2021 that may be closed by this pull request

Upload wheel to PyPI #664

Closed

tcuongd mentioned this pull request Oct 2, 2021

ModuleNotFoundError: No module named 'lunarcalendar' #2033

Open

tcuongd and others added 9 commits October 3, 2021 04:19

clean up functions, write documentation, remove build_model from mode…

b7be6fb

…ls code

Add unit tests to wheel building (#2020)

a76d33a

parallel wheel builds, fix triggers

3552e94

Merge branch 'tcuongd-cmdstanpy-wheel' of https://github.com/facebook…

b996ff1

…/prophet into tcuongd-cmdstanpy-wheel

update holidays req from master

3d6fade

bring back build and test

a971493

Merge branch 'master' into tcuongd-cmdstanpy-wheel

eabe76f

import extension from setuptools

461c050

use cache for build and test, clean up workflow yamls

180d081

tcuongd marked this pull request as ready for review October 3, 2021 00:59

tcuongd merged commit 4fb577e into master Oct 3, 2021

tcuongd deleted the tcuongd-cmdstanpy-wheel branch October 3, 2021 01:03

akosfurton mentioned this pull request Nov 10, 2021

Release a new version to pypi #2064

Closed

WardBrian mentioned this pull request Dec 13, 2021

Remove pystan backend, move to cmdstanpy #2041

Closed

3 tasks

ankane mentioned this pull request Apr 28, 2022

Precompiled models #2162

Closed

mdeshmu mentioned this pull request Jul 28, 2022

[prophet] prophet installation fails with ModuleNotFoundError: No module named 'lunarcalendar' apache/superset#20901

Closed

3 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Draft] Python Wheels for PyPi #2010

[Draft] Python Wheels for PyPi #2010

tcuongd commented Aug 28, 2021 •

edited

Loading

tcuongd Aug 29, 2021

tcuongd Aug 29, 2021

tcuongd Aug 29, 2021

freddyaboulton commented Sep 3, 2021

tcuongd commented Sep 4, 2021 •

edited

Loading

freddyaboulton commented Sep 5, 2021

freddyaboulton commented Sep 18, 2021

tcuongd commented Oct 2, 2021 •

edited

Loading

tcuongd commented Oct 3, 2021

freddyaboulton commented Feb 8, 2022

octavd commented Feb 9, 2022

lmmentel commented Feb 10, 2022

akosfurton commented Feb 14, 2022

freddyaboulton commented Mar 23, 2022 •

edited

Loading

akosfurton commented Mar 29, 2022


		def get_backends_from_env() -> List[str]:
		from prophet.models import StanBackendEnum

[Draft] Python Wheels for PyPi #2010

[Draft] Python Wheels for PyPi #2010

Conversation

tcuongd commented Aug 28, 2021 • edited Loading

Tasks

Open questions

tcuongd Aug 29, 2021

Choose a reason for hiding this comment

tcuongd Aug 29, 2021

Choose a reason for hiding this comment

tcuongd Aug 29, 2021

Choose a reason for hiding this comment

freddyaboulton commented Sep 3, 2021

tcuongd commented Sep 4, 2021 • edited Loading

freddyaboulton commented Sep 5, 2021

freddyaboulton commented Sep 18, 2021

tcuongd commented Oct 2, 2021 • edited Loading

tcuongd commented Oct 3, 2021

freddyaboulton commented Feb 8, 2022

octavd commented Feb 9, 2022

lmmentel commented Feb 10, 2022

akosfurton commented Feb 14, 2022

freddyaboulton commented Mar 23, 2022 • edited Loading

akosfurton commented Mar 29, 2022

tcuongd commented Aug 28, 2021 •

edited

Loading

tcuongd commented Sep 4, 2021 •

edited

Loading

tcuongd commented Oct 2, 2021 •

edited

Loading

freddyaboulton commented Mar 23, 2022 •

edited

Loading