Move s3fs to cloud extras #15729

lantiga · 2022-11-18T18:29:35Z

What does this PR do?

Since we are not using s3fs locally (s3fs is used for Drive, Path and Payload through fsspec), we can move it to cloud extras. This way we avoid getting boto and botocore when we pip install lightning locally.

Does your PR introduce any breaking changes? If yes, please list them.

None

Before submitting

Was this discussed/approved via a GitHub issue? (not for typos and docs)
Did you read the contributor guideline, Pull Request section?
Did you make sure your PR does only one thing, instead of bundling different changes together?
Did you make sure to update the documentation with your changes? (if necessary)
Did you write any new necessary tests? (not for typos and docs)
Did you verify new and existing tests pass locally with your changes?
Did you list all the breaking changes introduced by this pull request?
Did you update the CHANGELOG? (not for typos, docs, test updates, or minor internal changes/refactors)

PR review

Anyone in the community is welcome to review the PR.
Before you start reviewing, make sure you have read the review guidelines. In short, see the following bullet-list:

Is this pull request ready for review? (if not, please submit in draft mode)
Check that all items from Before submitting are resolved
Make sure the title is self-explanatory and the description concisely explains the PR
Add labels and milestones (and optionally projects) to the PR so it can be classified

Did you have fun?

Make sure you had fun coding 🙃

cc @Borda

github-actions · 2022-11-21T19:35:47Z

⚡ Required checks status: All passing 🟢

Groups summary

🟢 lightning_app: Tests workflow

Check ID	Status
app-pytest (macOS-11, app, 3.8, latest)	success	✅
app-pytest (macOS-11, app, 3.8, oldest)	success	✅
app-pytest (macOS-11, lightning, 3.9, latest)	success	✅
app-pytest (ubuntu-20.04, app, 3.8, latest)	success	✅
app-pytest (ubuntu-20.04, app, 3.8, oldest)	success	✅
app-pytest (ubuntu-20.04, lightning, 3.9, latest)	success	✅
app-pytest (windows-2022, app, 3.8, latest)	success	✅
app-pytest (windows-2022, app, 3.8, oldest)	success	✅
app-pytest (windows-2022, lightning, 3.8, latest)	success	✅

🟢 lightning_app: Examples

Check ID	Status
app-examples (macOS-11, app, 3.9, latest)	success	✅
app-examples (macOS-11, app, 3.9, oldest)	success	✅
app-examples (macOS-11, lightning, 3.9, latest)	success	✅
app-examples (ubuntu-20.04, app, 3.9, latest)	success	✅
app-examples (ubuntu-20.04, app, 3.9, oldest)	success	✅
app-examples (ubuntu-20.04, lightning, 3.9, latest)	success	✅
app-examples (windows-2022, app, 3.9, latest)	success	✅
app-examples (windows-2022, app, 3.9, oldest)	success	✅
app-examples (windows-2022, lightning, 3.9, latest)	success	✅

🟢 lightning_app: Azure

Check ID	Status
App.cloud-e2e	success	✅

🟢 lightning_app: Docs

Check ID	Status
make-doctest (app)	success	✅
make-html (app)	success	✅

🟢 mypy

Check ID	Status
mypy	success	✅

🟢 install

Check ID	Status
install-pkg (ubuntu-22.04, app, 3.7)	success	✅
install-pkg (ubuntu-22.04, app, 3.10)	success	✅
install-pkg (ubuntu-22.04, lite, 3.7)	success	✅
install-pkg (ubuntu-22.04, lite, 3.10)	success	✅
install-pkg (ubuntu-22.04, pytorch, 3.7)	success	✅
install-pkg (ubuntu-22.04, pytorch, 3.10)	success	✅
install-pkg (ubuntu-22.04, lightning, 3.7)	success	✅
install-pkg (ubuntu-22.04, lightning, 3.10)	success	✅
install-pkg (macOS-12, app, 3.7)	success	✅
install-pkg (macOS-12, app, 3.10)	success	✅
install-pkg (macOS-12, lite, 3.7)	success	✅
install-pkg (macOS-12, lite, 3.10)	success	✅
install-pkg (macOS-12, pytorch, 3.7)	success	✅
install-pkg (macOS-12, pytorch, 3.10)	success	✅
install-pkg (macOS-12, lightning, 3.7)	success	✅
install-pkg (macOS-12, lightning, 3.10)	success	✅
install-pkg (windows-2022, app, 3.7)	success	✅
install-pkg (windows-2022, app, 3.10)	success	✅
install-pkg (windows-2022, lite, 3.7)	success	✅
install-pkg (windows-2022, lite, 3.10)	success	✅
install-pkg (windows-2022, pytorch, 3.7)	success	✅
install-pkg (windows-2022, pytorch, 3.10)	success	✅
install-pkg (windows-2022, lightning, 3.7)	success	✅
install-pkg (windows-2022, lightning, 3.10)	success	✅

This comment was automatically generated and updates for 60 minutes every 180 seconds.

Thank you for your contribution! 💜

Co-authored-by: Luca Antiga <luca@lightning.ai> (cherry picked from commit dd75906)

@lantiga

* chlog update * Fix typo in script name (#15724) (cherry picked from commit d925077) * Torch inference mode for prediction (#15719) torch inference mode for prediction (cherry picked from commit 08d14ec) * [App] Update multi-node examples (#15700) Co-authored-by: Jirka Borovec <6035284+Borda@users.noreply.github.com> Co-authored-by: Carlos Mocholí <carlossmocholi@gmail.com> (cherry picked from commit 8306797) * feature(docs/app/lit_tabs): add works (#15731) (cherry picked from commit 1a31d13) * [App] Fix VSCode IDE debugger (#15747) (cherry picked from commit 6714ca7) * Update tensorboard requirement from <2.11.0,>=2.9.1 to >=2.9.1,<2.12.0 in /requirements (#15746) Update tensorboard requirement in /requirements Updates the requirements on [tensorboard](https://github.com/tensorflow/tensorboard) to permit the latest version. - [Release notes](https://github.com/tensorflow/tensorboard/releases) - [Changelog](https://github.com/tensorflow/tensorboard/blob/master/RELEASE.md) - [Commits](tensorflow/tensorboard@2.9.1...2.11.0) --- updated-dependencies: - dependency-name: tensorboard dependency-type: direct:production ... Signed-off-by: dependabot[bot] <support@github.com> Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> (cherry picked from commit 0b58b69) * Update beautifulsoup4 requirement from <=4.8.2 to <4.11.2 in /requirements (#15745) * Update beautifulsoup4 requirement in /requirements Updates the requirements on [beautifulsoup4](https://www.crummy.com/software/BeautifulSoup/bs4/) to permit the latest version. --- updated-dependencies: - dependency-name: beautifulsoup4 dependency-type: direct:production ... Signed-off-by: dependabot[bot] <support@github.com> * Apply suggestions from code review Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Jirka Borovec <6035284+Borda@users.noreply.github.com> (cherry picked from commit 1ffbe1b) * [App] Fix multi-node pytorch example CI (#15753) (cherry picked from commit bc797fd) * [App] Improve `LightningTrainerScript` start-up time (#15751) (cherry picked from commit c2c1974) * Enable Probot CheckGroup v5 (#15670) (cherry picked from commit 6c8ee01) * [App] Enable properties for the Lightning flow (#15750) (cherry picked from commit 5cfb176) * test for Enable setting property (#15755) Co-authored-by: thomas chaton <thomas@grid.ai> Co-authored-by: Ethan Harris <ethanwharris@gmail.com> (cherry picked from commit ba14038) * Move s3fs to cloud extras (#15729) Co-authored-by: Luca Antiga <luca@lightning.ai> (cherry picked from commit dd75906) * Revert new Hydra launch behavior (#15737) * revert new hydra cwd behavior * remove debug statements * changelog Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> Co-authored-by: Jirka <jirka.borovec@seznam.cz> Co-authored-by: Jirka Borovec <6035284+Borda@users.noreply.github.com> (cherry picked from commit 88b2e5a) * FCCV Docs (#15598) * add custom data iter docs * add custom data iter docs * Update docs/source-pytorch/data/custom_data_iterables.rst * remove ToDevice * nit * Update docs/source-pytorch/data/custom_data_iterables.rst Co-authored-by: Luca Antiga <luca.antiga@gmail.com> * clarification for @lantiga * typo * Update docs/source-pytorch/data/custom_data_iterables.rst * Update docs/source-pytorch/data/custom_data_iterables.rst * Update docs/source-pytorch/data/custom_data_iterables.rst Co-authored-by: Jirka Borovec <6035284+Borda@users.noreply.github.com> Co-authored-by: Akihiro Nitta <nitta@akihironitta.com> Co-authored-by: Luca Antiga <luca.antiga@gmail.com> (cherry picked from commit 006fde9) * Switch from tensorboard to tensorboardx in logger (#15728) * Switch from tensorboard to tensorboardx in logger * Warn if log_graph is set to True but tensorboard is not installed * Fix warning message formatting * Apply suggestions from code review * simplify for TBX as required pkg * docs example * chlog * tbx 2.2 Co-authored-by: Luca Antiga <luca@lightning.ai> Co-authored-by: William Falcon <waf2107@columbia.edu> Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> Co-authored-by: Carlos Mocholí <carlossmocholi@gmail.com> Co-authored-by: Jirka Borovec <6035284+Borda@users.noreply.github.com> Co-authored-by: Jirka <jirka.borovec@seznam.cz> (cherry picked from commit 9c2eb52) * resolve conflicts * Fix azure path excludes (#15756) Co-authored-by: Jirka Borovec <6035284+Borda@users.noreply.github.com> (cherry picked from commit aef94ce) * Disable XSRF protection in StreamlitFrontend to support upload in localhost (#15684) * Enable CORS in StreamlitFrontend to support upload * Only disable XSRF when running on localhost * Update test * Use utility fn to detect if localhost Co-authored-by: Luca Antiga <luca@lightning.ai> (cherry picked from commit ed3eef0) * Enable Probot CheckGroup v5.1 (#15763) (cherry picked from commit c55f80f) * Bump pytest from 7.1.3 to 7.2.0 in /requirements (#15677) Bumps [pytest](https://github.com/pytest-dev/pytest) from 7.1.3 to 7.2.0. - [Release notes](https://github.com/pytest-dev/pytest/releases) - [Changelog](https://github.com/pytest-dev/pytest/blob/main/CHANGELOG.rst) - [Commits](pytest-dev/pytest@7.1.3...7.2.0) --- updated-dependencies: - dependency-name: pytest dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com> Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> (cherry picked from commit cfb27bd) * Fix the `examples/app_dag` App (#14359) * Fix app dag example * Add test * Update doc * Update tests/tests_app_examples/test_app_dag.py Co-authored-by: Sherin Thomas <sherin@grid.ai> (cherry picked from commit 2b61c92) * mergify: drop ready for draft (#15766) (cherry picked from commit 1a07a9c) * lightning delete cluster CLI command help text update (#15760) * updated the lighting delete cluster CLI command help text output * updated changelog * typo fix * Apply suggestions from code review Co-authored-by: Jirka Borovec <6035284+Borda@users.noreply.github.com> (cherry picked from commit 75b0573) * Deduplicate top level lighting CLI command groups (#15761) * unify remove and delete command groups & the add and delete command groups * added changelog * fix tests * Apply suggestions from code review Co-authored-by: Jirka Borovec <6035284+Borda@users.noreply.github.com> (cherry picked from commit 7b2788e) * releasing 1.8.3 * CI: lite on GPU * Fix App Docs for lightning ssh-keys command (#15773) fixed ssh-keys docs (cherry picked from commit 317591d) Co-authored-by: thomas chaton <thomas@grid.ai> Co-authored-by: yiftachbeer <yiftach.beer@gmail.com> Co-authored-by: Sherin Thomas <sherin@lightning.ai> Co-authored-by: Ethan Harris <ethanwharris@gmail.com> Co-authored-by: Yurij Mikhalevich <yurij@grid.ai> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Carlos Mocholí <carlossmocholi@gmail.com> Co-authored-by: Luca Antiga <luca.antiga@gmail.com> Co-authored-by: Adrian Wälchli <aedu.waelchli@gmail.com> Co-authored-by: Justus Schock <12886177+justusschock@users.noreply.github.com> Co-authored-by: Kaushik B <45285388+kaushikb11@users.noreply.github.com> Co-authored-by: Rick Izzo <rlizzo@users.noreply.github.com>

Move s3fs to cloud extras

2c5c224

lantiga requested review from williamFalcon, Borda, tchaton and awaelchli as code owners November 18, 2022 18:29

lantiga added this to the v1.8.x milestone Nov 18, 2022

github-actions bot added the app (removed) Generic label for Lightning App package label Nov 18, 2022

Borda approved these changes Nov 18, 2022

View reviewed changes

awaelchli approved these changes Nov 19, 2022

View reviewed changes

Merge branch 'master' into s3fs-dependency

a5af2ff

mergify bot added the ready PRs ready to be merged label Nov 19, 2022

awaelchli enabled auto-merge (squash) November 19, 2022 11:28

lantiga and others added 2 commits November 20, 2022 17:55

Merge branch 'master' into s3fs-dependency

b19e884

Merge branch 'master' into s3fs-dependency

a766b8f

ethanwharris approved these changes Nov 21, 2022

View reviewed changes

awaelchli merged commit dd75906 into master Nov 21, 2022

awaelchli deleted the s3fs-dependency branch November 21, 2022 19:57

Borda pushed a commit that referenced this pull request Nov 21, 2022

Move s3fs to cloud extras (#15729)

23ec3c4

Co-authored-by: Luca Antiga <luca@lightning.ai> (cherry picked from commit dd75906)

Borda mentioned this pull request Nov 23, 2022

Revert/s3fs #15792

Merged

12 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Move s3fs to cloud extras #15729

Move s3fs to cloud extras #15729

lantiga commented Nov 18, 2022 •

edited by github-actions bot

Loading

github-actions bot commented Nov 21, 2022 •

edited

Loading

Move s3fs to cloud extras #15729

Move s3fs to cloud extras #15729

Conversation

lantiga commented Nov 18, 2022 • edited by github-actions bot Loading

What does this PR do?

Does your PR introduce any breaking changes? If yes, please list them.

Before submitting

PR review

Did you have fun?

github-actions bot commented Nov 21, 2022 • edited Loading

⚡ Required checks status: All passing 🟢

Groups summary

lantiga commented Nov 18, 2022 •

edited by github-actions bot

Loading

github-actions bot commented Nov 21, 2022 •

edited

Loading