Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Publish on PyPI #211

Merged
merged 9 commits into from
Jul 17, 2024
Merged

Publish on PyPI #211

merged 9 commits into from
Jul 17, 2024

Conversation

izzet
Copy link
Contributor

@izzet izzet commented Jul 11, 2024 •
edited
Loading

  • Built on top of DFTracer changes (most updated branch)
  • Tested CI via testpypi.org on Polaris
  • Tested Unet3D and Cosmoflow on Polaris via testpypi.org
  • Add CI tests that run DLIO via requirements.txt installation
  • Cleared redundant CI matrix options.

zhenghh04
zhenghh04 reviewed Jul 11, 2024
View reviewed changes
Copy link
Member

@zhenghh04 zhenghh04 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Could you check whether the package list is complete in requirements.txt? For example, I do not see tensorflow there.

requirements.txt Outdated Show resolved Hide resolved
@zhenghh04
Copy link
Member

Also, why there is no CI-tests running?

@izzet
Copy link
Contributor Author

izzet commented Jul 15, 2024

Also, why there is no CI-tests running?

CI will work when this branch is merged into dependency/dftracer -- because #208 is targeted to be merged on main.

@zhenghh04
Copy link
Member

Also, why there is no CI-tests running?

CI will work when this branch is merged into dependency/dftracer -- because #208 is targeted to be merged on main.

It is better to trigger this when new commit is pushed. Because we have to know whether it pass the tests or not before merging, right?

@izzet
Copy link
Contributor Author

izzet commented Jul 15, 2024

Also, why there is no CI-tests running?

CI will work when this branch is merged into dependency/dftracer -- because #208 is targeted to be merged on main.

It is better to trigger this when new commit is pushed. Because we have to know whether it pass the tests or not before merging, right?

Current CI script (below) works only when the PR is targeted towards main or dev -- this is usually to avoid wasting CI time when the PR is for a side branch.

on:
  pull_request:
    branches: [main, dev]

@zhenghh04 zhenghh04 merged commit f459b17 into argonne-lcf:dependency/dftracer Jul 17, 2024
zhenghh04 added a commit that referenced this pull request Jul 19, 2024
@hariharan-devarajan @izzet @zhenghh04
Switch DLIO Profiler to DFTracer. (#208)
cc4a1f5 
* Switch DLIO Profiler to DFTracer.

* Github workflow fixed.

* DFTracer pip package address fixed.

* switch mpi to openmpi

* added build debug so that symbols are shown.

* added DEBUG logging for DFTRACER

* removed explicit install of DFTracer as it is done during pip.

* added clanup code.

* ci cleanup

* Updated for DFTracer changes

* switched to 1.0.1

Due to a simple pypi bug

* Switch to release 1.0.2

* Package info updated. (#210)

* Update dali_tfrecord_reader.py to import PerfTrace and Profile from utils.utility

* Update dali_npy_reader.py

* Update dali_image_reader.py

* Update custom_npz_reader.py

* Update custom_torch_data_loader.py [ci-skip]

* Update pytorch_checkpointing.py [ci-skip]

* Update native_dali_data_loader.py [ci-skip]

* Publish on PyPI (#211)

* PyPI publish workflow added.

* `requirements.txt` cleaned up.

* Action names refactored.

* Missing requirements fixed.

* `setup.py` requirement versions refactored.

* CI support for running via `requirements.txt`.

* `VENV_PATH` fixed in CI script.

* CI script conditional executable fix.

* Cleared redundant CI matrix options.

---------

Co-authored-by: Izzet Yildirim <iyildirim@anl.gov>
Co-authored-by: Izzet Yildirim <izzet@izzet.dev>
Co-authored-by: Huihuo Zheng <zhenghh04@gmail.com>
zhenghh04 added a commit that referenced this pull request Aug 6, 2024
@zhenghh04 @johnugeorge
@hariharan-devarajan @izzet
OOM Fix (#217)
2fcdbfa 
* Fixed the MPI initialization issue (#207)

* Bring v1.0 to the most recent commit  (#202)

* Request changes from MLPerf Storage (#199)

* added au metric to the configuration file; set shuffling and shuffle buffer size to be 2 for cosmoflow

* removed dependencies on dlioprofiler

* fixed bugs

* Fixed potential insufficient samples due to num_files is not divisible by comm.size (#200)

* added au metric to the configuration file; set shuffling and shuffle buffer size to be 2 for cosmoflow

* removed dependencies on dlioprofiler

* fixed bugs

* recovered back dlio_profiler

* fixed potential not enough samples

* Update tf_reader.py

* Mlperf requests (#201)

* added au metric to the configuration file; set shuffling and shuffle buffer size to be 2 for cosmoflow

* removed dependencies on dlioprofiler

* fixed bugs

* fixed issue with dlio_profiler

* bring back dlio_profiler_py

* sync up (#205)

* Request changes from MLPerf Storage (#199)

* added au metric to the configuration file; set shuffling and shuffle buffer size to be 2 for cosmoflow

* removed dependencies on dlioprofiler

* fixed bugs

* Fixed potential insufficient samples due to num_files is not divisible by comm.size (#200)

* added au metric to the configuration file; set shuffling and shuffle buffer size to be 2 for cosmoflow

* removed dependencies on dlioprofiler

* fixed bugs

* recovered back dlio_profiler

* fixed potential not enough samples

* Update tf_reader.py

* Mlperf requests (#201)

* added au metric to the configuration file; set shuffling and shuffle buffer size to be 2 for cosmoflow

* removed dependencies on dlioprofiler

* fixed bugs

* fixed issue with dlio_profiler

* bring back dlio_profiler_py

* Bring v1.0 to the most recent commit  (#202) (#203)

* Request changes from MLPerf Storage (#199)

* added au metric to the configuration file; set shuffling and shuffle buffer size to be 2 for cosmoflow

* removed dependencies on dlioprofiler

* fixed bugs

* Fixed potential insufficient samples due to num_files is not divisible by comm.size (#200)

* added au metric to the configuration file; set shuffling and shuffle buffer size to be 2 for cosmoflow

* removed dependencies on dlioprofiler

* fixed bugs

* recovered back dlio_profiler

* fixed potential not enough samples

* Update tf_reader.py

* Mlperf requests (#201)

* added au metric to the configuration file; set shuffling and shuffle buffer size to be 2 for cosmoflow

* removed dependencies on dlioprofiler

* fixed bugs

* fixed issue with dlio_profiler

* bring back dlio_profiler_py

* Fix requirements file (#204)

Signed-off-by: Johnu George <johnugeorge109@gmail.com>

---------

Signed-off-by: Johnu George <johnugeorge109@gmail.com>
Co-authored-by: Johnu George <johnugeorge109@gmail.com>

* barrier in the beginning

* fixed bugs

* fixed MPI initilization issue

---------

Signed-off-by: Johnu George <johnugeorge109@gmail.com>
Co-authored-by: Johnu George <johnugeorge109@gmail.com>

* Switch DLIO Profiler to DFTracer. (#208)

* Switch DLIO Profiler to DFTracer.

* Github workflow fixed.

* DFTracer pip package address fixed.

* switch mpi to openmpi

* added build debug so that symbols are shown.

* added DEBUG logging for DFTRACER

* removed explicit install of DFTracer as it is done during pip.

* added clanup code.

* ci cleanup

* Updated for DFTracer changes

* switched to 1.0.1

Due to a simple pypi bug

* Switch to release 1.0.2

* Package info updated. (#210)

* Update dali_tfrecord_reader.py to import PerfTrace and Profile from utils.utility

* Update dali_npy_reader.py

* Update dali_image_reader.py

* Update custom_npz_reader.py

* Update custom_torch_data_loader.py [ci-skip]

* Update pytorch_checkpointing.py [ci-skip]

* Update native_dali_data_loader.py [ci-skip]

* Publish on PyPI (#211)

* PyPI publish workflow added.

* `requirements.txt` cleaned up.

* Action names refactored.

* Missing requirements fixed.

* `setup.py` requirement versions refactored.

* CI support for running via `requirements.txt`.

* `VENV_PATH` fixed in CI script.

* CI script conditional executable fix.

* Cleared redundant CI matrix options.

---------

Co-authored-by: Izzet Yildirim <iyildirim@anl.gov>
Co-authored-by: Izzet Yildirim <izzet@izzet.dev>
Co-authored-by: Huihuo Zheng <zhenghh04@gmail.com>

* Fix README CI badge (#212)

* Ignore file indexing for native data loader.

The sample building and native data loader case is needed only for DLIO created data loaders. For native data loaders which provide their own API;s they provide their own indexing and there this sampling can be ignored.

---------

Signed-off-by: Johnu George <johnugeorge109@gmail.com>
Co-authored-by: Johnu George <johnugeorge109@gmail.com>
Co-authored-by: Hariharan Devarajan <mani.hariharan@gmail.com>
Co-authored-by: Izzet Yildirim <iyildirim@anl.gov>
Co-authored-by: Izzet Yildirim <izzet@izzet.dev>
Co-authored-by: hariharandev1 <hariharandev1@llnl.gov>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@zhenghh04 zhenghh04 zhenghh04 left review comments

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
2 participants
@izzet @zhenghh04