Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Refactor digital_fingerprinting stages and add morpheus-split conda recipe (core, dfp, llm) #1897

Merged
merged 17 commits into from
Oct 4, 2024
Merged

Refactor digital_fingerprinting stages and add morpheus-split conda recipe (core, dfp, llm) #1897

merged 17 commits into from
Oct 4, 2024

Conversation

AnuradhaKaruppiah
Copy link
Contributor

@AnuradhaKaruppiah AnuradhaKaruppiah commented Sep 18, 2024 •
edited
Loading

  • Move dfp stages to python/morpheus_dfp
  • Create morpheus_dfp python package
  • Rename morpheus-core conda recipe to morpheus-libs, a split package conda recipe that generates morpheus-core, morpheus-dfp and morpheus-llm
  • Update github workflow to upload the morpheus-dfp conda package to anaconda

Other pending tasks, including test and doc refactoring, is tracked via python/morpheus_dfp/todo.md

Dev install now requires -

pip install -e python/morpheus
pip install -e python/morpheus_dfp
pip install -e python/morpheus_llm

@AnuradhaKaruppiah AnuradhaKaruppiah added non-breaking Non-breaking change feature request New feature or request DO NOT MERGE PR should not be merged; see PR for details labels Sep 18, 2024
@copy-pr-bot copy-pr-bot
Copy link

copy-pr-bot bot commented Sep 18, 2024

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

@review-notebook-app Review Notebook App
Copy link

Check out this pull request on  ReviewNB

See visual diffs & provide feedback on Jupyter Notebooks.

Powered by ReviewNB

@AnuradhaKaruppiah AnuradhaKaruppiah marked this pull request as draft September 18, 2024 17:21
@AnuradhaKaruppiah AnuradhaKaruppiah self-assigned this Sep 18, 2024
@AnuradhaKaruppiah AnuradhaKaruppiah changed the title [draft] Refactor the digitalfingerprinting source [draft] Refactor digital_fingerprinting stages and create morpheus_dfp conda package Sep 18, 2024
@AnuradhaKaruppiah
Copy link
Contributor Author

/ok to test

@AnuradhaKaruppiah AnuradhaKaruppiah removed DO NOT MERGE PR should not be merged; see PR for details labels Sep 18, 2024
@AnuradhaKaruppiah
Copy link
Contributor Author

/ok to test

@AnuradhaKaruppiah AnuradhaKaruppiah added conda-build Enables running the conda-build step on a PR conda-core-build labels Sep 19, 2024
@AnuradhaKaruppiah
Copy link
Contributor Author

@mdemoret-nv I have merged the morpheus-llm packaging recipe here. It was challenging to fixup the recipe across two PRs. I have also merged the individual recipes into a split package one for all the libraries (morpheus-libs). The old recipe, for building everything, is still present (morpheus). I will remove this once the morpheus container is updated to stop using it.

@AnuradhaKaruppiah
Copy link
Contributor Author

/ok to test

@AnuradhaKaruppiah
Maintain a todo.md for DFP refactoring.
9879a38 
Signed-off-by: Anuradha Karuppiah <anuradhak@nvidia.com>
@AnuradhaKaruppiah
Move production dfp stages/modules, messages and utils to morpheus_dfp
f0cabce 
Signed-off-by: Anuradha Karuppiah <anuradhak@nvidia.com>
@AnuradhaKaruppiah
Update import paths within the morpheus_dfp module
d460afe 
Style used for purposes of consistency -
1. Imports in __init__ and pipeline are absolute
2. All other imports (withing morpheus_dfp) are relative

Signed-off-by: Anuradha Karuppiah <anuradhak@nvidia.com>
@AnuradhaKaruppiah
Setup files for morpheus_dfp python package
a40590c 
Signed-off-by: Anuradha Karuppiah <anuradhak@nvidia.com>
@AnuradhaKaruppiah
Build morpheus_dfp python package via cmake
b43416a 
The morpheus_dfp package is entirely python and setup.py can be used directly to
create this package. However for the sake of consistency cmake with
mopheus_utils_build_python_package is used.

Signed-off-by: Anuradha Karuppiah <anuradhak@nvidia.com>
@AnuradhaKaruppiah
Update dfp imports in examples and tests
c82801e 
Signed-off-by: Anuradha Karuppiah <anuradhak@nvidia.com>
@AnuradhaKaruppiah
morpheus_dfp conda recipe and CI updates
8be1302 
1. Add a build flag to disable compiling morpheus-core
2. Add a build flag for morpheus_dfp
3. Add morpheus_dfp conda recipe

Cleanup:
1. Avoid enabling morpheus extended package flags in the runner container.
Instead set them via the build scripts. This is mainly to avoid the
confusion in managing the flags.
2. Removed some of the build flags passed to the conda recipes to avoid
confusion.
@AnuradhaKaruppiah
Update github workflow to build/upload the morpheus_dfp conda package
b409f59 
Signed-off-by: Anuradha Karuppiah <anuradhak@nvidia.com>
@AnuradhaKaruppiah
Drop conda-lib-build label
6136c0f 
1. conda-build is used for building conda pacakge for the morpheus libraries.
2. Dropped the conda package with "all components". This will be replaced with
the individual conda packages.

Signed-off-by: Anuradha Karuppiah <anuradhak@nvidia.com>
@AnuradhaKaruppiah
Build morpheus_llm without building morpheus_core
5839e59 
Signed-off-by: Anuradha Karuppiah <anuradhak@nvidia.com>
@AnuradhaKaruppiah
Convert the morpheus-libs conda recipe to a split package format
5a6726a 
That generates morpheus-core, morpheus-dfp, morpheus-llm

Signed-off-by: Anuradha Karuppiah <anuradhak@nvidia.com>
@AnuradhaKaruppiah
Update the github workflow to use the conda split package script
cf957c3 
Signed-off-by: Anuradha Karuppiah <anuradhak@nvidia.com>
@AnuradhaKaruppiah
Generate conda packages with output type .conda instead of .tar.bz2
6827b51 
Using .conda format can improve download and extract times.

Signed-off-by: Anuradha Karuppiah <anuradhak@nvidia.com>
@AnuradhaKaruppiah
Copy link
Contributor Author

/ok to test

cwharris
cwharris approved these changes Oct 3, 2024
View reviewed changes
ci/conda/recipes/morpheus-libs/morpheus_dfp_build.sh Show resolved Hide resolved
python/CMakeLists.txt Outdated Show resolved Hide resolved
dagardner-nv
dagardner-nv approved these changes Oct 3, 2024
View reviewed changes
Copy link
Contributor

@dagardner-nv dagardner-nv left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM, just a few minor things

examples/digital_fingerprinting/visualization/dfp_viz_duo_pipeline.py Outdated Show resolved Hide resolved
python/morpheus_dfp/morpheus_dfp/modules/dfp_rolling_window.py Outdated Show resolved Hide resolved
python/morpheus_dfp/todo.md Outdated Show resolved Hide resolved
tests/examples/digital_fingerprinting/test_dfp_mlflow_model_writer.py Outdated Show resolved Hide resolved
@AnuradhaKaruppiah
Fix indentation in the cmake file.
913a461 
Signed-off-by: Anuradha Karuppiah <anuradhak@nvidia.com>
@AnuradhaKaruppiah
Update year in the copyright header.
a665dd9 
Signed-off-by: Anuradha Karuppiah <anuradhak@nvidia.com>
@AnuradhaKaruppiah
Update the conda build scripts to remove the old build dir entirely
0e4f6ba 
Signed-off-by: Anuradha Karuppiah <anuradhak@nvidia.com>
@AnuradhaKaruppiah
Copy link
Contributor Author

/ok to test

@AnuradhaKaruppiah
Remove relative imports within morpheus_dfp
6ec6dc1 
Signed-off-by: Anuradha Karuppiah <anuradhak@nvidia.com>
@AnuradhaKaruppiah
Copy link
Contributor Author

/ok to test

@AnuradhaKaruppiah
Copy link
Contributor Author

/merge

@rapids-bot rapids-bot bot merged commit 502cbf3 into nv-morpheus:branch-24.10 Oct 4, 2024
11 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@mdemoret-nv mdemoret-nv mdemoret-nv left review comments

@dagardner-nv dagardner-nv dagardner-nv approved these changes

@cwharris cwharris cwharris approved these changes

Assignees

@AnuradhaKaruppiah AnuradhaKaruppiah

Labels
conda-build Enables running the conda-build step on a PR feature request New feature or request non-breaking Non-breaking change
Projects
Morpheus Boards
Status: Done
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
4 participants
@AnuradhaKaruppiah @cwharris @mdemoret-nv @dagardner-nv