Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Scripts for building and uploading the morpheus-core conda package #1883

Merged

Conversation

AnuradhaKaruppiah
Copy link
Contributor

@AnuradhaKaruppiah AnuradhaKaruppiah commented Sep 5, 2024

With this change two conda packages are generated -

  1. morpheus - with all the components (core + llm + examples + tests). This is existing code.
  2. morpheus-core - with only the core (sources, stages and sinks)
    The morpheus-core package is uploaded to anaconda.org/nvidia for use by applications.

A new PR label "conda-core-build" is available to trigger building the morpheus-core conda package as a part of the CI pipeline.

Testing done:
Generated the morpheus-core conda package via -
a. local ci runner -

ci/scripts/run_ci_local.sh conda_core

b. docker build -

DOCKER_IMAGE_TAG=latest DOCKER_IMAGE_NAME=conda_bld_morpheus_core DOCKER_TARGET=conda_bld_morpheus_core DOCKER_EXTRA_ARGS="--build-arg CONDA_UPLOAD=upload --build-arg CONDA_TOKEN=<my-token> --progress plain" ./docker/build_container.sh

And verified an application that uses the core stages.

Signed-off-by: Anuradha Karuppiah <anuradhak@nvidia.com>
Signed-off-by: Anuradha Karuppiah <anuradhak@nvidia.com>
This is a workaround to the following error -
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
CMake Error at
build-conda/_deps/rapids-cmake-src/rapids-cmake/cmake/make_global.cmake:59
(set_target_properties):^M
  Attempt to promote imported target "gflags" to global scope (by
setting^M
  IMPORTED_GLOBAL) which is not built in this directory.^M
Call Stack (most recent call first):^M
  build-conda/_deps/rapids-cmake-src/rapids-cmake/find/package.cmake:129
(rapids_cmake_make_global)^M
  /Morpheus/.cache/cpm/mrc/9f8a50df0cf94cf470a5cdc01861ed63f701e27f/cmake/dependencies.cmake:53
(rapids_find_package)^M
  /Morpheus/.cache/cpm/mrc/9f8a50df0cf94cf470a5cdc01861ed63f701e27f/CMakeLists.txt:148
(include)
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>

Signed-off-by: Anuradha Karuppiah <anuradhak@nvidia.com>
Signed-off-by: Anuradha Karuppiah <anuradhak@nvidia.com>
@AnuradhaKaruppiah AnuradhaKaruppiah requested review from a team as code owners September 5, 2024 20:58
Copy link

copy-pr-bot bot commented Sep 5, 2024

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

@AnuradhaKaruppiah AnuradhaKaruppiah added non-breaking Non-breaking change feature request New feature or request conda-build Enables running the conda-build step on a PR conda-core-build labels Sep 5, 2024
@AnuradhaKaruppiah
Copy link
Contributor Author

/ok to test

1 similar comment
@AnuradhaKaruppiah
Copy link
Contributor Author

/ok to test

@AnuradhaKaruppiah
Copy link
Contributor Author

/ok to test

1. The morpheus-core conda package is built for main/dev branches and
   when "conda-core-build" label is added to the PR.
2. Enabled upload of morpheus-core conda package.

The conda package with all morpheus compoments (core+llm) can still be
built via the usual triggers. Uploading that package remains disabled.

Signed-off-by: Anuradha Karuppiah <anuradhak@nvidia.com>
@AnuradhaKaruppiah
Copy link
Contributor Author

/ok to test

Copy link
Contributor

@dagardner-nv dagardner-nv left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM, just a few comments/questions

.github/workflows/ci_pipe.yml Outdated Show resolved Hide resolved
.github/workflows/ci_pipe.yml Show resolved Hide resolved
.github/workflows/pr.yaml Outdated Show resolved Hide resolved
Updated the "morpheus" conda-package job to not upload (it was earlier
relying on conda_upload_label being empty to skip upload).

Signed-off-by: Anuradha Karuppiah <anuradhak@nvidia.com>
@AnuradhaKaruppiah
Copy link
Contributor Author

/ok to test

@AnuradhaKaruppiah
Copy link
Contributor Author

/merge

@rapids-bot rapids-bot bot merged commit 070d168 into nv-morpheus:branch-24.10 Sep 6, 2024
12 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
conda-build Enables running the conda-build step on a PR feature request New feature or request non-breaking Non-breaking change
Projects
Archived in project
Development

Successfully merging this pull request may close these issues.

2 participants