Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Docker release image build failure #7

Closed
pdmack opened this issue Apr 22, 2022 · 1 comment · Fixed by #11
Closed

Docker release image build failure #7

pdmack opened this issue Apr 22, 2022 · 1 comment · Fixed by #11

Comments

@pdmack
Copy link
Contributor

pdmack commented Apr 22, 2022 

docker buildx build -t nvcr.io/nvidia/morpheus/morpheus:latest --target runtime --build-arg FROM_IMAGE=gpuci/miniforge-cuda --build-arg CUDA_VER=11.4 --build-arg LINUX_DISTRO=ubuntu --build-arg LINUX_VER=20.04 --build-arg RAPIDS_VER=21.10 --build-arg PYTHON_VER=3.8 --build-arg TENSORRT_VERSION=8.2.1.3 --build-arg NEO_GIT_URL=REDACTED --network=host --ssh default --load  -f docker/Dockerfile .
[+] Building 4.5s (23/31)
 => [internal] load build definition from Dockerfile                                                                                                                    0.0s
 => => transferring dockerfile: 32B                                                                                                                                     0.0s
 => [internal] load .dockerignore                                                                                                                                       0.0s
 => => transferring context: 35B                                                                                                                                        0.0s
 => resolve image config for docker.io/docker/dockerfile:1.3                                                                                                            0.7s
 => CACHED docker-image://docker.io/docker/dockerfile:1.3@sha256:42399d4635eddd7a9b8a24be879d2f9a930d0ed040a61324cfdf59ef1357b3b2                                       0.0s
 => [internal] load build definition from Dockerfile                                                                                                                    0.0s
 => [internal] load .dockerignore                                                                                                                                       0.0s
 => [internal] load metadata for docker.io/gpuci/miniforge-cuda:11.4-devel-ubuntu20.04                                                                                  0.0s
 => [base 1/4] FROM docker.io/gpuci/miniforge-cuda:11.4-devel-ubuntu20.04                                                                                               0.0s
 => [internal] load build context                                                                                                                                       0.1s
 => => transferring context: 73.92kB                                                                                                                                    0.1s
 => CACHED [base 2/4] RUN apt-get update &&    apt-get upgrade -y &&    curl -sL https://deb.nodesource.com/setup_12.x | bash - &&    apt-get install --no-install-rec  0.0s
 => CACHED [base 3/4] WORKDIR /workspace                                                                                                                                0.0s
 => CACHED [base 4/4] RUN conda config --set ssl_verify false &&    conda config --add pkgs_dirs /opt/conda/pkgs &&    conda config --env --add channels conda-forge &  0.0s
 => CACHED [conda_bld_deps 1/4] COPY ci/conda/recipes/cudf/ ./ci/conda/recipes/cudf/                                                                                    0.0s
 => CACHED [conda_bld_deps 2/4] COPY ci/conda/recipes/libcudf/ ./ci/conda/recipes/libcudf/                                                                              0.0s
 => CACHED [conda_bld_deps 3/4] COPY ci/conda/recipes/run_conda_build.sh ./ci/conda/recipes/run_conda_build.sh                                                          0.0s
 => CACHED [conda_bld_deps 4/4] RUN --mount=type=ssh     --mount=type=cache,id=workspace_cache,target=/workspace/.cache,sharing=locked     --mount=type=cache,id=conda  0.0s
 => CACHED [conda_env 1/3] RUN --mount=type=bind,from=conda_bld_deps,source=/opt/conda/conda-bld,target=/opt/conda/conda-bld     --mount=type=cache,id=conda_pkgs,targ  0.0s
 => CACHED [conda_env 2/3] RUN source activate morpheus &&    conda config --env --add channels conda-forge &&    conda config --env --add channels nvidia &&    conda  0.0s
 => CACHED [conda_env 3/3] COPY docker/entrypoint.sh ./docker/                                                                                                          0.0s
 => CACHED [runtime  1/10] COPY docker/conda/environments/requirements.txt ./docker/conda/environments/                                                                 0.0s
 => CACHED [runtime  2/10] COPY docker/conda/environments/cuda11.4_runtime.yml ./docker/conda/environments/                                                             0.0s
 => CACHED [conda_bld_morpheus 1/2] COPY . ./                                                                                                                           0.0s
 => ERROR [conda_bld_morpheus 2/2] RUN --mount=type=ssh     --mount=type=cache,id=workspace_cache,target=/workspace/.cache,sharing=locked     --mount=type=cache,id=co  3.3s
------
 > [conda_bld_morpheus 2/2] RUN --mount=type=ssh     --mount=type=cache,id=workspace_cache,target=/workspace/.cache,sharing=locked     --mount=type=cache,id=conda_pkgs,target=/opt/conda/pkgs,sharing=locked     source activate base &&    MORPHEUS_ROOT=/workspace CONDA_BLD_DIR=/opt/conda/conda-bld CONDA_ARGS="--no-test" ./ci/conda/recipes/run_conda_build.sh morpheus:
#23 0.969 CUDA        : 11.4.1
#23 0.969 PYTHON_VER  : 3.8
#23 0.969 NEO_GIT_TAG : 5b55e37c6320c1a5747311a1e29e7ebb049d12bc
#23 0.969
#23 0.999 fatal: No names found, cannot describe anything.
#23 1.001 Running conda-build for morpheus...
#23 1.001 ++ conda mambabuild --use-local --build-id-pat '{n}-{v}' --variants '{python: 3.8}' -c rapidsai -c nvidia -c nvidia/label/dev -c conda-forge --no-test ci/conda/recipes/morpheus
#23 1.717 No numpy version specified in conda_build_config.yaml.  Falling back to default numpy value of 1.16
#23 1.717 WARNING:conda_build.metadata:No numpy version specified in conda_build_config.yaml.  Falling back to default numpy value of 1.16
#23 1.764 Cloning into bare repository '/opt/conda/conda-bld/git_cache/workspace'...
#23 1.778 done.
#23 1.785 Cloning into '/opt/conda/conda-bld/morpheus-split-None/work'...
#23 1.792 done.
#23 1.890 Your branch is up to date with 'origin/branch-22.04'.
#23 1.990 fatal: No names found, cannot describe anything.
#23 2.043 commit 09eb17f105dd0d719da26e8817892d89c0f33334
#23 2.043 Author: Michael Demoret <mdemoret@nvidia.com>
#23 2.043 Date:   Thu Apr 21 17:20:55 2022 -0700
#23 2.043
#23 2.043     Initial Commit
#23 2.043
#23 2.043 commit 09eb17f105dd0d719da26e8817892d89c0f33334
#23 2.043 Author: Michael Demoret <mdemoret@nvidia.com>
#23 2.043 Date:   Thu Apr 21 17:20:55 2022 -0700
#23 2.043
#23 2.043     Initial Commit
#23 2.043
#23 2.043 On branch branch-22.04
#23 2.043 Your branch is up to date with 'origin/branch-22.04'.
#23 2.043
#23 2.043 nothing to commit, working tree clean
#23 2.043
#23 2.043 Updating build index: /opt/conda/conda-bld
#23 2.043
#23 2.043 checkout: 'HEAD'
#23 2.043 ==> /usr/bin/git log -n1 <==
#23 2.043
#23 2.043 ==> /usr/bin/git describe --tags --dirty <==
#23 2.043
#23 2.043 ==> /usr/bin/git status <==
#23 2.043
#23 2.043 Adding in variants from internal_defaults
#23 2.043 INFO:conda_build.variants:Adding in variants from internal_defaults
#23 2.043 Adding in variants from /workspace/ci/conda/recipes/morpheus/conda_build_config.yaml
#23 2.043 INFO:conda_build.variants:Adding in variants from /workspace/ci/conda/recipes/morpheus/conda_build_config.yaml
#23 2.043 Adding in variants from argument_variants
#23 2.043 INFO:conda_build.variants:Adding in variants from argument_variants
#23 3.031 Attempting to finalize metadata for morpheus
#23 3.031 /opt/conda/lib/python3.8/site-packages/conda_build/environ.py:444: UserWarning: The environment variable 'CMAKE_CUDA_ARCHITECTURES' is being passed through with value 'ALL'.  If you are splitting build and test phases with --no-test, please ensure that this value is also set similarly at test time.
#23 3.031   warnings.warn(
#23 3.031 /opt/conda/lib/python3.8/site-packages/conda_build/environ.py:444: UserWarning: The environment variable 'MORPHEUS_CACHE_DIR' is being passed through with value '/workspace/.cache'.  If you are splitting build and test phases with --no-test, please ensure that this value is also set similarly at test time.
#23 3.031   warnings.warn(
#23 3.031 /opt/conda/lib/python3.8/site-packages/conda_build/environ.py:444: UserWarning: The environment variable 'PARALLEL_LEVEL' is being passed through with value '80'.  If you are splitting build and test phases with --no-test, please ensure that this value is also set similarly at test time.
#23 3.031   warnings.warn(
#23 3.031 INFO:conda_build.metadata:Attempting to finalize metadata for morpheus
#23 3.108 Error: Failed to render jinja template in /workspace/ci/conda/recipes/morpheus/meta.yaml:
#23 3.108 list object has no element 1
------
error: failed to solve: executor failed running [/bin/bash -c source activate base &&    MORPHEUS_ROOT=/workspace CONDA_BLD_DIR=/opt/conda/conda-bld CONDA_ARGS="--no-test" ./ci/conda/recipes/run_conda_build.sh morpheus]: exit code: 1
@cwharris
Copy link
Contributor

Can you give some more context? In what environment is this command being run?

@mdemoret-nv mdemoret-nv mentioned this issue Apr 22, 2022
Merged
rapids-bot bot pushed a commit that referenced this issue Mar 24, 2023
@drobison00
Merge pull request #7 from bsuryadevara/bhargav_integrated_training_716
b228419 
Bhargav integrated training
rapids-bot bot pushed a commit that referenced this issue Aug 2, 2023
@cwharris
Reorganize C++ Tests (#1095)
b228389 
I have a few changes here. Some of them might not make it depending on how others feel.

- Move test utilities in to a library to share across tests.
- Move tests in to individual executable to enforce individual tests to not sharing code.
- Use `gtest_discover_tests` to let `ctest` run and show all tests individually.
- Use `gtest::gtest_main` and remove explicit main entry file.
- ~Build and run tests by default.~
- Ability to filter tests by `ctest --test-dir build -R "TestCuda.*"`

```
+ echo 'Running CTest...'
Running CTest...
+ ctest --test-dir build
Internal ctest changing into directory: /workspaces/morpheus/build
Test project /workspaces/morpheus/build
      Start  1: TestCuda.LargeShape
 1/35 Test  #1: TestCuda.LargeShape .......................................   Passed    0.17 sec
      Start  2: TestDataLoader.DataLoaderInitializationTest
 2/35 Test  #2: TestDataLoader.DataLoaderInitializationTest ...............   Passed    0.07 sec
      Start  3: TestDataLoader.DataLoaderRegisterLoaderTest
 3/35 Test  #3: TestDataLoader.DataLoaderRegisterLoaderTest ...............   Passed    0.07 sec
      Start  4: TestDataLoader.DataLoaderRemoveLoaderTest
 4/35 Test  #4: TestDataLoader.DataLoaderRemoveLoaderTest .................   Passed    0.07 sec
      Start  5: TestDataLoader.PayloadLoaderTest
 5/35 Test  #5: TestDataLoader.PayloadLoaderTest ..........................   Passed    1.42 sec
      Start  6: TestDataLoader.FileLoaderTest
 6/35 Test  #6: TestDataLoader.FileLoaderTest .............................   Passed    2.65 sec
      Start  7: TestControlMessage.InitializationTest
 7/35 Test  #7: TestControlMessage.InitializationTest .....................   Passed    0.07 sec
      Start  8: TestControlMessage.SetMessageTest
 8/35 Test  #8: TestControlMessage.SetMessageTest .........................   Passed    0.07 sec
      Start  9: TestControlMessage.TaskTest
 9/35 Test  #9: TestControlMessage.TaskTest ...............................   Passed    0.07 sec
      Start 10: TestControlMessage.PayloadTest
10/35 Test #10: TestControlMessage.PayloadTest ............................   Passed    1.45 sec
      Start 11: TestDataLoaderModule.DataLoaderModuleInitializationTest
11/35 Test #11: TestDataLoaderModule.DataLoaderModuleInitializationTest ...   Passed    0.07 sec
      Start 12: TestDataLoaderModule.EndToEndPayloadDataLoaderTest
12/35 Test #12: TestDataLoaderModule.EndToEndPayloadDataLoaderTest ........   Passed    0.12 sec
      Start 13: TestDeserializers.GetIndexColCountNoIdxFromFile
13/35 Test #13: TestDeserializers.GetIndexColCountNoIdxFromFile ...........   Passed    1.39 sec
      Start 14: TestDeserializers.GetIndexColCountWithIdxFromFile
14/35 Test #14: TestDeserializers.GetIndexColCountWithIdxFromFile .........   Passed    1.37 sec
      Start 15: TestDeserializers.GetIndexColCountNoIdxSimilarName
15/35 Test #15: TestDeserializers.GetIndexColCountNoIdxSimilarName ........   Passed    0.19 sec
      Start 16: TestDeserializers.GetIndexColCountIdx
16/35 Test #16: TestDeserializers.GetIndexColCountIdx .....................   Passed    0.18 sec
      Start 17: TestDeserializers.GetIndexColCountValidNameInvalidType
17/35 Test #17: TestDeserializers.GetIndexColCountValidNameInvalidType ....   Passed    0.16 sec
      Start 18: TestDevMemInfo.RmmBufferConstructor
18/35 Test #18: TestDevMemInfo.RmmBufferConstructor .......................   Passed    0.16 sec
      Start 19: TestDevMemInfo.VoidPtrConstructor
19/35 Test #19: TestDevMemInfo.VoidPtrConstructor .........................   Passed    0.25 sec
      Start 20: TestDevMemInfo.MakeNewBuffer
20/35 Test #20: TestDevMemInfo.MakeNewBuffer ..............................   Passed    0.25 sec
      Start 21: TestFileInOut.RoundTripCSV
21/35 Test #21: TestFileInOut.RoundTripCSV ................................   Passed    2.67 sec
      Start 22: TestFileInOut.RoundTripJSON
22/35 Test #22: TestFileInOut.RoundTripJSON ...............................   Passed    2.69 sec
      Start 23: TestMatxUtil.ReduceMax1d
23/35 Test #23: TestMatxUtil.ReduceMax1d ..................................   Passed    0.17 sec
      Start 24: TestMatxUtil.ReduceMax2dRowMajor
24/35 Test #24: TestMatxUtil.ReduceMax2dRowMajor ..........................   Passed    0.17 sec
      Start 25: TestMatxUtil.ReduceMax2dColMajor
25/35 Test #25: TestMatxUtil.ReduceMax2dColMajor ..........................   Passed    1.40 sec
      Start 26: TestMatxUtil.Cast
26/35 Test #26: TestMatxUtil.Cast .........................................   Passed    0.16 sec
      Start 27: TestMatxUtil.Threshold
27/35 Test #27: TestMatxUtil.Threshold ....................................   Passed    0.16 sec
      Start 28: TestMatxUtil.ThresholdByRow
28/35 Test #28: TestMatxUtil.ThresholdByRow ...............................   Passed    0.19 sec
      Start 29: TestMultiSlices.Ranges
29/35 Test #29: TestMultiSlices.Ranges ....................................   Passed    1.37 sec
      Start 30: TestTensor.UtilsShapeString
30/35 Test #30: TestTensor.UtilsShapeString ...............................   Passed    0.06 sec
      Start 31: TestTensor.GetElementStride
31/35 Test #31: TestTensor.GetElementStride ...............................   Passed    0.06 sec
      Start 32: TestTensor.AsType
32/35 Test #32: TestTensor.AsType .........................................   Passed    0.16 sec
      Start 33: TestTensor.Create
33/35 Test #33: TestTensor.Create .........................................   Passed    0.16 sec
      Start 34: TestTensor.UtilsValidateShapeAndStride
34/35 Test #34: TestTensor.UtilsValidateShapeAndStride ....................   Passed    0.16 sec
      Start 35: TestTypeUtils.DTypeCopy
35/35 Test #35: TestTypeUtils.DTypeCopy ...................................   Passed    0.06 sec

100% tests passed, 0 tests failed out of 35

Total Test time (real) =  19.91 sec
```

Authors:
  - Christopher Harris (https://github.com/cwharris)

Approvers:
  - David Gardner (https://github.com/dagardner-nv)

URL: #1095
rapids-bot bot pushed a commit that referenced this issue Aug 28, 2023
@tzemicheal
Merge pull request #7 from dagardner-nv/david-fsi_dgl-patch7
6c8fc27 
Update expected data and re-enable df check
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Fix Docker Build by Ensuring Cache Directories Exist mdemoret-nv/Morpheus
2 participants
@cwharris @pdmack