Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Restructure container builds to use multi-stage builds and a Merlin base image #234

Merged
merged 28 commits into from
Apr 26, 2022

Conversation

karlhigley
Copy link
Contributor

@karlhigley karlhigley commented Apr 25, 2022

This also adds container tests with container-structure-test.

Final sizes of the combined training/inference containers for each framework:

Container Tag Size
merlin/base 22.05-pre 8.88GB
merlin/hugectr 22.05-pre 10.6GB
merlin/tf 22.05-pre 13.2GB
merlin/torch 22.05-pre 13.6GB

@karlhigley karlhigley added this to the Merlin 22.05 milestone Apr 25, 2022
@karlhigley karlhigley self-assigned this Apr 25, 2022
@karlhigley karlhigley linked an issue Apr 25, 2022 that may be closed by this pull request
3 tasks
@github-actions
Copy link

Documentation preview

https://nvidia-merlin.github.io/Merlin/review/pr-234

@nvidia-merlin-bot
Copy link
Contributor

Click to view CI Results
GitHub pull request #234 of commit 83307dba3e23fae9f1a68f30b840c44dc9d4369f, no merge conflicts.
Running as SYSTEM
Setting status of 83307dba3e23fae9f1a68f30b840c44dc9d4369f to PENDING with url https://10.20.13.93:8080/job/merlin_merlin/47/console and message: 'Pending'
Using context: Jenkins
Building on master in workspace /var/jenkins_home/workspace/merlin_merlin
using credential systems-login
 > git rev-parse --is-inside-work-tree # timeout=10
Fetching changes from the remote Git repository
 > git config remote.origin.url https://github.com/NVIDIA-Merlin/Merlin # timeout=10
Fetching upstream changes from https://github.com/NVIDIA-Merlin/Merlin
 > git --version # timeout=10
using GIT_ASKPASS to set credentials login for merlin-systems
 > git fetch --tags --force --progress -- https://github.com/NVIDIA-Merlin/Merlin +refs/pull/234/*:refs/remotes/origin/pr/234/* # timeout=10
 > git rev-parse 83307dba3e23fae9f1a68f30b840c44dc9d4369f^{commit} # timeout=10
Checking out Revision 83307dba3e23fae9f1a68f30b840c44dc9d4369f (detached)
 > git config core.sparsecheckout # timeout=10
 > git checkout -f 83307dba3e23fae9f1a68f30b840c44dc9d4369f # timeout=10
Commit message: "Add a container test for HugeCTR"
 > git rev-list --no-walk 9f88235c76a3e8eaf14142179b2422ba2737a9a6 # timeout=10
[merlin_merlin] $ /bin/bash /tmp/jenkins8788969305069239975.sh
============================= test session starts ==============================
platform linux -- Python 3.8.10, pytest-7.1.1, pluggy-1.0.0
rootdir: /var/jenkins_home/workspace/merlin_merlin/merlin
plugins: xdist-2.5.0, forked-1.4.0, cov-3.0.0
collected 1 item

tests/unit/test_version.py . [100%]

============================== 1 passed in 0.01s ===============================
Performing Post build task...
Match found for : : True
Logical operation result is TRUE
Running script : #!/bin/bash
cd /var/jenkins_home/
CUDA_VISIBLE_DEVICES=1 python test_res_push.py "https://api.GitHub.com/repos/NVIDIA-Merlin/Merlin/issues/$ghprbPullId/comments" "/var/jenkins_home/jobs/$JOB_NAME/builds/$BUILD_NUMBER/log"
[merlin_merlin] $ /bin/bash /tmp/jenkins4300395400609222406.sh

@karlhigley karlhigley changed the title Refactor container builds to use multi-stage builds and a Merlin base image Restructure container builds to use multi-stage builds and a Merlin base image Apr 26, 2022
@jperez999 jperez999 merged commit c2c429f into main Apr 26, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Use multi-stage builds to create our containers
3 participants