Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add Arrow S3 support to the tensorflow-training container #217

Merged
merged 1 commit into from
Apr 18, 2022

Conversation

karlhigley
Copy link
Contributor

This makes it possible to export an ensemble config from the training container, since Feast requires Arrow to have S3 support.

This makes it possible to export an ensemble config from the training container, since Feast requires Arrow to have S3 support.
@karlhigley karlhigley added the enhancement New feature or request label Apr 17, 2022
@karlhigley karlhigley self-assigned this Apr 17, 2022
@karlhigley
Copy link
Contributor Author

This is related to #216. It looks like we can do it and still sneak in under the 15GB cap:

REPOSITORY TAG IMAGE ID CREATED SIZE
nvcr.io/nvidia/merlin/merlin-tensorflow-training arrow-s3 4bc70b8f0130 11 seconds ago 14.6GB
nvcr.io/nvidia/merlin/merlin-tensorflow-training 22.04 8a9d8ddb1aa1 10 days ago 14.3GB

@nvidia-merlin-bot
Copy link
Contributor

Click to view CI Results
GitHub pull request #217 of commit 8828ae38cd80839610c038720de747c7baed6af3, no merge conflicts.
Running as SYSTEM
Setting status of 8828ae38cd80839610c038720de747c7baed6af3 to PENDING with url https://10.20.13.93:8080/job/merlin_merlin/42/console and message: 'Pending'
Using context: Jenkins
Building on master in workspace /var/jenkins_home/workspace/merlin_merlin
using credential systems-login
 > git rev-parse --is-inside-work-tree # timeout=10
Fetching changes from the remote Git repository
 > git config remote.origin.url https://github.com/NVIDIA-Merlin/Merlin # timeout=10
Fetching upstream changes from https://github.com/NVIDIA-Merlin/Merlin
 > git --version # timeout=10
using GIT_ASKPASS to set credentials login for merlin-systems
 > git fetch --tags --force --progress -- https://github.com/NVIDIA-Merlin/Merlin +refs/pull/217/*:refs/remotes/origin/pr/217/* # timeout=10
 > git rev-parse 8828ae38cd80839610c038720de747c7baed6af3^{commit} # timeout=10
Checking out Revision 8828ae38cd80839610c038720de747c7baed6af3 (detached)
 > git config core.sparsecheckout # timeout=10
 > git checkout -f 8828ae38cd80839610c038720de747c7baed6af3 # timeout=10
Commit message: "Add Arrow S3 support to the `tensorflow-training` container"
 > git rev-list --no-walk 01d7726bf777a303d96998a2457afdd5d7ec36e5 # timeout=10
[merlin_merlin] $ /bin/bash /tmp/jenkins6955755692178741795.sh
============================= test session starts ==============================
platform linux -- Python 3.8.10, pytest-7.1.1, pluggy-1.0.0
rootdir: /var/jenkins_home/workspace/merlin_merlin/merlin
plugins: xdist-2.5.0, forked-1.4.0, cov-3.0.0
collected 1 item

tests/unit/test_version.py . [100%]

============================== 1 passed in 0.01s ===============================
Performing Post build task...
Match found for : : True
Logical operation result is TRUE
Running script : #!/bin/bash
cd /var/jenkins_home/
CUDA_VISIBLE_DEVICES=1 python test_res_push.py "https://api.GitHub.com/repos/NVIDIA-Merlin/Merlin/issues/$ghprbPullId/comments" "/var/jenkins_home/jobs/$JOB_NAME/builds/$BUILD_NUMBER/log"
[merlin_merlin] $ /bin/bash /tmp/jenkins8374834697445817492.sh

@github-actions
Copy link

Documentation preview

https://nvidia-merlin.github.io/Merlin/review/pr-217

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants