Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Use pip-compile to help with consistent Python dependency resolution #371

Merged
merged 151 commits into from
Nov 21, 2023
Merged
Changes from 1 commit
Commits
Show all changes
151 commits
Select commit Hold shift + click to select a range
abb6f97
wip
yhtang Oct 23, 2023
068aab9
fix typo
yhtang Oct 23, 2023
fb7cf0b
wip
yhtang Oct 24, 2023
4b0c406
wip
yhtang Oct 24, 2023
4044450
wip
yhtang Oct 24, 2023
38da8a1
wip
yhtang Oct 24, 2023
9ad94c8
wip
yhtang Oct 24, 2023
1aac03a
use full clone
yhtang Oct 25, 2023
42ec9bb
update pip-tools script
yhtang Oct 25, 2023
2f02023
update pip-tools script
yhtang Oct 25, 2023
ad12e78
update pip-tools script
yhtang Oct 25, 2023
bdc34c8
fix t5x dockerfile
yhtang Oct 25, 2023
a5c478e
fix t5x dockerfile
yhtang Oct 25, 2023
b4fd2d8
test flax hack
yhtang Oct 25, 2023
fe6e3d8
flax hack
yhtang Oct 25, 2023
ad83f21
hack for git top of tree Flax dependency
yhtang Nov 2, 2023
b7e1a6c
update URL req
yhtang Nov 3, 2023
0dd9617
update
yhtang Nov 3, 2023
dc44fe4
editability
yhtang Nov 3, 2023
acf2acf
editability
yhtang Nov 3, 2023
027cd63
editability
yhtang Nov 3, 2023
7c10ffa
editability
yhtang Nov 3, 2023
f1046d2
editability
yhtang Nov 3, 2023
bbf2c21
wip
yhtang Nov 3, 2023
cd0d5d1
wip
yhtang Nov 6, 2023
e973c9a
wip
yhtang Nov 6, 2023
1f6d4f1
fix shell
yhtang Nov 6, 2023
6d9ae00
fix arg order
yhtang Nov 6, 2023
cc66ce6
remove standalone TE build
yhtang Nov 6, 2023
fe8708a
build TE wheel in JAX
yhtang Nov 6, 2023
b1e332e
pax wip
yhtang Nov 6, 2023
62ca85b
add pax build
yhtang Nov 6, 2023
bc6c702
add pax build
yhtang Nov 6, 2023
ff6ec2a
fix CI
yhtang Nov 6, 2023
67df9b8
debug pax build
yhtang Nov 6, 2023
d67966a
debug pax build
yhtang Nov 6, 2023
1ba1bff
debug pax build
yhtang Nov 6, 2023
e8f87d2
fix EOF
yhtang Nov 6, 2023
24b4931
redesign workflow
yhtang Nov 7, 2023
e2c34b4
fix job step id
yhtang Nov 7, 2023
7ee441b
arm64 build
yhtang Nov 7, 2023
4f4d909
arm64 build
yhtang Nov 7, 2023
b777244
arm64 build
yhtang Nov 8, 2023
c4c22af
arm64 build
yhtang Nov 8, 2023
736246a
add sitrep to base build
yhtang Nov 8, 2023
bc4b6db
lingvo
yhtang Nov 8, 2023
ce1cf94
lingvo
yhtang Nov 8, 2023
9ac7367
lingvo
yhtang Nov 8, 2023
44d3026
refactor pax arm64 build
yhtang Nov 8, 2023
5c47fcb
refactor pax arm64 build wip
yhtang Nov 8, 2023
139e539
refactor pax arm64 build wip
yhtang Nov 8, 2023
5e2a5ad
refactor pax arm64 build wip
yhtang Nov 8, 2023
cb99d71
refactor pax arm64 build wip
yhtang Nov 9, 2023
0959264
pax arm64
yhtang Nov 9, 2023
314db99
redesign CI
yhtang Nov 9, 2023
8943e9f
redesign CI
yhtang Nov 9, 2023
94115ba
refactor CI
yhtang Nov 9, 2023
ee11851
refactor CI
yhtang Nov 9, 2023
6b6fc92
refactor CI
yhtang Nov 9, 2023
cf66cfd
refactor CI
yhtang Nov 9, 2023
9fd4503
refactor CI
yhtang Nov 9, 2023
f2e80a1
file permission
yhtang Nov 9, 2023
7db2f84
refactor CI
yhtang Nov 9, 2023
7090349
refactor CI
yhtang Nov 9, 2023
6900d86
refactor CI
yhtang Nov 9, 2023
06327c1
refactor CI
yhtang Nov 9, 2023
618a3f5
refactor CI
yhtang Nov 9, 2023
c659d3c
refactor CI
yhtang Nov 9, 2023
8395604
refactor CI
yhtang Nov 9, 2023
2ab0cc9
refactor CI
yhtang Nov 9, 2023
69c17fc
refactor CI
yhtang Nov 9, 2023
d9400d3
refactor CI
yhtang Nov 9, 2023
33dc9ac
refactor CI
yhtang Nov 9, 2023
4fc18dc
fix output tag order
yhtang Nov 9, 2023
b806a5a
t5x arm64 build not ready yet
yhtang Nov 9, 2023
8391f74
nightly T5X build
yhtang Nov 9, 2023
984a19a
nightly T5X build
yhtang Nov 9, 2023
715f62c
fix TE/T5X bug
yhtang Nov 9, 2023
7f06cb8
add TE examples and tests to wheel
yhtang Nov 9, 2023
92b6d0a
allow TE parallel build
yhtang Nov 9, 2023
c4f5b84
jax publish
yhtang Nov 9, 2023
bcfd0e4
rename staging to mealkit
yhtang Nov 9, 2023
12a2fd6
fix nightly
yhtang Nov 9, 2023
1925b14
fix nightly
yhtang Nov 9, 2023
6e71c64
bug fix
yhtang Nov 10, 2023
adb10da
bug fix
yhtang Nov 10, 2023
b956b30
bug fix
yhtang Nov 10, 2023
c727607
fix
yhtang Nov 10, 2023
d450ceb
fix TE test
yhtang Nov 10, 2023
fc2c6e6
fix pax test
yhtang Nov 10, 2023
f9c6cd3
fix TE test
yhtang Nov 10, 2023
7230589
merge CI yaml
yhtang Nov 10, 2023
429ae4d
fix arg
yhtang Nov 10, 2023
de32e4f
rerun TE/PAX test
yhtang Nov 10, 2023
43a57c6
Merge branch 'main' of github.com:NVIDIA/JAX-Toolbox into add-pip-com…
yhtang Nov 10, 2023
ab73f6b
fix TE multi-device test
yhtang Nov 10, 2023
9eb97e8
fix lzma build issue
yhtang Nov 10, 2023
772f606
edit TE test name
yhtang Nov 11, 2023
fcb29b4
fix TE arm64 test install error
yhtang Nov 13, 2023
22d400b
remove --install option from get-source.sh
yhtang Nov 13, 2023
e9f074f
fix TE arm64 test install error
yhtang Nov 13, 2023
602002f
disable sandbox
yhtang Nov 13, 2023
12a57eb
i'm jet-lagged
yhtang Nov 13, 2023
dbaba5b
use Pax image for TE testing
yhtang Nov 13, 2023
ccafb52
Fix job dependency
yhtang Nov 13, 2023
6974a3a
Add nightly rosetta build and test
DwarKapex Nov 16, 2023
c5f8f23
wip: fix typo
DwarKapex Nov 16, 2023
1d9d282
wip: build sandbox on dispatch
DwarKapex Nov 16, 2023
e6bf405
wip: fix build_rosetta
DwarKapex Nov 16, 2023
4c58356
wip: update yaml structure for nightly build of rosseta y5x
DwarKapex Nov 16, 2023
1fe5f69
wip: update yaml structure for nightly build of rosseta t5x
DwarKapex Nov 16, 2023
95f5729
wip: fix yaml structure for nightly build of rosseta t5x
DwarKapex Nov 16, 2023
d89f93f
wip: fix yaml structure for nightly build of rosseta pax
DwarKapex Nov 16, 2023
08fb7f9
wip: fix typo in yaml structure for nightly build of rosseta pax
DwarKapex Nov 16, 2023
a7f95f1
wip: fix typo in yaml structure for nightly build of rosseta pax 2
DwarKapex Nov 16, 2023
2d35714
wip
DwarKapex Nov 16, 2023
5ae2e79
wip:
DwarKapex Nov 16, 2023
ce8ba2d
use the _publish_container reusable workflow for base container weekl…
yhtang Nov 16, 2023
6501ae9
fix base build output arg name error
yhtang Nov 16, 2023
aae5865
wip: add base build
DwarKapex Nov 16, 2023
a98bf46
Merge branch 'add-pip-compile' of github.com:NVIDIA/JAX-Toolbox into …
DwarKapex Nov 16, 2023
94d2db7
wip: add base build
DwarKapex Nov 16, 2023
9bb80b4
wip: add base build
DwarKapex Nov 16, 2023
3442369
wip: build t5x and rosetta/pax
DwarKapex Nov 16, 2023
f567e2c
wip: build all in question
DwarKapex Nov 16, 2023
efb339c
wip: build all in question 2
DwarKapex Nov 16, 2023
dca817c
Build whole pipeline
DwarKapex Nov 16, 2023
356adbf
Merge origin/main
DwarKapex Nov 16, 2023
44d7897
Build rosetta in CI pipeline
DwarKapex Nov 16, 2023
420162f
Build rosetta in CI pipeline
DwarKapex Nov 16, 2023
2a12b05
Debug rosetta build
DwarKapex Nov 16, 2023
507c6c1
Debug rosetta build: correct container
DwarKapex Nov 16, 2023
81d03ef
Debug rosetta build: fix odd issue from docker file
DwarKapex Nov 17, 2023
f546c9b
Full pipelilne 2
DwarKapex Nov 17, 2023
85a8817
Addressed Ann's comments
DwarKapex Nov 17, 2023
630d1df
Full pipelilne: revert _sandbox.yaml
DwarKapex Nov 17, 2023
dc37804
Merge branch 'add-pip-compile' of github.com:NVIDIA/JAX-Toolbox into …
DwarKapex Nov 17, 2023
1cf80c2
Rename nightly rosseta to be able to sun nigthly rosseta pax build ma…
DwarKapex Nov 17, 2023
0885302
Add arch to rosetta-pax build
DwarKapex Nov 17, 2023
a97659e
Add arch to rosetta-pax build
DwarKapex Nov 17, 2023
2744e67
Add arch to rosetta-pax build: fix typo
DwarKapex Nov 17, 2023
858881b
Add arch to rosetta-pax build: fix typo
DwarKapex Nov 17, 2023
2d6a11c
Add arch to rosetta-pax build: fix typo
DwarKapex Nov 17, 2023
332bf98
Addressed Terry's comments
DwarKapex Nov 20, 2023
2bd1999
Pass git username and email thru params
DwarKapex Nov 20, 2023
873b5a5
Address Terry's final comments
DwarKapex Nov 20, 2023
770990d
Address Terry's LGTM comments
DwarKapex Nov 21, 2023
4ae4189
Rosetta dockerfiles to have 2 stages: mealkit and final
DwarKapex Nov 21, 2023
85182ef
Rosetta dockerfiles: remore __pychache__ deletion
DwarKapex Nov 21, 2023
6e7b418
Merge origin/main
DwarKapex Nov 21, 2023
4ea1742
Reverte test_destribution
DwarKapex Nov 21, 2023
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
wip: build t5x and rosetta/pax
DwarKapex committed Nov 16, 2023
commit 3442369981e50a672519df52880c2f376fec97ae
56 changes: 32 additions & 24 deletions .github/workflows/_sandbox.yaml
Original file line number Diff line number Diff line change
@@ -11,42 +11,50 @@ permissions:

jobs:

build-base:
uses: ./.github/workflows/_build_base.yaml
with:
ARCHITECTURE: amd64
secrets: inherit
# build-base:
# uses: ./.github/workflows/_build_base.yaml
# with:
# ARCHITECTURE: amd64
# secrets: inherit

build-jax:
needs: [build-base]
uses: ./.github/workflows/_build_jax.yaml
with:
ARCHITECTURE: amd64
BASE_IMAGE: ${{ needs.build-base.outputs.DOCKER_TAG }}
secrets: inherit
# build-jax:
# needs: [build-base]
# uses: ./.github/workflows/_build_jax.yaml
# with:
# ARCHITECTURE: amd64
# BASE_IMAGE: ${{ needs.build-base.outputs.DOCKER_TAG }}
# secrets: inherit

build-pax:
needs: [build-jax]
uses: ./.github/workflows/_build_pax.yaml
# build-pax:
# needs: [build-jax]
# uses: ./.github/workflows/_build_pax.yaml
# with:
# ARCHITECTURE: amd64
# BASE_IMAGE: ${{ needs.build-jax.outputs.DOCKER_TAG_MEALKIT }}
# secrets: inherit

build-t5x:
# needs: [build-jax]
uses: ./.github/workflows/_build_t5x.yaml
with:
ARCHITECTURE: amd64
BASE_IMAGE: ${{ needs.build-jax.outputs.DOCKER_TAG_MEALKIT }}
BASE_IMAGE: ghcr.io/nvidia/jax-toolbox-internal:6894053657-jax-amd64
secrets: inherit

test-te:
needs: [build-pax]
uses: ./.github/workflows/_test_te.yaml
with:
TE_IMAGE: ${{ needs.build-pax.outputs.DOCKER_TAG_FINAL }}
secrets: inherit
# test-te:
# needs: [build-pax]
# uses: ./.github/workflows/_test_te.yaml
# with:
# TE_IMAGE: ${{ needs.build-pax.outputs.DOCKER_TAG_FINAL }}
# secrets: inherit

build-rosetta:
needs: [build-pax]
# needs: [build-pax]
uses: ./.github/workflows/_build_rosetta.yaml
with:
ARCHITECTURE: amd64
BASE_LIBRARY: pax
BASE_IMAGE: ${{ needs.build-pax.outputs.DOCKER_TAG_FINAL }}
BASE_IMAGE: ghcr.io/nvidia/jax-toolbox-internal:6894053657-upstream-pax-amd64

# test-pax:
# needs: [build-pax]