Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[BE week] - huggingface repo support for datasets_download.py #2130

Merged
merged 23 commits into from
Jul 17, 2023
+236 −85
Merged

[BE week] - huggingface repo support for datasets_download.py #2130

Show file tree
Hide file tree
Changes from 18 commits
Commits
Show all changes
23 commits
Select commit Hold shift + click to select a range
d5ba914
first draft of huggingface repo support for datasets_download.py with…
aclegg3 Jun 13, 2023
792308c
split zip/tar and git clone flows into functions. Add some typing and…
aclegg3 Jun 14, 2023
79b84e7
Merge remote-tracking branch 'origin/main' into datasets_download-hug…
aclegg3 Jul 10, 2023
3d851a5
add ReplicaCAD public HF repos. Add tag cloning instead of main.
aclegg3 Jul 10, 2023
49d6328
add git-lfs to requirements.txt
aclegg3 Jul 10, 2023
6106eff
install lfs before using datasets_download.py
aclegg3 Jul 11, 2023
566fa6d
install git on CI machine
aclegg3 Jul 11, 2023
d5e45a6
guard clean against missing symbolic link or directory. Add special h…
aclegg3 Jul 11, 2023
1557fb1
install git-lfs before downloading data
aclegg3 Jul 11, 2023
74da7f7
install gitpython
aclegg3 Jul 11, 2023
759d72f
conda install -y
aclegg3 Jul 11, 2023
afa42c3
try to upgrade git on CI
aclegg3 Jul 11, 2023
76b7cbe
make prune optional for compat. with older CI machine git version
aclegg3 Jul 12, 2023
f58b619
update downloader to reference new HF datasources
aclegg3 Jul 12, 2023
54b7407
test against lab branch
aclegg3 Jul 12, 2023
67c95a4
fetch all branches in lab clone for testing
aclegg3 Jul 12, 2023
0e9277e
refactor repo handling to replace symblink as necessary when checking…
aclegg3 Jul 12, 2023
0f01b0c
cleanup
aclegg3 Jul 13, 2023
b4c5125
add checking for git-lfs with warnings and assertions. Pull repos wit…
aclegg3 Jul 13, 2023
1487274
Merge remote-tracking branch 'origin/main' into datasets_download-hug…
aclegg3 Jul 14, 2023
fbc572a
update README and DATASETS documentation to reflect huggingface migra…
aclegg3 Jul 14, 2023
7151e62
doc change
aclegg3 Jul 14, 2023
7e3ab99
revert testing specific changes
aclegg3 Jul 17, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
13 changes: 11 additions & 2 deletions .circleci/config.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -400,13 +400,18 @@ jobs:
- run: &download_test_data
name: Download test data
command: |
sudo apt install git-lfs
git --version
git-lfs --version
export PATH=$HOME/miniconda/bin:/usr/local/cuda/bin:$PATH
. activate habitat;
git lfs install
conda install -y gitpython

if [ ! -d ./data/scene_datasets/habitat-test-scenes/van-gogh-room.glb ]
then
cd habitat-sim
python src_python/habitat_sim/utils/datasets_download.py --uids ci_test_assets --replace --data-path data/
python src_python/habitat_sim/utils/datasets_download.py --uids ci_test_assets --replace --data-path data/ --no-prune
fi

- run:
Expand Down Expand Up @@ -445,9 +450,13 @@ jobs:
cd ..
if [ ! -d ./habitat-lab ]
then
git clone -q --depth 1 https://github.com/facebookresearch/habitat-lab.git
git clone -q https://github.com/facebookresearch/habitat-lab.git
fi
cd habitat-lab

# test against the modified branch
git checkout --track origin/huggingface-data-migration

pip install -r habitat-lab/requirements.txt --progress-bar off
ln -s ../habitat-sim/data data
touch ~/miniconda/pip_deps_installed
Expand Down
Loading