Skip to content

Commit cca59ce

Browse files
anton-lpcuenca
anton-l
and
pcuenca
authored
Add Apple M1 tests (open-mmlab#796)
* [CI] Add Apple M1 tests * setup-python * python build * conda install * remove branch * only 3.8 is built for osx-arm * try fetching prebuilt tokenizers * use user cache * update shells * Reports and cleanup * -> MPS * Disable parallel tests * Better naming * investigate worker crash * return xdist * restart * num_workers=2 * still crashing? * faulthandler for segfaults * faulthandler for segfaults * remove restarts, stop on segfault * torch version * change installation order * Use pre-RC version of PyTorch. To be updated when it is released. * Skip crashing test on MPS, add new one that works. * Skip cuda tests in mps device. * Actually use generator in test. I think this was a typo. * make style Co-authored-by: Pedro Cuenca <pedro@huggingface.co>
1 parent 627ad6e commit cca59ce

File tree

7 files changed

+235
-12
lines changed

7 files changed

+235
-12
lines changed

‎.github/actions/setup-miniconda/action.yml

+146
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,146 @@
1+
name: Set up conda environment for testing
2+
3+
description: Sets up miniconda in your ${RUNNER_TEMP} environment and gives you the ${CONDA_RUN} environment variable so you don't have to worry about polluting non-empeheral runners anymore
4+
5+
inputs:
6+
python-version:
7+
description: If set to any value, dont use sudo to clean the workspace
8+
required: false
9+
type: string
10+
default: "3.9"
11+
miniconda-version:
12+
description: Miniconda version to install
13+
required: false
14+
type: string
15+
default: "4.12.0"
16+
environment-file:
17+
description: Environment file to install dependencies from
18+
required: false
19+
type: string
20+
default: ""
21+
22+
runs:
23+
using: composite
24+
steps:
25+
# Use the same trick from https://github.com/marketplace/actions/setup-miniconda
26+
# to refresh the cache daily. This is kind of optional though
27+
- name: Get date
28+
id: get-date
29+
shell: bash
30+
run: echo "::set-output name=today::$(/bin/date -u '+%Y%m%d')d"
31+
- name: Setup miniconda cache
32+
id: miniconda-cache
33+
uses: actions/cache@v2
34+
with:
35+
path: ${{ runner.temp }}/miniconda
36+
key: miniconda-${{ runner.os }}-${{ runner.arch }}-${{ inputs.python-version }}-${{ steps.get-date.outputs.today }}
37+
- name: Install miniconda (${{ inputs.miniconda-version }})
38+
if: steps.miniconda-cache.outputs.cache-hit != 'true'
39+
env:
40+
MINICONDA_VERSION: ${{ inputs.miniconda-version }}
41+
shell: bash -l {0}
42+
run: |
43+
MINICONDA_INSTALL_PATH="${RUNNER_TEMP}/miniconda"
44+
mkdir -p "${MINICONDA_INSTALL_PATH}"
45+
case ${RUNNER_OS}-${RUNNER_ARCH} in
46+
Linux-X64)
47+
MINICONDA_ARCH="Linux-x86_64"
48+
;;
49+
macOS-ARM64)
50+
MINICONDA_ARCH="MacOSX-arm64"
51+
;;
52+
macOS-X64)
53+
MINICONDA_ARCH="MacOSX-x86_64"
54+
;;
55+
*)
56+
echo "::error::Platform ${RUNNER_OS}-${RUNNER_ARCH} currently unsupported using this action"
57+
exit 1
58+
;;
59+
esac
60+
MINICONDA_URL="https://repo.anaconda.com/miniconda/Miniconda3-py39_${MINICONDA_VERSION}-${MINICONDA_ARCH}.sh"
61+
curl -fsSL "${MINICONDA_URL}" -o "${MINICONDA_INSTALL_PATH}/miniconda.sh"
62+
bash "${MINICONDA_INSTALL_PATH}/miniconda.sh" -b -u -p "${MINICONDA_INSTALL_PATH}"
63+
rm -rf "${MINICONDA_INSTALL_PATH}/miniconda.sh"
64+
- name: Update GitHub path to include miniconda install
65+
shell: bash
66+
run: |
67+
MINICONDA_INSTALL_PATH="${RUNNER_TEMP}/miniconda"
68+
echo "${MINICONDA_INSTALL_PATH}/bin" >> $GITHUB_PATH
69+
- name: Setup miniconda env cache (with env file)
70+
id: miniconda-env-cache-env-file
71+
if: ${{ runner.os }} == 'macOS' && ${{ inputs.environment-file }} != ''
72+
uses: actions/cache@v2
73+
with:
74+
path: ${{ runner.temp }}/conda-python-${{ inputs.python-version }}
75+
key: miniconda-env-${{ runner.os }}-${{ runner.arch }}-${{ inputs.python-version }}-${{ steps.get-date.outputs.today }}-${{ hashFiles(inputs.environment-file) }}
76+
- name: Setup miniconda env cache (without env file)
77+
id: miniconda-env-cache
78+
if: ${{ runner.os }} == 'macOS' && ${{ inputs.environment-file }} == ''
79+
uses: actions/cache@v2
80+
with:
81+
path: ${{ runner.temp }}/conda-python-${{ inputs.python-version }}
82+
key: miniconda-env-${{ runner.os }}-${{ runner.arch }}-${{ inputs.python-version }}-${{ steps.get-date.outputs.today }}
83+
- name: Setup conda environment with python (v${{ inputs.python-version }})
84+
if: steps.miniconda-env-cache-env-file.outputs.cache-hit != 'true' && steps.miniconda-env-cache.outputs.cache-hit != 'true'
85+
shell: bash
86+
env:
87+
PYTHON_VERSION: ${{ inputs.python-version }}
88+
ENV_FILE: ${{ inputs.environment-file }}
89+
run: |
90+
CONDA_BASE_ENV="${RUNNER_TEMP}/conda-python-${PYTHON_VERSION}"
91+
ENV_FILE_FLAG=""
92+
if [[ -f "${ENV_FILE}" ]]; then
93+
ENV_FILE_FLAG="--file ${ENV_FILE}"
94+
elif [[ -n "${ENV_FILE}" ]]; then
95+
echo "::warning::Specified env file (${ENV_FILE}) not found, not going to include it"
96+
fi
97+
conda create \
98+
--yes \
99+
--prefix "${CONDA_BASE_ENV}" \
100+
"python=${PYTHON_VERSION}" \
101+
${ENV_FILE_FLAG} \
102+
cmake=3.22 \
103+
conda-build=3.21 \
104+
ninja=1.10 \
105+
pkg-config=0.29 \
106+
wheel=0.37
107+
- name: Clone the base conda environment and update GitHub env
108+
shell: bash
109+
env:
110+
PYTHON_VERSION: ${{ inputs.python-version }}
111+
CONDA_BASE_ENV: ${{ runner.temp }}/conda-python-${{ inputs.python-version }}
112+
run: |
113+
CONDA_ENV="${RUNNER_TEMP}/conda_environment_${GITHUB_RUN_ID}"
114+
conda create \
115+
--yes \
116+
--prefix "${CONDA_ENV}" \
117+
--clone "${CONDA_BASE_ENV}"
118+
# TODO: conda-build could not be cloned because it hardcodes the path, so it
119+
# could not be cached
120+
conda install --yes -p ${CONDA_ENV} conda-build=3.21
121+
echo "CONDA_ENV=${CONDA_ENV}" >> "${GITHUB_ENV}"
122+
echo "CONDA_RUN=conda run -p ${CONDA_ENV} --no-capture-output" >> "${GITHUB_ENV}"
123+
echo "CONDA_BUILD=conda run -p ${CONDA_ENV} conda-build" >> "${GITHUB_ENV}"
124+
echo "CONDA_INSTALL=conda install -p ${CONDA_ENV}" >> "${GITHUB_ENV}"
125+
- name: Get disk space usage and throw an error for low disk space
126+
shell: bash
127+
run: |
128+
echo "Print the available disk space for manual inspection"
129+
df -h
130+
# Set the minimum requirement space to 4GB
131+
MINIMUM_AVAILABLE_SPACE_IN_GB=4
132+
MINIMUM_AVAILABLE_SPACE_IN_KB=$(($MINIMUM_AVAILABLE_SPACE_IN_GB * 1024 * 1024))
133+
# Use KB to avoid floating point warning like 3.1GB
134+
df -k | tr -s ' ' | cut -d' ' -f 4,9 | while read -r LINE;
135+
do
136+
AVAIL=$(echo $LINE | cut -f1 -d' ')
137+
MOUNT=$(echo $LINE | cut -f2 -d' ')
138+
if [ "$MOUNT" = "/" ]; then
139+
if [ "$AVAIL" -lt "$MINIMUM_AVAILABLE_SPACE_IN_KB" ]; then
140+
echo "There is only ${AVAIL}KB free space left in $MOUNT, which is less than the minimum requirement of ${MINIMUM_AVAILABLE_SPACE_IN_KB}KB. Please help create an issue to PyTorch Release Engineering via https://github.com/pytorch/test-infra/issues and provide the link to the workflow run."
141+
exit 1;
142+
else
143+
echo "There is ${AVAIL}KB free space left in $MOUNT, continue"
144+
fi
145+
fi
146+
done

‎.github/workflows/pr_tests.yml

+53-5
Original file line numberDiff line numberDiff line change
@@ -1,4 +1,4 @@
1-
name: Run non-slow tests
1+
name: Run fast tests
22

33
on:
44
pull_request:
@@ -10,14 +10,14 @@ concurrency:
1010
cancel-in-progress: true
1111

1212
env:
13-
HF_HOME: /mnt/cache
1413
OMP_NUM_THREADS: 8
1514
MKL_NUM_THREADS: 8
1615
PYTEST_TIMEOUT: 60
16+
MPS_TORCH_VERSION: 1.13.0
1717

1818
jobs:
1919
run_tests_cpu:
20-
name: Diffusers tests
20+
name: CPU tests on Ubuntu
2121
runs-on: [ self-hosted, docker-gpu ]
2222
container:
2323
image: python:3.7
@@ -39,7 +39,7 @@ jobs:
3939
run: |
4040
python utils/print_env.py
4141
42-
- name: Run all non-slow selected tests on CPU
42+
- name: Run all fast tests on CPU
4343
run: |
4444
python -m pytest -n 2 --max-worker-restart=0 --dist=loadfile -s -v --make-reports=tests_torch_cpu tests/
4545
@@ -51,5 +51,53 @@ jobs:
5151
if: ${{ always() }}
5252
uses: actions/upload-artifact@v2
5353
with:
54-
name: pr_torch_test_reports
54+
name: pr_torch_cpu_test_reports
55+
path: reports
56+
57+
run_tests_apple_m1:
58+
name: MPS tests on Apple M1
59+
runs-on: [ self-hosted, apple-m1 ]
60+
61+
steps:
62+
- name: Checkout diffusers
63+
uses: actions/checkout@v3
64+
with:
65+
fetch-depth: 2
66+
67+
- name: Clean checkout
68+
shell: arch -arch arm64 bash {0}
69+
run: |
70+
git clean -fxd
71+
72+
- name: Setup miniconda
73+
uses: ./.github/actions/setup-miniconda
74+
with:
75+
python-version: 3.9
76+
77+
- name: Install dependencies
78+
shell: arch -arch arm64 bash {0}
79+
run: |
80+
${CONDA_RUN} python -m pip install --upgrade pip
81+
${CONDA_RUN} python -m pip install -e .[quality,test]
82+
${CONDA_RUN} python -m pip install --pre torch==${MPS_TORCH_VERSION} --extra-index-url https://download.pytorch.org/whl/test/cpu
83+
84+
- name: Environment
85+
shell: arch -arch arm64 bash {0}
86+
run: |
87+
${CONDA_RUN} python utils/print_env.py
88+
89+
- name: Run all fast tests on MPS
90+
shell: arch -arch arm64 bash {0}
91+
run: |
92+
${CONDA_RUN} python -m pytest -n 4 --max-worker-restart=0 --dist=loadfile -s -v --make-reports=tests_torch_mps tests/
93+
94+
- name: Failure short reports
95+
if: ${{ failure() }}
96+
run: cat reports/tests_torch_mps_failures_short.txt
97+
98+
- name: Test suite reports artifacts
99+
if: ${{ always() }}
100+
uses: actions/upload-artifact@v2
101+
with:
102+
name: pr_torch_mps_test_reports
55103
path: reports

‎tests/test_layers_utils.py

+27
Original file line numberDiff line numberDiff line change
@@ -221,6 +221,9 @@ def test_downsample_with_conv_out_dim(self):
221221

222222

223223
class AttentionBlockTests(unittest.TestCase):
224+
@unittest.skipIf(
225+
torch_device == "mps", "Matmul crashes on MPS, see https://github.com/pytorch/pytorch/issues/84039"
226+
)
224227
def test_attention_block_default(self):
225228
torch.manual_seed(0)
226229
if torch.cuda.is_available():
@@ -245,6 +248,30 @@ def test_attention_block_default(self):
245248
)
246249
assert torch.allclose(output_slice.flatten(), expected_slice, atol=1e-3)
247250

251+
def test_attention_block_sd(self):
252+
# This version uses SD params and is compatible with mps
253+
torch.manual_seed(0)
254+
if torch.cuda.is_available():
255+
torch.cuda.manual_seed_all(0)
256+
257+
sample = torch.randn(1, 512, 64, 64).to(torch_device)
258+
attentionBlock = AttentionBlock(
259+
channels=512,
260+
rescale_output_factor=1.0,
261+
eps=1e-6,
262+
num_groups=32,
263+
).to(torch_device)
264+
with torch.no_grad():
265+
attention_scores = attentionBlock(sample)
266+
267+
assert attention_scores.shape == (1, 512, 64, 64)
268+
output_slice = attention_scores[0, -1, -3:, -3:]
269+
270+
expected_slice = torch.tensor(
271+
[-0.6621, -0.0156, -3.2766, 0.8025, -0.8609, 0.2820, 0.0905, -1.1179, -3.2126], device=torch_device
272+
)
273+
assert torch.allclose(output_slice.flatten(), expected_slice, atol=1e-3)
274+
248275

249276
class SpatialTransformerTests(unittest.TestCase):
250277
def test_spatial_transformer_default(self):

‎tests/test_modeling_common.py

+1
Original file line numberDiff line numberDiff line change
@@ -247,6 +247,7 @@ def recursive_check(tuple_object, dict_object):
247247

248248
recursive_check(outputs_tuple, outputs_dict)
249249

250+
@unittest.skipIf(torch_device == "mps", "Gradient checkpointing skipped on MPS")
250251
def test_enable_disable_gradient_checkpointing(self):
251252
if not self.model_class._supports_gradient_checkpointing:
252253
return # Skip test if model does not support gradient checkpointing

‎tests/test_models_unet.py

+4-3
Original file line numberDiff line numberDiff line change
@@ -135,7 +135,7 @@ def test_from_pretrained_hub(self):
135135

136136
assert image is not None, "Make sure output is not None"
137137

138-
@unittest.skipIf(torch_device == "cpu", "This test is supposed to run on GPU")
138+
@unittest.skipIf(torch_device != "cuda", "This test is supposed to run on GPU")
139139
def test_from_pretrained_accelerate(self):
140140
model, _ = UNet2DModel.from_pretrained(
141141
"fusing/unet-ldm-dummy-update", output_loading_info=True, device_map="auto"
@@ -145,7 +145,7 @@ def test_from_pretrained_accelerate(self):
145145

146146
assert image is not None, "Make sure output is not None"
147147

148-
@unittest.skipIf(torch_device == "cpu", "This test is supposed to run on GPU")
148+
@unittest.skipIf(torch_device != "cuda", "This test is supposed to run on GPU")
149149
def test_from_pretrained_accelerate_wont_change_results(self):
150150
model_accelerate, _ = UNet2DModel.from_pretrained(
151151
"fusing/unet-ldm-dummy-update", output_loading_info=True, device_map="auto"
@@ -177,7 +177,7 @@ def test_from_pretrained_accelerate_wont_change_results(self):
177177

178178
assert torch.allclose(arr_accelerate, arr_normal_load, rtol=1e-3)
179179

180-
@unittest.skipIf(torch_device == "cpu", "This test is supposed to run on GPU")
180+
@unittest.skipIf(torch_device != "cuda", "This test is supposed to run on GPU")
181181
def test_memory_footprint_gets_reduced(self):
182182
torch.cuda.empty_cache()
183183
gc.collect()
@@ -267,6 +267,7 @@ def prepare_init_args_and_inputs_for_common(self):
267267
inputs_dict = self.dummy_input
268268
return init_dict, inputs_dict
269269

270+
@unittest.skipIf(torch_device == "mps", "Gradient checkpointing skipped on MPS")
270271
def test_gradient_checkpointing(self):
271272
# enable deterministic behavior for gradient checkpointing
272273
init_dict, inputs_dict = self.prepare_init_args_and_inputs_for_common()

‎tests/test_models_vae.py

+1-1
Original file line numberDiff line numberDiff line change
@@ -96,7 +96,7 @@ def test_output_pretrained(self):
9696
model.config.in_channels,
9797
model.config.sample_size,
9898
model.config.sample_size,
99-
generator=torch.manual_seed(0),
99+
generator=generator,
100100
)
101101
image = image.to(torch_device)
102102
with torch.no_grad():

‎tests/test_pipelines.py

+3-3
Original file line numberDiff line numberDiff line change
@@ -1196,7 +1196,7 @@ def test_stable_diffusion_inpaint_num_images_per_prompt(self):
11961196

11971197
assert images.shape == (batch_size * num_images_per_prompt, 32, 32, 3)
11981198

1199-
@unittest.skipIf(torch_device == "cpu", "This test requires a GPU")
1199+
@unittest.skipIf(torch_device != "cuda", "This test requires a GPU")
12001200
def test_stable_diffusion_fp16(self):
12011201
"""Test that stable diffusion works with fp16"""
12021202
unet = self.dummy_cond_unet
@@ -1229,7 +1229,7 @@ def test_stable_diffusion_fp16(self):
12291229

12301230
assert image.shape == (1, 128, 128, 3)
12311231

1232-
@unittest.skipIf(torch_device == "cpu", "This test requires a GPU")
1232+
@unittest.skipIf(torch_device != "cuda", "This test requires a GPU")
12331233
def test_stable_diffusion_img2img_fp16(self):
12341234
"""Test that stable diffusion img2img works with fp16"""
12351235
unet = self.dummy_cond_unet
@@ -1270,7 +1270,7 @@ def test_stable_diffusion_img2img_fp16(self):
12701270

12711271
assert image.shape == (1, 32, 32, 3)
12721272

1273-
@unittest.skipIf(torch_device == "cpu", "This test requires a GPU")
1273+
@unittest.skipIf(torch_device != "cuda", "This test requires a GPU")
12741274
def test_stable_diffusion_inpaint_fp16(self):
12751275
"""Test that stable diffusion inpaint works with fp16"""
12761276
unet = self.dummy_cond_unet

0 commit comments

Comments
 (0)