Skip to content

Commit cec7fb7

Browse files
mauwiimauwii
authored
squash merge update-gh-actions into fix-gh-actions
* fix mkdocs deployment * update path to python bin * add trigger for current branch * change path to python_bin for mac as well * try to use setup-python@v4 instead of setting env * remove setup conda action * try to use $CONDA * remove overseen action * change branch from master to main * sort out if then else for faster syntax * remove more if functions * add updates to create-caches as well * eliminate the rest of if functions * try to unpin pytorch and torchvision * restore pinned versions * try switching from set-output to use env * update test-invoke-conda as well * fix env var creation * quote variable * add second equal to compare * try another way to use outputs * fix outputs * pip install for mac before creating conda env * fix output variable * fix python bin path * remove pip install for before creating conda env * unpin streamlit version in conda mac env * try to make git-workflows better readable * remove 4gotten trigger * Update-gh-actions (#6) * fix mkdocs deployment * update path to python bin * add trigger for current branch * change path to python_bin for mac as well * try to use setup-python@v4 instead of setting env * remove setup conda action * try to use $CONDA * remove overseen action * change branch from master to main * sort out if then else for faster syntax * remove more if functions * add updates to create-caches as well * eliminate the rest of if functions * try to unpin pytorch and torchvision * restore pinned versions * try switching from set-output to use env * update test-invoke-conda as well * fix env var creation * quote variable * add second equal to compare * try another way to use outputs * fix outputs * pip install for mac before creating conda env * fix output variable * fix python bin path * remove pip install for before creating conda env * unpin streamlit version in conda mac env * try to make git-workflows better readable * use macos-latest * try to update conda before creating mac env * better conda update trial * re-pin streamlit version * re-added trigger to run workflow in current branch * try to find out if conda mac env could be updated * install cmake, protobuf and rust b4 conda * add yes to conda update * lets try anaconda3-2022.05 * try environment.yml for mac as well * reenable conda mac env, add pip install also fix gitignore by changing from dream to invoke * remove - unecesary virtualenv creation - conda update change != macos back to == linux * remove cmake from brew install since pre-installed * disable opencv-python pip requirement * fixed commands to find latest package versions * update requirements for mac env * back to the roots - only install conda env depending on runner_os with or without extra env variables * check out macOS in azure-pipelines since becoming kind of tired of the GitHub Runner which is broken as ... * let's try to setup python and update conda env * initialize conda before using it * add trigger in azure-pipelines.yml * And another go for update first .... * update azure-pipelines.yml - add caching - add checkpoint download - add paths to trigger and more * unquote checkpoint-url * fix chekpoint-url variable * mkdir before downloading model * set pr trigger to main, rename anaconda cache * unique cacheHitVariables * try to use macos-latest instead of macos-12 * update test-invoke-conda.yml: - remove unecesarry echo step - use s-weigand/setup-conda@v1 - remove conda update from install deps step since updated with action * update test-invoke-conda.yml: - rename conda env cache from ldm to invokeai - reorder steps: 1. checkout sources 2. setup python 3. setup conda 4. keep order after set platform variables * change macos back to 12 since also fails with 11 * update condition in run the tests make difference between main or not main * fix path to cache invokeai conda env * fix invokeai conda env cache path * update mkdocs-flow.yml * change conda-channel priority * update create-caches * update conda env also when cache was used * os dependend conda env cache path * use existing CONDA env pointing to conda root * create CONDA_ROOT output from $CONDA * use output variable to define test prompts * use setup-python v4, get rid of PYTHON_BIN env * add runner.os to result artifacts name * update test-invoke-conda.yml: - reuse macos-latest - disable setup python 3.9 - setup conda with default python version - create or update conda environment depending on cache success - remove name parameter from conda update since name is set in env yml * improve mkdocs-flow.yml * disable cache-hugginface-torch since preload_models.py downloads to more than one location * update mkdocs-flow.yml with new name * rename mkdocs action to mkdocs-material * try to ignore error when creating conda env maybe it would still be usable, lets see ;P * remove bloat * update environment-mac.yml to match dependencies of invoke-ai/InvokeAI's main branch * disable conda update, tweak prompt condition * try to set some env vars for macOS to fix conda * stop ignoring error, use env instead of outputs * tweak `[[` connditions * update python and pip dependencie makes a difference of 1 sec per itteration compared to 3.9!!! also I see no reason why using a old pip version would be beneficial * remove unecesarry env for macOS everything was pre-tested on my MacBook Air 2020 with M1 * update conda env in setup step * activate conda env after installation * update test-invoke-conda.yml - set conda env dependent on matrix.os - set CONDA_ENV_NAME to prevent breaking action when renaming conda env - fix conda env activation * fix activate conda env * set bash -l as default shell * use action to activate conda env * add conda env file to env activation * try to replace s-weigeand with conda-incubator * remove azure-pipelines.yml funniest part is that the macos runner is the same as the one on github! * include environment-file in matrix - also disable auto-activate-base and auto-update-conda - include macos-latest and macos-12 for debugging purpose - set miniforge-version in matrix * fix miniforge-variant, set fail-fast to false * add step to setup miniconda - make default shell a matrix variable - remove bloat * use a mac env yml without pinned versions * unpin nomkl, pytorch and torchvision also removed opencv-pyhton * cache conda pkgs dir instead of conda env * use python 3.10, exclude macos-12 from cache * fix expression * prepare for PR * fix doubled id * reuse pinned versions in mac conda env - updated python pip version - unpined pytorch and torchvision - removed opencv-python - updated versions to most recent (tested locally) * fix classical copy/paste error * remove unused env from shell-block comment * fix hashFiles function to determine restore-keys * reenable caching `~.cache`, update create-caches * unpin all versions in mac conda env file this was the only way I got it working in the action, also works locally tested on MacBook Air 2020 M1 remove environment-mac-unpinned.yml * prepare merge by removing this branch from trigger * include pull_request trigger for main and dev * remove pull_request trigger
1 parent 57a3ea9 commit cec7fb7

File tree

6 files changed

+227
-161
lines changed

6 files changed

+227
-161
lines changed

‎.github/workflows/create-caches.yml‎

Lines changed: 76 additions & 39 deletions
Original file line numberDiff line numberDiff line change
@@ -1,26 +1,43 @@
11
name: Create Caches
2-
on:
3-
workflow_dispatch
2+
3+
on: workflow_dispatch
4+
45
jobs:
5-
build:
6+
os_matrix:
67
strategy:
78
matrix:
8-
os: [ ubuntu-latest, macos-12 ]
9-
name: Create Caches on ${{ matrix.os }} conda
9+
os: [ubuntu-latest, macos-latest]
10+
include:
11+
- os: ubuntu-latest
12+
environment-file: environment.yml
13+
default-shell: bash -l {0}
14+
- os: macos-latest
15+
environment-file: environment-mac.yml
16+
default-shell: bash -l {0}
17+
name: Test invoke.py on ${{ matrix.os }} with conda
1018
runs-on: ${{ matrix.os }}
19+
defaults:
20+
run:
21+
shell: ${{ matrix.default-shell }}
1122
steps:
12-
- name: Set platform variables
13-
id: vars
14-
run: |
15-
if [ "$RUNNER_OS" = "macOS" ]; then
16-
echo "::set-output name=ENV_FILE::environment-mac.yml"
17-
echo "::set-output name=PYTHON_BIN::/usr/local/miniconda/envs/ldm/bin/python"
18-
elif [ "$RUNNER_OS" = "Linux" ]; then
19-
echo "::set-output name=ENV_FILE::environment.yml"
20-
echo "::set-output name=PYTHON_BIN::/usr/share/miniconda/envs/ldm/bin/python"
21-
fi
2223
- name: Checkout sources
2324
uses: actions/checkout@v3
25+
26+
- name: setup miniconda
27+
uses: conda-incubator/setup-miniconda@v2
28+
with:
29+
auto-activate-base: false
30+
auto-update-conda: false
31+
miniconda-version: latest
32+
33+
- name: set environment
34+
run: |
35+
[[ "$GITHUB_REF" == 'refs/heads/main' ]] \
36+
&& echo "TEST_PROMPTS=tests/preflight_prompts.txt" >> $GITHUB_ENV \
37+
|| echo "TEST_PROMPTS=tests/dev_prompts.txt" >> $GITHUB_ENV
38+
echo "CONDA_ROOT=$CONDA" >> $GITHUB_ENV
39+
echo "CONDA_ENV_NAME=invokeai" >> $GITHUB_ENV
40+
2441
- name: Use Cached Stable Diffusion v1.4 Model
2542
id: cache-sd-v1-4
2643
uses: actions/cache@v3
@@ -29,42 +46,62 @@ jobs:
2946
with:
3047
path: models/ldm/stable-diffusion-v1/model.ckpt
3148
key: ${{ env.cache-name }}
32-
restore-keys: |
33-
${{ env.cache-name }}
49+
restore-keys: ${{ env.cache-name }}
50+
51+
- name: Use Cached Stable Diffusion v1.4 Model
52+
id: cache-sd-v1-4
53+
uses: actions/cache@v3
54+
env:
55+
cache-name: cache-sd-v1-4
56+
with:
57+
path: models/ldm/stable-diffusion-v1/model.ckpt
58+
key: ${{ env.cache-name }}
59+
restore-keys: ${{ env.cache-name }}
60+
3461
- name: Download Stable Diffusion v1.4 Model
3562
if: ${{ steps.cache-sd-v1-4.outputs.cache-hit != 'true' }}
3663
run: |
37-
if [ ! -e models/ldm/stable-diffusion-v1 ]; then
38-
mkdir -p models/ldm/stable-diffusion-v1
39-
fi
40-
if [ ! -e models/ldm/stable-diffusion-v1/model.ckpt ]; then
41-
curl -o models/ldm/stable-diffusion-v1/model.ckpt ${{ secrets.SD_V1_4_URL }}
42-
fi
43-
- name: Use Cached Dependencies
44-
id: cache-conda-env-ldm
64+
[[ -d models/ldm/stable-diffusion-v1 ]] \
65+
|| mkdir -p models/ldm/stable-diffusion-v1
66+
[[ -r models/ldm/stable-diffusion-v1/model.ckpt ]] \
67+
|| curl -o models/ldm/stable-diffusion-v1/model.ckpt ${{ secrets.SD_V1_4_URL }}
68+
69+
- name: Use cached Conda Environment
4570
uses: actions/cache@v3
4671
env:
47-
cache-name: cache-conda-env-ldm
72+
cache-name: cache-conda-env-${{ env.CONDA_ENV_NAME }}
73+
conda-env-file: ${{ matrix.environment-file }}
4874
with:
49-
path: ~/.conda/envs/ldm
75+
path: ${{ env.CONDA_ROOT }}/envs/${{ env.CONDA_ENV_NAME }}
5076
key: ${{ env.cache-name }}
51-
restore-keys: |
52-
${{ env.cache-name }}-${{ runner.os }}-${{ hashFiles(steps.vars.outputs.ENV_FILE) }}
53-
- name: Install Dependencies
54-
if: ${{ steps.cache-conda-env-ldm.outputs.cache-hit != 'true' }}
55-
run: |
56-
conda env create -f ${{ steps.vars.outputs.ENV_FILE }}
77+
restore-keys: ${{ env.cache-name }}-${{ runner.os }}-${{ hashFiles(env.conda-env-file) }}
78+
79+
- name: Use cached Conda Packages
80+
uses: actions/cache@v3
81+
env:
82+
cache-name: cache-conda-env-${{ env.CONDA_ENV_NAME }}
83+
conda-env-file: ${{ matrix.environment-file }}
84+
with:
85+
path: ${{ env.CONDA_PKGS_DIR }}
86+
key: ${{ env.cache-name }}
87+
restore-keys: ${{ env.cache-name }}-${{ runner.os }}-${{ hashFiles(env.conda-env-file) }}
88+
89+
- name: Activate Conda Env
90+
uses: conda-incubator/setup-miniconda@v2
91+
with:
92+
activate-environment: ${{ env.CONDA_ENV_NAME }}
93+
environment-file: ${{ matrix.environment-file }}
94+
5795
- name: Use Cached Huggingface and Torch models
58-
id: cache-huggingface-torch
96+
id: cache-hugginface-torch
5997
uses: actions/cache@v3
6098
env:
61-
cache-name: cache-huggingface-torch
99+
cache-name: cache-hugginface-torch
62100
with:
63101
path: ~/.cache
64102
key: ${{ env.cache-name }}
65103
restore-keys: |
66104
${{ env.cache-name }}-${{ hashFiles('scripts/preload_models.py') }}
67-
- name: Download Huggingface and Torch models
68-
if: ${{ steps.cache-huggingface-torch.outputs.cache-hit != 'true' }}
69-
run: |
70-
${{ steps.vars.outputs.PYTHON_BIN }} scripts/preload_models.py
105+
106+
- name: run preload_models.py
107+
run: python scripts/preload_models.py

‎.github/workflows/mkdocs-flow.yml‎

Lines changed: 0 additions & 28 deletions
This file was deleted.

‎.github/workflows/mkdocs-material.yml‎

Lines changed: 44 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,44 @@
1+
name: mkdocs-material
2+
on:
3+
push:
4+
branches:
5+
- 'main'
6+
- 'development'
7+
pull_request:
8+
branches:
9+
- 'main'
10+
- 'development'
11+
12+
jobs:
13+
mkdocs-material:
14+
runs-on: ubuntu-latest
15+
steps:
16+
- name: checkout sources
17+
uses: actions/checkout@v3
18+
with:
19+
fetch-depth: 0
20+
21+
- name: setup python
22+
uses: actions/setup-python@v4
23+
with:
24+
python-version: '3.10'
25+
26+
- name: install requirements
27+
run: |
28+
python -m \
29+
pip install -r requirements-mkdocs.txt
30+
31+
- name: confirm buildability
32+
run: |
33+
python -m \
34+
mkdocs build \
35+
--clean \
36+
--verbose
37+
38+
- name: deploy to gh-pages
39+
if: ${{ github.ref == 'refs/heads/main' }}
40+
run: |
41+
python -m \
42+
mkdocs gh-deploy \
43+
--clean \
44+
--force

‎.github/workflows/test-invoke-conda.yml‎

Lines changed: 65 additions & 50 deletions
Original file line numberDiff line numberDiff line change
@@ -4,29 +4,43 @@ on:
44
branches:
55
- 'main'
66
- 'development'
7+
78
jobs:
89
os_matrix:
910
strategy:
1011
matrix:
11-
os: [ ubuntu-latest, macos-12 ]
12+
os: [ubuntu-latest, macos-latest]
13+
include:
14+
- os: ubuntu-latest
15+
environment-file: environment.yml
16+
default-shell: bash -l {0}
17+
- os: macos-latest
18+
environment-file: environment-mac.yml
19+
default-shell: bash -l {0}
1220
name: Test invoke.py on ${{ matrix.os }} with conda
1321
runs-on: ${{ matrix.os }}
22+
defaults:
23+
run:
24+
shell: ${{ matrix.default-shell }}
1425
steps:
15-
- run: |
16-
echo The PR was merged
17-
- name: Set platform variables
18-
id: vars
19-
run: |
20-
# Note, can't "activate" via github action; specifying the env's python has the same effect
21-
if [ "$RUNNER_OS" = "macOS" ]; then
22-
echo "::set-output name=ENV_FILE::environment-mac.yml"
23-
echo "::set-output name=PYTHON_BIN::/usr/local/miniconda/envs/ldm/bin/python"
24-
elif [ "$RUNNER_OS" = "Linux" ]; then
25-
echo "::set-output name=ENV_FILE::environment.yml"
26-
echo "::set-output name=PYTHON_BIN::/usr/share/miniconda/envs/ldm/bin/python"
27-
fi
2826
- name: Checkout sources
2927
uses: actions/checkout@v3
28+
29+
- name: setup miniconda
30+
uses: conda-incubator/setup-miniconda@v2
31+
with:
32+
auto-activate-base: false
33+
auto-update-conda: false
34+
miniconda-version: latest
35+
36+
- name: set environment
37+
run: |
38+
[[ "$GITHUB_REF" == 'refs/heads/main' ]] \
39+
&& echo "TEST_PROMPTS=tests/preflight_prompts.txt" >> $GITHUB_ENV \
40+
|| echo "TEST_PROMPTS=tests/dev_prompts.txt" >> $GITHUB_ENV
41+
echo "CONDA_ROOT=$CONDA" >> $GITHUB_ENV
42+
echo "CONDA_ENV_NAME=invokeai" >> $GITHUB_ENV
43+
3044
- name: Use Cached Stable Diffusion v1.4 Model
3145
id: cache-sd-v1-4
3246
uses: actions/cache@v3
@@ -35,31 +49,42 @@ jobs:
3549
with:
3650
path: models/ldm/stable-diffusion-v1/model.ckpt
3751
key: ${{ env.cache-name }}
38-
restore-keys: |
39-
${{ env.cache-name }}
52+
restore-keys: ${{ env.cache-name }}
53+
4054
- name: Download Stable Diffusion v1.4 Model
4155
if: ${{ steps.cache-sd-v1-4.outputs.cache-hit != 'true' }}
4256
run: |
43-
if [ ! -e models/ldm/stable-diffusion-v1 ]; then
44-
mkdir -p models/ldm/stable-diffusion-v1
45-
fi
46-
if [ ! -e models/ldm/stable-diffusion-v1/model.ckpt ]; then
47-
curl -o models/ldm/stable-diffusion-v1/model.ckpt ${{ secrets.SD_V1_4_URL }}
48-
fi
49-
- name: Use Cached Dependencies
50-
id: cache-conda-env-ldm
57+
[[ -d models/ldm/stable-diffusion-v1 ]] \
58+
|| mkdir -p models/ldm/stable-diffusion-v1
59+
[[ -r models/ldm/stable-diffusion-v1/model.ckpt ]] \
60+
|| curl -o models/ldm/stable-diffusion-v1/model.ckpt ${{ secrets.SD_V1_4_URL }}
61+
62+
- name: Use cached Conda Environment
5163
uses: actions/cache@v3
5264
env:
53-
cache-name: cache-conda-env-ldm
65+
cache-name: cache-conda-env-${{ env.CONDA_ENV_NAME }}
66+
conda-env-file: ${{ matrix.environment-file }}
5467
with:
55-
path: ~/.conda/envs/ldm
68+
path: ${{ env.CONDA_ROOT }}/envs/${{ env.CONDA_ENV_NAME }}
5669
key: ${{ env.cache-name }}
57-
restore-keys: |
58-
${{ env.cache-name }}-${{ runner.os }}-${{ hashFiles(steps.vars.outputs.ENV_FILE) }}
59-
- name: Install Dependencies
60-
if: ${{ steps.cache-conda-env-ldm.outputs.cache-hit != 'true' }}
61-
run: |
62-
conda env create -f ${{ steps.vars.outputs.ENV_FILE }}
70+
restore-keys: ${{ env.cache-name }}-${{ runner.os }}-${{ hashFiles(env.conda-env-file) }}
71+
72+
- name: Use cached Conda Packages
73+
uses: actions/cache@v3
74+
env:
75+
cache-name: cache-conda-env-${{ env.CONDA_ENV_NAME }}
76+
conda-env-file: ${{ matrix.environment-file }}
77+
with:
78+
path: ${{ env.CONDA_PKGS_DIR }}
79+
key: ${{ env.cache-name }}
80+
restore-keys: ${{ env.cache-name }}-${{ runner.os }}-${{ hashFiles(env.conda-env-file) }}
81+
82+
- name: Activate Conda Env
83+
uses: conda-incubator/setup-miniconda@v2
84+
with:
85+
activate-environment: ${{ env.CONDA_ENV_NAME }}
86+
environment-file: ${{ matrix.environment-file }}
87+
6388
- name: Use Cached Huggingface and Torch models
6489
id: cache-hugginface-torch
6590
uses: actions/cache@v3
@@ -70,28 +95,18 @@ jobs:
7095
key: ${{ env.cache-name }}
7196
restore-keys: |
7297
${{ env.cache-name }}-${{ hashFiles('scripts/preload_models.py') }}
73-
- name: Download Huggingface and Torch models
74-
if: ${{ steps.cache-hugginface-torch.outputs.cache-hit != 'true' }}
75-
run: |
76-
${{ steps.vars.outputs.PYTHON_BIN }} scripts/preload_models.py
77-
# - name: Run tmate
78-
# uses: mxschmitt/action-tmate@v3
79-
# timeout-minutes: 30
98+
99+
- name: run preload_models.py
100+
run: python scripts/preload_models.py
101+
80102
- name: Run the tests
81103
run: |
82-
# Note, can't "activate" via github action; specifying the env's python has the same effect
83-
if [ $(uname) = "Darwin" ]; then
84-
export PYTORCH_ENABLE_MPS_FALLBACK=1
85-
fi
86-
# Utterly hacky, but I don't know how else to do this
87-
if [[ ${{ github.ref }} == 'refs/heads/master' ]]; then
88-
time ${{ steps.vars.outputs.PYTHON_BIN }} scripts/invoke.py --from_file tests/preflight_prompts.txt
89-
elif [[ ${{ github.ref }} == 'refs/heads/development' ]]; then
90-
time ${{ steps.vars.outputs.PYTHON_BIN }} scripts/invoke.py --from_file tests/dev_prompts.txt
91-
fi
104+
time python scripts/invoke.py \
105+
--from_file ${{ env.TEST_PROMPTS }}
92106
mkdir -p outputs/img-samples
107+
93108
- name: Archive results
94109
uses: actions/upload-artifact@v3
95110
with:
96-
name: results
111+
name: results_${{ matrix.os }}
97112
path: outputs/img-samples

‎.gitignore‎

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -1,7 +1,7 @@
11
# ignore default image save location and model symbolic link
22
outputs/
33
models/ldm/stable-diffusion-v1/model.ckpt
4-
ldm/dream/restoration/codeformer/weights
4+
ldm/invoke/restoration/codeformer/weights
55

66
# ignore the Anaconda/Miniconda installer used while building Docker image
77
anaconda.sh
@@ -180,7 +180,7 @@ src
180180
**/__pycache__/
181181
outputs
182182

183-
# Logs and associated folders
183+
# Logs and associated folders
184184
# created from generated embeddings.
185185
logs
186186
testtube

0 commit comments

Comments
 (0)