Putting install scripts into dockerfile #822

shimwell · 2022-07-25T23:01:05Z

work in progress, not ready for review

Description

As mentioned in project https://github.com/svalinn/DAGMC/projects/6 it has been proposed to move the install scripts into the dockerfile.

Motivation and Context

This simplifies the passing of environments to cmake and other commands as they are all visible in one file and no longer creating in one script to be used in another.

Changes

What kind of change does this PR introduce? (Bug fix, feature, documentation update...)
refactored dockerfile to include more ARGs ENVs and install commands.

Behavior

What is the current behavior? What is the new behavior?
old = dockerfile copies in sh scripts and runs them, new = dockerfile runs install commands directly

shimwell · 2022-07-25T23:19:40Z

When building this new dockerfile with the defaults (moab 5.4.0) I am getting a Pymoab / PYTHONPATH error which is similar to the error I am seeing on PR #816

 58%] Building CXX object test/CMakeFiles/intx_in_plane_test.dir/intx_in_plane_test.cpp.o
    Complete output from command /usr/bin/python3 -c "import setuptools, tokenize;__file__='/root/build_dir/moab/bld/pymoab/setup.py';f=getattr(tokenize, 'open', open)(__file__);code=f.read().replace('\r\n', '\n');f.close();exec(compile(code, __file__, 'exec'))" develop --no-deps --prefix=/root/build_dir/moab/bld/pymoab:
    running develop
    Checking .pth file support in /root/build_dir/moab/bld/pymoab/lib/python3.6/site-packages
    /usr/bin/python3 -E -c pass
    TEST FAILED: /root/build_dir/moab/bld/pymoab/lib/python3.6/site-packages does NOT support .pth files
    error: bad install directory or PYTHONPATH
    
    You are attempting to install a package to a directory that is not
    on PYTHONPATH and which Python does not read ".pth" files from.  The
    installation directory you specified (via --install-dir, --prefix, or
    the distutils default setting) was:
    
        /root/build_dir/moab/bld/pymoab/lib/python3.6/site-packages
    
    and your PYTHONPATH environment variable currently contains:
    
        ''
    
    Here are some of your options for correcting the problem:
    
    * You can choose a different installation directory, i.e., one that is
      on PYTHONPATH or supports .pth files
    
    * You can add the installation directory to the PYTHONPATH environment
      variable.  (It must then also be on PYTHONPATH whenever you run
      Python and want to use the package(s) you are installing.)
    
    * You can set up the installation directory to support ".pth" files by
      using one of the approaches described here:
    
      https://setuptools.readthedocs.io/en/latest/easy_install.html#custom-installation-locations
    
    
    Please make the appropriate changes for your system and try again.
    
    ----------------------------------------
  Can't roll back pymoab; was not uninstalled
Command "/usr/bin/python3 -c "import setuptools, tokenize;__file__='/root/build_dir/moab/bld/pymoab/setup.py';f=getattr(tokenize, 'open', open)(__file__);code=f.read().replace('\r\n', '\n');f.close();exec(compile(code, __file__, 'exec'))" develop --no-deps --prefix=/root/build_dir/moab/bld/pymoab" failed with error code 1 in /root/build_dir/moab/bld/pymoab/
pymoab/CMakeFiles/pymoab-local-install.dir/build.make:58: recipe for target 'pymoab/CMakeFiles/pymoab-local-install' failed
make[2]: *** [pymoab/CMakeFiles/pymoab-local-install] Error 1
CMakeFiles/Makefile2:1238: recipe for target 'pymoab/CMakeFiles/pymoab-local-install.dir/all' failed
make[1]: *** [pymoab/CMakeFiles/pymoab-local-install.dir/all] Error 2
make[1]: *** Waiting for unfinished jobs....

gonuke

Thanks @shimwell - this looks like a good step forward!

I recognize that some of these suggestions represent changes of intent that we may want to leave for a separate PR & discussion.

CI/Dockerfile

Co-authored-by: Paul Wilson <paul.wilson@wisc.edu>

gonuke

A few issues introduced with the embree directory variables

CI/Dockerfile

gonuke

I think we're getting closer....

gonuke · 2022-08-19T16:24:18Z

CI/Dockerfile

+RUN mkdir -p ${embree_build_dir}/build && \
+    cd ${embree_build_dir} && \
+    git clone -b ${EMBREE_BRANCH} https://github.com/embree/embree && \
+    mkdir build && \


No longer needed - done in #103

Sorry I am not quite sure which part of this embree clone/install was done in #103

I have removed the 2nd mkdir build if that is what was needed

gonuke · 2022-08-19T16:25:26Z

CI/Dockerfile

-        /root/etc/CI/docker/build_moab.sh; \
-    fi;
+# Set the hdf5 install dir as this is also needed by moab compile
+ENV hdf5_install_dir=${install_dir}/hdf5


This is done in line 130 isn't it?

Raising this one again...

I thought we needed to declare ENV in each build stage they are used as they were not carried between build stages.

CI/Dockerfile

shimwell · 2022-08-19T16:49:28Z

A moab compile error has appeared when trying to build this docker file locally with moab 5.4.0.

Building CXX object test/CMakeFiles/intx_in_plane_test.dir/intx_in_plane_test.cpp.o
    Complete output from command /usr/bin/python3 -c "import setuptools, tokenize;__file__='/root/build_dir/moab/build/pymoab/setup.py';f=getattr(tokenize, 'open', open)(__file__);code=f.read().replace('\r\n', '\n');f.close();exec(compile(code, __file__, 'exec'))" develop --no-deps --prefix=/root/build_dir/moab/build/pymoab:
    running develop
    Checking .pth file support in /root/build_dir/moab/build/pymoab/lib/python3.6/site-packages
    /usr/bin/python3 -E -c pass
    TEST FAILED: /root/build_dir/moab/build/pymoab/lib/python3.6/site-packages does NOT support .pth files
    error: bad install directory or PYTHONPATH
    
    You are attempting to install a package to a directory that is not
    on PYTHONPATH and which Python does not read ".pth" files from.  The
    installation directory you specified (via --install-dir, --prefix, or
    the distutils default setting) was:
    
        /root/build_dir/moab/build/pymoab/lib/python3.6/site-packages
    
    and your PYTHONPATH environment variable currently contains:
    
        ''
    
    Here are some of your options for correcting the problem:
    
    * You can choose a different installation directory, i.e., one that is
      on PYTHONPATH or supports .pth files
    
    * You can add the installation directory to the PYTHONPATH environment
      variable.  (It must then also be on PYTHONPATH whenever you run
      Python and want to use the package(s) you are installing.)
    
    * You can set up the installation directory to support ".pth" files by
      using one of the approaches described here:
    
      https://setuptools.readthedocs.io/en/latest/easy_install.html#custom-installation-locations
    
    
    Please make the appropriate changes for your system and try again.
    
    ----------------------------------------
  Can't roll back pymoab; was not uninstalled

CI/Dockerfile

gonuke · 2022-08-27T14:03:26Z

For some reason, these latest commits didn't seem to trigger rebuild of the images on your fork???

shimwell · 2022-08-27T15:37:35Z

Sorry about that, I've just manually trigger it
https://github.com/shimwell/DAGMC/actions/runs/2939772739

shimwell · 2023-02-08T18:13:18Z

🙏 work this time. Latest action link is here

shimwell · 2023-02-10T12:09:48Z

Got to the stage where it fails dagmc tests on my CI fork actions

Anyone got any ideas on this output error, looks like it can't find some h5m files (iter_imprinted.h5m and bllite30matls.h5m) which I also can't find on the repo

#17 266.9 23/26 Test #23: make_watertight_regression_tests ........***Failed    0.01 sec
#17 266.9 [==========] Running 3 tests from 3 test cases.
#17 266.9 [----------] Global test environment set-up.
#17 266.9 [----------] 1 test from ITERMakeWatertightRegressionTest
#17 266.9 [ RUN      ] ITERMakeWatertightRegressionTest.ITERRegressionTest
#17 266.9 MOAB ERROR: --------------------- Error Message ------------------------------------
#17 266.9 MOAB ERROR: iter_imprinted.h5m: No such file or directory!
#17 266.9 MOAB ERROR: serial_load_file() line 518 in Core.cpp
#17 266.9 MOAB ERROR: load_file() line 467 in Core.cpp
#17 266.9 /root/build_dir/dagmc/src/make_watertight/tests/test_classes.cpp:27: Failure
#17 266.9 Expected equality of these values:
#17 266.9   result
#17 266.9     Which is: 7
#17 266.9   moab::MB_SUCCESS
#17 266.9     Which is: 0
#17 266.9 /root/build_dir/dagmc/src/make_watertight/tests/test_classes.cpp:45: Failure
#17 266.9 Expected equality of these values:
#17 266.9   result
#17 266.9     Which is: 6
#17 266.9   moab::MB_SUCCESS
#17 266.9     Which is: 0
#17 266.9 /root/build_dir/dagmc/src/make_watertight/tests/test_classes.cpp:51: Failure
#17 266.9 Expected: (0) != (num_meshsets), actual: 0 vs 0
#17 266.9 MOAB ERROR: --------------------- Error Message ------------------------------------
#17 266.9 MOAB ERROR: file set not found!
#17 266.9 MOAB ERROR: get_sealing_mesh_tags() line 1976 in Gen.cpp
#17 266.9 MOAB ERROR: --------------------- Error Message ------------------------------------
#17 266.9 MOAB ERROR: could not get the mesh tags!
#17 266.9 MOAB ERROR: make_mesh_watertight() line 1803 in MakeWatertight.cpp
#17 266.9 MOAB ERROR: --------------------- Error Message ------------------------------------
#17 266.9 MOAB ERROR: could not make the mesh watertight!
#17 266.9 MOAB ERROR: seal_and_check() line 355 in test_classes.cpp
#17 266.9 [  FAILED  ] ITERMakeWatertightRegressionTest.ITERRegressionTest (1 ms)
#17 266.9 [----------] 1 test from ITERMakeWatertightRegressionTest (1 ms total)
#17 266.9 
#17 266.9 [----------] 1 test from FNSFMakeWatertightRegressionTest
#17 266.9 [ RUN      ] FNSFMakeWatertightRegressionTest.FNSFRegressionTest
#17 266.9 MOAB ERROR: --------------------- Error Message ------------------------------------
#17 266.9 MOAB ERROR: model1_360_1.h5m: No such file or directory!
#17 266.9 MOAB ERROR: serial_load_file() line 518 in Core.cpp
#17 266.9 MOAB ERROR: load_file() line 467 in Core.cpp
#17 266.9 /root/build_dir/dagmc/src/make_watertight/tests/test_classes.cpp:27: Failure
#17 266.9 Expected equality of these values:
#17 266.9   result
#17 266.9     Which is: 7
#17 266.9   moab::MB_SUCCESS
#17 266.9     Which is: 0
#17 266.9 /root/build_dir/dagmc/src/make_watertight/tests/test_classes.cpp:45: Failure
#17 266.9 Expected equality of these values:
#17 266.9   result
#17 266.9     Which is: 6
#17 266.9   moab::MB_SUCCESS
#17 266.9     Which is: 0
#17 266.9 /root/build_dir/dagmc/src/make_watertight/tests/test_classes.cpp:51: Failure
#17 266.9 Expected: (0) != (num_meshsets), actual: 0 vs 0
#17 266.9 MOAB ERROR: --------------------- Error Message ------------------------------------
#17 266.9 MOAB ERROR: file set not found!
#17 266.9 MOAB ERROR: get_sealing_mesh_tags() line 1976 in Gen.cpp
#17 266.9 MOAB ERROR: --------------------- Error Message ------------------------------------
#17 266.9 MOAB ERROR: could not get the mesh tags!
#17 266.9 MOAB ERROR: make_mesh_watertight() line 1803 in MakeWatertight.cpp
#17 266.9 MOAB ERROR: --------------------- Error Message ------------------------------------
#17 266.9 MOAB ERROR: could not make the mesh watertight!
#17 266.9 MOAB ERROR: seal_and_check() line 355 in test_classes.cpp
#17 266.9 [  FAILED  ] FNSFMakeWatertightRegressionTest.FNSFRegressionTest (0 ms)
#17 266.9 [----------] 1 test from FNSFMakeWatertightRegressionTest (0 ms total)
#17 266.9 
#17 266.9 [----------] 1 test from BLiteMakeWatertightRegressionTest
#17 266.9 [ RUN      ] BLiteMakeWatertightRegressionTest.BLiteRegressionTest
#17 266.9 MOAB ERROR: --------------------- Error Message ------------------------------------
#17 266.9 MOAB ERROR: bllite30matls.h5m: No such file or directory!
#17 266.9 MOAB ERROR: serial_load_file() line 518 in Core.cpp
#17 266.9 MOAB ERROR: load_file() line 467 in Core.cpp
#17 266.9 /root/build_dir/dagmc/src/make_watertight/tests/test_classes.cpp:27: Failure
#17 266.9 Expected equality of these values:
#17 266.9   result
#17 266.9     Which is: 7
#17 266.9   moab::MB_SUCCESS
#17 266.9     Which is: 0
#17 266.9 /root/build_dir/dagmc/src/make_watertight/tests/test_classes.cpp:45: Failure
#17 266.9 Expected equality of these values:
#17 266.9   result
#17 266.9     Which is: 6
#17 266.9   moab::MB_SUCCESS
#17 266.9     Which is: 0
#17 266.9 /root/build_dir/dagmc/src/make_watertight/tests/test_classes.cpp:51: Failure
#17 266.9 Expected: (0) != (num_meshsets), actual: 0 vs 0
#17 266.9 MOAB ERROR: --------------------- Error Message ------------------------------------
#17 266.9 MOAB ERROR: file set not found!
#17 266.9 MOAB ERROR: get_sealing_mesh_tags() line 1976 in Gen.cpp
#17 266.9 MOAB ERROR: --------------------- Error Message ------------------------------------
#17 266.9 MOAB ERROR: could not get the mesh tags!
#17 266.9 MOAB ERROR: make_mesh_watertight() line 1803 in MakeWatertight.cpp
#17 266.9 MOAB ERROR: --------------------- Error Message ------------------------------------
#17 266.9 MOAB ERROR: could not make the mesh watertight!
#17 266.9 MOAB ERROR: seal_and_check() line 355 in test_classes.cpp
#17 266.9 [  FAILED  ] BLiteMakeWatertightRegressionTest.BLiteRegressionTest (1 ms)
#17 266.9 [----------] 1 test from BLiteMakeWatertightRegressionTest (1 ms total)
#17 266.9 
#17 266.9 [----------] Global test environment tear-down
#17 266.9 [==========] 3 tests from 3 test cases ran. (2 ms total)
#17 266.9 [  PASSED  ] 0 tests.
#17 266.9 [  FAILED  ] 3 tests, listed below:
#17 266.9 [  FAILED  ] ITERMakeWatertightRegressionTest.ITERRegressionTest
#17 266.9 [  FAILED  ] FNSFMakeWatertightRegressionTest.FNSFRegressionTest
#17 266.9 [  FAILED  ] BLiteMakeWatertightRegressionTest.BLiteRegressionTest
#17 266.9 
#17 266.9  3 FAILED TESTS
#17 266.9

bam241 · 2023-02-11T08:28:39Z

maybe they are created by the tests.

But were the tests are running and where the tests used to be run are not the same location, so the files are not created where the tests are looking for them ?

bam241 · 2023-02-11T08:31:35Z

or do the test have permission to write new file in those location ?

gonuke · 2023-02-11T14:14:00Z

I think I found it! There is a whole logic in the current system about whether it is a pull request or not (then a merge) that determines whether or not to run this regression test. It should only run on merge. Simultaneously, if it is not a pull request, there are lines to wget the missing files!!

shimwell · 2023-02-12T22:44:39Z

Shall we simplify that logic so that it always downloads the model and runs the test

I can see this wget command in the install.sh script

  wget ${MW_REG_TEST_MODELS_URL} -O mw_reg_test_files.tar.gz -o wget.out

I did a quick grep for the MW_REG_TEST_MODELS_URL string but I can't see where it is set.

makeclean · 2023-02-12T22:46:35Z

${docker-env} file?

makeclean · 2023-02-12T22:50:56Z

All these bash scripts should have set -u in them for safety for undefined variables and '-e' if we want to exit on errors.

gonuke · 2023-02-13T12:36:38Z

This PR is aiming to get rid of the scripts!

gonuke · 2023-02-13T12:43:31Z

Shall we simplify that logic so that it always downloads the model and runs the test

I can see this wget command in the install.sh script
  wget ${MW_REG_TEST_MODELS_URL} -O mw_reg_test_files.tar.gz -o wget.out
I did a quick grep for the MW_REG_TEST_MODELS_URL string but I can't see where it is set.

Hmmmm.... it appears to have been lost in CI transitions...

shimwell · 2023-02-16T14:29:44Z

CI/Dockerfile

+                      -DBUILD_GEANT4=ON \
+                      -DGEANT4_DIR=${geant4_install_dir} \
+                      -DBUILD_CI_TESTS=ON \
+                      -DBUILD_MW_REG_TESTS=${build_mw_reg_tests} \


this line could be changed to avoid running these tests.

The tests download a tar.gz file from a url that we can't find anymore

-DBUILD_MW_REG_TESTS=OFF \

Discussed this at the DAGMC meeting yesterday and this is the preferred route, I've opened issue #858 to track the missing tar.gz file with the models in it.

shimwell · 2023-02-17T12:37:14Z

Latest attempt to get the dockerfile building is running on my forks actions over here

shimwell · 2023-02-17T20:09:21Z

Action on my fork is passing 🎉

shimwell · 2023-02-22T22:26:28Z

If this one gets merged can I make a follow up one that removes the old scripts or is that a step too far

gonuke · 2023-02-23T16:57:54Z

Sorry @shimwell , but I'm just noticing that none of the incremental stages are using any of the prior dockerfiles at all. You are building from scratch at each stage (and thus wasting a lot of CI resources :) ). Alas, I think we lost this without noticing in #813

gonuke · 2023-02-23T17:00:20Z

Prior to #813, we had different dockerfiles for each stage, and each of those files used a different container that had already been pushed to GHCR. With a single dockerfile, we have stages that build on each other. At each step, we aim to build a particular stage, but with no caching, the CI runner has no idea that we've already built that stage. As a result, every step is rebuilding GEANT, HDF5, etc... including the step where you test the DAGMC install.

gonuke · 2023-02-23T17:04:27Z

The right way to fix this is probably introduce all the right caching, etc, but perhaps the multistage-docker-build-action is what we need to simplify this.

shimwell · 2023-02-23T19:24:32Z

Good spot yes, I can start implementing the multistage-docker-build-action on this open PR

gonuke · 2023-04-30T15:02:04Z

I think this has been replaced by #863 - do you want to close this one @shimwell ?

shimwell · 2023-04-30T18:25:24Z

Closing in favour of #863

shimwell added 4 commits July 25, 2022 23:26

moved install scripts into Dockerfile

24c6467

improved docs/comments

c2dba7a

added PR 817

5255950

Merge branch 'develop' into moving_scripts_into_dockerfile

2f558e4

Merge branch 'develop' into moving_scripts_into_dockerfile

11b4be9

shimwell added the Work in progress label Aug 18, 2022

gonuke requested changes Aug 18, 2022

View reviewed changes

CI/Dockerfile Show resolved Hide resolved

CI/Dockerfile Outdated Show resolved Hide resolved

CI/Dockerfile Outdated Show resolved Hide resolved

CI/Dockerfile Show resolved Hide resolved

CI/Dockerfile Outdated Show resolved Hide resolved

shimwell and others added 6 commits August 18, 2022 16:07

[skip ci] compile cores review suggestion by @gonuke

14a9291

Co-authored-by: Paul Wilson <paul.wilson@wisc.edu>

[skip ci] removed compiler arg fully

ad88096

more consistent dirs for build

8c8d579

ordering and intenting args

21af7e7

reduced layers

5571c7a

added missing hdf5 install dir env

5cdad9a

gonuke requested changes Aug 18, 2022

View reviewed changes

CI/Dockerfile Outdated Show resolved Hide resolved

CI/Dockerfile Outdated Show resolved Hide resolved

shimwell added 4 commits August 18, 2022 16:41

commented out moab static build

e331db4

corrected embree dirs

8507e48

using embree dir in rm -rf cmd

75d2419

cd to safe place

3881de6

gonuke requested changes Aug 19, 2022

View reviewed changes

shimwell commented Aug 19, 2022

View reviewed changes

CI/Dockerfile Outdated Show resolved Hide resolved

removed 2nd mkdir build in embree compile

85884d6

bam241 reviewed Aug 21, 2022

View reviewed changes

CI/Dockerfile Show resolved Hide resolved

shimwell commented Aug 21, 2022

View reviewed changes

CI/Dockerfile Outdated Show resolved Hide resolved

shimwell commented Aug 21, 2022

View reviewed changes

CI/Dockerfile Outdated Show resolved Hide resolved

shimwell added 2 commits August 23, 2022 20:43

remove env for dag

d4723f8

Remove making python path folders

7c93d5d

shimwell added 2 commits February 8, 2023 18:10

housebuilding only if build-test works

8e4da44

housebuilding only if build-test works

5ddc8f8

shimwell added 2 commits February 9, 2023 09:40

docker qemu usage in dag step same as other stages

4a123b5

not running dag intsall step in container

eb91b60

shimwell commented Feb 16, 2023

View reviewed changes

shimwell mentioned this pull request Feb 17, 2023

DBUILD_MW_REG_TESTS requires h5m models that can't be found as URL missing #858

Open

disable reg tests as models missing

d3edab8

shimwell removed the Work in progress label Feb 22, 2023

shimwell mentioned this pull request Mar 2, 2023

single dockerfile and single step in the docker building CI #863

Merged

bquan0 mentioned this pull request Mar 31, 2023

Use multistage docker build action in docker_publish.yml #875

Closed

shimwell closed this Apr 30, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Putting install scripts into dockerfile #822

Putting install scripts into dockerfile #822

shimwell commented Jul 25, 2022

shimwell commented Jul 25, 2022 •

edited

Loading

gonuke left a comment

gonuke left a comment

gonuke left a comment

gonuke Aug 19, 2022

shimwell Aug 22, 2022

gonuke Aug 19, 2022 •

edited

Loading

gonuke Dec 31, 2022

shimwell Jan 1, 2023

shimwell commented Aug 19, 2022

gonuke commented Aug 27, 2022

shimwell commented Aug 27, 2022

shimwell commented Feb 8, 2023

shimwell commented Feb 10, 2023 •

edited

Loading

bam241 commented Feb 11, 2023

bam241 commented Feb 11, 2023

gonuke commented Feb 11, 2023

shimwell commented Feb 12, 2023

makeclean commented Feb 12, 2023

makeclean commented Feb 12, 2023

gonuke commented Feb 13, 2023

gonuke commented Feb 13, 2023

shimwell Feb 16, 2023 •

edited

Loading

shimwell Feb 17, 2023

shimwell commented Feb 17, 2023

shimwell commented Feb 17, 2023

shimwell commented Feb 22, 2023

gonuke commented Feb 23, 2023

gonuke commented Feb 23, 2023

gonuke commented Feb 23, 2023

shimwell commented Feb 23, 2023

gonuke commented Apr 30, 2023

shimwell commented Apr 30, 2023

Putting install scripts into dockerfile #822

Putting install scripts into dockerfile #822

Conversation

shimwell commented Jul 25, 2022

Description

Motivation and Context

Changes

Behavior

shimwell commented Jul 25, 2022 • edited Loading

gonuke left a comment

Choose a reason for hiding this comment

gonuke left a comment

Choose a reason for hiding this comment

gonuke left a comment

Choose a reason for hiding this comment

gonuke Aug 19, 2022

Choose a reason for hiding this comment

shimwell Aug 22, 2022

Choose a reason for hiding this comment

gonuke Aug 19, 2022 • edited Loading

Choose a reason for hiding this comment

gonuke Dec 31, 2022

Choose a reason for hiding this comment

shimwell Jan 1, 2023

Choose a reason for hiding this comment

shimwell commented Aug 19, 2022

gonuke commented Aug 27, 2022

shimwell commented Aug 27, 2022

shimwell commented Feb 8, 2023

shimwell commented Feb 10, 2023 • edited Loading

bam241 commented Feb 11, 2023

bam241 commented Feb 11, 2023

gonuke commented Feb 11, 2023

shimwell commented Feb 12, 2023

makeclean commented Feb 12, 2023

makeclean commented Feb 12, 2023

gonuke commented Feb 13, 2023

gonuke commented Feb 13, 2023

shimwell Feb 16, 2023 • edited Loading

Choose a reason for hiding this comment

shimwell Feb 17, 2023

Choose a reason for hiding this comment

shimwell commented Feb 17, 2023

shimwell commented Feb 17, 2023

shimwell commented Feb 22, 2023

gonuke commented Feb 23, 2023

gonuke commented Feb 23, 2023

gonuke commented Feb 23, 2023

shimwell commented Feb 23, 2023

gonuke commented Apr 30, 2023

shimwell commented Apr 30, 2023

shimwell commented Jul 25, 2022 •

edited

Loading

gonuke Aug 19, 2022 •

edited

Loading

shimwell commented Feb 10, 2023 •

edited

Loading

shimwell Feb 16, 2023 •

edited

Loading