Creating virtual environment for each test

[sungeunbae]

Creates virtual environment at /tmp/qcore/#hash#, installs requirements.
Runs test, then removes after.

Docker images all removed from the server to release space.
Binaries are kept under /nesi/project/nesi00213/tools

Note: For virtual environments, pytest and pytest-black should be included in requirements.txt for the auto-tests to run

[jasonmotha]

The change itself is looking good! Glad you got it working

@jpa198 Thoughts about compiling emod3d-tools as part of environment setup. Just because the binaries may change and hence break the tests?

[jasonmotha]

If we're using these as debug output for the logs, it would be nice to wrap it with "Current working directory:" etc.

[sungeunbae]

You mean as a separate "stage"? Sure - can do.

[sungeunbae]

Done in the latest patch

[jpa198]

@jpa198 Thoughts about compiling emod3d-tools as part of environment setup. Just because the binaries may change and hence break the tests?

I think pulling EMOD3D and compiling every time we do a test is too much. I think the responsibility for updating the binaries should be on the EMOD3D repo.
So that probably needs a jenkins file to build the tools on PR creation or merge.
EMOD3D is normally low traffic, so it shouldn't be an issue too often.

[jasonmotha]

@jpa198 Thoughts about compiling emod3d-tools as part of environment setup. Just because the binaries may change and hence break the tests?

I think pulling EMOD3D and compiling every time we do a test is too much. I think the responsibility for updating the binaries should be on the EMOD3D repo.
So that probably needs a jenkins file to build the tools on PR creation or merge.
EMOD3D is normally low traffic, so it shouldn't be an issue too often.

Sweet. Agreed with all your thoughts.

[sungeunbae]

@jpa198 Thoughts about compiling emod3d-tools as part of environment setup. Just because the binaries may change and hence break the tests?

I think pulling EMOD3D and compiling every time we do a test is too much. I think the responsibility for updating the binaries should be on the EMOD3D repo.
So that probably needs a jenkins file to build the tools on PR creation or merge.
EMOD3D is normally low traffic, so it shouldn't be an issue too often.

Sweet. Agreed with all your thoughts.

I only managed to build a couple of tools on ec2 - had to temporarily install openmpi and FFTW3, and it crashed at Fractal3D. At the point, I had all the binaries needed for the test, so I didn't go further to fix the build issues.

[jpa198]

@jpa198 Thoughts about compiling emod3d-tools as part of environment setup. Just because the binaries may change and hence break the tests?

I think pulling EMOD3D and compiling every time we do a test is too much. I think the responsibility for updating the binaries should be on the EMOD3D repo.
So that probably needs a jenkins file to build the tools on PR creation or merge.
EMOD3D is normally low traffic, so it shouldn't be an issue too often.

Sweet. Agreed with all your thoughts.

I only managed to build a couple of tools on ec2 - had to temporarily install openmpi and FFTW3, and it crashed at Fractal3D. At the point, I had all the binaries needed for the test, so I didn't go further to fix the build issues.

If we do want to take that path, I can make a build target for just tools we need for testing

[sungeunbae]

@jpa198 Thoughts about compiling emod3d-tools as part of environment setup. Just because the binaries may change and hence break the tests?

I think pulling EMOD3D and compiling every time we do a test is too much. I think the responsibility for updating the binaries should be on the EMOD3D repo.
So that probably needs a jenkins file to build the tools on PR creation or merge.
EMOD3D is normally low traffic, so it shouldn't be an issue too often.

Sweet. Agreed with all your thoughts.

I only managed to build a couple of tools on ec2 - had to temporarily install openmpi and FFTW3, and it crashed at Fractal3D. At the point, I had all the binaries needed for the test, so I didn't go further to fix the build issues.

If we do want to take that path, I can make a build target for just tools we need for testing

No need for now - I will move on to other repos and convert to virtual envs and will see if I need more binaries.

[jasonmotha]

Cool. Looks good.