Make it possible to create multiple independent sit-environments at the same time

[xhernandez]

This PR makes it possible to define and use multiple independent environments at the same time.

There are basically three changes:

1. A python virtual environment is created for each sit-environment.

   This provides a way to have different versions of python and/or ansible collections, which may be required by some backends. It's also a first step into making required ansible and python components available without polluting the local system, which may be useful to get rid of the setup machine in the future.

2. All configuration and generated files are stored inside the python virtual environment.

   Everything susceptible to be modified or generated by sit-environment has been moved inside the virtual environment. This was, the same playbooks and tree can be used to manage different environments at the same time without interferences.

3. A new site command has been created to manage the environments.

   The new command replaces the Makefile in the root directory (for now, the Makefile has been converted to a wrapper around this command). It makes it easy to create, use, destroy and check multiple environments. If the Makefile is used instead of the site command, it will work normally but using a default environment named test.

   Main commands:

   • site create <name>

     Creates a new virtual environment named <name>. These environments are created in ~/.site/. The current value of EXTRA_VARS is saved inside the environment, and that value will always be used for any command run in the environment.

   • site build <name>

     Builds an environment. This is equivalent to what happened previously when executing make in the root directory.

   • site clean <name>

     It's equivalent to the previous make clean command. After clean, the environment can be reconfigured and built again.

   • site destroy <name>

     Cleans the environment and completely deletes it.

   • site shell <name>

     Opens a bash shell inside the specified environment. It can be useful to quickly access the configuration files and modify, or to manually execute some commands making sure that the environment is correctly configured (like the EXTRA_VARS variable).

   • site show

     Lists all the defined environments and the EXTRA_VARS associated to each one.

   • site yamllint

     It's equivalent to the previous make yamllint command.

   • site statedump

     It's equivalent to the previous make statedump command.

[anoopcs9]

This is something which will definitely be helpful in future. Thanks for taking time to come up with the change.

[Shwetha-Acharya]

just wondering if we can use something like $which python | xargs -i{} bash -c '{} --version' instead of listing all the PATH variables

[phlogistonjohn]

This may be a good idea in general, but please don't use which - which is common but not non-standard and not part of a base system. Use command, particularly command -v instead. It's a shell builtin.

[xhernandez]

The idea here is to use the highest available version of python, and this may not be what python command points to (assuming that it exists because in some cases it even doesn't exist). Different versions of python have different executable names, so using which or similar commands won't be very useful unless we do a brute-force search of the expected executable names.

[Shwetha-Acharya]

I got quite puzzled when I tried to execute make on this branch:

shwethaacharya:sit-environment$ EXTRA_VARS="backend=xfs" make
Python 2.7.18
Python 2.7.18
Python 3.4+ is required
make: *** [Makefile:4: test] Error 1
shwethaacharya:sit-environment$ 

Basically python2 package versions are getting formatted differently than python3,

shwethaacharya:sit-environment$ echo -n "${PATH}" |tr ':' '\n' |xargs -i{} find {} -maxdepth 1 -type f -regex '.*/python[0-9.]*' 2>/dev/null | xargs -i{} bash -c "echo \$({} --version |  tr -dc '0-9.') {}"|LC_ALL=C sort -Vr
Python 2.7.18
/usr/bin/python2.7
3.13.02 /usr/bin/python3.13
3.12.0 /usr/bin/python3.12
3.11.6 /usr/bin/python3.11
3.10.13 /usr/bin/python3.10
3.9.18 /usr/bin/python3.9
3.8.18 /usr/bin/python3.8
3.7.17 /usr/bin/python3.7
3.6.15 /usr/bin/python3.6
shwethaacharya:sit-environment$

and hence, a wrong python version appears for the command head -n 1:

shwethaacharya:sit-environment$ PYTHON="$(echo -n "${PATH}" |tr ':' '\n' |xargs -i{} find {} -maxdepth 1 -type f -regex '.*/python[0-9.]*' 2>/dev/null |xargs -i{} bash -c "echo \$({} --version | tr -dc '0-9.') {}" |LC_ALL=C sort -Vr |head -n 1 |cut -d" " -f 2-)"
Python 2.7.18
shwethaacharya:sit-environment$

I am not sure if we want to handle this scenario with python2 or any future such possible occurences or should ignore it considering python2 is EOLed.

Just adding my observation here.

[xhernandez]

That's interesting. I do want to handle those cases for sure. The problem is that the format is the expected one, so this shouldn't happen.

I've tested what happens and it seems that python 2.x shows the version info in the stderr instead of the stdout, so the version parsed is actually an empty string.

I'll update it. Thanks for catching it !

[xhernandez]

/retest all

[xhernandez]

/retest centos-ci/xfs