tox with shared workdir across multiple platforms and repositories

[ssbarnea]

There is one use-case where tox does not work well, or at least not without extensive changes made in tox.ini which make its its maintenance hard to understand and use. I will try to describe the use case hoping that we endup with a good solution, one that we also document and one that can easily be adopted. Ideally we should aim to find a solution that could address this topic even without having to alter most tox.ini find in the wild.

Multi-platform use case and shared filesystem

I am developer which works on a python project using tox which has tests which need to pass not only on various python versions, but also on multiple platforms, like linux/osx/macos for example. As is always recommended to test changes locally before create a change-request, I do have a set of hosts where I run tox on.

To avoid having to rsync my changes among all these machines I do keep my source code in a shared disk (NFS, SAMBA,...).

This means that I should be able to run tox from each of these machines and get the results. Well, not really because the .tox environments are not isolated between machines, and they are clearly not portable.

In the past, on few personal projects I used a trick to add the platform in environment name and use a launcher (makefile) that passed correct name to tox. Not only ugly, inconvenient but also a deal breaker due to the modification of default env names for all the other developers which may don't care about other-platforms. I kinda see their point: why to worsen the UX for most people just to cover for such an advanced use-case.

Idea 1

I observed that one tool is not affected at all about this issue: pre-commit. This is because pre-commit creates its temporary environments in a location that is outside the repository. This means that this is unlikely to clash among platforms. In fact even if the entire user home directory would be network based the tool could easily include the platform name as part of the hashing when creating its environments. In fact the use of official caching folder has other advantages as this could easily be wiped, or stored on faster filesystems than the code itself.

One fun aspect of this approach is that we could experimental add a TOX_SHARED_WORKDIR variable that would optionally enable the new location of the work directory. Just putting a custom directory for tox work dir would not work because the tool itself needs to be aware that multiple code repositories are using the same caching folder, only a name or hash based solution could avoid clashing inside the "centralized workdir" (likely centralized not between machines but between repositories).

other ideas?

[gaborbernat]

I'm agreeable to the concept here, but not within the core. Sounds something that can be added as a plugin on top of tox 4.

[gaborbernat]

My available efforts at the moment are aimed at fixing this as part of #1394, but that probably will take a while (ETA September). Closing as there's no action from our side, and no one else seems to request this yet.