Python: Location of algorithm files

[Lestropie]

With the FHS re-arrangement of the repository, we now have the following situation:

bin/5ttgen
lib/mrtrix3/app.py (and others)
lib/mrtrix3/_5ttgen/fsl.py
lib/mrtrix3/_5ttgen/freesurfer.py

That placement was maybe not super-critically thought out with the FHS changes. I'm now not sold on it: The files in lib/mrtrix3/_5ttgen/ aren't really a part of the "library", since they're by design intended to be used by a single script only. Additionally, if someone were to write a standalone script against the MRtrix3 Python libraries (which they can, and I myself do), they would not be able to make use of the algorithm module, since the corresponding module files need to go into MRtrix3's lib/mrtrix3/ directory, regardless of where the executable script resides.

They clearly can't reside directly in bin/, since they're not executable in and of themselves, and that could lead to naming conflicts. Likewise they don't really belong in core/, since a packaged installation needs to contain these files yet that whole directory doesn't exist in the case of packaged installs.

I'm wondering if they should instead reside in sub-directories of bin/? One sub-directory per module-based script as is currently done, with the name of the sub-directory being the executable script's name, somehow modified to prevent naming conflicts. So we'd have e.g. bin/_5ttgen/ and bin/_dwi2response/. The algorithm module would then find them based on the real path (i.e. follow symlinks) of the executed script, rather than the location of the library files.

[thijsdhollander]

That placement was maybe not super-critically thought out with the FHS changes. I'm now not sold on it

Yes, I fully agree with this. But the problem is then indeed what makes more sense (I'm not sure myself...). I understand the original reasons for having the "main" scripts (i.e. not talking about the algos here) in bin/, but I have to admit this still jars a bit for me. Moving them there has somewhat contributed to the issue voiced here now as well. I've got a similar feeling that the main scripts (e.g. dwi2response) should be sticking closer to their algorithms. But where I'm undecided is whether that should be both in bin/ (as you suggest; I wouldn't be against this in fact), or somewhere else. And if somewhere else, where that should be then of course...

I can see the appeal of going all-in on the FHS philosophy (and there's definitely a lot of good aspects to FHS, not denying that at all), but as most of you probably already know, I'm for a system that serves the user (the user also possibly being the developer; let's call it "the human"), not the other way around. What would be e.g. a convincing (practical) argument against having a scripts/ folder (at the top level)? It still wouldn't include the actual script library of course. Just the scripts that now sit in bin/ together with sub-folders for algorithms in the way that @Lestropie proposes here. All the rest remains FHS of course as well. And then bin/ are only "real" binaries again, and scripts are scripts. 🤷‍♂️

[jdtournier]

OK, this is something that we discussed quite a bit previously in the context of #864 and the associated pull request #896. In particular, I'd commented on most of the concerns expressed here in this post - worth a re-read to re-acquaint yourselves with the arguments I'd put forward at the time to support these changes, and I think they're all still entirely relevant as far as I can tell.

Just as another example, I've just had a look around for well-known applications actually written in Python (there's remarkably few well-known ones, surprisingly - lots of libraries, but few actual applications). One of them is scons, a tool similar to cmake. On my system, its main application is in bin/, and its modules reside in /usr/lib/python2.7/site-packages/SCons/ - entirely consistent with our approach. I also note they use the same trick that we use to add the location of their modules to the system search path - see these lines in particular.

So I get that the fact that the scripts are now in bin/ is not as 'clean' as if they were to reside in their own scripts folder, but as far as users (and sysadmins) are concerned that's by far the simplest and most obvious location for an executable application to be located. As I've said before, having the library modules and algorithms in lib/mrtrix3, and any non-code data in share/mrtrix3 is completely consistent with the expected convention for an Unix system. I've listed a few relevant examples in that previous post (linked to above), and they're far from the exception...

Happy to discuss, of course, but in this instance I really don't think there is a good argument for changing things around yet again, and in my opinion the current locations are fact the most appropriate ones...

[Lestropie]

Ah OK, so it was in fact an informed decision and not just a random choice on my part. I'll just have to accept not being able to use this on my stand-alones then. Sorry for the time waste.