Reorganise GRASS installation to comply with Filesystem Hierarchy Standard (FHS)

[nilason]

In grass-dev ML thread the issue was raised on the reorganisation of the GRASS installation according to Filesystem Hierarchy Standard (FHS). The present work with enabling CMake-based building/installation present an opportunity to adopt such a change. If the community will reach consensus on this goal, this may be the place to discuss the changes that need to be made.

See also FHS on linuxfoundation.org.

This is the current (default) installation structure:

└── usr
    ├── bin
    │   └── grass
    └── grass84                   <- $GISBASE
        ├── AUTHORS
        ├── CHANGES
        ├── CITING
        ├── COPYING
        ├── GPL.TXT
        ├── INSTALL.md
        ├── REQUIREMENTS.md
        ├── bin
        ├── contributors.csv
        ├── contributors_extra.csv
        ├── demolocation
        ├── docs
        ├── driver
        ├── etc
        ├── fonts
        ├── gui
        ├── include
        ├── lib
        ├── locale
        ├── scripts
        ├── share
        ├── translation_status.json
        ├── translators.csv
        └── utils

Current (reorganised) Debian installation structure

└── usr
    ├── bin
    │   ├── grass
    │   └── x-grass
    ├── lib
    │   └── grass84             <- $GISBASE
    │       ├── AUTHORS
    │       ├── CHANGES
    │       ├── CITING
    │       ├── COPYING
    │       ├── GPL.TXT
    │       ├── INSTALL.md
    │       ├── REQUIREMENTS.md
    │       ├── bin
    │       ├── contributors.csv
    │       ├── contributors_extra.csv
    │       ├── demolocation
    │       ├── driver
    │       ├── etc
    │       ├── fonts
    │       ├── gui
    │       ├── include
    │       ├── lib
    │       ├── locale
    │       ├── scripts
    │       ├── translation_status.json
    │       ├── translators.csv
    │       └── utils
    └── share
        ├── doc
        │   ├── grass-dev-doc
        │   └── grass-doc
        ├── grass
        │   ├── applications
        │   ├── icons
        │   └── metainfo
        └── man
            └── man1

My proposition for future GRASS structure, based on the above Debian structure and discussion on the ML:

└── usr
    ├── bin
    │   └── grass
    ├── lib
    │   └── grass               <- $GISBASE
    │   │   ├── AUTHORS
    │   │   ├── CHANGES
    │   │   ├── CITING
    │   │   ├── COPYING
    │   │   ├── GPL.TXT
    │   │   ├── INSTALL.md
    │   │   ├── REQUIREMENTS.md
    │   │   ├── contributors.csv
    │   │   ├── contributors_extra.csv
    │   │   ├── demolocation
    │   │   ├── driver
    │   │   ├── etc
    │   │   ├── fonts
    │   │   ├── gui
    │   │   ├── include
    │   │   ├── locale
    │   │   ├── translation_status.json
    │   │   ├── translators.csv
    │   │   └── utils
    │   ├── libgrass_arraystats.8.4.dylib
    │   ├── ...
    │   └── libgrass_vedit.8.4.dylib
    ├── libexec
    │   └── grass               <- Current $GISBASE/bin + $GISBASE/scripts 
    │       ├── d.background
    │       ├── d.colorlist
    │       ├── ...
    │       ├── v.what
    │       └── v.what.rast
    └── share
        ├── doc
        │   ├── grass-dev-doc
        │   └── grass-doc
        ├── grass
        │   ├── applications
        │   ├── icons
        │   └── metainfo
        └── man
            └── man1

[HuidaeCho]

This is what I propose:

└── usr
    ├── bin
    │   └── grass
    ├── lib
    │   ├── libgrass_arraystats.8.4.dylib
    │   ├── ...
    │   └── libgrass_vedit.8.4.dylib
    ├── libexec
    │   └── grass               <- Current $GISBASE/bin + $GISBASE/scripts 
    │       ├── d.background
    │       ├── d.colorlist
    │       ├── ...
    │       ├── v.what
    │       ├── v.what.rast
    │       ├── wxpython        <- Current $GISBASE/gui/wxpython
    │       ├── utils           <- Current $GISBASE/utils
    │       ├── drivers         <- Current $GISBASE/driver
    │       └── etc             <- Excutables only from current $GISBASE/etc
    ├── include
    ├── share
    │   ├── grass
    │   │   ├── doc
    │   │   │   ├── AUTHORS
    │   │   │   ├── CHANGES
    │   │   │   ├── CITING
    │   │   │   ├── COPYING
    │   │   │   ├── GPL.TXT
    │   │   │   ├── INSTALL.md
    │   │   │   ├── REQUIREMENTS.md
    │   │   │   ├── contributors.csv
    │   │   │   ├── contributors_extra.csv
    │   │   │   ├── translation_status.json
    │   │   │   ├── translators.csv
    │   │   │   ├── fonts
    │   │   │   ├── etc         <- Non-executables from current $GISBASE/etc
    │   │   │   ├── demolocation
    │   │   │   ├── ...         <- User documentation
    │   │   │   └── dev         <- Developer documentation
    │   │   ├── applications
    │   │   ├── icons           <- Current $GISBASE/gui/icons and other icons
    │   │   └── metainfo        <- What is this for? Distribution-specific?
    │   └── man
    │       └── man1
    └── locale

[wenzeslaus]

/usr/lib/{libraries} and /usr/libexec/grass/{tools} make sense to me. I'm sort of unclear about wxpython. Setting up the path is needed there, but then if you want to do it, you are in a situation where you don't care much about one extra path setup. No idea how this fits with FHS.

[nilason]

Thanks for your input. I was following the least-resistance path and didn't go all-in breaking up the current $GISBASE, but this might be worth persuing. I post here a suggestion of a bit reworked structure, and iteration of Huidae's version.

└── usr
    ├── bin
    │   └── grass
    ├── lib
    │   ├── libgrass_arraystats.8.4.dylib
    │   ├── ...
    │   └── libgrass_vedit.8.4.dylib
    ├── libexec
    │   └── grass               <- New quasi $GISBASE
    │       ├── bin             <- Current $GISBASE/bin + $GISBASE/scripts
    │       │   ├── d.background
    │       │   ├── d.colorlist
    │       │   ├── ...
    │       │   ├── v.what
    │       │   └── v.what.rast
    │       ├── driver          <- Current $GISBASE/driver
    │       ├── etc             <- Excutables only from current $GISBASE/etc
    │       ├── include
    │       │   └── Make        <- Current $GISBASE/include/Make
    │       └── utils           <- Current $GISBASE/utils
    ├── "Python 'site-packages' dir"
    │   └── grass               <- Current $GISBASE/etc/python/grass + $GISBASE/gui
    ├── include
    │   └── grass               <- Current $GISBASE/include/grass
    ├── share
    │   ├── doc
    │   │   ├── grass-dev-doc
    │   │   │   └── programming-manual  <- Current $GISBASE/html (if `make htmldocs-single`)
    │   │   └── grass-doc               <- Current $GISBASE/docs/html
    │   ├── grass
    │   │   ├── AUTHORS
    │   │   ├── CHANGES
    │   │   ├── CITING
    │   │   ├── COPYING
    │   │   ├── GPL.TXT
    │   │   ├── INSTALL.md
    │   │   ├── REQUIREMENTS.md
    │   │   ├── contributors.csv
    │   │   ├── contributors_extra.csv
    │   │   ├── translation_status.json
    │   │   ├── translators.csv
    │   │   ├── fonts
    │   │   ├── etc             <- Non-executables from current $GISBASE/etc
    │   │   ├── demolocation
    │   │   ├── ...
    │   │   ├── applications
    │   │   ├── icons           <- Current $GISBASE/gui/icons and other icons
    │   │   └── metainfo
    │   └── man
    │       └── man1            <- Current $GISBASE/docs/man/man1
    └── locale
        └── grass               <- Current $GISBASE/locale

The present form of our Python code doesn't fit anywhere, as far as I understand, in FHS. So I suggest to put it all under libexec/grass/python, both the library part and the gui.

We add a substructure to the libexec/grass with sub-directories for each specific part: bin, driver, etc. This will enable setting the $PATH correctly for the specific parts needed. The include/Make is a bit of outsider here, but is needed for addons (if I'm not mistaken) and the share directory wouldn't be optimal for this directory.

It is an established custom to put non-man page documentation in share/doc, so I'd insist we put the html documentations in there.

In general it would be worth studying the content of present $GISBASE/etc on what does what and where it should go. (There is a lot there of which I have no clue of its purpose.)

[wenzeslaus]

The present form of our Python code doesn't fit anywhere, as far as I understand, in FHS. So I suggest to put it all under libexec/grass/python, both the library part and the gui.

I don't know if this is according to FHS, but QGIS and other applications have their Python pieces in /usr/lib/python3/dist-packages/ on my Ubuntu machine.

It seems to me that /usr/lib/python3/dist-packages/ works for Python code and packages sort of like /usr/lib works for compiled code and dynamic libraries.

[echoix]

What would the organisation look like if we thought about GRASS as a pure Python package (application and library) that has natively compiled code. Does it bring the réflexion further?

[nilason]

A Python package would “normally” go to dist-packages or site-packages, depending on platform convention. Either way, it should be settable with configuration. GRASS Python code, as far as I understand, is not in present form a standard Python package. We’ll still need a “GISBASE” in either /usr/lib/grass or /usr/libexec/grass, doesn’t matter much (I think) which one, to which we can add the Python dirs (code).
Changing GRASS Python code, to be able to become a “proper” python module would be very difficult to implement to work side by side with the present code. If we want to change the Python code base into a Python module, that should preferably be made after complete transition to FHS.

[wenzeslaus]

GRASS Python code, as far as I understand, is not in present form a standard Python package. ... Changing GRASS Python code, to be able to become a “proper” python module would be very difficult to implement to work side by side with the present code.

What are you referring to? The following should work (assuming grass is on path and latest main):

import sys
import subprocess

sys.path.append(
    subprocess.check_output(["grass", "--config", "python_path"], text=True).strip()
)

import grass.script as gs

path = "/tmp/nc_spm/"
gs.create_project(path, epsg="3358")
with gs.setup.init(path):
    epsg = gs.parse_command("g.proj", flags="g")["srid"]
    assert epsg == "EPSG:3358"

[echoix]

I was trying to imagine if ever GRASS was able to compile the native parts through pip (and wheels), as many scientific apps do, what would the Python layout look like (where would it be). With cmake the newer package building toolchain makes it possible in the future

[nilason]

GRASS Python code, as far as I understand, is not in present form a standard Python package. ... Changing GRASS Python code, to be able to become a “proper” python module would be very difficult to implement to work side by side with the present code.

What are you referring to? The following should work (assuming grass is on path and latest main):

Maybe I misremember this, or mix it up with previous similar conversations. All the better. Let's try it out. I'll update my latest graph above. (I wonder only why we haven't made this move before?).
The question following this is: how can we integrate in one package the gui part with the other?

[echoix]

A dependency? Or like an optional feature, like black vs black[jupyter]

[echoix]

I didn't analyse thoroughly if it's a good move (I'm not sure), but it certainly helps to think about how we organize our stuff

[nilason]

Mapping of the current and the FHS-complying install directories, which should suffice for managing the the two structures:

Variable	Current definition	FSH definition
GISBASE_DIR	"grass84"	${CMAKE_INSTALL_LIBEXECDIR}/${PROJECT_NAME_LOWER}
GRASS_BIN_DIR	${GISBASE_DIR}/bin	${GISBASE_DIR}/bin
GRASS_LIB_DIR	${GISBASE_DIR}/lib	${CMAKE_INSTALL_LIBDIR}
GRASS_SCRIPTS_DIR	${GISBASE_DIR}/scripts	${GISBASE_DIR}/bin
GRASS_SHARE_DIR	${GISBASE_DIR}/share	${CMAKE_INSTALL_DATAROOTDIR}/${PROJECT_NAME_LOWER}
GRASS_ETC_DIR	${GISBASE_DIR}/etc	${GRASS_SHARE_DIR}/etc
GRASS_ETC_BIN_DIR	${GISBASE_DIR}/etc	${GISBASE_DIR}/etc
GRASS_PY_DIR	${GISBASE_DIR}/etc/python/grass	${Python_SITEARCH}/${PROJECT_NAME_LOWER}
GRASS_GUI_DIR	${GISBASE_DIR}/gui	${GRASS_PY_DIR}/???
GRASS_DRIVER_DIR	${GISBASE_DIR}/driver	${GISBASE_DIR}/driver
GRASS_FONTS_DIR	${GISBASE_DIR}/fonts	${GRASS_SHARE_DIR}/fonts
GRASS_UTILS_DIR	${GISBASE_DIR}/utils	${GISBASE_DIR}/utils
GRASS_INCLUDE_DIR	${GISBASE_DIR}/include	${CMAKE_INSTALL_INCLUDEDIR}
GRASS_DOC_DIR	${GISBASE_DIR}/docs/html	${CMAKE_INSTALL_DATAROOTDIR}/doc/${PROJECT_NAME_LOWER}-doc
GRASS_DEV_DOC_DIR	${GISBASE_DIR}/html	${CMAKE_INSTALL_DATAROOTDIR}/doc/${PROJECT_NAME_LOWER}-dev-doc
GRASS_MAN_DIR	${GISBASE_DIR}/docs/man	${CMAKE_INSTALL_MANDIR}
GRASS_DEMO_DIR	${GISBASE_DIR}/demolocation	${GRASS_SHARE_DIR}/demolocation
GRASS_MISC_DIR	${GISBASE_DIR}	${GRASS_SHARE_DIR}
GRASS_MAKEFILE_DIR	${GISBASE_DIR}/include/Make	${GISBASE_DIR}/include/Make
GRASS_LOCALE	${GISBASE_DIR}/locale	${CMAKE_INSTALL_LOCALEDIR}

Given this is set:

string(TOLOWER ${PROJECT_NAME} PROJECT_NAME_LOWER)

The FHS version resolves to the following (after include(GNUInstallDirs)):

-- GISBASE_DIR libexec/grass
-- GRASS_BIN_DIR libexec/grass/bin
-- GRASS_LIB_DIR lib
-- GRASS_SCRIPTS_DIR libexec/grass/bin
-- GRASS_SHARE_DIR share/grass
-- GRASS_ETC_DIR share/grass/etc
-- GRASS_ETC_BIN_DIR libexec/grass/etc
-- GRASS_PY_DIR [...]/lib/python3.11/site-packages/grass
-- GRASS_DRIVER_DIR libexec/grass/driver
-- GRASS_UTILS_DIR libexec/grass/utils
-- GRASS_INCLUDE_DIR include
-- GRASS_DOC_DIR share/doc/grass-doc
-- GRASS_DEVDOC_DIR share/doc/grass-dev-doc
-- GRASS_MAN_DIR share/man
-- GRASS_DEMO_DIR share/grass/demolocation
-- GRASS_MISC_DIR share/grass
-- GRASS_MAKEFILE_DIR libexec/grass/Make
-- GRASS_LOCALE share/locale

(GRASS_MISC_DIR is intended for various files now installed in $GISBASE like AUTHORS, CHANGES, CITING etc.)

[wenzeslaus]

As for GUI (${GISBASE_DIR}/gui - ${GRASS_PY_DIR}/???), putting it together with the rest of Python code under gui mentioned before seems like an obvious choice:

grass/script
grass/temporal
grass/gui

Then we can either rewrite imports in GUI or continue setting up the path manually for GUI. GUI is not part of our public API, so that would speak against such location, but we already have other things which are not part of public API for various reasons in sub-directories of python/grass. I don't know if it is common to mix two different directories in source into one directory in distribution especially when is optional.

Alternatives I can think of are:

2) grass/wxgui
3) grass_gui

[echoix]

The underscore might make matching Python imports a bit harder

[nilason]

@sebastic Could you please spare some minutes and give us some of your thought on this topic, while it is still in its embryonic stage?

[sebastic]

The grass Debian package currently abuses --prefix=/usr/lib and INST_DIR=/usr/lib/grass83 to install the files in non-standard directories.

The prefix should just be /usr for the Debian package and /usr/local by default for user builds.

The proposed structure from #3661 (comment) looks good at first glance and would do the right thing for both prefixes.

You probably need an additional directory for the different executables, #3661 (comment) doesn't contains UNIX_BIN for /usr/bin/grass (${CMAKE_INSTALL_BINDIR}), only GRASS_BIN_DIR for g.proj and friends.

[nilason]

The grass Debian package currently abuses --prefix=/usr/lib and INST_DIR=/usr/lib/grass83 to install the files in non-standard directories.

The prefix should just be /usr for the Debian package and /usr/local by default for user builds.

The proposed structure from #3661 (comment) looks good at first glance and would do the right thing for both prefixes.

Thanks for looking at this, greatly appreciate it! We then have a plan...

You probably need an additional directory for the different executables, #3661 (comment) doesn't contains UNIX_BIN for /usr/bin/grass (${CMAKE_INSTALL_BINDIR}), only GRASS_BIN_DIR for g.proj and friends.

I was thinking of using ${CMAKE_INSTALL_BINDIR} directly for the startup executable (/usr/bin/grass), but it wasn't perhaps obvious with the above proposition.

[wenzeslaus]

...the startup executable (/usr/bin/grass)...it wasn't perhaps obvious with the above proposition.

To ease communication, maybe we should use this opportunity to settle on one name. While it is placed in lib/init and it has a lot of initialization code, it does also the clean up and with --exec it executes subprocesses handling, again, everything from the beginning to the end. Startup or init script don't seem to cover it well. What about calling it the main executable?

[nilason]

main executable sounds ok for me. By the way, is the script file https://github.com/OSGeo/grass/blob/main/lib/init/grass.sh used anywhere?

[wenzeslaus]

I'm not aware of grass.sh being used. I noticed that the bat file is used on Windows. My Ubuntu install of GRASS GIS is grass.py, not the sh file. It looks like it might have been used for Python 3 in the transitional times, but grass.py and others have shebang. I would say it may go.

[nilason]

I have now implemented the proposed two-way build and install for both the current structure and for FHS enabled with a configuration flag. It is still missing some parts (eg. locale, documentation and man page generation), and have some temporary workarounds for the splitting up of GISBASE content in FHS mode, but it otherwise seems promising.

I'm not sure what to do with the GUI part yet.

As for GUI (${GISBASE_DIR}/gui - ${GRASS_PY_DIR}/???), putting it together with the rest of Python code under gui mentioned before seems like an obvious choice:

I think so too. This is what we now have:

${GISBASE}
├── ...
├── gui
│   ├── icons                         <- resource
│   ├── images                        <- resource
│   ├── scripts                       <- resource
│   ├── wxpython                      <- Python module ${GRASS_PY_DIR}/wxgui
│   │   ├── README
│   │   ├── animation
│   │   ├── core
│   │   ├── datacatalog
│   │   ├── dbmgr
│   │   ├── ...
│   │   ├── tplot
│   │   ├── vdigit
│   │   ├── vnet
│   │   ├── web_services
│   │   ├── wxgui.py
│   │   ├── wxplot
│   │   └── xml                     <- resource
│   │       ├── grass-gxm.dtd
│   │       ├── grass-gxw.dtd
│   │       ├── main_menu.dtd
│   │       ├── main_menu.xml
│   │       ├── ...
│   │       ├── toolboxes.xml
│   │       ├── wxgui_items.dtd
│   │       └── wxgui_items.xml
│   └── xml                           <- resource
│       └── grass-interface.dtd
├── ...

As you can see there are some GUI related resources, the question is where to but them.

This is a possible new destination/structure:

${Python_SITEARCH}
└── grass                               <- ${GRASS_PY_DIR}
    ├── __init__.py
    ├── app
    ├── benchmark
    ├── exceptions
    ├── grassdb
    ├── gunittest
    ├── imaging
    ├── jupyter
    ├── lib
    ├── pydispatch
    ├── pygrass
    ├── script
    ├── semantic_label
    ├── temporal
    ├── utils
    └── wxgui                           <- GUI Python module (current gui/wxpython)
        ├── animation
        ├── ...
        ├── wxplot
        └── resources                   <- resources ???
            ├── icons
            ├── images
            ├── scripts
            └── xml

An alternative is to put the resources under eg. ${GRASS_ETC_DIR}/gui.

[sebastic]

The resources seem to be architecture independent data which suggests ${GRASS_SHARE_DIR} (/usr/{,local/}share/grass) or a subdirectory thereof as an appropriate location.