add easyconfig for 'Python-core' built with GCCcore which can be used as a build dep

[boegel]

This is an alternate approach to #4962, the main differences being:

• using Python-core as software name, to avoid confusion with a full Python installation
• making the module hidden by default (since it's not intended to be used directly by users)
• including some extensions in it, rather than creating separate modules for each of them (this also avoids the need for the nasty RPATH hack afaik)
• (edit) also, you can load a full Python module to shadow a loaded Python-core (that was part of the reason why I renamed this Python built with GCCcore as Python-core, to avoid a name clash with Python)

The purpose of this Python-core is similar to what @JackPerdue had in mind: to be used as a build dependency for modules installed with GCCcore. If other software needs the Python bits provided by those modules, they'll either need to load Python-core as a build dep, or load a full Python as a dependency.

I consider this a blocker for EasyBuild v3.4.0, since I'm not in favour of including the 'bare' Python with GCCcore that got merged in #4962...

Thoughts @easybuilders/easybuild-easyconfigs-maintainers & @JackPerdue?

[verdurin]

Test report by @verdurin
SUCCESS
Build succeeded for 7 out of 7 (4 easyconfigs in this PR)
ca003.camp.thecrick.org - Linux centos linux 7.3.1611, Intel(R) Xeon(R) CPU E5-2640 v3 @ 2.60GHz, Python 2.7.5
See https://gist.github.com/6b22d514c425041dca39a43202d4bfb8 for a full test report.

[JackPerdue]

From #4962

Note: Here's a quick list of things I have working with Python-bare and GCCcore....

NTLR ASE asyncoro ATK cairo cairocffi cffi colorama Cycler Cython dbus-glib dispy Docutils FFmpeg FIAT FLTK FOX freeglut FTGL GConf Gdk-Pixbuf GEOS gettext Ghostscript GLEW GLib GObject-Introspection GST-plugins-base GStreamer GTK+ HarfBuzz ImageMagick jemalloc json2html libdap libdrm libepoxy libglade libGLU libICE libspatialite libxml2 libxslt LLVM Mako MarkupSafe matplotlib Mesa mpmath myriad Pango ply PostgreSQL psycopg2 PyCairo pycparser PyGObject PyGTK PyOpenGL PyQt PyQt5 PySlurm Python PyYAML PyZMQ QCA QJson QScintilla QScintilla5 Qt Qt5 Qwt QwtPolar rebuild.FLTK rebuild.PyGObject requests setuptools simplejson SIP six SWIG sympy Tk VTK wxPython X11 xprop

[JackPerdue]

Is this approach going to work for all those??? I don't want to expand the "core/base" GCCcore Python with all of that. How will they be added so that their Python version is clear?

[boegel]

@JackPerdue Hmm, that's a long list...

For some of them, like PyQt, basically all the ones that depend on additional stuff, it definitely does not make sense to include them as extensions in Python-core itself (I consider PyYAML an exception).

Stuff like simplejson, sympy, etc., sure, those should be included in Python-core imho.

If the core of what the software provides is a Python package (that's not the case for X11 imho, see discussion in #4979), then we should make the Python version for which it was built clear via versionsuffix (I'm still a bit concerned about possible confusion on automatic loading of Python as a dep though...).

[JackPerdue]

Well.. the point was to keep Python "bare".... e.g.

exts_list = []

It doesn't even have "setuptools" (I think)... by design.

It's much easier to manage extensions individually than trying to lump them all into a giant EasyConfig that won't build (like R).

[boegel]

@JackPerdue I disagree on the latter, because it also leads to an explosion of modules for Python packages. With your approach, it's even worse imho because the modules you obtain don't even provide a working installation unless you load an additional Python module with it.

The Python packages we'd include as extensions should remain fairly limited in number, certainly less than a full Python where we also include numpy, scipy & co.
Basically, we should only include those packages that are required as a build dep for something.

As soon as you need Python as a runtime dep, you need to use a full Python imho.

Also, including some Python packages as extensions doesn't mean you can't flesh out others as stand-alone modules, you can still do that. But, the whole Python-built-with-GCCcore-to-be-used-as-a-build-dep complicates that imho, since I don't see a good way of making it clear that loading that module implies that you need to load a full Python along with it...

[JackPerdue]

BTW (as mentioned in previous PR), I have a bunch for iompi/iimpi/gompi as well.

[vanzod]

I like the idea of Python-core, less the fact that we will have two installation of Python depending if it is used as build dependency or as interpreter for human consumption.

I know that this is not back-compatible, but how about the Python module being a bundle that has the hidden Python-core as dependency and, eventually, add additional packages?
In other words, something similar to GCCcore and GCC.

[mboisson]

Regarding @JackPerdue's comment

How will they be added so that their Python version is clear?

I would strongly suggest the EB community to adopt what we have been doing at Compute Canada. Our modules that provide python bindings or python packages work with all supported versions of python. You load the same module for mpi4py to use it with python/2.7, 3.5 or 3.6.

All that is needed for that is to forgo PYTHONPATH in favor of a sitecustomize.py file and an environment variable (which we named EBPYTHONPREFIXES). The only path that PYTHONPATH has is the one that contains sitecustomize.py

Our sitecustomize.py looks like this. It would have to be refined for the broader EB community.

import os
import site

# use prefixes from EBPYTHONPREFIXES, so they have lower priority than
# virtualenv-installed packages

if "EBPYTHONPREFIXES" in os.environ:
    postfix = os.path.join('lib', 'python'+sys.version[:3], 'site-packages')
    for prefix in os.environ["EBPYTHONPREFIXES"].split(os.pathsep):
        sitedir = os.path.join(prefix, postfix)
        if os.path.isdir(sitedir):
            site.addsitedir(sitedir)


# Append site-packages in Nix profile to sys.path so packages can be found there.
# This avoids needing to set PYTHONPATH to this directory.
# virtualenv uses a private site-packages directory so this should NOT be done inside
# a virtualenv; in that case
# /cvmfs/soft.computecanada.ca/nix/store/<hash>/lib/pythonx.y/site-packages
# is not in sys.path either

def get_base_prefix():
    """Get base prefix, valid for virtualenv, pyvenv, and python without virtualenv"""
    return getattr(sys, 'real_prefix', getattr(sys, 'base_prefix', sys.prefix))

if get_base_prefix().startswith('/cvmfs/soft.computecanada.ca/nix/store/'):
    newprefix = os.environ.get("EBROOTPYTHON")
    if newprefix is not None:
        postfix = os.path.join('lib', 'python'+sys.version[:3], 'site-packages')
        nixstoresitepackages = os.path.join(sys.prefix, postfix)
        if nixstoresitepackages in sys.path:
            site.addsitedir(os.path.join(newprefix, postfix))

[Micket]

This is no longer necessary

[boegel]

Indeed, cfr. #7821 and #7898 (+ #8065 & #8072)