From e30b6e166f2c909d7e5d7c467df754a51e4bfd75 Mon Sep 17 00:00:00 2001 From: Michael H Kelsey Date: Thu, 16 Apr 2020 23:08:16 -0500 Subject: [PATCH 01/16] Add paragraph explaining exts_list options. --- docs/version-specific/easyconfig_parameters.rst | 7 +++++++ 1 file changed, 7 insertions(+) diff --git a/docs/version-specific/easyconfig_parameters.rst b/docs/version-specific/easyconfig_parameters.rst index b57e93ac..5a108cb0 100644 --- a/docs/version-specific/easyconfig_parameters.rst +++ b/docs/version-specific/easyconfig_parameters.rst @@ -34,6 +34,7 @@ toolchain parameters ``toolchainopts`` Extra options for compilers None ================== ================================================================================================================================================================================================================================= ================= +.. _easyconfig_parameters_build_parameters: build parameters ---------------- @@ -137,6 +138,12 @@ extensions parameters ``exts_list`` List with extensions added to the base installation [] ========================== ========================================================================================================== ================= +Each entry in ``exts_list`` is a tuple with the extension name +(``ext_name``), version string (``ext_version``), and a dictionary of +source options. Any of the +:ref:`easyconfig_parameters_build_parameters` can be specified within +the options dictionary. + modules parameters ------------------ From ad027b1456039e591f12ccd3846472203a0e63a5 Mon Sep 17 00:00:00 2001 From: Michael H Kelsey Date: Tue, 21 Apr 2020 12:05:04 -0500 Subject: [PATCH 02/16] Remove modifications under version-specific. Start adding section on Extensions in Writing_easyconfig_fils. --- docs/Writing_easyconfig_files.rst | 9 +++++++++ docs/version-specific/easyconfig_parameters.rst | 7 ------- 2 files changed, 9 insertions(+), 7 deletions(-) diff --git a/docs/Writing_easyconfig_files.rst b/docs/Writing_easyconfig_files.rst index edd0aff4..a64f475b 100644 --- a/docs/Writing_easyconfig_files.rst +++ b/docs/Writing_easyconfig_files.rst @@ -506,12 +506,21 @@ which is equivalent to:: dependencies = [('PnMPI', '1.2.0', '', True)] + Using external modules as dependencies ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Since EasyBuild v2.1, specifying modules that are not provided via EasyBuild as dependencies is also supported. See :ref:`using_external_modules` for more information. + +Module Extensions +~~~~~~~~~~~~~~~~~ + +Besides dependencies, which are found outside the package being built but are part of the site's EasyBuild +installation, it is also possible to incorporate "dependencies" into the package build itself. This is typically used as part of a PythonPackage or PythonBundle, ... This is done via the `exts_list` array. + + .. _configure_build_install_command_options: Configure/build/install command options diff --git a/docs/version-specific/easyconfig_parameters.rst b/docs/version-specific/easyconfig_parameters.rst index 5a108cb0..b57e93ac 100644 --- a/docs/version-specific/easyconfig_parameters.rst +++ b/docs/version-specific/easyconfig_parameters.rst @@ -34,7 +34,6 @@ toolchain parameters ``toolchainopts`` Extra options for compilers None ================== ================================================================================================================================================================================================================================= ================= -.. _easyconfig_parameters_build_parameters: build parameters ---------------- @@ -138,12 +137,6 @@ extensions parameters ``exts_list`` List with extensions added to the base installation [] ========================== ========================================================================================================== ================= -Each entry in ``exts_list`` is a tuple with the extension name -(``ext_name``), version string (``ext_version``), and a dictionary of -source options. Any of the -:ref:`easyconfig_parameters_build_parameters` can be specified within -the options dictionary. - modules parameters ------------------ From 85a105f91a28c02df832c81105ec0dc4f51b898e Mon Sep 17 00:00:00 2001 From: Michael Kelsey Date: Tue, 21 Apr 2020 21:36:47 -0700 Subject: [PATCH 03/16] Expand new 'Module extensions' section with example code and cross-refernences to sources dictionary and git_config. --- docs/Writing_easyconfig_files.rst | 33 +++++++++++++++++++++++++++++-- 1 file changed, 31 insertions(+), 2 deletions(-) diff --git a/docs/Writing_easyconfig_files.rst b/docs/Writing_easyconfig_files.rst index a64f475b..a5d6d111 100644 --- a/docs/Writing_easyconfig_files.rst +++ b/docs/Writing_easyconfig_files.rst @@ -517,8 +517,37 @@ See :ref:`using_external_modules` for more information. Module Extensions ~~~~~~~~~~~~~~~~~ -Besides dependencies, which are found outside the package being built but are part of the site's EasyBuild -installation, it is also possible to incorporate "dependencies" into the package build itself. This is typically used as part of a PythonPackage or PythonBundle, ... This is done via the `exts_list` array. +Besides dependencies, which are found outside the module being built but are part of the site's EasyBuild installation, +it is also possible to incorporate extensions to the module within the build. This is typically used as part of a +PythonPackage or PythonBundle, and done via the `exts_list` array:: + +.. code:: python + + exts_list = [ + ('llvmlite', '0.26.0', { + 'source_urls': ['https://pypi.python.org/packages/source/l/llvmlite/'], + 'patches': ['llvmlite-0.26.0_fix-ffi-Makefile.patch'], + 'checksums': [ + '13e84fe6ebb0667233074b429fd44955f309dead3161ec89d9169145dbad2ebf', # llvmlite-0.26.0.tar.gz + '40e6fe6de48709b45daebf8082f65ac26f73a4afdf58fc1e8099b97c575fecae', # llvmlite-0.26.0_fix-ffi-Makefile.patch + ], + }), + ('singledispatch', '3.4.0.3', { + 'source_urls': ['https://pypi.python.org/packages/source/s/singledispatch/'], + 'checksums': ['5b06af87df13818d14f08a028e42f566640aef80805c3b50c5056b086e3c2b9c'], + }), + (name, version, { + 'source_urls': ['https://pypi.python.org/packages/source/n/numba/'], + 'checksums': ['c62121b2d384d8b4d244ef26c1cf8bb5cb819278a80b893bf41918ad6d391258'], + }), + ] + +Each entry in `exts_list` is a three-component tuple, with the name and version number, and a dictionary of source +information. The latter is essentially the same as the the dictionary structure of `sources`; see +:ref:`_common_easyconfig_param_sources_alt` for details. + +Since EasyBuild v4.2.1, the `git_config` dictionary (see :ref:`downloading-from-a-git-repository`) may be included in +the `exts_list` source definition. .. _configure_build_install_command_options: From fdd50c39b5ed40e637e960435ed05071512e664b Mon Sep 17 00:00:00 2001 From: Michael Kelsey Date: Wed, 22 Apr 2020 23:03:24 -0700 Subject: [PATCH 04/16] Clean up formatting of new 'Module extensions' Fix my various RST mistakes in the new 'Module extensions' section of Writing_easyconfig_files. Remove spurious double-colon, improve indentation in `exts_list` example, add specific example for using `git_config` here (coming as part of Framework PR #3294. --- docs/Writing_easyconfig_files.rst | 22 ++++++++++++++++++---- 1 file changed, 18 insertions(+), 4 deletions(-) diff --git a/docs/Writing_easyconfig_files.rst b/docs/Writing_easyconfig_files.rst index a5d6d111..c5e0c50c 100644 --- a/docs/Writing_easyconfig_files.rst +++ b/docs/Writing_easyconfig_files.rst @@ -519,7 +519,7 @@ Module Extensions Besides dependencies, which are found outside the module being built but are part of the site's EasyBuild installation, it is also possible to incorporate extensions to the module within the build. This is typically used as part of a -PythonPackage or PythonBundle, and done via the `exts_list` array:: +PythonPackage or PythonBundle, and done via the `exts_list` array: .. code:: python @@ -528,8 +528,8 @@ PythonPackage or PythonBundle, and done via the `exts_list` array:: 'source_urls': ['https://pypi.python.org/packages/source/l/llvmlite/'], 'patches': ['llvmlite-0.26.0_fix-ffi-Makefile.patch'], 'checksums': [ - '13e84fe6ebb0667233074b429fd44955f309dead3161ec89d9169145dbad2ebf', # llvmlite-0.26.0.tar.gz - '40e6fe6de48709b45daebf8082f65ac26f73a4afdf58fc1e8099b97c575fecae', # llvmlite-0.26.0_fix-ffi-Makefile.patch + '13e84fe6ebb0667233074b429fd44955f309dead3161ec89d9169145dbad2ebf', # llvmlite-0.26.0.tar.gz + '40e6fe6de48709b45daebf8082f65ac26f73a4afdf58fc1e8099b97c575fecae', # llvmlite-0.26.0_fix-ffi-Makefile.patch ], }), ('singledispatch', '3.4.0.3', { @@ -547,7 +547,21 @@ information. The latter is essentially the same as the the dictionary structure :ref:`_common_easyconfig_param_sources_alt` for details. Since EasyBuild v4.2.1, the `git_config` dictionary (see :ref:`downloading-from-a-git-repository`) may be included in -the `exts_list` source definition. +the `exts_list` source definition. For example, to download Git sources from a private repository and convert them to a tar-ball for installation: + +.. code:: python + + exts_defaultclass = 'PythonPackage' + exts_list = [ + ('pyCAP', 'master', { + 'filename': 'pyCAP-master.tar.gz', + 'git_config': { + 'url': 'ssh://nero.stanford.edu/data/git/Analysis', + 'repo_name': 'pyCAP', + 'tag': 'master', + } + }), + ] .. _configure_build_install_command_options: From 544c236393b197c62b689a905a2b9cedebb2c5cf Mon Sep 17 00:00:00 2001 From: Michael Kelsey Date: Fri, 24 Apr 2020 22:56:27 -0700 Subject: [PATCH 05/16] Update exts_list to be explicit about options. --- docs/Writing_easyconfig_files.rst | 28 +++++++++++++++++++--------- 1 file changed, 19 insertions(+), 9 deletions(-) diff --git a/docs/Writing_easyconfig_files.rst b/docs/Writing_easyconfig_files.rst index c5e0c50c..04cbc24e 100644 --- a/docs/Writing_easyconfig_files.rst +++ b/docs/Writing_easyconfig_files.rst @@ -197,7 +197,7 @@ Using the ``--inject-checksums`` command line option, you can let EasyBuild add in one or more easyconfig files (which is significantly more convenient than doing it manually). With ``--inject-checksums``, checksums are injected for all sources and patches (if any), -as well as for all sources & patches of every extension listed in ``exts_list`` (if any). +as well as for all sources & patches of every extension listed in ``exts_list`` (if any, see :ref:`module_extensions`). If the sources (& patches) are not available yet, EasyBuild will try to download them first; i.e., the ``fetch`` step is run prior to computing & injecting the checksums. @@ -518,8 +518,18 @@ Module Extensions ~~~~~~~~~~~~~~~~~ Besides dependencies, which are found outside the module being built but are part of the site's EasyBuild installation, -it is also possible to incorporate extensions to the module within the build. This is typically used as part of a -PythonPackage or PythonBundle, and done via the `exts_list` array: +it is also possible to incorporate extensions to the module within the build. This is done via the `exts_list` array. + +Each entry in `exts_list` is a three-component tuple, with the name and version number, and a dictionary of source +information. The latter consist of most of the allowed keys as in the dictionary structure of `sources` (see +:ref:`_common_easyconfig_param_sources_alt`): + +* **nosource**: If specified, no download will be done +* **source_tmpl**: Template string for the file to be downloaded + * default is `'%(name)s-%(version)s.tar.gz'` + * `%(name)s` and `%(version)s` come from the `exts_list` entry (above) +* **source_urls**, **patches**, **checksums**, **force_download**: as in `sources` +* **git_config**: Dictionary specifying download from a Git repository .. code:: python @@ -542,19 +552,16 @@ PythonPackage or PythonBundle, and done via the `exts_list` array: }), ] -Each entry in `exts_list` is a three-component tuple, with the name and version number, and a dictionary of source -information. The latter is essentially the same as the the dictionary structure of `sources`; see -:ref:`_common_easyconfig_param_sources_alt` for details. - Since EasyBuild v4.2.1, the `git_config` dictionary (see :ref:`downloading-from-a-git-repository`) may be included in -the `exts_list` source definition. For example, to download Git sources from a private repository and convert them to a tar-ball for installation: +the `exts_list` source definition. For example, to download Git sources from a private repository and convert them to a +tar-ball for installation: .. code:: python exts_defaultclass = 'PythonPackage' exts_list = [ ('pyCAP', 'master', { - 'filename': 'pyCAP-master.tar.gz', + 'source_tmpl': '%(name)s-%(version)s.tar.gz', 'git_config': { 'url': 'ssh://nero.stanford.edu/data/git/Analysis', 'repo_name': 'pyCAP', @@ -563,6 +570,9 @@ the `exts_list` source definition. For example, to download Git sources from a }), ] +Here, the template strings `%(name)s` and `%(version)s` will be substituted from the `exts_list` entry elements ("pyCAP" +and "master", respectively), not from the EasyConfigs values. + .. _configure_build_install_command_options: From 0a97a7d3a2213cc9c4163d72aeb1280f30bd59df Mon Sep 17 00:00:00 2001 From: Michael Kelsey Date: Sat, 25 Apr 2020 14:23:37 -0700 Subject: [PATCH 06/16] Improve description of exts_list options, and how they differ from main sources options. --- docs/Writing_easyconfig_files.rst | 26 ++++++++++++++++++-------- 1 file changed, 18 insertions(+), 8 deletions(-) diff --git a/docs/Writing_easyconfig_files.rst b/docs/Writing_easyconfig_files.rst index 04cbc24e..624fd116 100644 --- a/docs/Writing_easyconfig_files.rst +++ b/docs/Writing_easyconfig_files.rst @@ -520,16 +520,26 @@ Module Extensions Besides dependencies, which are found outside the module being built but are part of the site's EasyBuild installation, it is also possible to incorporate extensions to the module within the build. This is done via the `exts_list` array. -Each entry in `exts_list` is a three-component tuple, with the name and version number, and a dictionary of source -information. The latter consist of most of the allowed keys as in the dictionary structure of `sources` (see -:ref:`_common_easyconfig_param_sources_alt`): +Each entry in `exts_list` is a three-component tuple, with the name and +version number, and a dictionary of source options:: -* **nosource**: If specified, no download will be done +.. code:: python + + exts_list = [ + ('name', 'version', { 'option':'value', 'option':'value' }) + ] + +The latter consist of most of the allowed keys as in the dictionary structure of `sources` (see +:ref:`_common_easyconfig_param_sources_alt`). Among those options, the following exceptions and +special cases should be noted: + +* **nosource**: If set `True`, no download will be done +* **filename** and **download_filename** are ignored; use `'source_tmpl'` * **source_tmpl**: Template string for the file to be downloaded * default is `'%(name)s-%(version)s.tar.gz'` * `%(name)s` and `%(version)s` come from the `exts_list` entry (above) -* **source_urls**, **patches**, **checksums**, **force_download**: as in `sources` * **git_config**: Dictionary specifying download from a Git repository +* **start_dir**: If not set, will be derived; the EasyConfigs value will not be used .. code:: python @@ -552,9 +562,9 @@ information. The latter consist of most of the allowed keys as in the dictionar }), ] -Since EasyBuild v4.2.1, the `git_config` dictionary (see :ref:`downloading-from-a-git-repository`) may be included in -the `exts_list` source definition. For example, to download Git sources from a private repository and convert them to a -tar-ball for installation: +Since EasyBuild v4.2.1, the `git_config` dictionary (see :ref:`downloading-from-a-git-repository`) +may be included in the `exts_list` source definition. For example, to download Git sources from a +private repository and convert them to a tar-ball for installation: .. code:: python From 3a4d12cb2065d9fb2b3594d40345e724db1bdb4e Mon Sep 17 00:00:00 2001 From: Michael Kelsey Date: Sat, 25 Apr 2020 14:29:07 -0700 Subject: [PATCH 07/16] Fix literal quoting in Module Extensions section. --- docs/Writing_easyconfig_files.rst | 26 ++++++++++++++------------ 1 file changed, 14 insertions(+), 12 deletions(-) diff --git a/docs/Writing_easyconfig_files.rst b/docs/Writing_easyconfig_files.rst index 624fd116..c481e855 100644 --- a/docs/Writing_easyconfig_files.rst +++ b/docs/Writing_easyconfig_files.rst @@ -518,10 +518,10 @@ Module Extensions ~~~~~~~~~~~~~~~~~ Besides dependencies, which are found outside the module being built but are part of the site's EasyBuild installation, -it is also possible to incorporate extensions to the module within the build. This is done via the `exts_list` array. +it is also possible to incorporate extensions to the module within the build. This is done via the ``exts_list`` array. -Each entry in `exts_list` is a three-component tuple, with the name and -version number, and a dictionary of source options:: +Each entry in ``exts_list`` is a three-component tuple, with the name and +version number, and a dictionary of source options: .. code:: python @@ -529,15 +529,15 @@ version number, and a dictionary of source options:: ('name', 'version', { 'option':'value', 'option':'value' }) ] -The latter consist of most of the allowed keys as in the dictionary structure of `sources` (see +The latter consist of most of the allowed keys as in the dictionary structure of ``sources`` (see :ref:`_common_easyconfig_param_sources_alt`). Among those options, the following exceptions and special cases should be noted: -* **nosource**: If set `True`, no download will be done -* **filename** and **download_filename** are ignored; use `'source_tmpl'` +* **nosource**: If set ``True``, no download will be done +* **filename** and **download_filename** are ignored; use ``'source_tmpl'`` * **source_tmpl**: Template string for the file to be downloaded - * default is `'%(name)s-%(version)s.tar.gz'` - * `%(name)s` and `%(version)s` come from the `exts_list` entry (above) + * default is ``'%(name)s-%(version)s.tar.gz'`` + * ``%(name)s`` and ``%(version)s`` come from the ``exts_list`` entry (above) * **git_config**: Dictionary specifying download from a Git repository * **start_dir**: If not set, will be derived; the EasyConfigs value will not be used @@ -562,8 +562,10 @@ special cases should be noted: }), ] -Since EasyBuild v4.2.1, the `git_config` dictionary (see :ref:`downloading-from-a-git-repository`) -may be included in the `exts_list` source definition. For example, to download Git sources from a +That third instance uses the ``name`` and ``version`` variables defined in the EasyConfigs file. + +Since EasyBuild v4.2.1, the ``git_config`` dictionary (see :ref:`downloading-from-a-git-repository`) +may be included in the ``exts_list`` source definition. For example, to download Git sources from a private repository and convert them to a tar-ball for installation: .. code:: python @@ -580,8 +582,8 @@ private repository and convert them to a tar-ball for installation: }), ] -Here, the template strings `%(name)s` and `%(version)s` will be substituted from the `exts_list` entry elements ("pyCAP" -and "master", respectively), not from the EasyConfigs values. +Here, the template strings ``%(name)s`` and ``%(version)s`` will be substituted from the +``exts_list`` entry elements ("pyCAP" and "master", respectively), not from the EasyConfigs values. .. _configure_build_install_command_options: From b9701ae42ed4cbd0ee9f569589a108759aab1fcd Mon Sep 17 00:00:00 2001 From: Michael Kelsey Date: Sat, 25 Apr 2020 14:54:56 -0700 Subject: [PATCH 08/16] Add section explaining how to use self.cfg in an EasyBlock, for issue #618. --- docs/Implementing-easyblocks.rst | 31 +++++++++++++++++++++++++++++++ 1 file changed, 31 insertions(+) diff --git a/docs/Implementing-easyblocks.rst b/docs/Implementing-easyblocks.rst index 82cccc66..0484753e 100644 --- a/docs/Implementing-easyblocks.rst +++ b/docs/Implementing-easyblocks.rst @@ -254,6 +254,37 @@ redefining it entirely, since we call out to the original ``configure_step`` met Specific aspects of easyblocks ------------------------------ +.. _implementing_easyblocks_default_parameters: + +Default easyconfig parameters +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +All of the parameters which are "set" in an easyconfig file (see :ref:`vsd_avail_easyconfig_params`) +become key-value pairs in the ``self.cfg`` dictionary. For instance, where the easyconfig file might +specify + +.. code:: python + + name = 'example' + version = '2.5.3' + versionsuffix = '-Python-3.7.4' + +you could access these within an easyblock via + +.. code:: python + + ec_name = self.cfg['name'] + ec_ver = self.cfg['version'] + ec_suffix = self.cfg['versionsuffix'] + +For these particular "universal" arguments, this mapping is done for you in the ``EasyBlock`` base +class and generic subclasses, so most easyconfig parameters are directly available as local Python +variables (``name``, ``version``, etc.). + +Some variables, and in particular :ref:`custom parameters +` which you define for your own easyblock, will not be +automatically mapped. You will need to use ``self.cfg`` to access them in your code. + .. _implementing_easyblocks_custom_parameters: From 91b9697a78fae0ea3f1664bb85ef47a1b151cc4f Mon Sep 17 00:00:00 2001 From: Michael Kelsey Date: Fri, 8 May 2020 07:11:27 -0700 Subject: [PATCH 09/16] Expand explanation of self.cfg[] vs. local self.variable usage. --- docs/Implementing-easyblocks.rst | 22 ++++++++++++---------- 1 file changed, 12 insertions(+), 10 deletions(-) diff --git a/docs/Implementing-easyblocks.rst b/docs/Implementing-easyblocks.rst index 0484753e..37f30ee4 100644 --- a/docs/Implementing-easyblocks.rst +++ b/docs/Implementing-easyblocks.rst @@ -269,21 +269,23 @@ specify version = '2.5.3' versionsuffix = '-Python-3.7.4' -you could access these within an easyblock via +these are accessible within an easyblock via .. code:: python - ec_name = self.cfg['name'] - ec_ver = self.cfg['version'] - ec_suffix = self.cfg['versionsuffix'] + longform = ''.join(self.cfg['name'],'/',self.cfg['version'],self.cfg['versionsuffix']) -For these particular "universal" arguments, this mapping is done for you in the ``EasyBlock`` base -class and generic subclasses, so most easyconfig parameters are directly available as local Python -variables (``name``, ``version``, etc.). +For these "universal" arguments, this mapping is done for you in the ``EasyBlock`` base class, so +they are also available as local Python variables (``self.name``, ``self.version``, etc.): -Some variables, and in particular :ref:`custom parameters -` which you define for your own easyblock, will not be -automatically mapped. You will need to use ``self.cfg`` to access them in your code. +.. code:: python + + longform = ''.join(self.name,'/',self.version,self.versionsuffix) + +This is true for the variables listed in :ref:`vsd_avail_easyconfig_params`. Any additional +:ref:`custom parameters ` which you define for your own +easyblock, will not be automatically mapped. You will need to use ``self.cfg`` to access them in +your code. .. _implementing_easyblocks_custom_parameters: From bf4df40795a73f62e47330363ca69153abbb2730 Mon Sep 17 00:00:00 2001 From: Michael Kelsey Date: Fri, 8 May 2020 08:05:05 -0700 Subject: [PATCH 10/16] Use proper version number in 'pyCAP' example of downloading extension sources. --- docs/Writing_easyconfig_files.rst | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/docs/Writing_easyconfig_files.rst b/docs/Writing_easyconfig_files.rst index c481e855..5cd90579 100644 --- a/docs/Writing_easyconfig_files.rst +++ b/docs/Writing_easyconfig_files.rst @@ -572,7 +572,7 @@ private repository and convert them to a tar-ball for installation: exts_defaultclass = 'PythonPackage' exts_list = [ - ('pyCAP', 'master', { + ('pyCAP', '0.1', { 'source_tmpl': '%(name)s-%(version)s.tar.gz', 'git_config': { 'url': 'ssh://nero.stanford.edu/data/git/Analysis', @@ -583,7 +583,7 @@ private repository and convert them to a tar-ball for installation: ] Here, the template strings ``%(name)s`` and ``%(version)s`` will be substituted from the -``exts_list`` entry elements ("pyCAP" and "master", respectively), not from the EasyConfigs values. +``exts_list`` entry elements ("pyCAP" and "0.1", respectively), not from the EasyConfigs values. .. _configure_build_install_command_options: From 06b015e410b882dfdf49544f5a46c91ab0c2f37c Mon Sep 17 00:00:00 2001 From: Michael Kelsey Date: Fri, 8 May 2020 14:23:54 -0700 Subject: [PATCH 11/16] Use version-number with exaple of exts_list git_config. --- docs/Writing_easyconfig_files.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/Writing_easyconfig_files.rst b/docs/Writing_easyconfig_files.rst index 5cd90579..a97521ee 100644 --- a/docs/Writing_easyconfig_files.rst +++ b/docs/Writing_easyconfig_files.rst @@ -577,7 +577,7 @@ private repository and convert them to a tar-ball for installation: 'git_config': { 'url': 'ssh://nero.stanford.edu/data/git/Analysis', 'repo_name': 'pyCAP', - 'tag': 'master', + 'tag': '%(version)s', } }), ] From 6cfd6f8dded04e00fba9ab21ae4d666445774c42 Mon Sep 17 00:00:00 2001 From: Michael Kelsey Date: Tue, 9 Jun 2020 10:27:36 -0700 Subject: [PATCH 12/16] Updated extensions section to reflect PR #3294 using 'sources', not 'git_config' directly. --- docs/Writing_easyconfig_files.rst | 33 ++++++++++++++++--------------- 1 file changed, 17 insertions(+), 16 deletions(-) diff --git a/docs/Writing_easyconfig_files.rst b/docs/Writing_easyconfig_files.rst index d0fa83e1..e5f1064a 100644 --- a/docs/Writing_easyconfig_files.rst +++ b/docs/Writing_easyconfig_files.rst @@ -525,7 +525,7 @@ Besides dependencies, which are found outside the module being built but are par it is also possible to incorporate extensions to the module within the build. This is done via the ``exts_list`` array. Each entry in ``exts_list`` is a three-component tuple, with the name and -version number, and a dictionary of source options: +version number, and a dictionary of configuration options for the entry: .. code:: python @@ -533,16 +533,16 @@ version number, and a dictionary of source options: ('name', 'version', { 'option':'value', 'option':'value' }) ] -The latter consist of most of the allowed keys as in the dictionary structure of ``sources`` (see -:ref:`_common_easyconfig_param_sources_alt`). Among those options, the following exceptions and -special cases should be noted: +The latter may contain essentially any of the full EasyConfig parameters, including ``buildopts``, ``installopts``, etc. +Among those options, the following exceptions and special cases should be noted: * **nosource**: If set ``True``, no download will be done -* **filename** and **download_filename** are ignored; use ``'source_tmpl'`` * **source_tmpl**: Template string for the file to be downloaded * default is ``'%(name)s-%(version)s.tar.gz'`` * ``%(name)s`` and ``%(version)s`` come from the ``exts_list`` entry (above) -* **git_config**: Dictionary specifying download from a Git repository +* **sources**: Dictionary specifying details of where to download the extension + * equivalent to a single entry from the EasyConfig ``sources`` list + * preffered to use of ``source_tmpl`` * **start_dir**: If not set, will be derived; the EasyConfigs value will not be used .. code:: python @@ -566,22 +566,23 @@ special cases should be noted: }), ] -That third instance uses the ``name`` and ``version`` variables defined in the EasyConfigs file. - -Since EasyBuild v4.2.1, the ``git_config`` dictionary (see :ref:`downloading-from-a-git-repository`) -may be included in the ``exts_list`` source definition. For example, to download Git sources from a -private repository and convert them to a tar-ball for installation: +That third instance uses the ``name`` and ``version`` variables defined in the EasyConfigs file. Since EasyBuild +v4.2.2, a single-entry ``sources`` dictionary (see :ref:`_common_easyconfig_param_sources_alt`) may be included in an +``exts_list`` entry. For example, to download Git sources from a private repository and convert them to a tar-ball for +installation: .. code:: python exts_defaultclass = 'PythonPackage' exts_list = [ ('pyCAP', '0.1', { - 'source_tmpl': '%(name)s-%(version)s.tar.gz', - 'git_config': { - 'url': 'ssh://nero.stanford.edu/data/git/Analysis', - 'repo_name': 'pyCAP', - 'tag': '%(version)s', + 'sources': { + 'filename': '%(name)s-%(version)s.tar.gz', + 'git_config': { + 'url': 'ssh://nero.stanford.edu/data/git/Analysis', + 'repo_name': 'pyCAP', + 'tag': '%(version)s', + } } }), ] From 03b3ee9b239cd15637d5c2428c3f250852facf02 Mon Sep 17 00:00:00 2001 From: Michael Kelsey Date: Tue, 9 Jun 2020 13:00:36 -0700 Subject: [PATCH 13/16] Fix misspelling of 'preferred'. --- docs/Writing_easyconfig_files.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/Writing_easyconfig_files.rst b/docs/Writing_easyconfig_files.rst index e5f1064a..ce9ba488 100644 --- a/docs/Writing_easyconfig_files.rst +++ b/docs/Writing_easyconfig_files.rst @@ -542,7 +542,7 @@ Among those options, the following exceptions and special cases should be noted: * ``%(name)s`` and ``%(version)s`` come from the ``exts_list`` entry (above) * **sources**: Dictionary specifying details of where to download the extension * equivalent to a single entry from the EasyConfig ``sources`` list - * preffered to use of ``source_tmpl`` + * preferred to use of ``source_tmpl`` * **start_dir**: If not set, will be derived; the EasyConfigs value will not be used .. code:: python From 25ff2e29239b00276addb24406d20ba249a6b09d Mon Sep 17 00:00:00 2001 From: Michael Kelsey Date: Tue, 9 Jun 2020 13:12:41 -0700 Subject: [PATCH 14/16] Per @ocaisa, add 'namd' to Codespell 'ignore-words-list'. --- .github/workflows/check_with_codespell.sh | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/.github/workflows/check_with_codespell.sh b/.github/workflows/check_with_codespell.sh index 4d8ed081..c1fa4b05 100644 --- a/.github/workflows/check_with_codespell.sh +++ b/.github/workflows/check_with_codespell.sh @@ -1 +1 @@ -codespell --skip='.git,version-specific,scripts' --ignore-words-list=atleast,ninjs,simpy,proovread --quiet-level=2; retVal=$?; if [ $retVal -eq 1 ]; then echo 'Got one expected warning, so success!'; else $(exit $retVal); fi; +codespell --skip='.git,version-specific,scripts' --ignore-words-list=atleast,ninjs,simpy,proovread,namd --quiet-level=2; retVal=$?; if [ $retVal -eq 1 ]; then echo 'Got one expected warning, so success!'; else $(exit $retVal); fi; From ba320e2b9298a0c06fcdea87e80c3eb5184e0608 Mon Sep 17 00:00:00 2001 From: Michael Kelsey Date: Tue, 9 Jun 2020 14:46:04 -0700 Subject: [PATCH 15/16] Add 'namd' to igore-words-list for Travis CI as well, per @ocaisa. --- .travis.yml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/.travis.yml b/.travis.yml index a330ae7b..eac21d13 100644 --- a/.travis.yml +++ b/.travis.yml @@ -13,7 +13,7 @@ script: # still added to the exit code (which is a just a count of spelling # suggestions). Since we know we have one of these, we allow for that. # see https://github.com/easybuilders/easybuild/pull/485 for details - - codespell --skip=".git,version-specific,scripts" --ignore-words-list=atleast,ninjs,simpy,proovread --quiet-level=2; retVal=$?; if [ $retVal -eq 1 ]; then echo "Got one expected warning, so success!"; else $(exit $retVal); fi; + - codespell --skip=".git,version-specific,scripts" --ignore-words-list=atleast,ninjs,simpy,proovread,namd --quiet-level=2; retVal=$?; if [ $retVal -eq 1 ]; then echo "Got one expected warning, so success!"; else $(exit $retVal); fi; - flake8 . --count --select=E9,F63,F7,F82 --show-source --statistics - READTHEDOCS=1 sphinx-build docs build # test installation too (using options that don't require to have a modules tool installed) From fb06f793e90bcf81ae369e7ddab9a1288fb4e060 Mon Sep 17 00:00:00 2001 From: Michael Kelsey Date: Mon, 22 Jun 2020 10:45:26 -0700 Subject: [PATCH 16/16] Improve explanation of using self.cfg['key'] vs. self.key; replace EasyConfig with easyconfig in new documentation sections. --- docs/Implementing-easyblocks.rst | 23 ++++++++++++++--------- docs/Writing_easyconfig_files.rst | 12 ++++++------ 2 files changed, 20 insertions(+), 15 deletions(-) diff --git a/docs/Implementing-easyblocks.rst b/docs/Implementing-easyblocks.rst index 37f30ee4..692a2496 100644 --- a/docs/Implementing-easyblocks.rst +++ b/docs/Implementing-easyblocks.rst @@ -260,8 +260,7 @@ Default easyconfig parameters ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ All of the parameters which are "set" in an easyconfig file (see :ref:`vsd_avail_easyconfig_params`) -become key-value pairs in the ``self.cfg`` dictionary. For instance, where the easyconfig file might -specify +become key-value pairs in the ``self.cfg`` dictionary. For instance, if the easyconfig file specifies .. code:: python @@ -269,23 +268,29 @@ specify version = '2.5.3' versionsuffix = '-Python-3.7.4' -these are accessible within an easyblock via +then these three parameters are accessible within an easyblock via .. code:: python longform = ''.join(self.cfg['name'],'/',self.cfg['version'],self.cfg['versionsuffix']) -For these "universal" arguments, this mapping is done for you in the ``EasyBlock`` base class, so -they are also available as local Python variables (``self.name``, ``self.version``, etc.): +You can use this notation successfully in your easyblock. For all of the variables listed +:ref:`vsd_avail_easyconfig_params`, the ``EasyBlock`` base class copies those dictionary entries into local Python +variables: + +* ``self.name = self.cfg['name']`` +* ``self.version = self.cfg['version']`` +* ``self.versionsuffix = self.cfg['versionsuffix']`` + +and so on. So in your easyblock code, you may replace the above expression +with .. code:: python longform = ''.join(self.name,'/',self.version,self.versionsuffix) -This is true for the variables listed in :ref:`vsd_avail_easyconfig_params`. Any additional -:ref:`custom parameters ` which you define for your own -easyblock, will not be automatically mapped. You will need to use ``self.cfg`` to access them in -your code. +Any additional :ref:`custom parameters ` which you define for your own +easyblock, will not be automatically mapped. You will need to use ``self.cfg`` to access them in your code. .. _implementing_easyblocks_custom_parameters: diff --git a/docs/Writing_easyconfig_files.rst b/docs/Writing_easyconfig_files.rst index ce9ba488..c7f85f9b 100644 --- a/docs/Writing_easyconfig_files.rst +++ b/docs/Writing_easyconfig_files.rst @@ -326,7 +326,7 @@ For example, to inject checksums in *every* easyconfig file required to build HP ... .. note:: We are clearing ``$MODULEPATH`` and specifying a custom (empty) location to ``--installpath`` to - avoid that EasyBuild skips any easyconfigs because a corresponding module is already available. + avoid that EasyBuild skips any easyconfig because a corresponding module is already available. .. _inject_checksums_type: @@ -533,7 +533,7 @@ version number, and a dictionary of configuration options for the entry: ('name', 'version', { 'option':'value', 'option':'value' }) ] -The latter may contain essentially any of the full EasyConfig parameters, including ``buildopts``, ``installopts``, etc. +The latter may contain essentially any of the full easyconfig parameters, including ``buildopts``, ``installopts``, etc. Among those options, the following exceptions and special cases should be noted: * **nosource**: If set ``True``, no download will be done @@ -541,9 +541,9 @@ Among those options, the following exceptions and special cases should be noted: * default is ``'%(name)s-%(version)s.tar.gz'`` * ``%(name)s`` and ``%(version)s`` come from the ``exts_list`` entry (above) * **sources**: Dictionary specifying details of where to download the extension - * equivalent to a single entry from the EasyConfig ``sources`` list + * equivalent to a single entry from the easyconfig ``sources`` list * preferred to use of ``source_tmpl`` -* **start_dir**: If not set, will be derived; the EasyConfigs value will not be used +* **start_dir**: If not set, will be derived; the easyconfig value will not be used .. code:: python @@ -566,7 +566,7 @@ Among those options, the following exceptions and special cases should be noted: }), ] -That third instance uses the ``name`` and ``version`` variables defined in the EasyConfigs file. Since EasyBuild +That third instance uses the ``name`` and ``version`` variables defined in the easyconfig file. Since EasyBuild v4.2.2, a single-entry ``sources`` dictionary (see :ref:`_common_easyconfig_param_sources_alt`) may be included in an ``exts_list`` entry. For example, to download Git sources from a private repository and convert them to a tar-ball for installation: @@ -588,7 +588,7 @@ installation: ] Here, the template strings ``%(name)s`` and ``%(version)s`` will be substituted from the -``exts_list`` entry elements ("pyCAP" and "0.1", respectively), not from the EasyConfigs values. +``exts_list`` entry elements ("pyCAP" and "0.1", respectively), not from the easyconfig values. .. _configure_build_install_command_options: