Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

reduce the number of command line options for 'cmake' command in CMakeMake generic easyblock #2514

Merged
merged 5 commits into from
Jul 27, 2022

Conversation

Flamefire
Copy link
Contributor

Especially to make logs more readable use the following:

  • CMake uses the compiler env vars since basically forever, so only set them if we need absolute paths
  • Since at least 2.8.0 the CMAKE_*_FLAGS are also taken from the environment
  • If the user specifies configopts we can omit our argument for the same option

Note that CMake reads the environment only during first configure. Hence if CMakeCache.txt exists the args still need to be passed, but that should be very rare as we usually build in a clean dir

@Flamefire Flamefire force-pushed the cmake_flags branch 4 times, most recently from bb37039 to f0047ce Compare July 16, 2021 11:15
@Flamefire
Copy link
Contributor Author

Test report by @Flamefire

Overview of tested easyconfigs (in order)

  • SUCCESS OpenCV-3.3.0-foss-2017b-Python-3.6.3.eb

Build succeeded for 1 out of 1 (1 easyconfigs in total)
taurusi6607.taurus.hrsk.tu-dresden.de - Linux RHEL 7.9, x86_64, Intel(R) Xeon(R) CPU E5-2680 v3 @ 2.50GHz (haswell), Python 2.7.5
See https://gist.github.com/3eb1ad7e8adddac80ba5b62c49fe0653 for a full test report.

@akesandgren
Copy link
Contributor

@Flamefire conflict resolution needed

@boegel boegel modified the milestones: 4.5.1, release after 4.5.1 Dec 7, 2021
@boegel boegel modified the milestones: 4.5.2, release after 4.5.2 Jan 14, 2022
@boegel boegel modified the milestones: 4.5.3, release after 4.5.3 Feb 9, 2022
@smoors
Copy link
Contributor

smoors commented Apr 1, 2022

@Flamefire ping about the merge conflicts so we can submit test reports?

I'd really like to get this merged :)

@Flamefire
Copy link
Contributor Author

@smoors Rebased, but with no testing as I'm currently OoO, so check if everything looks good. Haven't found any differences in the new diff (before and after force-push) so it should be good.

Use `' '.join` to add a space only if both config option strings are non-empty
@Flamefire
Copy link
Contributor Author

Test report by @Flamefire

Overview of tested easyconfigs (in order)

  • SUCCESS hwloc-1.11.7-GCCcore-6.4.0.eb
  • SUCCESS OpenMPI-2.1.1-GCC-6.4.0-2.28.eb
  • SUCCESS JUnit-4.12-Java-1.8.eb
  • SUCCESS gompi-2017b.eb
  • SUCCESS ant-1.10.1-Java-1.8.eb
  • SUCCESS FFTW-3.3.6-gompi-2017b.eb
  • SUCCESS ScaLAPACK-2.0.2-gompi-2017b-OpenBLAS-0.2.20.eb
  • SUCCESS foss-2017b.eb
  • SUCCESS LibTIFF-4.0.9-foss-2017b.eb
  • SUCCESS x264-20171217-foss-2017b.eb
  • SUCCESS Python-3.6.3-foss-2017b.eb
  • SUCCESS util-linux-2.31-GCCcore-6.4.0.eb
  • SUCCESS GLib-2.53.5-GCCcore-6.4.0.eb
  • SUCCESS freetype-2.8-GCCcore-6.4.0.eb
  • SUCCESS expat-2.2.4-GCCcore-6.4.0.eb
  • SUCCESS Perl-5.26.0-GCCcore-6.4.0.eb
  • SUCCESS fontconfig-2.12.4-GCCcore-6.4.0.eb
  • SUCCESS XML-Parser-2.44_01-GCCcore-6.4.0-Perl-5.26.0.eb
  • SUCCESS intltool-0.51.0-GCCcore-6.4.0-Perl-5.26.0.eb
  • SUCCESS X11-20171023-GCCcore-6.4.0.eb
  • SUCCESS FFmpeg-3.4.1-foss-2017b.eb
  • SUCCESS cairo-1.14.10-GCCcore-6.4.0.eb
  • SUCCESS Python-2.7.14-foss-2017b.eb
  • SUCCESS GObject-Introspection-1.53.5-foss-2017b-Python-2.7.14.eb
  • SUCCESS ATK-2.27.1-foss-2017b.eb
  • SUCCESS Gdk-Pixbuf-2.36.11-foss-2017b.eb
  • SUCCESS HarfBuzz-1.7.1-foss-2017b.eb
  • SUCCESS Pango-1.41.0-foss-2017b.eb
  • SUCCESS GTK+-2.24.32-foss-2017b.eb
  • SUCCESS OpenCV-3.3.0-foss-2017b-Python-3.6.3.eb

Build succeeded for 30 out of 30 (1 easyconfigs in total)
taurusa14 - Linux CentOS Linux 7.7.1908, x86_64, Intel(R) Xeon(R) CPU E5-2603 v4 @ 1.70GHz (broadwell), 3 x NVIDIA GeForce GTX 1080 Ti, 460.32.03, Python 2.7.5
See https://gist.github.com/0b08049e193e7daa2b5f39dd8f3f2294 for a full test report.

@branfosj
Copy link
Member

Test report by @branfosj

Overview of tested easyconfigs (in order)

  • SUCCESS BamTools-2.5.1-GCC-8.3.0.eb
  • SUCCESS BamTools-2.5.2-GCC-11.2.0.eb
  • SUCCESS Eigen-3.4.0-GCCcore-11.2.0.eb
  • SUCCESS Eigen-3.3.7-GCCcore-9.3.0.eb
  • SUCCESS LLVM-8.0.1-GCCcore-8.3.0.eb
  • SUCCESS netCDF-4.9.0-gompi-2022a.eb
  • SUCCESS MPFR-4.1.0-GCCcore-10.3.0.eb
  • SUCCESS make-4.3-GCCcore-10.2.0.eb
  • SUCCESS Boost-1.74.0-GCC-10.2.0.eb
  • SUCCESS Mako-1.1.4-GCCcore-10.3.0.eb
  • SUCCESS libdrm-2.4.106-GCCcore-10.3.0.eb
  • SUCCESS libpng-1.6.37-GCCcore-10.2.0.eb
  • SUCCESS libglvnd-1.3.3-GCCcore-10.3.0.eb
  • SUCCESS Python-2.7.18-GCCcore-10.3.0-bare.eb
  • SUCCESS Bison-3.0.5.eb
  • SUCCESS binutils-2.31.1.eb
  • SUCCESS GCCcore-8.2.0.eb
  • SUCCESS zlib-1.2.11-GCCcore-8.2.0.eb
  • SUCCESS M4-1.4.18-GCCcore-8.2.0.eb
  • SUCCESS help2man-1.47.7-GCCcore-8.2.0.eb
  • SUCCESS Bison-3.0.5-GCCcore-8.2.0.eb
  • SUCCESS flex-2.6.4-GCCcore-8.2.0.eb
  • SUCCESS binutils-2.31.1-GCCcore-8.2.0.eb
  • SUCCESS ncurses-6.1-GCCcore-8.2.0.eb
  • SUCCESS bzip2-1.0.6-GCCcore-8.2.0.eb
  • SUCCESS libtool-2.4.6-GCCcore-8.2.0.eb
  • SUCCESS pkg-config-0.29.2-GCCcore-8.2.0.eb
  • SUCCESS cURL-7.63.0-GCCcore-8.2.0.eb
  • SUCCESS XZ-5.2.4-GCCcore-8.2.0.eb
  • SUCCESS libreadline-8.0-GCCcore-8.2.0.eb
  • SUCCESS expat-2.2.6-GCCcore-8.2.0.eb
  • SUCCESS GCC-8.2.0-2.31.1.eb
  • SUCCESS CMake-3.13.3-GCCcore-8.2.0.eb
  • SUCCESS PnetCDF-1.12.2-gompi-2020b.eb
  • SUCCESS libunwind-1.4.0-GCCcore-10.3.0.eb
  • SUCCESS DB-18.1.32-GCCcore-8.2.0.eb
  • SUCCESS re2c-2.1.1-GCCcore-10.3.0.eb
  • SUCCESS Brotli-1.0.9-GCCcore-10.2.0.eb
  • SUCCESS libiconv-1.16-GCCcore-10.2.0.eb
  • SUCCESS Mesa-21.1.1-GCCcore-10.3.0.eb
  • SUCCESS libGLU-9.0.1-GCCcore-10.3.0.eb
  • SUCCESS Perl-5.28.1-GCCcore-8.2.0.eb
  • SUCCESS freetype-2.10.3-GCCcore-10.2.0.eb
  • SUCCESS Doxygen-1.8.20-GCCcore-10.2.0.eb
  • SUCCESS Autoconf-2.69-GCCcore-8.2.0.eb
  • SUCCESS Automake-1.16.1-GCCcore-8.2.0.eb
  • SUCCESS netCDF-4.7.4-gompi-2020b.eb
  • SUCCESS netCDF-Fortran-4.5.3-gompi-2020b.eb
  • SUCCESS Autotools-20180311-GCCcore-8.2.0.eb
  • SUCCESS GMP-6.1.2-GCCcore-8.2.0.eb
  • SUCCESS libffi-3.2.1-GCCcore-8.2.0.eb
  • SUCCESS OpenBLAS-0.3.5-GCC-8.2.0-2.31.1.eb
  • SUCCESS double-conversion-3.1.5-GCCcore-10.3.0.eb
  • SUCCESS Tcl-8.6.9-GCCcore-8.2.0.eb
  • SUCCESS SQLite-3.27.2-GCCcore-8.2.0.eb
  • SUCCESS Python-2.7.15-GCCcore-8.2.0.eb
  • SUCCESS intltool-0.51.0-GCCcore-10.2.0.eb
  • SUCCESS numactl-2.0.12-GCCcore-8.2.0.eb
  • SUCCESS PCRE2-10.36-GCCcore-10.3.0.eb
  • SUCCESS Meson-0.55.3-GCCcore-10.2.0.eb
  • SUCCESS libxml2-2.9.8-GCCcore-8.2.0.eb
  • SUCCESS NASM-2.15.05-GCCcore-10.2.0.eb
  • SUCCESS GLib-2.68.2-GCCcore-10.3.0.eb
  • SUCCESS libjpeg-turbo-2.0.5-GCCcore-10.2.0.eb
  • SUCCESS DBus-1.13.18-GCCcore-10.3.0.eb
  • SUCCESS jbigkit-2.1-GCCcore-10.2.0.eb
  • SUCCESS snappy-1.1.8-GCCcore-10.3.0.eb
  • SUCCESS gperf-3.1-GCCcore-10.2.0.eb
  • SUCCESS xorg-macros-1.19.2-GCCcore-8.2.0.eb
  • SUCCESS libpciaccess-0.14-GCCcore-8.2.0.eb
  • SUCCESS hwloc-1.11.11-GCCcore-8.2.0.eb
  • SUCCESS OpenMPI-3.1.3-GCC-8.2.0-2.31.1.eb
  • SUCCESS gompi-2019a.eb
  • SUCCESS FFTW-3.3.8-gompi-2019a.eb
  • SUCCESS ScaLAPACK-2.0.2-gompi-2019a-OpenBLAS-0.3.5.eb
  • SUCCESS Boost-1.70.0-gompi-2019a.eb
  • SUCCESS foss-2019a.eb
  • SUCCESS Libint-2.5.0-gompi-2019a.eb
  • SUCCESS arpack-ng-3.7.0-foss-2019a.eb
  • FAIL (build issue) Armadillo-9.700.2-foss-2019a.eb (partial log available at https://gist.github.com/afa3527db67ef09d392759b66273baff)
  • SUCCESS JasPer-2.0.28-GCCcore-10.3.0.eb
  • SUCCESS util-linux-2.36-GCCcore-10.2.0.eb
  • SUCCESS gzip-1.10-GCCcore-10.2.0.eb
  • SUCCESS Qt5-5.15.2-GCCcore-10.3.0.eb
  • SUCCESS fontconfig-2.13.92-GCCcore-10.2.0.eb
  • SUCCESS CGAL-4.14.3-gompi-2021a.eb
  • SUCCESS X11-20201008-GCCcore-10.2.0.eb
  • SUCCESS Tk-8.6.10-GCCcore-10.2.0.eb
  • SUCCESS Tkinter-3.8.6-GCCcore-10.2.0.eb
  • SUCCESS lz4-1.9.2-GCCcore-10.2.0.eb
  • SUCCESS zstd-1.4.5-GCCcore-10.2.0.eb
  • SUCCESS LibTIFF-4.1.0-GCCcore-10.2.0.eb
  • SUCCESS Pillow-8.0.1-GCCcore-10.2.0.eb
  • SUCCESS matplotlib-3.3.3-foss-2020b.eb
  • FAIL (build issue) Amber-20.11-foss-2020b-AmberTools-21.3.eb (partial log available at https://gist.github.com/2c57d6a48aee7f9ea3cc133df17635c3)

Build succeeded for 93 out of 95 (10 easyconfigs in total)
bear-pg0211u03a.bear.cluster - Linux RHEL 8.6, x86_64, Intel(R) Xeon(R) Gold 6248 CPU @ 2.50GHz (cascadelake), Python 3.6.8
See https://gist.github.com/ad4e20d393ff6dd9b8b5f37690c4be7c for a full test report.

Copy link
Member

@branfosj branfosj left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

lgtm

The two failures above are download failures.

@branfosj
Copy link
Member

Test report by @branfosj

Overview of tested easyconfigs (in order)

  • SUCCESS arpack-ng-3.8.0-foss-2021a.eb
  • SUCCESS Armadillo-10.7.5-foss-2021a.eb

Build succeeded for 2 out of 2 (1 easyconfigs in total)
bear-pg0211u03a.bear.cluster - Linux RHEL 8.6, x86_64, Intel(R) Xeon(R) Gold 6248 CPU @ 2.50GHz (cascadelake), Python 3.6.8
See https://gist.github.com/b7082f436e616d69905769ad4c4d7ca2 for a full test report.

@branfosj
Copy link
Member

Going in, thanks @Flamefire!

@branfosj branfosj merged commit dc8d14d into easybuilders:develop Jul 27, 2022
@Flamefire Flamefire deleted the cmake_flags branch July 28, 2022 07:13
@boegel boegel changed the title Reduce the number of command line args for CMake configure reduce the number of command line args for CMake configure Aug 3, 2022
@boegel boegel changed the title reduce the number of command line args for CMake configure reduce the number of command line options for 'cmake' command in CMakeMake generic easyblock Aug 3, 2022
@boegel
Copy link
Member

boegel commented Aug 3, 2022

Note that CMake reads the environment only during first configure. Hence if CMakeCache.txt exists the args still need to be passed, but that should be very rare as we usually build in a clean dir

@Flamefire Shouldn't we check for an existing CMakeCache.txt to avoid this?
Maybe print a warning, or even make it a hard failure?
Or just revert to explicitly speciyfing those cmake options again to ensure that the correct settings are used?

@Flamefire
Copy link
Contributor Author

@Flamefire Shouldn't we check for an existing CMakeCache.txt to avoid this? Maybe print a warning, or even make it a hard failure? Or just revert to explicitly speciyfing those cmake options again to ensure that the correct settings are used?

That is already done: https://github.com/easybuilders/easybuild-easyblocks/pull/2514/files#diff-73f272366b8e2ec9f6f8730a2e4b13aca5692579faa17b0f8c9b5d4abf2021a4R202

@boegel
Copy link
Member

boegel commented Aug 4, 2022

@Flamefire Shouldn't we check for an existing CMakeCache.txt to avoid this? Maybe print a warning, or even make it a hard failure? Or just revert to explicitly speciyfing those cmake options again to ensure that the correct settings are used?

That is already done: https://github.com/easybuilders/easybuild-easyblocks/pull/2514/files#diff-73f272366b8e2ec9f6f8730a2e4b13aca5692579faa17b0f8c9b5d4abf2021a4R202

Sorry, I overlooked that...

setvar('FC', fc)

# Flags are read from environment variables already since at least CMake 2.8.0
if LooseVersion(get_software_version('CMake')) < LooseVersion('2.8.0') or cache_exists:
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We're making the assumption here that CMake is indeed listed as a (build) dependency, which was not the case before.

Prior to these changes, the configure step was failing with "cmake: command not found" (unless cmake was also available in the OS), but now we get a nasty crash:

ERROR: Traceback (most recent call last):
  File "/data/gent/400/vsc40023/easybuild_easy_installed/lib/python3.6/site-packages/easybuild_framework-4.6.1.dev0-py3.6.egg/easybuild/main.py", line 128, in build_and_install_software
    (ec_res['success'], app_log, err) = build_and_install_one(ec, init_env)
  File "/data/gent/400/vsc40023/easybuild_easy_installed/lib/python3.6/site-packages/easybuild_framework-4.6.1.dev0-py3.6.egg/easybuild/framework/easyblock.py", line 4077, in build_and_install_one
    result = app.run_all_steps(run_test_cases=run_test_cases)
  File "/data/gent/400/vsc40023/easybuild_easy_installed/lib/python3.6/site-packages/easybuild_framework-4.6.1.dev0-py3.6.egg/easybuild/framework/easyblock.py", line 3960, in run_all_steps
    self.run_step(step_name, step_methods)
  File "/data/gent/400/vsc40023/easybuild_easy_installed/lib/python3.6/site-packages/easybuild_framework-4.6.1.dev0-py3.6.egg/easybuild/framework/easyblock.py", line 3795, in run_step
    step_method(self)()
  File "/data/gent/400/vsc40023/easybuild_easy_installed/lib/python3.6/site-packages/easybuild_easyblocks-4.6.1.dev0-py3.6.egg/easybuild/easyblocks/generic/cmakemake.py", line 221, in configure_step
    if LooseVersion(get_software_version('CMake')) < LooseVersion('2.8.0') or cache_exists:
  File "/usr/lib64/python3.6/distutils/version.py", line 52, in __lt__
    c = self._cmp(other)
  File "/usr/lib64/python3.6/distutils/version.py", line 335, in _cmp
    if self.version == other.version:
AttributeError: 'LooseVersion' object has no attribute 'version'

We're also not taking into account that CMake may be filtered out as a dependency via --filter-deps (some people may prefer using the CMake that is provided via the OS), so we should have a fallback to determining the CMake version via cmake --version if get_software_version returns None.

@Flamefire Are you up for looking into this? If not, we should open an issue to avoid forgetting about this (we should handle this prior to the next EasyBuild release that includes the changes in this PR)

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I looked into this myself, see #2772

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

6 participants