Fortran support

[albertziegenhagel]

Just recently PGI released the free community edition of their compiler suite for windows [1], which makes a free fortran compiler available on windows that is compatible with VC++.

In this merge request I implemented support to use new triplets which enable additional toolchains for the PGI and the Intel compiler suites if available.

As a demo port file, I added netlib's reference lapack (including reference blas, cblas and lapacke).

One could now use

vcpkg install lapack:x64-windows-pgi

or

vcpkg install lapack:x64-windows-intel

to compile lapack with one of those toolchains.

This merge request should be considered as work in progress. The following questions/issues/comments are still to be resolved:

1. Support for PGI on windows in CMake is not yet complete. One patch [2] has been merged already and another one is currently in the last stage before merging [3]. Those changes should be ready for CMake 3.9. There may be the need for some additional changes in CMake to fix libraries that link object files generated from pgfortran and Visual C++ and we will try to have these changes ready for CMake 3.9 as well.
2. PGI has to be installed by the user first. I don't think it would be possible to make vcpkg download and install PGI automatically. The same is true for the Intel compilers.
3. Currently detecting the PGI installation is hard coded. We should find a way to detect this automatically.
4. Intel does not support VS2017 yet. That is why I added a way to force vcpkg to use VS2015 through triplets.
5. PGI does not support VS2017 officially yet as well. But I think this is only because their Visual Studio plugin is not yet available for VS2017, which is not included in the free community edition anyways. I tested using the VS2017 compilers together with pgfortran and everything seems to work without any problems. Nevertheless the user needs to have VS2015 installed to be able to install PGI.
6. Currently only the fortran compilers are used from PGI/Intel. C and C++ code is still compiled with Visual C++. PGI does not come with a C++ compiler on windows but ships a C compiler. Intel has both, a C and a C++ compiler. One could think about allowing to use the Intel C and C++ compilers to be used as full alternative toolchains.

I would be glad to here your comments on this. And please keep in mind that this is a first prove of concept implementation. If there are any ideas on how to improve this, I would be very willing to incorporate those changes.

[1] http://www.pgroup.com/products/community.htm
[2] https://gitlab.kitware.com/cmake/cmake/merge_requests/802
[3] https://gitlab.kitware.com/cmake/cmake/merge_requests/794

[KindDragon]

Related to #485

[albertziegenhagel]

3. Currently detecting the PGI installation is hard coded. We should find a way to detect this automatically.

I looked through the Internet and the PGI documentation and couldn't find any official way to detect the PGI installation path. The only information I could find in the PGI documentation is that the default installation paths are %SYSTEMDRIVE%\Program Files\PGI and %SYSTEMDRIVE%\Program Files\PGICE for the payed and the Community Edition respectively. I changed the detection algorithm to look into those paths first and introduced an environment variable VCPKG_PGI_ROOT that can be set by the user if he installed PGI to a non-default location.

[KindDragon]

The only information I could find in the PGI documentation is that the default installation paths are %SYSTEMDRIVE%\Program Files\PGI and %SYSTEMDRIVE%\Program Files\PGICE for the payed and the Community Edition respectively.

The installer doesn't allow to change the path. So you can just check %ProgramFiles%\PGI and %ProgramFiles%\PGICE paths

[ras0219-msft]

It would be incredible to get a solid, working fortran compiler 🌈!

Let me post some thoughts in a mostly-unsorted order:

• The license[1] for PGI CE is somewhat problematic due to the 1 year time bomb. I think there is an implication that the compiler (even if you already downloaded it!) will check the datetime of your system and simply stop working after a year. This, of course, plays havoc with the idea of "frozen offline reproducible" builds. 😢
• It's clear from the official license text [2] that PG intends merely downloading the archive to indicate consent; thus we probably shouldn't download it on the user's behalf.
• We could probably do tricks such as msiexec /a to unpack any MSIs without installing them system-wide once they're downloaded (intel or pgi).
• It's a bad omen that VS2017 isn't yet supported directly by either of these compilers. What happens for VSnext? It isn't a total deal-breaker, but it does mean we will need to keep these at arms length to avoid upgrade problems; we'll have to keep the number of packages that depend on them to an absolute minimum. 👻
• Nice job plumbing VCPKG_USE_INTEL_TOOLCHAIN et al; overall looks good! When we get closer to closure on this, we'll do a more thorough code review
• As we add more possible triplet settings, we don't really want to add the entire combinatoric space. Additionally, it seems to me (at first glance) that a fortran compiler is really something that we always want to have a setting for.
  • So, something like a VCPKG_FORTRAN_COMPILER setting that all triplets can set. For all the default triplets, we pick one that works for most users.
  • Users can either alter the existing triplets or make their own that change any of these settings. This does mean we need much better documentation about the triplet files and valid settings.
  • Ports that actually need the fortran compilers will call a vcpkg helper (perhaps vcpkg_fortran_compiler(FORTRAN)) which will do whatever is needed to pull down the fortran compiler on demand. This might be to fail the build with a message instructing the user on how to install it themselves. It would decide the compiler to look for based on the above VCPKG_FORTRAN_COMPILER setting -- if it is blank, always fail the build with "No fortran configured. Please see [docs] for valid fortran settings.".
  • This also has the mixed benefit of moving configuration out of the vcpkg.exe and into CMake scripts. Pros: easier to modify, especially for downstream users. Cons: CMake sometimes feels more difficult to use than C++ 😉.
• Downloading an x86 ninja from Kitware actually has some nice side benefits -- we could use Ninja on x86 machines! Currently we are restricted because the prebuilt official ninja is only available for x64. I've thought of building from source, but we haven't done it.
• The PREFER_NMAKE option is probably better served by simply using GENERATOR "NMake Makefiles". The reason for PREFER_NINJA is to imply that it won't always be used -- just prefer it when it's an option.

[1] http://www.pgroup.com/support/community-faq.htm#diff
[2] http://www.pgroup.com/doc/LICENSE.txt

[albertziegenhagel]

The license[1] for PGI CE is somewhat problematic due to the 1 year time bomb. I think there is an implication that the compiler (even if you already downloaded it!) will check the datetime of your system and simply stop working after a year. This, of course, plays havoc with the idea of "frozen offline reproducible" builds. 😢

This is true. Ideally we should have a 100% free fortran compiler. The only ones that come to my mind are gfortran and maybe sometime in the future flang. I have a second branch [1] that enables fortran support in vcpkg via gfortran and mingw based on the work of @traversaro [2]. The problem that remains here is that MinGW does not yet support the UCRT. I found a fork of MinGW where someone tried to port MinGW to the UCRT [3] but it is not yet finished and it does not seem to be developed actively anymore.

It's a bad omen that VS2017 isn't yet supported directly by either of these compilers. What happens for VSnext? It isn't a total deal-breaker, but it does mean we will need to keep these at arms length to avoid upgrade problems; we'll have to keep the number of packages that depend on them to an absolute minimum. 👻

Intel actually added support for Visual Studio 2017 in its latest releases (Parallel Studio XE 2017 Update 4 and Parallel Studio XE 2018 beta). I think PGI will be a little bit more difficult since they usually release the Community Edition once or twice a year only. I assume VS2017 support to be present in the payed version more early.

As we add more possible triplet settings, we don't really want to add the entire combinatoric space. Additionally, it seems to me (at first glance) that a fortran compiler is really something that we always want to have a setting for. [...]

The VCPKG_FORTRAN_COMPILER option sounds good. I will try to implement something like this as soon as I can find some spare time again.
About pulling the fortran compiler in the portfiles: initially I wanted to do exactly that (similar to how the environment is changed in the MinGW branch mentioned above) but then I decided to implement it analogous to how you currently setup the environment for VS via the vsvarsall.bat since both Intel and PGI have similar batch files (which actually makes it more easy to pull in the environment, since we can just call the environment batch scripts). Nevertheless, I see the benefits of having the code in the CMake scripts.

The PREFER_NMAKE option is probably better served by simply using GENERATOR "NMake Makefiles". The reason for PREFER_NINJA is to imply that it won't always be used -- just prefer it when it's an option.

The NMAKE support may be removed by now. When I started to work on the fortran support there have been some issues with CMake, Ninja and PGI which have been fixed in CMake by now. Nevertheless, it may make sense to keep the support in vcpkg_build_cmake and vcpkg_install_cmake and only remove the parameter in vcpkg_configure_cmake.

[1] https://github.com/albertziegenhagel/vcpkg/tree/add-lapack-mingw-fortran
[2] https://github.com/traversaro/vcpkg/tree/add-lapack
[3] https://github.com/nak5124/MinGW-w64

[albertziegenhagel]

I've implemented support for Fortran in pure CMake via an additional function called vcpkg_enable_fortran that can be called in portfiles that require a Fortran compiler.

This function reads a variable called VCPKG_FORTRAN_COMPILER that can be set in the triplet file by a user. If the variable is not set at all, the user will get an error. Supported values for that variable are Intel and PGI (names are chosen to reflect the supported values in CMAKE_<LANG>_COMPILER_ID [1]).

The function vcpkg_enable_fortran then tries to find the compilervars.bat (Intel) or pgi_env.bat (PGI), figures out which environment variables are modified by those batch scripts, and applies those changes to the current environment.

I will remove the old implementation in the vcpkg C++ code in an other commit.

Some remaining questions to be answered:

• Should I keep the support to set VCPKG_PLATFORM_TOOLSET in the triplets to a specific version so that one can force vcpkg to use an other VC++ toolchain than the newest one?
• Where should we add the documentation about that feature (having [vcpkg] Documentation revamp using MkDocs & readthedocs #1111 in mind)?
• I've added some helper functions in scripts/cmake/vcpkg_enable_fortran.cmake that start with an underscore. Should I move those functions to a different place or is it okay to have them in there?
• Anything else that can be improved?

[1] https://cmake.org/cmake/help/latest/variable/CMAKE_LANG_COMPILER_ID.html

[ChrisTX]

The license[1] for PGI CE is somewhat problematic due to the 1 year time bomb. I think there is an implication that the compiler (even if you already downloaded it!) will check the datetime of your system and simply stop working after a year. This, of course, plays havoc with the idea of "frozen offline reproducible" builds. 😢

It certainly does, but keep in mind that if you want to work with Fortran on Windows there's not that many options: There's Intel, PGI, MinGW, Absoft and NAG. Out of these, all but MinGW are proprietary software and only PGI offers a free edition in the form of the time-bombed PGI CE. If you're looking for VS compatibility you're already down to PGI and Intel. It's unlikely MinGW will support VS in the future, and even if it eventually did, MinGW has never offered reasonably performing binaries - given that Fortran is a language only suitable for computing, that's by no means optimal.

If you're looking for "frozen offline reproducible" builds with Fortran on Windows, the answer is mostly that there's no free option for doing that. As of such, PGI CE is the best "free" offer there realistically is going to be in the predictable future.

Not to mention that when working with scientific or computational software, you will eventually require Fortran binaries, and will probably have either switched off Windows in order to use GFortran, or use Cygwin or MSYS for doing so - or have purchased a license for Intel or PGI.

At the moment, if you want to "bother" with Windows here, you have to compile all dependencies yourself, and given the poor state of some CMake scripts shipped by popular Fortran projects (have a look at ScaLAPACK if you like a demonstration), compiling dependencies is a pain. Any improvement you can make to that situation by adding Fortran support in vcpkg is going to make this less painful. Even if just those with commercial licenses for Intel could install Fortran packages the way it's possible to install other packages with vcpkg is going to make this less of an ordeal for a significant group of users who are using Fortran on Windows right now.

It's a bad omen that VS2017 isn't yet supported directly by either of these compilers. What happens for VSnext? It isn't a total deal-breaker, but it does mean we will need to keep these at arms length to avoid upgrade problems; we'll have to keep the number of packages that depend on them to an absolute minimum. 👻

You can't really expect third party commercial compilers to offer support for a new Visual Studio release on day one. Intel usually takes 1-2 months after a VS release to update their toolset, both C/C++ and Fortran, to support the version - this time around it took them about 1.5 months. Faster than that just isn't going to be possible if they need to adapt to changes - not only concerning their VS plugins, but also the C++ STL and more - and verify it works.

As for PGI, I can't say how long it takes them, but the delay is going to be longer in the CE. However, it would only need VS2015 to be present upon installation and can be mixed with VS2017-built code. It mostly boils down to PGI using link.exe from VS, and I think it's always going to look for the 2015 one here.

We could probably do tricks such as msiexec /a to unpack any MSIs without installing them system-wide once they're downloaded (intel or pgi).

For commercial versions of Intel and PGI that won't work because both require a license file, which is going to be generated upon installation time. For Intel, you can automatize the installation, see this piece of documentation, but getting around an installation won't be possible. Not to mention that Intel registers with the various Visual Studio versions found on the system (it supports VS2012-2017) and an extracted install will not work. Even on Linux that's not supported, but by extracting the files to /opt you can make it mostly work, assuming you have already generated a .lic file.

Lastly, for PGI CE that might sound reasonable, but for the commercial versions the question remains why you would want to not install a product system-wide that you've purchased for a serious amount of money implicitly by vcpkg? I'm really not seeing the usage scenario here, and I would suggest leaving this part to the user, just like installing Visual Studio is the user's responsibility.

PGI CE itself is an InstallShield wizard wrapped around an MSI package that contains a pre-generated license.dat. This might be possible to extract but I'm not sure how that would work or whether PGI CE could even function afterwards.

[albertziegenhagel]

I've cleaned everything up, removed all unnecessary changes in the C++ code, rebased on master and updated the CMake version to 3.9-rc1, which includes all the modifications required for PGI support.

The only remaining part in my opinion is documentation, which is blocked by #1185, where I don't think that I am the right person to write it.

Another thing is that this PR adds a fortran lapack port, which will mean we will have two lapack implementations: lapack and clapack. In my opinion #164 should be fixed before using lapack as dependency in other ports. If you prefer, I can remove the lapack port from this PR until #164 gets fixed.

[ras0219-msft]

I've added a documentation page for triplet options in 20af69a, which I think is the part of #1185 that was blocking you.

I think it is fine to add the lapack port with this PR, but I agree we should avoid it as a dependency. Actually, based on the discussion in #164, perhaps this package should be renamed to something like lapack-reference or fortran-lapack? It seems like we'd really want to use lapack to refer to the abstract ideal of "lapack" (unless I'm totally mistaken about the relationship between mpi/mkl/blas/lapack/etc -- I haven't done very much scientific computing!).

I think all the major blockers are therefore gone, so I'll start reviewing this for merging.

not relevant to this immediate PR, just some thoughts for the future
Looking at the vcpkg_enable_fortran() code I'm really sorry to make you write that! We had similar cmake code to adapt vcvarsall.bat and it was really painful to author. In the future, maybe we can lift the nastiest parts into C++ -- perhaps call back into vcpkg and have it run/parse the bat file, then write out a temporary cmake script containing all the calls to set(ENV{X} ...)? This would still allow the target batch file to be controlled from cmake, but would avoid the nasty semicolon+backslash manipulation.

We could also possibly lift vcpkg_enable_fortran() into the CONTROL file (Requires-Fortran: yes or Build-Depends: fortran-compiler) in the future, which would allow us to short-circuit even before running the main portfile. This is a very similar situation to the CUDA compiler, which we currently attempt to solve with a stub portfile[1]. I think there should be a single solution that works perfectly for both of these.

[1] https://github.com/Microsoft/vcpkg/blob/357db2295892fa49c351bd1eb31cefcfc283fda0/ports/cuda/portfile.cmake

[albertziegenhagel]

I've added some documentation for the VCPKG_FORTRAN_COMPILER triplet setting, rebased on master and renamed the lapack port to reference-lapack.

Looking at the vcpkg_enable_fortran() code I'm really sorry to make you write that! We had similar cmake code to adapt vcvarsall.bat and it was really painful to author. In the future, maybe we can lift the nastiest parts into C++ -- perhaps call back into vcpkg and have it run/parse the bat file, then write out a temporary cmake script containing all the calls to set(ENV{X} ...)? This would still allow the target batch file to be controlled from cmake, but would avoid the nasty semicolon+backslash manipulation.

We could also possibly lift vcpkg_enable_fortran() into the CONTROL file (Requires-Fortran: yes or Build-Depends: fortran-compiler) in the future, which would allow us to short-circuit even before running the main portfile. This is a very similar situation to the CUDA compiler, which we currently attempt to solve with a stub portfile[1]. I think there should be a single solution that works perfectly for both of these.

This sounds reasonable. Besides the problems that may arise while setting-up the environment in the current implementation (handling of \ and \n), the current implementation re-detects the Fortran compiler for each port file that needs it. This could be optimized to detecting it only once for all ports when stating the dependency in the CONTROL file. Additionally the just recently added support for the selection of toolset versions may be nice to have for the Fortran toolset as well (maybe through a variableVCPKG_FORTRAN_PLATFORM_TOOLSET?!).

If one is interested in implementing some of those changes, my old C++ implementation can be found at https://github.com/albertziegenhagel/vcpkg/tree/fortran-support-backup

[MVoz]

надо как то поднять тему, особенно про PETSc

It should be like that raise the topic, especially about PETSc

https://www.gridpack.org/wiki/index.php/Building_on_Windows
https://github.com/INMOST-DEV/INMOST/wiki/0206-Compilation-PETSc-Windows
http://openmdao.readthedocs.io/en/latest/getting-started/mpi_windows.html
https://chaste.cs.ox.ac.uk/trac/wiki/InstallGuides/Windows
http://ftp.mcs.anl.gov/pub/petsc/software_old/ ... petsc-lib-win32_ms-2.1.0.1.tar.gz

[albertziegenhagel]

@ras0219-msft I've resolved the merge conflicts and added some additional fixes. Is there any chance this could be merged, or do you prefer to have the changes implemented that you mentioned in your last comment?

[MVoz]

:x64-windows-static
спасибо за поддержку fortran, но у меня почему то не находит компилятор Intel, а вот PGI (PGI Community Edition 17.4) находит и удачно компилирует

путь к Intel C:\Program Files (x86)\IntelSWTools\compilers_and_libraries_2017.4.210\windows
либо его символическая ссылка

C:\Program Files (x86)\IntelSWTools\compilers_and_libraries_2017
C:\Program Files (x86)\IntelSWTools\compilers_and_libraries

прописывал пути в файле scripts/cmake/vcpkg_enable_fortran.cmake
set(POTENTIAL_PATHS "C:/Program Files (x86)/IntelSWTools/compilers_and_libraries_2017.4.210/windows")
компилятор Intel находится, но тогда vs2017 не распознается

compilervars.bat

:ParseArgs
:: Parse the incoming arguments
if /i "%1"==""        goto Build
if /i "%1"=="ia32"         (set TARGET_ARCH=ia32)         & (set C_TARGET_ARCH=ia32)    & shift & goto ParseArgs
if /i "%1"=="intel64"      (set TARGET_ARCH=intel64)      & (set C_TARGET_ARCH=intel64) & shift & goto ParseArgs
if /i "%1"=="vs2012"       (set TARGET_VS=vs2012)                                       & shift & goto ParseArgs
if /i "%1"=="vs2013"       (set TARGET_VS=vs2013)                                       & shift & goto ParseArgs
if /i "%1"=="vs2013shell"  (set TARGET_VS=vs2013)                                       & shift & goto ParseArgs
if /i "%1"=="vs2015"       (set TARGET_VS=vs2015)                                       & shift & goto ParseArgs
if /i "%1"=="vs2017"       (set TARGET_VS=vs2017)                                       & shift & goto ParseArgs
if /i "%1"=="lp64"         (set LP64_ILP64=lp64)                                        & shift & goto ParseArgs
if /i "%1"=="ilp64"        (set LP64_ILP64=ilp64)                                       & shift & goto ParseArgs
if /i "%1"=="quiet"        (set QUIET=quiet)                                            & shift & goto ParseArgs
if /i "%1"=="-arch"        shift & goto ParseArgs
if /i "%1"=="-platform"    shift & goto ParseArgsPlatform 

:Syntax
echo Syntax:
echo  %SCRIPT_NAME% [-arch] ^<arch^> [vs] [-platform ^<platform^>] 
echo.
echo    ^<arch^> must be is one of the following
echo        ia32         : Set up for IA-32 target
echo        intel64      : Set up for Intel(R) 64 target
echo    If specified, ^<vs^> must be one of the following
echo        vs2012      : Set to use Microsoft Visual Studio* 2012
echo        vs2013      : Set to use Microsoft Visual Studio* 2013
echo        vs2013shell : Set to use Microsoft Visual Studio* Shell 2013
echo        vs2015      : Set to use Microsoft Visual Studio* 2015
echo        vs2017      : Set to use Microsoft Visual Studio* 2017
echo    If ^<vs^> is not specified, the version of Visual Studio* detected at install
echo    time is used.
echo    ^<platform^> must be of the following.  
echo        linux          : Set to Linux* target.      
echo        android        : Set to Android* target.
echo.
exit /B 1

Thanks for the support, fortran but I cannot find the Intel compiler, but the PGI finds and successfully compiles

Intel path ' C:\Program Files (x 86) \IntelSWTools\compilers_and_libraries_2017.4.210\windows '
or a symbolic link

C:\Program Files (x 86) \IntelSWTools\compilers_and_libraries_2017
C:\Program Files (x 86) \IntelSWTools\compilers_and_libraries

stipulate the way in the file ' scripts/cmake/vcpkg_enable_fortran.cmake '
' set ' (POTENTIAL_PATHS ' C:/Program Files (x 86)/IntelSWTools/compilers_and_libraries_2017.4.210/windows ") '
the Intel compiler is, but then vs2017 is not recognized

дошел до PETSc и не смог осилить ошибки, как я понял нужен python2 и только с состава cygwin

собранные сырцы, вдруг кому пригодится ссылка
https://anaconda.org/ESSS/petsc/files

came to PETSc and could not overpower the error, as I understand it you need python2 and only part of cygwin

[MVoz]

@albertziegenhagel 1.8.2 Ninja to support Fortran
https://github.com/Kitware/ninja/releases

PGI Community Edition Version 18.4 (released April 26, 2018)
https://www.pgroup.com/support/downloader.php?file=pgi-community-windows-x64

[MVoz]

@albertziegenhagel
If not difficult, add a port

https://github.com/opencollab/arpack-ng

[cbezault]

Please move any work to be done for Fortran support to #6939.