Bump conan dependencies (which will allow MSVC 2022 and C++20) and clang 13.1+ fixes by jmarrec · Pull Request #4596 · NatLabRockies/OpenStudio

jmarrec · 2022-05-24T00:21:30Z

Pull request overview

Fix Update conan packages on our NREL conan Artifactory remote (for MSVC 2022 and mac M1 in particular) #4511 (once we've test it works for all platform, we'll vendor everything to NREL's conan remote)
Bump all conan dependencies
Companion PR: Bumps deps conan-openstudio-ruby#39

Boost msut be bumped to 1.78+ for C++20

Pull Request Author

Model API Changes / Additions
Any new or modified fields have been implemented in the EnergyPlus ForwardTranslator (and ReverseTranslator as appropriate)
Model API methods are tested (in src/model/test)
EnergyPlus ForwardTranslator Tests (in src/energyplus/Test)
If a new object or method, added a test in NREL/OpenStudio-resources: Add Link
If needed, added VersionTranslation rules for the objects (src/osversion/VersionTranslator.cpp)
Verified that C# bindings built fine on Windows, partial classes used as needed, etc.
All new and existing tests passes
If methods have been deprecated, update rest of code to use the new methods

Labels:

If change to an IDD file, add the label IDDChange
If breaking existing API, add the label APIChange
If deemed ready, add label Pull Request - Ready for CI so that CI builds your PR

Review Checklist

This will not be exhaustively relevant to every PR.

Perform a Code Review on GitHub
Code Style, strip trailing whitespace, etc.
All related changes have been implemented: model changes, model tests, FT changes, FT tests, VersionTranslation, OS App
Labeling is ok
If defect, verify by running develop branch and reproducing defect, then running PR and reproducing fix
If feature, test running new feature, try creative ways to break it
CI status: all green or justified

…plement workaround

Similar to the C++ 20 branch commit: 1a6c628

… in python 3.12

… am applying it to conan-openstudio-ruby at the same time)

… called with a boost::optional ``` 1596 | BOOST_STATIC_ASSERT_MSG(sizeof(CharType) == 0, "If you want to output boost::optional, include header <boost/optional/optional_io.hpp>"); ```

…reated as error)

…:filesystem (which is worse), and doesn't treat a directory with a trailing separator the same way cf https://www.boost.org/doc/libs/1_78_0/libs/filesystem/doc/v4.html

jmarrec · 2022-05-24T00:23:08Z

.github/workflows/dependabot_conan.yml

+name: "Conan Dependabot"
+on:
+#  schedule:
+#    # Every week on Thursdays
+#    - cron: '23 10 * * 4'
+  push:
+    branches: [ bump_deps, dependabot_conan ]
+
+jobs:


I added a dependabot for conan...

jmarrec · 2022-05-24T00:27:43Z

ConanInstall.cmake

+    set(CONAN_RUBY
+        "openstudio_ruby/2.7.2@jmarrec/testing#98444b7bc8d391ea1521d7f79d4d4926"
+    ) # TODO: pending https://github.com/NREL/conan-openstudio-ruby/pull/39


Pending NatLabRockies/conan-openstudio-ruby#39

jmarrec · 2022-05-24T00:28:00Z

ConanInstall.cmake

+    "openssl/1.1.1o#213dbdeb846a4b40b4dec36cf2e673d7" # force every package to align on the same as our conan-openstudio-ruby
+    "boost/1.79.0#f664bfe40e2245fa9baf1c742591d582"
+    "pugixml/1.12.1#5a39f82651eba3e7d6197903a3202e21"
+    "jsoncpp/1.9.5#536d080aa154e5853332339bf576747c"
+    "minizip/1.2.12#0b5296887a2558500d0323c6c94c8d02" # This depends on zlib/1.2.11, and basically patches it
+    "zlib/1.2.12#3b9e037ae1c615d045a06c67d88491ae" # Also needed, so we can find zlib.h and co (+ pinning exactly is good)
+    "fmt/8.1.1#b3e969f8561a85087bd0365c09bbf4fb"
+    "sqlite3/3.38.5#68f3bf289cde01176e39355869c39ac0"
+    "cpprestsdk/2.10.18#df2f6ac88e47cadd9c9e8e0971e00d89"
+    "websocketpp/0.8.2#3fd704c4c5388d9c08b11af86f79f616"
+    "geographiclib/1.52#76536a9315a003ef3511919310b2fe37"
+    "swig/4.0.2#9fcccb1e39eed9acd53a4363d8129be5"


bump bump bump

jmarrec · 2022-05-24T00:28:20Z

developer/conan/dependabot_conan.py

+import re
+import shlex
+import subprocess
+from pathlib import Path


the dependabot

jmarrec · 2022-05-24T00:30:05Z

src/energyplus/Test/CentralHeatPumpSystem_GTest.cpp

+  ASSERT_EQ(i_central_hp.getString(CentralHeatPumpSystemFields::CoolingLoopInletNodeName).get(),
+            central_hp.supplyInletModelObject().get().nameString());


Lots of these, when EXPECT_EQ is used with one of two arguments being a boost::optional and not a T

1596 | BOOST_STATIC_ASSERT_MSG(sizeof(CharType) == 0, "If you want to output boost::optional, include header <boost/optional/optional_io.hpp>");

jmarrec · 2022-05-24T00:31:23Z

src/utilities/geometry/Intersection.cpp

+// Boost 1.73.0 has this bit in has_self_intersections.hpp:
+//
+// For backward compatibility
+// template <typename Geometry>
+// inline bool has_self_intersections(Geometry const& geometry, bool throw_on_self_intersection = true) {
+//   typedef typename geometry::point_type<Geometry>::type point_type;
+//   typedef typename geometry::rescale_policy_type<point_type>::type rescale_policy_type;
+
+//   typename strategy::intersection::services::default_strategy<typename cs_tag<Geometry>::type>::type strategy;
+
+//   rescale_policy_type robust_policy = geometry::get_rescale_policy<rescale_policy_type>(geometry, strategy);
+
+//   return has_self_intersections(geometry, strategy, robust_policy, throw_on_self_intersection);
+// }
+
+// For backward compatibility
+template <typename Geometry>
+inline bool has_self_intersections(Geometry const& geometry, bool throw_on_self_intersection = true) {
+  using point_type = typename boost::geometry::point_type<Geometry>::type;
+  using rescale_policy_type = typename boost::geometry::rescale_policy_type<point_type>::type;
+
+  typename boost::geometry::strategies::relate::services::default_strategy<Geometry, Geometry>::type strategy;
+
+  // typename boost::geometry::strategy::intersection::services::default_strategy<typename boost::geometry::cs_tag<Geometry>::type>::type strategy;
+
+  auto robust_policy = boost::geometry::get_rescale_policy<rescale_policy_type>(geometry, strategy);
+
+  return boost::geometry::detail::overlay::has_self_intersections(geometry, strategy, robust_policy, throw_on_self_intersection);
+}
+


An actual change. has_self_intersections overload we're using was finally removed in 1.76. implement workaround

jmarrec · 2022-05-24T00:31:48Z

src/utilities/core/test/Zip_Benchmark.cpp


  openstudio::UnzipFile uf(p);
-  uf.setChunksize(state.range(0));
+  uf.setChunksize(static_cast<unsigned long>(state.range(0)));


MSVC 2022 was complaining here

jmarrec · 2022-05-24T00:32:08Z

src/utilities/data/Variant.hpp

 #include "../core/Logger.hpp"
-#include "../core/Optional.hpp"

+#include <optional>


some includes here and there

jmarrec · 2022-05-24T00:32:42Z

src/utilities/filetypes/WorkflowJSON.cpp

    Json::Value defaultValue(".");
    Json::Value root = m_value.get("root", defaultValue);
-    return toPath(root.asString());
+    return toPath(root.asString()).remove_trailing_separator();


Boost filesystem in 1.78 is now V4, and it follows the C++17 STL std::filesystem (which is worse), and doesn't treat a directory with a trailing separator the same way

cf https://www.boost.org/doc/libs/1_78_0/libs/filesystem/doc/v4.html

ref c0248cf

tijcolem · 2022-05-24T22:42:40Z

okay, NatLabRockies/conan-openstudio-ruby#39 is now merged. Do we need to vendor any of deps up to NREL remote before merging this in? Since I don't have an m1 yet I can't build and push conan packages for m1.

…merged

… remote after bumping (I keep reinventing the wheel on that, so committing a clean notebook for future use)

jmarrec · 2022-06-14T00:43:55Z

Full build running at https://ci.openstudio.net/blue/organizations/jenkins/openstudio-develop-nightly/detail/openstudio-develop-nightly/1534/pipeline/

Platform	Build	Tests
macOS	OK	OK
Ubuntu 18.04	OK	OK
Ubuntu 20.04	OK	OK
Windows x86_64	OK	TDB
Windows x86 (nuget)	FAIL	TDB

…l be removed) and force a x86_64 arch **BUILD** profile even when targeting x86 (for the win32 nuget...)

…te compiler.cppstd (windows)i

jmarrec · 2022-06-14T14:27:05Z

Reruning at https://ci.openstudio.net/blue/organizations/jenkins/openstudio-develop-nightly/detail/openstudio-develop-nightly/1537/pipeline after making significant changes to how we use cmake-conan so we can force a build profile with a settings.arch=x86_64 while the host profile is at settings.arch=x86 so we can build the csharp nuget in win 32bit mode. msys2 is particular (which is a build_requires of a LOT of recipes like m4, bison, etc) completely dropped support for 32bit due to the cygwin upstream dropping it in the first place

cf https://www.msys2.org/news/#2020-05-17-32-bit-msys2-no-longer-actively-supported

2020-05-17 - 32-bit MSYS2 no longer actively supported
32-bit mingw-w64 packages are still supported, this is about the POSIX emulation layer, i.e. the runtime, Bash, MinTTY...
After this date, we don't plan on building updated msys-i686 packages nor releasing i686 installers anymore. This is due to increasingly frustrating difficulties with limited 32-bit address space, high penetration of 64-bit systems and Cygwin (our upstream) starting their way to drop 32-bit support as well.

Platform	Build	Tests
macOS	OK	OK
Ubuntu 18.04	OK	OK
Ubuntu 20.04	OK	OK
Windows x86_64	OK	TDB
Windows x86 (nuget)	FAIL	TDB

Before it was doing `ASSERT_EQ(cooledBeam.supplyAirVolumetricFlowRate(), cloneBeam.supplyAirVolumetricFlowRate());` when really it didn't have a value. I changed that to ASSERT_EQ(lhs.get(), rhs.get()) and logically it started throwing because it's not initialized

jmarrec · 2022-06-16T10:05:26Z

Rerunning again at https://ci.openstudio.net/blue/organizations/jenkins/openstudio-develop-nightly/detail/openstudio-develop-nightly/1540/pipeline after I manually built all win32 (x86) deps and uploaded them to NREL's remote. I sincerely hope it works this time...

I'm going to be building binary packages with MSVC 2022 and upload them as well in the meantime.

Why was openssl not building?

It seems it's due to the fact the DevTools were already initialized to x64 in the command prompt. OpenSSL starts by calling vcvarsall.bat

openssl/1.1.1o: Calling build()
**********************************************************************
** Visual Studio 2022 Developer Command Prompt v17.1.6
** Copyright (c) 2022 Microsoft Corporation
**********************************************************************
[vcvarsall.bat] Environment initialized for: 'x86'

As a result later you end up with:

        cl  /Zi /Fdossl_static.pdb /MD /Zl /Gs0 /GF /Gy -O2 -Ob2 -MD /W3 /wd4090 /nologo /O2 /I "." /I "include" -D"L_ENDIAN" -D"OPENSSL_PIC" -D"OPENSSL_CPUID_OBJ" -D"OPENSSL_BN_ASM_PART_WORDS" -D"OPENSSL_IA32_SSE2" -D"OPENSSL_BN_ASM_MONT" -D"OPENSSL_BN_ASM_GF2m" -D"SHA1_ASM" -D"SHA256_ASM" -D"SHA512_ASM" -D"RC4_ASM" -D"MD5_ASM" -D"RMD160_ASM" -D"AESNI_ASM" -D"VPAES_ASM" -D"WHIRLPOOL_ASM" -D"GHASH_ASM" -D"ECP_NISTZ256_ASM" -D"POLY1305_ASM" -D"OPENSSLDIR=\"C:\\Users\\julien\\.conan\\bump_deps\\openssl\\1.1.1o\\_\\_\\package\\70711aff723b218b02483e600038031bf91007bc\\res\"" -D"ENGINESDIR=\"C:\\Users\\julien\\.conan\\bump_deps\\openssl\\1.1.1o\\_\\_\\package\\70711aff723b218b02483e600038031bf91007bc\\lib\\engines-1_1\"" -D"OPENSSL_SYS_WIN32" -D"WIN32_LEAN_AND_MEAN" -D"UNICODE" -D"_UNICODE" -D"_CRT_SECURE_NO_DEPRECATE" -D"_WINSOCK_DEPRECATED_NO_WARNINGS" -D"NDEBUG" -D"NDEBUG"  /Zs /showIncludes "engines\e_padlock.c" 2>&1 > engines\e_padlock.d
        lib /nologo /out:libcrypto.lib @C:\Users\julien\AppData\Local\Temp\nm3CBB.tmp
crypto\aes\aesni-x86.obj : fatal error LNK1112: module machine type 'x86' conflicts with target machine type 'x64'
NMAKE : fatal error U1077: '"C:\Program Files\Microsoft Visual Studio\2022\Community\VC\Tools\MSVC\14.31.31103\bin\HostX64\x64\lib.EXE"' : return code '0x458'
Stop.
NMAKE : fatal error U1077: '"C:\Program Files\Microsoft Visual Studio\2022\Community\VC\Tools\MSVC\14.31.31103\bin\HostX64\x64\nmake.exe"' : return code '0x2'
Stop.
openssl/1.1.1o:
openssl/1.1.1o: ERROR: Package '70711aff723b218b02483e600038031bf91007bc' build failed
openssl/1.1.1o: WARN: Build folder C:\Users\julien\.conan\bump_deps\openssl\1.1.1o\_\_\build\70711aff723b218b02483e600038031bf91007bc
ERROR: openssl/1.1.1o: Error in build() method, line 758
        self._make()
while calling '_make', line 698
        self._run_make()
while calling '_run_make', line 653
        self.run(" ".join(command), win_bash=self._win_bash)
        ConanException: Error 2 while executing nmake

I would get that error by doing:

call vcvarsall.bat x64
cmake -G "Visual Studio 16 2019" -A x64  -DCMAKE_BUILD_TYPE=Release `
  -DBUILD_CSHARP_BINDINGS:BOOL=ON  -DBUILD_NUGET_PACKAGE:BOOL=ON -DBUILD_RUBY_BINDINGS:BOOL=OFF -DBUILD_CLI:BOOL=OFF `
  -DBUILD_RADIANCE:BOOL=OFF -DBUILD_DOCUMENTATION:BOOL=OFF -DBUILD_TESTING:BOOL=OFF `
  -DCPACK_BINARY_DEB:BOOL=OFF -DCPACK_BINARY_IFW:BOOL=ON -DCPACK_BINARY_NSIS:BOOL=OFF -DCPACK_BINARY_RPM:BOOL=OFF -DCPACK_BINARY_STGZ:BOOL=OFF `
  -DCPACK_BINARY_TBZ2:BOOL=OFF -DCPACK_BINARY_TGZ:BOOL=ON -DCPACK_BINARY_TXZ:BOOL=OFF -DCPACK_BINARY_TZ:BOOL=OFF `
  ../

# HERE!!!

cmake --build . --config Release --target OS-32

If between both cmake command I exit the prompt, start a new one without calling vcvarsall.bat, openssl builds.

This problem should not arise when the dependencies are downloaded

…archx86, etc)

jmarrec · 2022-06-16T14:28:00Z

I uploaded all x86 (Release) and MSVC 17 (Release and Debug) packages I built on my machine to the NREL remote, so we should be in business!

BCL.obj : error LNK2038: mismatch detected for 'boost_log_abi': value 'v2s_mt_nt6' doesn't match value 'v2s_mt_nt62' in ruby_OpenStudioUtilitiesCore_wrap.obj [D:\OSN\build\src\cli\openstudio.vcxproj] cf boost: include/boost/winapi/config.hpp

@aae

https://ci.openstudio.net/blue/organizations/jenkins/openstudio-develop-nightly/detail/openstudio-develop-nightly/1544/pipeline/51 Logger.obj : error LNK2019: unresolved external symbol "private: static unsigned int __cdecl boost::log::v2s_mt_nt62::attribute_name::get_id_from_string(char const *)" (?get_id_from_string@attribute_name@v2s_mt_nt62@log@boost@@CAIPBD@Z) referenced in function "private: __thiscall openstudio::LoggerSingleton::LoggerSingleton(void)" (??0LoggerSingleton@openstudio@@aae@XZ) [D:\OSN\build\OS-32\src\lib\openstudiolib.vcxproj] [D:\OSN\build\csharp\OS-32.vcxproj]

…AND_MEAN)

jmarrec · 2022-06-20T14:25:43Z

@tijcolem It finally builds on https://ci.openstudio.net/blue/organizations/jenkins/openstudio-develop-nightly/detail/openstudio-develop-nightly/1550/pipeline/284

tijcolem

Great that things are building!

jmarrec and others added 30 commits April 4, 2022 10:56

Bump boost from 1.73.0 to 1.78.0 for MSVC 2022 support

4302b2a

has_self_intersections overload we're was finally removed in 1.76. im…

6432e60

…plement workaround

Fix a couple of includes

9d13622

Similar to the C++ 20 branch commit: 1a6c628

cmake format ConanInstall.cmake

4babc20

Write a dependabot-like utility to lookup conan package updates

7ac3c75

Try a github actions to run the dependabot

f4331ee

Add the nrel remote

6181a44

Pass the remote, and use the default remote name

da93dcf

Enable revisions

7f88729

replace distutil.version with packaging.version as it will be removed…

d3c14fb

… in python 3.12

Name the branch and commiter for pr

10b05fc

Need to add packaging in requirements.txt

5236b03

use PIN_OPENSSL_VERSION

3cb31c8

Improve dependabot_conan by supporting conanfile etc and use a OOP (I…

69d598e

… am applying it to conan-openstudio-ruby at the same time)

Automated checkin - update conan recipes

c57231e

Bump conan openstudio ruby to NatLabRockies/conan-openstudio-ruby#39

5f65193

Automated checkin - update conan recipes

e339548

Update conan.cmake to 0.18.0

b2b13d2

Fix precommit and update deps again

cd3f45e

conan.cmake sha is wrong on conan-cmake github

4c25161

Fix an include for MSVC

6420ca3

Compiler Driven Development: Fix problems in gtests when EXPECT_EQ is…

a529149

… called with a boost::optional ``` 1596 | BOOST_STATIC_ASSERT_MSG(sizeof(CharType) == 0, "If you want to output boost::optional, include header <boost/optional/optional_io.hpp>"); ```

Fix a build error in MSVC 2022 in Zip benchmark (conversion warning t…

0214fb6

…reated as error)

WIP: more test fixes

a1c6d63

Fix the remaining, it builds

cd1201a

Automated checkin - update conan recipes

1308a7d

Test error

bde5db8

fix a -Wunused-but-set-variable on apple clang 13.1 + dry up code

8f219a4

More fix a -Wunused-but-set-variable on apple clang 13.1.6

2efc0db

Boost filesystem in 1.78 is now V4, and it follows the C++17 STL std:…

c0248cf

…:filesystem (which is worse), and doesn't treat a directory with a trailing separator the same way cf https://www.boost.org/doc/libs/1_78_0/libs/filesystem/doc/v4.html

jmarrec requested a review from tijcolem May 24, 2022 00:21

jmarrec self-assigned this May 24, 2022

jmarrec mentioned this pull request May 24, 2022

Bumps deps NatLabRockies/conan-openstudio-ruby#39

Merged

jmarrec commented May 24, 2022

View reviewed changes

jmarrec and others added 4 commits June 13, 2022 20:59

Merge remote-tracking branch 'origin/develop' into bump_deps

85b77b3

Use nrel/testing now that NatLabRockies/conan-openstudio-ruby#39 was …

541d8dc

…merged

Automated checkin - update conan recipes

c52eb84

Commit a notebook I used to vendor the conan dependencies to the NREL…

e25147a

… remote after bumping (I keep reinventing the wheel on that, so committing a clean notebook for future use)

Update notebook to also skip gcc 5, and shared=True packages [skip ci]

c73a089

jmarrec mentioned this pull request Jun 14, 2022

Enable C++20 #4484

Merged

19 tasks

jmarrec added 3 commits June 14, 2022 15:03

Modernize usage of conan.cmake (conan_cmake_run is deprecated and wil…

23390ea

…l be removed) and force a x86_64 arch **BUILD** profile even when targeting x86 (for the win32 nuget...)

Dry up the code by sharing path being multi/non mult generators. Dele…

ebff05e

…te compiler.cppstd (windows)i

Update notebook

cdb0f3b

jmarrec added 2 commits June 16, 2022 15:21

Add script in notebook to deal with windows specific stuff (MSVC 17, …

7931f2a

…archx86, etc)

Move the notebook into the conan subfolder, makes more sense

3a01863

jmarrec added 3 commits June 17, 2022 14:51

Try to define use BOOST_WINAPI_DEFINE_VERSION_MACROS (and WIN32_LEAN_…

5fb7c4b

…AND_MEAN)

jmarrec force-pushed the bump_deps branch from 3ad9bff to 5fb7c4b Compare June 20, 2022 10:12

Merge remote-tracking branch 'origin/develop' into bump_deps

7dab55c

tijcolem approved these changes Jun 21, 2022

View reviewed changes

tijcolem merged commit 37364f9 into develop Jun 21, 2022

tijcolem deleted the bump_deps branch June 21, 2022 14:32

		ASSERT_EQ(i_central_hp.getString(CentralHeatPumpSystemFields::CoolingLoopInletNodeName).get(),
		central_hp.supplyInletModelObject().get().nameString());

Conversation

jmarrec commented May 24, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Pull request overview

Pull Request Author

Review Checklist

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

jmarrec May 24, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

tijcolem commented May 24, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

jmarrec commented Jun 14, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

jmarrec commented Jun 14, 2022

Uh oh!

jmarrec commented Jun 16, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Why was openssl not building?

Uh oh!

jmarrec commented Jun 16, 2022

Uh oh!

jmarrec commented Jun 20, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

tijcolem left a comment

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

jmarrec commented May 24, 2022 •

edited

Loading

jmarrec May 24, 2022 •

edited

Loading

tijcolem commented May 24, 2022 •

edited

Loading

jmarrec commented Jun 14, 2022 •

edited

Loading

jmarrec commented Jun 16, 2022 •

edited

Loading

jmarrec commented Jun 20, 2022 •

edited

Loading