(#9047) boost: Bump build_requires of 'b2' to '4.7.1' (was '4.5.0')

[accwebs]

This fixes #9047 ('build from source fails with MSVC 2022 due to outdated b2 dep')

---

• I've read the guidelines for contributing.
• I've followed the PEP8 style guides for Python code in the recipes.
• I've used the latest Conan client version.
• I've tried at least one configuration locally with the
  conan-center hook activated.

[CLAassistant]

All committers have signed the CLA.

[accwebs]

This change seemed to fix the issue but I'm not sure if there could be other breakages as a result. Any opinions?

I'm very much a Conan noob so please do double check me (or tell me what I should do to very this isn't gonna break stuff).

I did test with Visual Studio 2019 and Visual Studio 2022.

[ghost]

I detected other pull requests that are modifying boost/all recipe:

• boost: Port patches from 1.77.0 to 1.78.0 #9106
• boost: Fix path finder for python libraries #9059
• [boost] Apply patch to fix https://github.com/boostorg/graph/issues/268 #6132
• boost: use clang-win toolset when compiler=Visual Studio and compiler.toolset=ClangCL #4749

This message is automatically generated by https://github.com/ericLemanissier/conan-center-conflicting-prs so don't hesitate to report issues/improvements there.

[accwebs]

Hmm, I guess I didn't consider that the formula needs to keep working on older Boost versions. How do other formulae solve these types of issues? Would it be okay for newer versions to have a newer b2 build_requires but older ones retain older? Not sure if that's kosher.

Edit: had another idea. I suppose I could try to figure out the problem with older boost versions and write patch files for them?

Would appreciate feedback on the direction I should take.

[SSE4]

Hmm, I guess I didn't consider that the formula needs to keep working on older Boost versions. How do other formulae solve these types of issues? Would it be okay for newer versions to have a newer b2 build_requires but older ones retain older? Not sure if that's kosher.

Edit: had another idea. I suppose I could try to figure out the problem with older boost versions and write patch files for them?

Would appreciate feedback on the direction I should take.

I suppose you could use a conditional build requirement, like the following:

    def build_requirements(self):
        if not self.options.header_only:
            if tools.Version(self.version) >= "1.78.0":
                self.build_requires("b2/4.7.1")
            else:
                self.build_requires("b2/4.5.0")

this will use b2/4.7.1 for boost 1.78.0+, and keep 4.5.0 for older versions. perhaps, older boost versions won't support MSVC 2022 anyway, as code wasn't prepared for the compiler that didn't exist by that time.
/cc @grafikrobot

[accwebs]

Ok, thanks for the feedback; wasn't sure if that's acceptable.

Before I do that, let me tinker a little with the various boost versions to try to figure out A. why the build precisely breaks and B. what boost versions(s) do and don't work with newer b2.

Give me a day or 2.

[grafikrobot]

Upgrading the b2 version to latest for Boost should work all the way back to Boost 1.66.0. It's a use case I test for all B2 changes. I just started a fresh CI run of those tests you can take a look here https://dev.azure.com/bfgroup/B2/_build/results?buildId=408&view=results. I definitely do not see the same error you are seeing on this. I will be very curious at to what @accwebs finds out.

[accwebs]

Trying the build locally I get similar output about numpy and the 'rules are limited to 19 arguments' error:

notice: [zlib] zlib is already configured
notice: [bzip2] bzip is already configured
notice: iostreams: not using lzma compression
notice: iostreams: not using zstd compression
notice: [python-cfg] Configuring python...
notice: [python-cfg] Checking interpreter command "python"...
notice: [python-cfg] running command 'DIR /-C /A:S "C:\Program Files\Python310\python.exe" 2>&1'
notice: [python-cfg] running command 'python -c "from sys import *; print('version=%d.%d\nplatform=%s\nbase_prefix=%s\nbase_exec_prefix=%s\nexecutable=%s' % (version_info[0],version_info[1],platform,base_prefix,base_exec_prefix,executable))" 2>&1'
notice: [python-cfg] ...requested configuration matched!
notice: [python-cfg] Details of this Python configuration:
notice: [python-cfg]   interpreter command: "python"
notice: [python-cfg]   include path: "C:\Program Files\Python310\Include"
notice: [python-cfg]   library path: "C:\Program Files\Python310\libs"
notice: [python-cfg]   DLL search path: "C:\Program Files\Python310"
notice: [python-cfg] Checking for NumPy...
notice: [python-cfg] running command 'python -c "import sys; sys.stderr = sys.stdout; import numpy; print(numpy.get_include())"'
notice: [python-cfg] NumPy disabled. Reason:
notice: [python-cfg]   python -c "import sys; sys.stderr = sys.stdout; import numpy; print(numpy.get_include())" aborted with
notice: [python-cfg]   Traceback (most recent call last):
  File "<string>", line 1, in <module>
ModuleNotFoundError: No module named 'numpy'
ERROR: rules are limited to 19 arguments
C:/.conan/b35911/1/source_subfolder/tools/build/src/util\indirect.jam:105: in indirect.call
C:/.conan/b35911/1/source_subfolder/tools/build/src/build\property.jam:144: in property.evaluate-conditionals-in-context
C:/.conan/b35911/1/source_subfolder/tools/build/src/build\targets.jam:1087: in evaluate-requirements
C:/.conan/b35911/1/source_subfolder/tools/build/src/build\targets.jam:1121: in common-properties2
C:/.conan/b35911/1/source_subfolder/tools/build/src/build\targets.jam:1017: in targets.common-properties
C:/.conan/b35911/1/source_subfolder/tools/build/src/build\targets.jam:1313: in alias-target-class.generate
C:/.conan/b35911/1/source_subfolder\boostcpp.jam:403: in build-multiple
C:/.conan/b35911/1/source_subfolder\boostcpp.jam:393: in class@top-level-target.generate
C:/.conan/b35911/1/source_subfolder/tools/build/src/build\targets.jam:812: in generate-really
C:/.conan/b35911/1/source_subfolder/tools/build/src/build\targets.jam:784: in class@main-target.generate
C:/.conan/b35911/1/source_subfolder/tools/build/src\build-system.jam:797: in load
C:/.conan/b35911/1/source_subfolder/tools/build/src/kernel\modules.jam:295: in import
C:/.conan/b35911/1/source_subfolder/tools/build/src/kernel/bootstrap.jam:139: in boost-build
C:/.conan/b35911/1/source_subfolder/boost-build.jam:17: in module scope
boost/1.71.0:
boost/1.71.0: ERROR: Package 'bcd116a16984c90b61a251b4509e40e7bd3dc0d2' build failed
boost/1.71.0: WARN: Build folder C:\.conan\18c25b\1
ERROR: boost/1.71.0: Error in build() method, line 837
        self.run(full_command, run_environment=True)
        ConanException: Error 1 while executing b2.exe -q numa=on target-os=windows architecture=x86 address-model=32 binary-format=pe abi=ms --layout=system --user-config=C:\.conan\b35911\1\source_subfolder\tools\build\user-config.jam -sNO_ZLIB=0 -sNO_BZIP2=0 -sNO_LZMA=1 -sNO_ZSTD=1 boost.locale.icu=off --disable-icu boost.locale.iconv=off --disable-iconv runtime-link=static runtime-debugging=off threading=multi visibility=hidden link=static variant=release --with-atomic --with-chrono --with-container --with-context --with-contract --with-coroutine --with-date_time --with-exception --with-fiber --with-filesystem --with-graph --with-iostreams --with-locale --with-log --with-math --with-program_options --with-random --with-regex --with-serialization --with-stacktrace --with-system --with-test --with-thread --with-timer --with-type_erasure --with-wave toolset=msvc cxxflags=/std:c++14 pch=on  cxxflags="-fPIC /Z7" install --prefix=C:\.conan\933ff1\1 -j16 --abbreviate-paths -d0 --debug-configuration --build-dir="C:\.conan\18c25b\1"

[accwebs]

Hmm, I think perhaps this change is related to the problem?

boostorg/build@5731edb

(change happened between 4.5.0 and 4.7.1)

Likely before it was still broken, just not failing, thus hiding the error. Any thoughts @grafikrobot ?

Likely conan formula resulting in a large number of parameters or something? Still getting my bearings.

[accwebs]

Additional info: Only Boost 1.78.0 works. All prior versions fail with the same error when trying to build the Conan recipe. So something changed between 1.77.0 -> 1.78.0 in boost that - combined with Conan recipe - triggers the problem.

[grafikrobot]

Likely before it was still broken, just not failing, thus hiding the error. Any thoughts @grafikrobot ?

Right.

Likely conan formula resulting in a large number of parameters or something? Still getting my bearings.

Right. Since it works fine in the B2 building Boost Ci tests.

[accwebs]

Ok, I've dug into it for a while and still am stuck.

Its like b2 + boost prior to 1.78.0 is getting confused and thus somehow exceeds the parameter limit in the early pre-processing steps that b2 is doing. If I simply change to the extracted boost source directory (extracted there by Conan) and run 'b2.exe' without any args I get the same error.

At this point I'm pretty stumped so would welcome any ideas people have.

One thing I need to still look at is the patches Conan is applying to boost post extraction. I did a quick eyeball for anything obvious and nothing jumped out at me. Still, I should dig deeper there.

[SSE4]

there is debug_level option of b2, which may produce an additional debug output. it's cryptic, but for sure it may contain something helpful.

[accwebs]

Yeah, I was trying that last night. I was really struggling to understand what b2 is doing at the time that the processing stops. To be honest I think I'd have to be considerably more familiar with boost and b2 in order to make further progress with that route.

Instead, in order to make further progress, I might have to focus on the deltas between Conan vs not and also Boost 1.78 / prev versions. I'll try that either tonight or tomorrow night.

[accwebs]

Ok, I made some progress. The difference is that the Azure builds of b2 where it's building boost as part of the tests supply the "--boost-build=${env:BUILD_SOURCESDIRECTORY}/src" argument. Conan is not setting this switch when invoking b2. I manually invoked the command Conan was trying to run with that argument (pointing to a git clone of b2 v4.7.1) and it bypassed the problem.

I'm having trouble finding any documentation for that argument and why you might need it. But my guess would be that possibly the b2 executable and various related script files are intended to be tightly coupled. Thus Conan's pairing of b2.exe from v4.7.1 with older versions of boost (that have older b2 scripts inside) is the problem; however, up till now the Boost Conan formula has used an old enough version of b2 that this didn't matter (when combined with newer Boost sources having newer build scripts).

@grafikrobot sorry to bug you again, but guessing you can confirm this diagnosis (or set me straight if it's wrong :-P )

Even if this is correct, I'm not sure yet if the current b2 conan formula is packaging up all the files into a location that --boost-build= can be pointed at...checking into this next. If they ARE being packaged up with the compiled b2 then I think I just need to add this flag to the command line invocation of b2 in the boost formula :-)

[grafikrobot]

Ok, I made some progress. The difference is that the Azure builds of b2 where it's building boost as part of the tests supply the "--boost-build=${env:BUILD_SOURCESDIRECTORY}/src" argument. Conan is not setting this switch when invoking b2. I manually invoked the command Conan was trying to run with that argument (pointing to a git clone of b2 v4.7.1) and it bypassed the problem.

I'm having trouble finding any documentation for that argument and why you might need it. But my guess would be that possibly the b2 executable and various related script files are intended to be tightly coupled. Thus Conan's pairing of b2.exe from v4.7.1 with older versions of boost (that have older b2 scripts inside) is the problem; however, up till now the Boost Conan formula has used an old enough version of b2 that this didn't matter (when combined with newer Boost sources having newer build scripts).

@grafikrobot sorry to bug you again, but guessing you can confirm this diagnosis (or set me straight if it's wrong :-P )

That argument is not part of B2. It's part of Boost https://github.com/boostorg/boost/blob/master/boost-build.jam#L15. Hence why it's not documented in B2. It controls where to find the B2 build files, i.e. the *.jam files. And it makes sense now why having it on my Azure CI would circumvent the problem. As it means that, yes, something in the older B2 files included with the Boost releases is causing the error when used with the newer B2 engine exe.

Even if this is correct, I'm not sure yet if the current b2 conan formula is packaging up all the files into a location that --boost-build= can be pointed at...checking into this next. If they ARE being packaged up with the compiled b2 then I think I just need to add this flag to the command line invocation of b2 in the boost formula :-)

All the needed files are included in the B2 package, see

conan-center-index/recipes/b2/portable/conanfile.py

Line 86 in ebd98ec

self.copy(pattern="*.jam", dst="bin", src="output")

. And, yes, adding the --boost-build=?? option to the Boost build invocation should resolve the problem. Not sure if it will create other problems. But it will let you make some progress.

[accwebs]

That argument is not part of B2. It's part of Boost https://github.com/boostorg/boost/blob/master/boost-build.jam#L15. Hence why it's not documented in B2. It controls where to find the B2 build files, i.e. the *.jam files. And it makes sense now why having it on my Azure CI would circumvent the problem. As it means that, yes, something in the older B2 files included with the Boost releases is causing the error when used with the newer B2 engine exe.

Gotcha; thanks for the info @grafikrobot ! I guess this begs the question of 'should all older B2 files be compatible with newer B2 versions'? i.e. does B2 even really aim to support that? Or are you supposed to only build Boost with the B2 version that matches w it?

Not sure if it will create other problems.

Indeed. I'm not sure if I'm really solving the issue correctly by taking this path. Maybe I should be instead trying to create .patch files for all the older boost versions to fix up their .jam file(s) in the right places to work with newer B2?

[grafikrobot]

I guess this begs the question of 'should all older B2 files be compatible with newer B2 versions'? i.e. does B2 even really aim to support that?

Although not explicitly mentioned, the aim is for the B2 engine (exe) to only work with the B2 build system (*.jam) that is the same version. But for that to support building existing versions of project Jamfiles.

Or are you supposed to only build Boost with the B2 version that matches w it?

Boost itself is only tested with the B2 version that comes with it. But I do the additional testing of using the newest B2 release with old Boost versions.

Not sure if it will create other problems.

Indeed. I'm not sure if I'm really solving the issue correctly by taking this path. Maybe I should be instead trying to create .patch files for all the older boost versions to fix up their .jam file(s) in the right places to work with newer B2?

Not sure since I don't know what changes it would take. Another option is for me to also support old B2 build system files with the newest B2 engine exe. But that will make for significant complexity in B2 itself.

[accwebs]

Hey,

All useful thoughts and information, thanks!

I think the next steps are is for me to try:

1. To figure out if a simple tweak to older Boost .jam files can make it work with latest B2. If so I'll just add a .patch file that's applied to older Boost versions post-extract and pre-compile :-)

2. If approach 1 fails, then maybe the solution is indeed to simply set up the Boost recipe for older versions to use older B2 versions, and newer Boost to use newer B2.

Regarding:

Another option is for me to also support old B2 build system files with the newest B2 engine exe. But that will make for significant complexity in B2 itself.

Yeah, IMO that seems excessive and unnecessary given the current scope/objectives of B2. Sounds to me like this is more a bug in older Boost build files brought to the front by newer B2, a stance which is reinforced somewhat by the fact that current Boost version works just fine :-)

[accwebs]

Odd, the build failed with 1.74.0 but I don't see any way to get the detailed error message. I did now just re-test with 1.74.0 (albeit on Windows not MacOS) and it seems okay.

@SSE4 any ideas? Do I just need to re-run the build job? Or is there some way to review the error log (and I'm just being dense and missing it)?

[SSE4]

Odd, the build failed with 1.74.0 but I don't see any way to get the detailed error message. I did now just re-test with 1.74.0 (albeit on Windows not MacOS) and it seems okay.

@SSE4 any ideas? Do I just need to re-run the build job? Or is there some way to review the error log (and I'm just being dense and missing it)?

it's an internal CI error and has nothing to do with the recipe/library itself (might be a network error, for instance). but as I can see, you already have restarted a build.

[conan-center-bot]

All green in build 3 (10c1bea9e4b7b24e87728436fe990bfa16e8c93b):

• boost/1.74.0@:
  All packages built successfully! (All logs)

• boost/1.72.0@:
  All packages built successfully! (All logs)

• boost/1.69.0@:
  All packages built successfully! (All logs)

• boost/1.70.0@:
  All packages built successfully! (All logs)

• boost/1.73.0@:
  All packages built successfully! (All logs)

• boost/1.76.0@:
  All packages built successfully! (All logs)

• boost/1.75.0@:
  All packages built successfully! (All logs)

• boost/1.77.0@:
  All packages built successfully! (All logs)

• boost/1.78.0@:
  All packages built successfully! (All logs)

• boost/1.71.0@:
  All packages built successfully! (All logs)

[accwebs]

Ok, looks like everything passes now ( finally ;-) ). In the end I simply created .patch files for each of the supported older Boost versions to fix up their build scripts to work with newer B2. The patches were all very minor and basically the same. I ended up including 3 different .patch file versions to account for slight differences in the original files.

@grafikrobot thank you very much for your guidance; thanks to you I was able to avoid going down the wrong rabbit holes on multiple occasions. I'm very much a Boost and B2 noob.

@SSE4 thank you for your help as well.