[ComputeLibrary] Add new package for ARM-software/ComputeLibrary #19958

uilianries · 2023-09-19T13:32:46Z

Specify library name and version: compute_library/23.08

That's a requirement for #19681 (OpenVino)

The ComputeLibrary has experimental CMake support, but it's restricted to a single linux profile, so it does not work with Mac. The official documentation uses SCons: https://github.com/ARM-software/ComputeLibrary

The fPIC options is missing because is hardcoded in Scons script, so it would require a patch.
It's not possible to avoid building all tests, but we don't install them at least
The steps build and install are executed together, because install is a sub-step of build major step. We can not change without a patch, so we copy/clean during package() method
It builds both static and shared at same build, there is no option to control it.

I've read the contributing guidelines.
I've used a recent Conan client version close to the currently deployed.
I've tried at least one configuration locally with the conan-center hook activated.

Signed-off-by: Uilian Ries <uilianries@gmail.com>

github-actions · 2023-09-19T14:53:25Z

Hooks produced the following warnings for commit 7d86026

compute_library/23.08@#557d1db9635cd53cff3ff429a550ed93

post_source(): WARN: [SHORT_PATHS USAGE (KB-H066)] The file './src/core/NEON/kernels/arm_conv/depthwise/kernels/a64_u8s8u8q_packed_to_nhwc_generic_with_multiplier_output2x8_mla_depthfirst/generic.cpp' has a very long path and may exceed Windows max path length. Add 'short_paths = True' in your recipe.

ilya-lavrenov · 2023-09-19T15:32:56Z

recipes/compute_library/all/conanfile.py

+ yes_no = lambda v: "1" if v else "0"
+ debug = yes_no(self.settings.build_type == "Debug")
+ build_os = str(self.settings.os).lower()
+ arch = {"armv8": "armv8a", "x86": "x86_32", "armv7": "armv7a", "armv7hf": "armv7a-hf"}.get(str(self.settings.arch), str(self.settings.arch))


ARM compute allows to build code with armv8.2a (fp16 support), SVE and SME.
How to use it via this recipe?

for example Apple M1, M2 support fp16, so we need to enable it by default.

uilianries · 2023-09-19T19:28:23Z

recipes/compute_library/all/conanfile.py

+ minimum_version = self._compilers_minimum_version.get(str(self.settings.compiler), False)
+ if minimum_version and Version(self.settings.compiler.version) < minimum_version:
+ raise ConanInvalidConfiguration(f"{self.ref} requires C++{self._min_cppstd}, which your compiler does not support.")
+ supported_os = ["Android", "Linux", "OpenBSD", "Macos", "Tizen"]


Sorry, but it's not documented by the upstream: https://github.com/ARM-software/ComputeLibrary#supported-systems

https://arm-software.github.io/ComputeLibrary/latest/how_to_build.xhtml#S1_6_3_WoA

ilya-lavrenov · 2023-09-19T19:16:41Z

recipes/compute_library/all/conanfile.py

+ # INFO: Using scons to build the library we don't have control over shared/static and install step, it is done all together always
+ # INFO: https://arm-software.github.io/ComputeLibrary/latest/how_to_build.xhtml
+ yes_no = lambda v: "1" if v else "0"
+ debug = yes_no(self.settings.build_type == "Debug")


maybe we can also add these options for debug debug=1 asserts=1 logging=1

ilya-lavrenov · 2023-09-19T19:21:28Z

recipes/compute_library/all/conanfile.py

+ if str(self.settings.os) not in supported_os:
+ raise ConanInvalidConfiguration(f"{self.ref} does not support {self.settings.os}. It is only supported on {supported_os}.")
+ if "arm" not in str(self.settings.arch) and "x86" not in str(self.settings.arch):
+ raise ConanInvalidConfiguration(f"{self.ref} does not support {self.settings.arch}. It is only supported on arm and x86.")


arm and x86 usually imply 32bits
should we write arm / arm64 and x86_64 ?

I think that's a good clarification.

recipes/compute_library/all/conandata.yml

uilianries · 2023-09-19T19:28:23Z

recipes/compute_library/all/conanfile.py

+ minimum_version = self._compilers_minimum_version.get(str(self.settings.compiler), False)
+ if minimum_version and Version(self.settings.compiler.version) < minimum_version:
+ raise ConanInvalidConfiguration(f"{self.ref} requires C++{self._min_cppstd}, which your compiler does not support.")
+ supported_os = ["Android", "Linux", "OpenBSD", "Macos", "Tizen"]


Sorry, but it's not documented by the upstream: https://github.com/ARM-software/ComputeLibrary#supported-systems

recipes/compute_library/all/conanfile.py

Signed-off-by: Uilian Ries <uilianries@gmail.com>

github-actions · 2023-09-20T10:43:34Z

Hooks produced the following warnings for commit 158d0c7

compute_library/23.08@#bbfb9834bb59b8d05cadb58204eb4f74

post_source(): WARN: [SHORT_PATHS USAGE (KB-H066)] The file './src/core/NEON/kernels/arm_conv/depthwise/kernels/a64_u8s8u8q_packed_to_nhwc_generic_with_multiplier_output2x8_mla_depthfirst/generic.cpp' has a very long path and may exceed Windows max path length. Add 'short_paths = True' in your recipe.

Signed-off-by: Uilian Ries <uilianries@gmail.com>

github-actions · 2023-09-20T17:31:03Z

Hooks produced the following warnings for commit e3e2f36

compute_library/23.08@#77aaf6972aa3a2e9d1856df00349d77e

post_source(): WARN: [SHORT_PATHS USAGE (KB-H066)] The file './src/core/NEON/kernels/arm_conv/depthwise/kernels/a64_u8s8u8q_packed_to_nhwc_generic_with_multiplier_output2x8_mla_depthfirst/generic.cpp' has a very long path and may exceed Windows max path length. Add 'short_paths = True' in your recipe.

Signed-off-by: Uilian Ries <uilianries@gmail.com>

ilya-lavrenov · 2023-09-21T09:47:16Z

recipes/compute_library/all/conanfile.py

+ if "armv8" not in str(self.settings.arch) and self.options.enable_multi_isa:
+ raise ConanInvalidConfiguration(f"{self.ref} does not support multi_isa option for {self.settings.arch}. It is only supported on armv8.")
+ if self.settings.arch == "armv8" and self.build_settings.arch == "x86_64" and self.settings.os == "Macos":
+ raise ConanInvalidConfiguration(f"Mac Intel is not supported for armv8-a. Please, use Mac M1 as native build.")


actually, cross compilation from macOS arm64 to x86_64 (as vice versa) works well. You need to use -arch <arm64 | x86_64> option.

See https://github.com/openvinotoolkit/openvino/blob/master/src/plugins/intel_cpu/thirdparty/ACLConfig.cmake#L285-L288 as example

The problem is: I'm using scons and it changes to armv8-a that's not identified by apple clang and passing as -march.

I also use scons, but it is just run from cmake.

github-actions · 2023-09-21T12:55:45Z

Hooks produced the following warnings for commit ff185c2

compute_library/23.02.1@#47dafd2c82db97de0174fe779d93eb19

post_source(): WARN: [SHORT_PATHS USAGE (KB-H066)] The file './src/core/NEON/kernels/arm_conv/depthwise/kernels/a64_u8s8u8q_packed_to_nhwc_generic_with_multiplier_output2x8_mla_depthfirst/generic.cpp' has a very long path and may exceed Windows max path length. Add 'short_paths = True' in your recipe.
post_package(): WARN: [MISSING SYSTEM LIBS (KB-H043)] Library './lib/libarm_compute.so' links to system library 'dl' but it is not in cpp_info.system_libs.
post_package(): WARN: [MISSING SYSTEM LIBS (KB-H043)] Library './lib/libarm_compute_graph.so' links to system library 'dl' but it is not in cpp_info.system_libs.
post_package(): WARN: [MISSING SYSTEM LIBS (KB-H043)] Library './lib/libarm_compute_core.so' links to system library 'dl' but it is not in cpp_info.system_libs.
post_package(): WARN: [APPLE RELOCATABLE SHARED LIBS (KB-H077)] install_name dir of these shared libs is not @rpath: libarm_compute.dylib, libarm_compute_core.dylib, libarm_compute_graph.dylib

compute_library/23.08@#418fdff5101d761b59739ceaf986b2c5

post_source(): WARN: [SHORT_PATHS USAGE (KB-H066)] The file './src/core/NEON/kernels/arm_conv/depthwise/kernels/a64_u8s8u8q_packed_to_nhwc_generic_with_multiplier_output2x8_mla_depthfirst/generic.cpp' has a very long path and may exceed Windows max path length. Add 'short_paths = True' in your recipe.
post_package(): WARN: [MISSING SYSTEM LIBS (KB-H043)] Library './lib/libarm_compute.so' links to system library 'dl' but it is not in cpp_info.system_libs.
post_package(): WARN: [MISSING SYSTEM LIBS (KB-H043)] Library './lib/libarm_compute_graph.so' links to system library 'dl' but it is not in cpp_info.system_libs.
post_package(): WARN: [MISSING SYSTEM LIBS (KB-H043)] Library './lib/libarm_compute_core.so' links to system library 'dl' but it is not in cpp_info.system_libs.
post_package(): WARN: [APPLE RELOCATABLE SHARED LIBS (KB-H077)] install_name dir of these shared libs is not @rpath: libarm_compute.dylib, libarm_compute_core.dylib, libarm_compute_graph.dylib

ilya-lavrenov · 2023-09-21T17:13:30Z

recipes/compute_library/all/conanfile.py

+ multi_isa = yes_no(self.options.enable_multi_isa)
+ build = "cross_compile" if cross_building(self) else "native"
+ with chdir(self, self.source_folder):
+ self.run(f"scons Werror=0 validation_tests=0 examples=0 gemm_tuner=0 multi_isa={multi_isa} openmp={openmp} debug={debug} neon={neon} opencl={opencl} os={build_os} arch={arch} build={build} build_dir={self.build_folder} install_dir={self.package_folder} -j{build_jobs(self)} toolchain_prefix=''", env="conanbuild")


conan's arch and compute library archs are different. Compute Library allows to specify exact optimizations which are required. Without it, it does not give proper performance.
So, bypassing conan's arch to compute library is not enough. Conan's arch can be used as initial guess, while users still need to have an ability to provide an option for arch to enable specific optimizations like SVE, SME

As I wrote - Apple M1 must set armv8.2-a to enable FP16 optimization. Currently, it's not set and performance is poor.

No problem, you can customize your settings.yml, providing not only new archs, but also new subsettings: https://docs.conan.io/2/reference/config_files/settings.html#customizing-settings

Conan covers most generic and known archs, but when talking about optimization, is better having a specific profile adapted, so you can be sure about what's in your package.

I disagree with such approach.

If we put it on x64 arch, what I am asking is to provide a way to select between SSE, AVX, AMX optimizations, while you want me to customize whole settings file and add SSE, AVX and so on to be able to properly compile the library. Don't mix different platforms (settings) against different library optimizations capabilities (the arm option of ARM Compute Library).

Any reason why on Apple ARM64 people would not have FP16 support (which is available out of box) and have to change settings to get suitable performance?

Signed-off-by: Uilian Ries <uilianries@gmail.com>

conan-center-bot · 2023-09-25T15:12:39Z

Conan v1 pipeline ✔️

All green in build 8 (8064b14d181f682f6398af943ba6467271bf5670):

compute_library/23.02.1:
All packages built successfully! (All logs)
compute_library/23.08:
All packages built successfully! (All logs)

Conan v2 pipeline ✔️

Note: Conan v2 builds are now mandatory. Please read our discussion about it.

All green in build 8 (8064b14d181f682f6398af943ba6467271bf5670):

compute_library/23.08:
All packages built successfully! (All logs)
compute_library/23.02.1:
All packages built successfully! (All logs)

github-actions · 2023-09-25T15:13:18Z

Hooks produced the following warnings for commit 8064b14

compute_library/23.02.1@#f6f4e87eeace0a9e27d5728842898b34

post_source(): WARN: [SHORT_PATHS USAGE (KB-H066)] The file './src/core/NEON/kernels/arm_conv/depthwise/kernels/a64_u8s8u8q_packed_to_nhwc_generic_with_multiplier_output2x8_mla_depthfirst/generic.cpp' has a very long path and may exceed Windows max path length. Add 'short_paths = True' in your recipe.
post_package(): WARN: [MISSING SYSTEM LIBS (KB-H043)] Library './lib/libarm_compute.so' links to system library 'dl' but it is not in cpp_info.system_libs.
post_package(): WARN: [MISSING SYSTEM LIBS (KB-H043)] Library './lib/libarm_compute_graph.so' links to system library 'dl' but it is not in cpp_info.system_libs.
post_package(): WARN: [MISSING SYSTEM LIBS (KB-H043)] Library './lib/libarm_compute_core.so' links to system library 'dl' but it is not in cpp_info.system_libs.
post_package(): WARN: [APPLE RELOCATABLE SHARED LIBS (KB-H077)] install_name dir of these shared libs is not @rpath: libarm_compute.dylib, libarm_compute_core.dylib, libarm_compute_graph.dylib

compute_library/23.08@#552c581b77110a37876484b05ed29789

post_source(): WARN: [SHORT_PATHS USAGE (KB-H066)] The file './src/core/NEON/kernels/arm_conv/depthwise/kernels/a64_u8s8u8q_packed_to_nhwc_generic_with_multiplier_output2x8_mla_depthfirst/generic.cpp' has a very long path and may exceed Windows max path length. Add 'short_paths = True' in your recipe.
post_package(): WARN: [MISSING SYSTEM LIBS (KB-H043)] Library './lib/libarm_compute.so' links to system library 'dl' but it is not in cpp_info.system_libs.
post_package(): WARN: [MISSING SYSTEM LIBS (KB-H043)] Library './lib/libarm_compute_graph.so' links to system library 'dl' but it is not in cpp_info.system_libs.
post_package(): WARN: [MISSING SYSTEM LIBS (KB-H043)] Library './lib/libarm_compute_core.so' links to system library 'dl' but it is not in cpp_info.system_libs.
post_package(): WARN: [APPLE RELOCATABLE SHARED LIBS (KB-H077)] install_name dir of these shared libs is not @rpath: libarm_compute.dylib, libarm_compute_core.dylib, libarm_compute_graph.dylib

jwillikers · 2023-10-27T11:49:01Z

recipes/compute_library/all/conanfile.py

+ if str(self.settings.os) not in supported_os:
+ raise ConanInvalidConfiguration(f"{self.ref} does not support {self.settings.os}. It is only supported on {supported_os}.")
+ if "arm" not in str(self.settings.arch) and "x86" not in str(self.settings.arch):
+ raise ConanInvalidConfiguration(f"{self.ref} does not support {self.settings.arch}. It is only supported on arm and x86.")


I think that's a good clarification.

uilianries added 2 commits September 19, 2023 14:42

Add ComputeLibrary

34022db

Signed-off-by: Uilian Ries <uilianries@gmail.com>

Fix Macos build

7d86026

Signed-off-by: Uilian Ries <uilianries@gmail.com>

uilianries requested a review from AbrilRBS September 19, 2023 13:32

AbrilRBS assigned uilianries Sep 19, 2023

prince-chrismc mentioned this pull request Sep 19, 2023

Pull Requests Ready to be Reviewed prince-chrismc/conan-center-index-pending-review#1

Open

conan-center-bot added the Failed label Sep 19, 2023

This comment has been minimized.

Sign in to view

ilya-lavrenov reviewed Sep 19, 2023

View reviewed changes

recipes/compute_library/all/conandata.yml Show resolved Hide resolved

uilianries commented Sep 19, 2023

View reviewed changes

ilya-lavrenov reviewed Sep 19, 2023

View reviewed changes

recipes/compute_library/all/conanfile.py Show resolved Hide resolved

Add version 23.02.1

c98c305

Signed-off-by: Uilian Ries <uilianries@gmail.com>

This comment has been minimized.

Sign in to view

Fix package sha-256

158d0c7

Signed-off-by: Uilian Ries <uilianries@gmail.com>

This comment has been minimized.

Sign in to view

uilianries added 2 commits September 20, 2023 14:12

add gemm_tunner=0

fdc9cdf

Signed-off-by: Uilian Ries <uilianries@gmail.com>

do not enforce toolchain prefix

e3e2f36

Signed-off-by: Uilian Ries <uilianries@gmail.com>

This comment has been minimized.

Sign in to view

uilianries added 5 commits September 21, 2023 11:29

Add multi_isa option

9198d4c

Signed-off-by: Uilian Ries <uilianries@gmail.com>

restrict mac cross-building

315f906

Signed-off-by: Uilian Ries <uilianries@gmail.com>

remove unused import

030977f

Signed-off-by: Uilian Ries <uilianries@gmail.com>

fix settings build

e3a6018

Signed-off-by: Uilian Ries <uilianries@gmail.com>

Improve mac restriction

ff185c2

Signed-off-by: Uilian Ries <uilianries@gmail.com>

ilya-lavrenov reviewed Sep 21, 2023

View reviewed changes

conan-center-bot removed the Failed label Sep 21, 2023

This comment has been minimized.

Sign in to view

ilya-lavrenov reviewed Sep 21, 2023

View reviewed changes

Support gcc 6 actually

8064b14

Signed-off-by: Uilian Ries <uilianries@gmail.com>

conan-center-bot requested a review from danimtb October 19, 2023 12:45

jwillikers suggested changes Oct 27, 2023

View reviewed changes

conan-center-bot requested a review from jcar87 December 19, 2023 11:44

conan-center-bot requested a review from franramirez688 January 15, 2024 11:51

ErniGH mentioned this pull request Jun 14, 2024

Prioritization of new recipe PRs #24327

Open

danimtb approved these changes Aug 19, 2024

View reviewed changes

danimtb self-assigned this Aug 19, 2024

AbrilRBS approved these changes Aug 22, 2024

View reviewed changes

conan-center-bot merged commit 5fb1c3b into conan-io:master Aug 22, 2024
18 checks passed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[ComputeLibrary] Add new package for ARM-software/ComputeLibrary #19958

[ComputeLibrary] Add new package for ARM-software/ComputeLibrary #19958

uilianries commented Sep 19, 2023 •

edited

Loading

This comment has been minimized.

github-actions bot commented Sep 19, 2023

ilya-lavrenov Sep 19, 2023

ilya-lavrenov Sep 19, 2023

uilianries Sep 19, 2023

ilya-lavrenov Sep 19, 2023

ilya-lavrenov Sep 19, 2023

ilya-lavrenov Sep 19, 2023

jwillikers Oct 27, 2023

uilianries Sep 19, 2023

This comment has been minimized.

This comment has been minimized.

github-actions bot commented Sep 20, 2023

This comment has been minimized.

github-actions bot commented Sep 20, 2023

ilya-lavrenov Sep 21, 2023

uilianries Sep 21, 2023 •

edited

Loading

ilya-lavrenov Sep 21, 2023

This comment has been minimized.

github-actions bot commented Sep 21, 2023

ilya-lavrenov Sep 21, 2023 •

edited

Loading

uilianries Sep 25, 2023

ilya-lavrenov Sep 25, 2023

conan-center-bot commented Sep 25, 2023

github-actions bot commented Sep 25, 2023

jwillikers Oct 27, 2023

[ComputeLibrary] Add new package for ARM-software/ComputeLibrary #19958

[ComputeLibrary] Add new package for ARM-software/ComputeLibrary #19958

Conversation

uilianries commented Sep 19, 2023 • edited Loading

This comment has been minimized.

github-actions bot commented Sep 19, 2023

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

This comment has been minimized.

This comment has been minimized.

github-actions bot commented Sep 20, 2023

This comment has been minimized.

github-actions bot commented Sep 20, 2023

Choose a reason for hiding this comment

uilianries Sep 21, 2023 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

This comment has been minimized.

github-actions bot commented Sep 21, 2023

ilya-lavrenov Sep 21, 2023 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

conan-center-bot commented Sep 25, 2023

Conan v1 pipeline ✔️

Conan v2 pipeline ✔️

github-actions bot commented Sep 25, 2023

Choose a reason for hiding this comment

uilianries commented Sep 19, 2023 •

edited

Loading

uilianries Sep 21, 2023 •

edited

Loading

ilya-lavrenov Sep 21, 2023 •

edited

Loading