Defining custom settings.yml for sanitizers

[memsharded]

I already configured our settings.yml with sanitizer options per the doc a while back. (BTW, why not just include these sanitization options in the default conan.io settings.yml?)

compiler:
    gcc: &gcc
        address_sanitizer: [None, True]
        hwaddress_sanitizer: [None, True]
        kernel_address_sanitizer: [None, True]
        thread_sanitizer: [None, True]
        undefined_behavior_sanitizer: [None, True]
        pointer_compare_sanitizer: [None, True] # only valid with address or kernel_address
        pointer_subtract_sanitizer: [None, True] # only valid with address or kernel_address
        shadow_call_stack_sanitizer: [None, True]
        leak_sanitizer: [None, True]
        stack_protector: [None, True]
        stack_check: [None, True]
    clang:
        address_sanitizer: [None, True]
        hwaddress_sanitizer: [None, True]
        thread_sanitizer: [None, True]
        memory_sanitizer: [None, True]
        undefined_behavior_sanitizer: [None, True]

But, I still don't know how to manage my .conan/data output. As it is now, conan install just overwrites the dependency binaries per sanitization setting. How does a settings_user.yml fix this? Are you saying to upload different binaries to artifactory on a per-settings.yml-basis? What does that look like? I could in-theory upload the binaries to artifactory, but nobody else needs them other than me.

Oh, and CONAN_SETTINGS_COMPILER_SANITIZER seems not to be propagated for the conan 1.60 generator. Is there something else I have to do to make that work?

Even if I could do something with settings.yml, uploading and then downloading 65-package binaries that I already compiled on my macbook just because I changed a compiler flag seems wrong too.

Originally posted by @hoyhoy in #5295 (comment)

[memsharded]

As it is now, conan install just overwrites the dependency binaries per sanitization setting. How does a settings_user.yml fix this?

Defining the -s compiler.address_sanitizer=True in the command line or in your profile will result in a new package_id, and then the binaries are not overwritten, but stored in a different folder. Please try the conan create with the different values for the settings, and do a conan search (in Conan 1) or conan list (Conan 2) to list the created binaries and see their different package_id with their different settings,not overwriting each other.

Are you saying to upload different binaries to artifactory on a per-settings.yml-basis? What does that look like? I could in-theory upload the binaries to artifactory, but nobody else needs them other than me.

You can upload them to another repository if you want to, that only you have permissions. But you can upload them to the main repo used by others, if other users don't need those binaries because they don't do sanitized builds, they will not be downloaded at all. Only package binaries that are necessary are downloaded, the others are not.

And in any case, as I commented, it is not mandatory to upload them if you don't want to upload them. They will still be handled correctly in the cache without overwriting the main packages and not requiring another folder or storage.

Oh, and CONAN_SETTINGS_COMPILER_SANITIZER seems not to be propagated for the conan 1.60 generator. Is there something else I have to do to make that work?

The CONAN_SETTINGS_XXX only exists in legacy cmake generator. With the new generator, if you want to do something specific, you do it in the CMakeToolchain, translating the Conan settings to the CMake variables that are needed. The idea is that CMakeLists.txt are completely Conan-agnostic, the integration can be fully transparent. The CMakeToolchain also have other mechanisms, like the ability to inject your own cmake toolchain with tools.cmake.cmaketoolchain:user_toolchain new conf.

Even if I could do something with settings.yml, uploading and then downloading 65-package binaries that I already compiled on my macbook just because I changed a compiler flag seems wrong too.

If you compiled the packages in your laptop they will not be the re-downloaded. Conan cache mechanism avoid doing that, a conan install will find the package binaries in the cache and used them, it will not download anything from the server. Conan can work fully offline if it finds the binaries it needs in the cache. Not sure what you mean by that.

[leha-bot]

So, is there any updates about this topic? It seems that the defining setting in compiler is the most efficient way to ensure that all libraries are built with sanitizers as if any library didn't build with sanitizers, then they won't able to detect problems inside their code

[hoyhoy]

I ended up mostly using elaborate profiles. The old profile language had a ton of bugs though. Now there's a new profile Jinja2 language, but it's INSANELY hard to use. Imagine PHP, but like if PHP was written in 1991... Oh, and you can't use Python f-strings at all anywhere. I've already managed to crash the new j2 profile interpreter a couple of times. Try accidentally double-specifying the same profile if you want a repro @memsharded!

I used to have fifty individual profile files, but I'm trying to share a main included compiler one for all the sanitizers. This is working for me now. You don't need to modify setting.yml. I did modify mine, but then ended up not using it.

Here is a detailed example of what is working for me with conan2 using j2 profiles and the detect_api.

https://gist.github.com/hoyhoy/dc46c8edbf442fd51f44e51d542d185c

[paulharris]

@memsharded I see you added address_sanitizer into the settings.yml, but does that translate into compiler flags? I see @hoyhoy has an elaborate gist for his profiles, I was looking for something simpler to start with.

[memsharded]

The settings that I added to the profile were user settings, not built-ins, and as such Conan is not aware of them. Recipes still need to read them and apply them, with user logic, or the adequate flags should also be added to the profile, like via tools.build:cxxflags

I think the profiles of @hoyhoy are for a very flexible and complete case, including different dependencies configurations, ccache and others. Just hardcoding some sanitizer flags into tools.build:cxxflags for one compiler should be relatively easy, if you know what flags are necessary to enable that sanitizer, of course.

[hoyhoy]

My profile is meant for compiling a lot of legacy packages and even then it doesn't work reliably. I just hit a case with libcurl where conan is setting --with-ssl=$CONAN_HOME/p/b/opens7b45e8b4b84d5/p for the configure, but autoconf is still using the /usr/local/include for the ssl include. It's more of an autoconf issue really. But, I ended up just having to delete my /usr/local openssl deployment. There are a number of issues like that with everything that's built without CMake actually.

@memsharded is there any way to pass-in options from the command-line to jinja? It seems like environment variables are the only way, and, on msvc, that really isn't convenient.

The other issue I'm having is that I want to use build_vars for output, but there's no way to bring in various booleans for the sanitizer flags.

[paulharris]

Thanks for the clarifications.

I've had that /usr/local/include issue in the past with conan 1 and some older packages.
I ended up "poisoning" the header files by adding #error "something" to help guarantee the build isn't sucking in system includes.