[question] How to build a toolchain package (or how to fix / debug)

[hcw70]

What is your question?

I am a conan2 starter, so some concepts may not be clear yet to me.

I am using conan 2.20.1 and trying to build a conan package for our toolchains.

The toolchains should be identified by os.distro = "SW39344E" meaning a part number for a tar.gz containing a buildroot based toolchain.

So os.distro should be the primary key to identify the cross compilation target.

However i am failing to create a toolchain package, since the test_package step always fails:

deuta-toolchain/1.0.0: package(): Packaged 243 '.hpp' files
deuta-toolchain/1.0.0: Created package revision c3dcd9364f09cfa21acbb537e7082d51
deuta-toolchain/1.0.0: Package '9512ff797f287a1542c98c751aa30dd495b02fa4' created
deuta-toolchain/1.0.0: Full package reference: deuta-toolchain/1.0.0#65cf19a8800606e5e842c6ace20cc080:9512ff797f287a1542c98c751aa30dd495b02fa4#c3dcd9364f09cfa21acbb537e7082d51
deuta-toolchain/1.0.0: Package folder /home/hcw/.conan2/p/b/deuta64c91524e200a/p

======== Launching test_package ========

======== Computing dependency graph ========
deuta-toolchain/1.0.0 (test package): Using toolchain build-require: deuta-toolchain/1.0.0#65cf19a8800606e5e842c6ace20cc080%1757490224.9919577 for Linux-SW39-347E-x86_64
Graph root
    deuta-toolchain/1.0.0 (test package): /home/hcw/work/Deuta/deuta-packaging/conan-recipies/toolchain/test_package/conanfile.py
Build requirements
    deuta-toolchain/1.0.0#65cf19a8800606e5e842c6ace20cc080 - Cache

======== Computing necessary packages ========
Build requirements
    deuta-toolchain/1.0.0#65cf19a8800606e5e842c6ace20cc080:63fead0844576fc02943e16909f08fcdddd6f44b - Invalid
ERROR: There are invalid packages:
deuta-toolchain/1.0.0: Invalid: 'settings.os.distro' value not defined

It looks like the test package (or the toolchain package) does not define os.distro, although it does it...?

Maybe i am setting the package_id for the toolchain wrong, so the test_package is unable to find it?
Looks like the package is being built, but later not found when consuming by the test_package.

Here is my

settings.yml:

os:
  Linux:
    distro: [Ubuntu-22.04, Ubuntu-24.04, SW39-347E, SW39-373A, SW39-382A, SW39-388A]
  Windows:
    distro: [SW39-409A]

Profile SW39-347E:

[settings]
os=Linux
os.distro=SW39-347E
arch=armv7
compiler=gcc
compiler.version=11
compiler.libcxx=libstdc++11
build_type=Release

[conf]
tools.cmake.cmaketoolchain:generator=Ninja
tools.cmake.cmaketoolchain:system_name=SW39-347E
tools.cmake.cmaketoolchain:toolchain_file=/opt/deuta/SW39-347E/sdk/share/buildroot/toolchainfile.cmake

my toolchain/conanfile.py:

import os
from conan import ConanFile
from conan.tools.files import get, copy, download
from conan.errors import ConanInvalidConfiguration
from conan.tools.scm import Version

class ToolchainInfo:
    def __init__(self, url, sdk_sysroot_rel, toolchain_file_rel):
        self.url = url
        self.sdk_sysroot_rel = sdk_sysroot_rel
        self.toolchain_file_rel = toolchain_file_rel

# Where to find a toolchain and what properties it has
DISTROS = {
    "SW39-347E": ToolchainInfo("https://XXX.de/repository/device-firmware/DEUTA/DMI%20Image/Linux%204.14.108-ti-rt-r130/released/SW39-347E/sdk/arm-buildroot-linux-gnueabihf_sdk-buildroot.tar.gz",
                                "arm-buildroot-linux-gnueabihf/sysroot",
                                "share/buildroot/toolchainfile.cmake"
                               ),
}

class DeutaToolchain(ConanFile):
    name = "deuta-toolchain"
    version = "1.0.0"

    license = "GPL-3.0-only"
    homepage = "https://git.XXX.de/shared-tools/deuta-packaging"
    description = "Conan package for different toolchains."
    settings = "os", "arch"
    package_type = "application"

    def validate(self):
        if not hasattr(self.settings, "os"):
            raise ConanInvalidConfiguration("'settings.os' needs to be specified")
        if not hasattr(self.settings.os, "distro"):
            raise ConanInvalidConfiguration("'settings.os.distro' needs to be specified")

        if self.settings.os == "Linux" or self.settings.os == "Windows":
            if not hasattr(self.settings.os, 'distro'):
                raise ConanInvalidConfiguration("'settings.os.distro' needs to be specified")
        else:
            raise ConanInvalidConfiguration("'settings.os' must be Linux or Windows")

        if self.settings.os != "Linux":
            raise ConanInvalidConfiguration(f"This toolchain is not compatible with {self.settings.os}-{self.settings.arch}. "
                                            "It can only run on Linux-x86_64.")

        distro = str(self.settings.os.distro);

        if distro not in DISTROS:
            raise ConanInvalidConfiguration(f"Cross toolchain not available for {distro}")



    def source(self):
        pass


    def build(self):
        self.output.info(f"os={self.settings.os}")
        if hasattr(self.settings.os, "distro"):
            self.output.info(f"os.distro={self.settings.os.distro}")
        else:
            self.output.info("no os.distro available")

        toolchain = DISTROS[str(self.settings.os.distro)]

        tgz_url = toolchain.url

        get(self, tgz_url, destination=self.source_folder, strip_root=True, verify=False)

        # Run the relocate script if it exists
        relocate_script = os.path.join(self.source_folder, "relocate-sdk.sh")
        if os.path.exists(relocate_script):
            # Run the relocate script
            self.run(f"cd {self.source_folder} && ./relocate-sdk.sh")


    def package(self):
        # Copy everything to the package folder
        copy(self, "*",
             src=self.source_folder,
             dst=self.package_folder,
             keep_path=True)


    def package_id(self):
        # For toolchain packages, we care about target architecture and specific Linux distro/SDK
        # Remove settings that don't affect the toolchain binary compatibility

        self.info.settings.rm_safe("compiler")
        self.info.settings.rm_safe("build_type")
        # Keep arch as toolchains are architecture-specific
        # Keep os.distro as it encodes the concrete Linux image/SDK
        # Remove other os sub-settings that aren't relevant
        if hasattr(self.info.settings, "os"):
            self.info.settings.os.rm_safe("version")

    def package_info(self):
        distro = str(self.settings.os.distro)
        if distro not in DISTROS:
            raise ConanInvalidConfiguration(f"No toolchain info for distro {distro}")

        toolchain = DISTROS[str(self.settings.os.distro)]

        # Add the bin directory to make executables available
        self.cpp_info.bindirs.append("bin")

        # Set the CMake toolchain file location
        cmake_toolchain_path = os.path.join(self.package_folder, toolchain.toolchain_file_rel)

        self.conf_info.define("tools.cmake.cmaketoolchain:toolchain_file", cmake_toolchain_path)

        # Set environment variables for the toolchain
        self.buildenv_info.define_path("TOOLCHAIN_ROOT", self.package_folder)

    def deploy(self):
        toolchain = DISTROS[str()]
        # Deploy to the specific target directory
        target_path = f"/opt/deuta/{self.settings.os.distro}/sdk"

        # Ensure target directory exists
        os.makedirs(target_path, exist_ok=True)

        # Copy package contents to target location
        copy(self, "*",
             src=self.package_folder,
             dst=target_path,
             keep_path=True)

my toolchain/test_package/conanfile.py:

from io import StringIO
from conan import ConanFile
from conan.errors import ConanInvalidConfiguration
from conan.tools.cmake import CMake, cmake_layout
import os


class TestPackageConan(ConanFile):
    settings = "os", "arch", "compiler", "build_type"
    generators = "CMakeToolchain", "VirtualBuildEnv"

    def validate(self):
        if self.settings.os == "Linux":
            if not hasattr(self.settings.os, "distro"):
                raise ConanInvalidConfiguration("'settings.os.distro' must be defined")

    def configure(self):
        if not hasattr(self.settings.os, "distro"):
            raise ConanInvalidConfiguration("'settings.os.distro' must be defined")
        if self.settings.os == "Linux":
            _ = self.settings.os.distro  # touch the subsetting
        if self.settings.os == "Windows":
            _ = self.settings.os.distro

    def build_requirements(self):
        if not hasattr(self.settings.os, "distro"):
            raise ConanInvalidConfiguration("'settings.os.distro' must be defined")
        distro = str(self.settings.os.distro)
        self.output.info(
            f"Using toolchain build-require: {self.tested_reference_str} "
            f"for {self.settings.os}-{distro}-{self.settings.arch}"
        )
        self.tool_requires(self.tested_reference_str)


    def layout(self):
        cmake_layout(self)

    def build(self):
        if not hasattr(self.settings.os, "distro"):
            raise ConanInvalidConfiguration("'settings.os.distro' must be defined")
        distro = str(self.settings.os.distro)
        self.output.info(f"Building test_package for {self.settings.os}-{distro}-{self.settings.arch}")

        cmake = CMake(self)
        cmake.configure()
        cmake.build()

    def test(self):
        if not hasattr(self.settings.os, "distro"):
            raise ConanInvalidConfiguration("'settings.os.distro' must be defined")
        if self.settings.arch in ["armv6", "armv7", "armv7hf"]:
            toolchain = "arm-none-linux-gnueabihf"
        else:
            toolchain = "aarch64-none-linux-gnu"
        self.run(f"{toolchain}-gcc --version")
        test_file = os.path.join(self.cpp.build.bindirs[0], "test_package")
        stdout = StringIO()
        self.run(f"file {test_file}", stdout=stdout)
        if toolchain == "aarch64-none-linux-gnu":
            assert "ELF 64-bit" in stdout.getvalue()
        else:
            assert "ELF 32-bit" in stdout.getvalue()

Aim is to set up something similair to https://docs.conan.io/2/examples/cross_build/toolchain_packages.html
adapted to our toolchain, and our aim to have os.distro as the primary key to identify cross targets.

Have you read the CONTRIBUTING guide?

• [X ] I've read the CONTRIBUTING guide

[memsharded]

Hi @hcw70

Thanks for your question.

It seems the main issue is that you are creating a tool, something aimed to be used as a tool_require.
Your test_package is clearly testing it as self.tool_requires(self.tested_reference_str), which is good.

But the problem seems to be that you are defining the os.distro only in your "host" profile.
Recall that in Conan there are 2 profiles, the "build" profile for the "build" context, where the tools live (as they need to run in the current build machine), and the "host" context, where the final produced packages will run (that can be different than the "build" context, for example when crosss-compiling). You can read more about the 2 profiles in https://docs.conan.io/2/tutorial/consuming_packages/cross_building_with_conan.html

Now, if you defined in your settings_user.yml something like:

distro: [Ubuntu-22.04, Ubuntu-24.04, SW39-347E, SW39-373A, SW39-382A, SW39-388A]

it means that the os.distro will always be mandatory for any package, no matter if it is a tool or a regular library.
Imagine just a cmake/version Conan package, that means that it will require a different binary for every one of those distros, even if it is just a build tool.

There are 2 possibilities here:

• Build tools dont depend on the os.distro

You don't really care about the os.distro for build tools like cmake/version Conan package. Then you need to introduce the distro: [null, Ubuntu-22.04, ...] in the settings to determine that it is ok to not define it.
Then, the "host" profile will define it, but the "build" profile will not.

If you still need your toolchain tool_requires package to model its package_id after the "host" binary that will be produced, you need to introduce the self.settings_target.os.distro into the package_id, like what the recipe for the toolchain in the https://docs.conan.io/2/examples/cross_build/toolchain_packages.html example is doing:

    def package_id(self):
        self.info.settings_target = self.settings_target
        ...

• Build tools depend on the os.distro

In this case, what is missing is to define the os.distro in the "build" profile.

Please let me know if this helps, and if you have any other question. Thanks!

[hcw70]

@memsharded Thanks for the quick answer.

I now have added "null" to the os.distro setting and added the
self.info.settings_target = self.settings_target
to the package id.

which now looks like:

    def package_id(self):
        # For toolchain packages, we care about target architecture and specific Linux distro/SDK
        # Remove settings that don't affect the toolchain binary compatibility
        self.info.settings_target = self.settings_target

        self.info.settings.rm_safe("compiler")
        self.info.settings.rm_safe("build_type")
        # Keep arch as toolchains are architecture-specific
        # Keep os.distro as it encodes the concrete Linux image/SDK
        # Remove other os sub-settings that aren't relevant
        if hasattr(self.info.settings, "os"):
            self.info.settings.os.rm_safe("version")

Now i invoke

conan create -s os=Linux -s os.distro=SW39-347E -s arch=x86_64 -pr:b=default -pr:h=SW39-347E .
where profile SW39-347E sets os.distro to SW39-347E.

but now i get:

======== Launching test_package ========

======== Computing dependency graph ========
deuta-toolchain/1.0.0 (test package): Using toolchain build-require: deuta-toolchain/1.0.0#bf4f8988abe6c5ebf243655987a2c62b%1757496647.5956788 for Linux-SW39-347E-x86_64
Graph root
    deuta-toolchain/1.0.0 (test package): /home/hcw/work/Deuta/deuta-packaging/conan-recipies/toolchain/test_package/conanfile.py
Build requirements
    deuta-toolchain/1.0.0#bf4f8988abe6c5ebf243655987a2c62b - Cache

======== Computing necessary packages ========
Connecting to remote 'deuta-dmi-base' with user 'wintjenh'
Connecting to remote 'deuta-dmi-base-dev' with user 'wintjenh'
Build requirements
    deuta-toolchain/1.0.0#bf4f8988abe6c5ebf243655987a2c62b:5b85c747fcc0c9c6c62bfc0f5743ea8083476535 - Missing
ERROR: Missing binary: deuta-toolchain/1.0.0:5b85c747fcc0c9c6c62bfc0f5743ea8083476535

deuta-toolchain/1.0.0: WARN: Can't find a 'deuta-toolchain/1.0.0' package binary '5b85c747fcc0c9c6c62bfc0f5743ea8083476535' for the configuration:
[settings]
arch=x86_64
os=Linux
[settings_target]
arch=x86_64
build_type=Release
compiler=gcc
compiler.libcxx=libstdc++11
compiler.version=11
os=Linux
os.distro=SW39-347E

ERROR: Missing prebuilt package for 'deuta-toolchain/1.0.0'. You can try:
    - List all available packages using 'conan list "deuta-toolchain/1.0.0:*" -r=remote'
    - Explain missing binaries: replace 'conan install ...' with 'conan graph explain ...'
    - This is a **test_package** missing binary. You can use --build (for all dependencies) or --build-test (exclusive for 'conan create' building test_package' dependencies) to define what can be built from sources

More Info at 'https://docs.conan.io/2/knowledge/faq.html#error-missing-prebuilt-package'

So i got a step further, and the settings_target look good, but still fails.

[memsharded]

Thanks for the feedback!

yes, let me clarify a bit about the conan create with test_package process.
It is really a 2 step process, that are mostly fully decoupled:

• First the package creation happens, equivalent to a conan export (to put the recipe in the cache) + conan install --requires=mypkg/version --build=mypkg/version to build in the cache
• After the package has been succesfully built in the cache, the test_package functionality is launched, to verify the created package. This is decoupled an equivalent to run a consumer project that uses the project. This is equivalent to running the conan test ... command.

Then, the second step is using the package as tool_requires, but there is nothing in the first step that tells it to build it in the "build" context as a tool, instead of building it in the "host" context. So by default it builds it for the "host" context, and when the test_package wants to use it, it will not find a binary, because it requires a tool_requires for the "build" context.

The solution is to explicitly tell in the conan create that you are creating the package to be a tool-require with:

conan create ... --build-require

That ensures the created package uses the "build" profile as a base to build its binary, then when the test_package kicks, it will find it correctly.

You might ask, why it doesn't take the information directly from the test_package, find there it is a tool_require and then force it to be built as a --build-require?
The reason is that the test_package is optional, it might not exist in many cases, still the package needs to be created for the "build" context. The conan create syntax should be the same and consistent, and create exactly the same binary irrespective of the existence of a test_package folder.

Please let me know if this works.

[hcw70]

Ok, i tried with:

conan create -s os=Linux -s os.distro=SW39-347E -s arch=x86_64 -pr:b=default -pr:h=SW39-347E --build-require .
and now it is able to at least go into the test_package testing, where it does not find the toolchain file, but that i can (hopefully) figure out.

What i did was replace most of the self.settings.os.distro with self.settings_target.os.distro, since that makes more sense,
since that is where the settings apply to. And add the --build-require.

May thanks for your quick and helpful answers!

[memsharded]

Always happy to help, thanks to you for the feedback! 🙂

Don't hesitate to re-open or create new tickets if you need further help or have any other question.

[memsharded]

I am doing PR conan-io/docs#4222 to the docs to try to make this --build-require a bit more visible.