Add tools.build:linker_scripts conf variable

[jwillikers]

Changelog: Feature: Adds a conf variable for supplying linker scripts to the linker using the -T flag.
Docs: conan-io/docs#2887

Fixes #9789

Linker scripts are required for bare metal development, i.e. when targeting microcontrollers.
They communicate the memory layout of the chip to the linker since there is no operating system available to handle such details.
Although tools.build:exelinkflags provides a mechanism that could be used to pass this flag, it is not as convenient as a dedicated option for something fundamental to bare metal development nor is it sufficient for use cases requiring more flexibility.
For instance, a project which wants to share a common set of linker flags for a variety of applications targeting the same microcontroller platform may want to modify the linker script on a per-application basis.
Common use cases for this include using a different linker script for a separate bootloader application.
Currently, a project which sets a default linker script in exelinkflags must overwrite all of the existing exelinkflags in order to change the linker script for a particular package.
This is prone to copy and paste errors and makes it more difficult to safely compose profiles.
A dedicated linker_scripts option also makes it possible for Conan to handle any peculiarities related to handling filesystem paths.
This PR includes support for the tools.build:linker_scripts conf variable in the CMakeToolchain, AutotoolsToolchain, and MesonToolchain toolchains.

Here are some notes on compatibility.
This conf option only works for GNU-compatible linkers that support the -T linker script argument which includes ld, gold, lld, and the mingw-w64 ld.
Since Conan doesn't have configuration options for the linker, these linkers correspond to the Clang, GCC, and mingw compilers.
The linker for Clang on Windows is compatible with the MSVC++ linker link.exe, so it does not support this option.
In general, the linker on most Unix systems is likely to support the -T flag.

• Refer to the issue that supports this Pull Request.
• If the issue has missing info, explain the purpose/use case/pain/need that covers this Pull Request.
• I've read the Contributing guide.
• I've followed the PEP8 style guides for Python code.
• I've opened another PR in the Conan docs repo to the develop branch, documenting this one.

^{Note: By default this PR will skip the slower tests and will use a limited set of python versions. Check here how to increase the testing level by writing some tags in the current PR body text.}

[memsharded]

Thanks @jwillikers for your contribution.
I think it seems this could make sense, and seems reasonable.
I'd like to understand a bit better your comments:

Common use cases for this include using a different linker script for a separate bootloader application.
Currently, a project which sets a default linker script in exelinkflags must overwrite all of the existing exelinkflags in order to change the linker script for a particular package.
This is prone to copy and paste errors and makes it more difficult to safely compose profiles.

What do you mean? Wouldn't it be equivalent? Something like:

pkg1*:tools.build:exelinkflags+="-T mybooloader1"
pkg2*:tools.build:exelinkflags+="-T myloaderboot2"
# and 
pkg1*:tools.build:linker_scripts=["mybootloader1"]
pkg2*:tools.build:linker_scripts=["myloaderboot2"]

It would be interesting to know better the pain that this aims to alleviate.

[jwillikers]

@memsharded It would be that simple if the only exelinkflags I had to set were for a single linker script flag. Unfortunately, it's necessary to set several additional linker flags for the particular architecture, standard library, and flags that help reduce the code size. I actually did end up making an extremely difficult to debug copy-paste error due to having to duplicate the rest of the linker flags. Here's a snippet of the current situation I'm describing.

# Common profile used by all applications built for this particular MCU
tools.build:exelinkflags+=["-mthumb", "-mfloat-abi=hard", "-march=armv7e-m+fp", "-ffunction-sections", "-fdata-sections", "-specs=nano.specs", "-specs=nosys.specs", "-Xlinker --gc-sections"]

# Separate profile that inherits the previous one and sets the linker scripts for the different apps

# In the default case, the linker script can just be appended to the inherited linker flags
tools.build:exelinkflags+=["-T/path/to/mcu_flash.ld"]

# Overwrite the linker script for two special apps that use other portions of the flash memory and therefore different linker scripts.
# All other linker flags are the exact same, but we must completely overwrite the existing flags with `=` to avoid the previous linker script being included and duplicating the `-specs` flags.
bootloader/*:tools.build:exelinkerflags=["-mthumb", "-mfloat-abi=hard", "-march=armv7e-m+fp", "-ffunction-sections", "-fdata-sections", "-specs=nano.specs", "-specs=nosys.specs", "-Xlinker --gc-sections", "-T/path/to/bootloader_flash.ld"]
upgrade/*:tools.build:exelinkerflags=["-mthumb", "-mfloat-abi=hard", "-march=armv7e-m+fp", "-ffunction-sections", "-fdata-sections", "-specs=nano.specs", "-specs=nosys.specs", "-Xlinker --gc-sections", "-T/path/to/upgrade_flash.ld"]

By having a separate conf variable for the linker scripts, this becomes:

# Default linker script
tools.build:linker_scripts=["/path/to/mcu_flash.ld"]

bootloader/*:tools.build:linker_scripts=["/path/to/bootloader_flash.ld"]
upgrade/*:tools.build:linker_scripts=["/path/to/upgrade_flash.ld"]

My copy-paste error was related to me duplicating these profiles for a very similar MCU.
For one of the app overrides, I missed updating the architecture flag from "-march=armv7e-m+fp" to "-march=armv7e-m+dp", literally a one letter difference that has very important but non-obvious consequences.
I think the ability to similarly configure / overwrite a linker_scripts conf variable in a project's generate method is also helpful in the common case where a project wants to keep their linker script with the project's source code.

[jwillikers]

I'll ping @kammce since I know he has shown interest in this feature in the past. He might have something to add to the conversation.

[memsharded]

Hi @jwillikers

I think the problem that you are describing is not really happening. I have done a test in #12862 to validate it. These are the profiles:

profile1:

[conf]
user.myteam:myconf=["a", "b", "c"]

profile2:

include(profile1)
[conf]
mypkg*:user.myteam:myconf+=["d"]
mydep*:user.myteam:myconf=+["e"]

Applying it to different packages works fine:

conan install . mypkg/0.1@ -pr=profile2
conanfile.py (mypkg/0.1): MYCONF: ['a', 'b', 'c', 'd']
conan install . mydep/0.1@ -pr=profile2
conanfile.py (mydep/0.1): MYCONF: ['e', 'a', 'b', 'c']

[jwillikers]

@memsharded Yes, your stated case works in my experience as well. My use case is slightly different than that because I have a linker script which is used in the default case for all packages:

# Default linker script
# In the default case, the linker script can just be appended to the inherited linker flags
tools.build:exelinkflags+=["-T/path/to/mcu_flash.ld"]

Later in the profile, I override the default linker script for two specific packages:

bootloader/*:tools.build:exelinkerflags=["-mthumb", "-mfloat-abi=hard", "-march=armv7e-m+fp", "-ffunction-sections", "-fdata-sections", "-specs=nano.specs", "-specs=nosys.specs", "-Xlinker --gc-sections", "-T/path/to/bootloader_flash.ld"]
upgrade/*:tools.build:exelinkerflags=["-mthumb", "-mfloat-abi=hard", "-march=armv7e-m+fp", "-ffunction-sections", "-fdata-sections", "-specs=nano.specs", "-specs=nosys.specs", "-Xlinker --gc-sections", "-T/path/to/upgrade_flash.ld"]

Unfortunately, I can't just append at this point since the default linker script has already been added to tools.build:exelinkflags.

I use a linker script for all cases for convenience and if I remember correctly, it's necessary for test packages to link successfully. For instance, if my package uses fmt, then I think in order for the fmt test_package to link correctly, there has to be a linker script defining the memory layout.

[memsharded]

Ok, perfect, thanks very much @jwillikers, that fully clarifies now the use case.
Lets ask for some extra review, but I'd say this seems doable.

[kammce]

So I'm still learning the different ways to use Conan so this may come off as a dumb question. Is there a way with a conan recipe to produce multiple package targets. Like, installing a package example-mcu-6xxx, which will have multiple microcontrollers in its family. For each MCU in that family, it would be helpful to have a separate library link target and separate tools.build.linker_script's for each. This would be a requirement as a single linker script cannot do when the size of RAM and Flash vary. You'd either use the smallest to be safe or the largest and maybe do some post process checks. But if we have this capability, then that leads to my second question.

Is it a choice that the user can use the conf variable or not? Because there are many times when a user will want to use their own linker script rather than the one handed to them. Most times to segment out sections of RAM or Flash for particular types of data.

So long as we have 1 package multiple subpackages and a way to opt in/out of using the package provided linker script, then this feature would be great. This would definitely help get rid of the copy & paste problem I've been dealing with. If I have a ton of apps using a library, and I want to use the same default linker script for everyone, I have to copy it N number of times

[memsharded]

Hi @kammce

So I'm still learning the different ways to use Conan so this may come off as a dumb question. Is there a way with a conan recipe to produce multiple package targets. Like, installing a package example-mcu-6xxx, which will have multiple microcontrollers in its family. For each MCU in that family, it would be helpful to have a separate library link target and separate tools.build.linker_script's for each. This would be a requirement as a single linker script cannot do when the size of RAM and Flash vary. You'd either use the smallest to be safe or the largest and maybe do some post process checks. But if we have this capability, then that leads to my second question.

I am not sure what you mean exactly, but it sounds that you want to extend the Conan settings.yml (https://docs.conan.io/en/latest/extending/custom_settings.html), to define different micros, as each one might result in different binaries as you describe. Then in the profile you would define the custom setting value, and that will give you a different package_id. This is the "binary model".

Is it a choice that the user can use the conf variable or not? Because there are many times when a user will want to use their own linker script rather than the one handed to them. Most times to segment out sections of RAM or Flash for particular types of data.

I know nothing about the linker scripts, but yes, the conf is defined by the users in the profile. So every user can provide their own linker script.

So long as we have 1 package multiple subpackages and a way to opt in/out of using the package provided linker script, then this feature would be great. This would definitely help get rid of the copy & paste problem I've been dealing with. If I have a ton of apps using a library, and I want to use the same default linker script for everyone, I have to copy it N number of times

I don't fully get it either. Maybe @jwillikers can give some more insights given he has the knowledge both about Conan and these linker scripts for micros.

[jwillikers]

@kammce Conan doesn't really have a concept for subpackages, so you would want to actually create individual builds of such a package for each microcontroller you want to target. I'm imagining your use case would be to create a HAL library like stm32f0xx for various MCU's in the STM32F0 family. You could add settings or options for your package to build for a particular microcontroller in that family, which would allow you to control the linker script used for individual MCU's.

I'm primarily controlling linker scripts through profiles at this point, outside of any package. If you'd like such a HAL library to configure the linker scripts for consumers' projects, they would need to consume it as a tool_requires in order to propagate the linker scripts, and also as a regular requires in order to use the provided library target.

I believe it's possible to opt out or override linker scripts provided through profiles, tool_requires, and the CLI in individual packages in the generate method:

self.conf.define("tools.build:linker_scripts", ["my_linker_script"])

[kammce]

@jwillikers Okay understood. For profiles, this seems like a good idea then. I'll look into settings/options for my driver libraries for setting up linker scripts.

[memsharded]

Sorry, closed by accident by another PR, re-opening