[question] How to custom generator for conan 2.x #13709

waruqi · 2023-04-17T12:23:10Z

What is your question?

I'm trying to get xmake support to integrate with conan 2.x, but I'm having some problems. (xmake-io/xmake#327)

For conan 1.x, we use a custom Xmake generator https://github.com/xmake-io/conan-xmake_generator

Then we use the conan export command to export it and configure it in conanfile.txt

[generators]
xmake

This works well with a custom generator to export package information.

But for conan 2.0 it doesn't work anymore. I tried writing new generator for conan 2.0

#!/usr/bin/env python
# -*- coding: utf-8 -*-

from conan import ConanFile
from conan.tools.files import save, load
from conan.tools.microsoft import unix_path, VCVars, is_msvc
from conan.errors import ConanInvalidConfiguration
from conan.errors import ConanException
from conans.model.build_info import CppInfo

class xmake(ConanFile):
    name = "xmake_generator"
    version = "2.0.1"
    url = "https://github.com/solvingj/conan-xmake_generator"
    description = "Conan build generator for xmake build system"
    topics = ("conan", "generator", "xmake", "xmake-io")
    homepage = "https://github.com/tboox/xmake"
    author = "ruki <waruqi@gmail.com>"
    license = "MIT"
    exports = ["LICENSE.md"]

    generators = "xmake_generator"

    def __init__(self, conanfile):
        super().__init__(conanfile)
        print("init")

    def generate(self):
        print("generate")

    @property
    def content(self):
        print("content")
        return {}

then run conan export --name xmake_generator /tmp/test/.xmake/linux/x86_64/conan/xmake_generator to export it.

But when I use conan to install package, it will report unknown xmake generator .

/usr/local/bin/conan export --name xmake_generator /tmp/test/.xmake/linux/x86_64/conan/xmake_generator

======== Exporting recipe to the cache ========
init
xmake_generator/2.0.1: Exporting package recipe: /tmp/test/.xmake/linux/x86_64/conan/xmake_generator/conanfile.py
xmake_generator/2.0.1: Copied 1 '.py' file: conanfile.py
xmake_generator/2.0.1: Copied 1 '.md' file: LICENSE.md
xmake_generator/2.0.1: Exported to cache folder: /home/ruki/.conan2/p/xmakea3ba34f62451c/e
xmake_generator/2.0.1: Exported: xmake_generator/2.0.1#c24b63fc7b4f7c16317f17dd7e0e8db9 (2023-04-17 12:14:50 UTC)
generate /tmp/test/build/.conan/fmt/conanfile.txt ..
/usr/local/bin/conan install . --profile:build=profile_build.txt --profile:host=profile_host.txt --build=missing -s os=Linux -s arch=x86_64 -s build_type=Release

======== Input profiles ========
Profile host:
[settings]
arch=x86_64
build_type=Release
compiler=gcc
compiler.cppstd=gnu17
compiler.libcxx=libstdc++11
compiler.version=11
os=Linux

Profile build:
[settings]
arch=x86_64
build_type=Release
compiler=gcc
compiler.cppstd=gnu17
compiler.libcxx=libstdc++11
compiler.version=11
os=Linux


======== Computing dependency graph ========
init
Graph root
    conanfile.txt: /tmp/test/build/.conan/fmt/conanfile.txt
Requirements
    fmt/9.1.0#e747928f85b03f48aaf227ff897d9634 - Cache
Build requirements
    xmake_generator/2.0.1#c24b63fc7b4f7c16317f17dd7e0e8db9 - Cache

======== Computing necessary packages ========
Requirements
    fmt/9.1.0#e747928f85b03f48aaf227ff897d9634:c9399c0423ddb210e7337ba6034ebe67ba8cd038#f1cfae685774fcf5a4c25bdf162542ae - Cache
Build requirements
    xmake_generator/2.0.1#c24b63fc7b4f7c16317f17dd7e0e8db9:da39a3ee5e6b4b0d3255bfef95601890afd80709 - Build

======== Installing packages ========
fmt/9.1.0: Already installed! (1 of 2)

-------- Installing package xmake_generator/2.0.1 (2 of 2) --------
xmake_generator/2.0.1: Building from source
xmake_generator/2.0.1: Package xmake_generator/2.0.1:da39a3ee5e6b4b0d3255bfef95601890afd80709
xmake_generator/2.0.1: Copying sources to build folder
xmake_generator/2.0.1: Building your package in /home/ruki/.conan2/p/t/xmake6187ab40167a8/b
xmake_generator/2.0.1: Writing generators to /home/ruki/.conan2/p/t/xmake6187ab40167a8/b
ERROR: Invalid generator 'xmake'. Available types: CMakeToolchain, CMakeDeps, MesonToolchain, MSBuildDeps, MSBuildToolchain, NMakeToolchain, NMakeDeps, VCVars, QbsProfile, VirtualRunEnv, VirtualBuildEnv, AutotoolsDeps, AutotoolsToolchain, PkgConfigDeps, BazelDeps, BazelToolchain, IntelCC, XcodeDeps, XcodeToolchain, PremakeDeps
error: @programdir/core/sandbox/modules/os.lua:378: execv(/usr/local/bin/conan install . --profile:build=profile_build.txt --profile:host=profile_host.txt --build=missing -s os=Linux -s arch=x86_64 -s build_type=Release) failed(1)

I don't know how to customize the generator and configure the correct generator name in conanfile.txt.

Have you read the CONTRIBUTING guide?

I've read the CONTRIBUTING guide

The text was updated successfully, but these errors were encountered:

memsharded · 2023-04-17T13:04:16Z

Hi @waruqi

Thanks for your question.
In Conan 2.0, the approach for custom generators is using explicit python_requires. It means that it is no longer possible to do a generators = "mycustomgeneratorname", but instead a explicit:

python_requires = "mygenerator/1.0"
def generate(self):
     mygenerator =self.python_requires["mygenerator"].module.MyGenerator(self)
     mygenerator.generate()

This was done because the global registry for custom generators was very problematic, for example, different packages requiring different versions of the generator were simply ovewriting each other, and the last one (pretty random) was the one being used for all packages, leading to total caos. The only possibility to handle this is to do a per-package explicit python_requires versioned inclusion.

If you would like to have a named generator, those have to be built-in in Conan 2.0

waruqi · 2023-04-17T13:33:37Z

In Conan 2.0, the approach for custom generators is using explicit python_requires

I don't really understand how to use this approach? How can I tell conan to call mygenerator/1.0 when I run the conan install fmt command?

If you would like to have a named generator, those have to be built-in in Conan 2.0

If you would consider accepting the new xmake generator as a built-in generator, I would try to open a new pr to do it.

waruqi · 2023-04-18T05:38:50Z

In Conan 2.0, the approach for custom generators is using explicit python_requires

I tried it, it does not work.

from conan import ConanFile
from conan.tools.files import save, load
from conan.tools.microsoft import unix_path, VCVars, is_msvc
from conan.errors import ConanInvalidConfiguration
from conan.errors import ConanException
from conans.model.build_info import CppInfo

class XmakeGenerator:
    common = None
    def __init__(self, conanfile):
        self.conanfile = conanfile
    def generate(self):
        self.conanfile.output.info("VALUE TOOL: {}!!!".format(XmakeGenerator.common.mycommon()))

class xmake(ConanFile):
    name = "xmake_generator"
    version = "2.0.1"
    python_requires = "xmake_generator/2.0.1"
    def generate(self):
        print("generate")
        mygenerator = self.python_requires["xmake_generator"].module.XmakeGenerator(self)
        mygenerator.generate()

/usr/local/bin/conan export --name xmake_generator /tmp/test/.xmake/linux/x86_64/conan/xmake_generator

======== Exporting recipe to the cache ========
xmake_generator/2.0.1: Not found in local cache, looking in remotes...
xmake_generator/2.0.1: Checking remote: conancenter
ERROR: Error loading conanfile at '/tmp/test/.xmake/linux/x86_64/conan/xmake_generator/conanfile.py':
 Cannot resolve python_requires 'xmake_generator/2.0.1': Unable to find 'xmake_generator/2.0.1' in remotes

memsharded · 2023-04-18T07:22:25Z

The generate() is in the consumer recipe of the generator, not in the generator itself, to use it.

In any case, could you please wait a bit, we will be discussing later other possibilities.

waruqi · 2023-04-18T07:31:18Z

The generate() is in the consumer recipe of the generator, not in the generator itself, to use it.

If this is the case, I cannot use conan install fmt to install existing libraries and get their package information, such as links, linkdirs, includedirs, etc. Because I can't modify the conanfile.py of these packages

In any case, could you please wait a bit, we will be discussing later other possibilities.

ok.

System-Arch · 2023-04-18T18:34:46Z

Hi @memsharded,
I'm glad to hear that the use of custom generators is being discussed as I have a different but related use case that seems like it is not addressed with Conan 2.0. More specifically, I am looking to create a custom generator that I can call with any recipe when running "conan install" with the -g option. The full picture is that I want to leverage Conan 2.0 packages in an existing build system by using both a custom deployer (to place shared libraries in shipping areas) and a custom generator to interface with the existing build infrastructure. Thus, it would be nice if I could place my custom generator in the "extensions" folder (say in a subdirectory called "generate") analogous to how I can place my custom deployer in extensions/deploy and have it invoked via the "conan install --deploy MyDeployer" syntax and managed via "conan config install". Has this functionality been considered as today the desired syntax only works with the built-in generators? Thanks

memsharded · 2023-04-18T20:23:15Z

Yes, @System-Arch @waruqi

We have discussed this, I have changed the tag to "feature", it means that we will implement this as global generators in the Conan home extensions folder.

waruqi · 2023-04-18T23:12:37Z

Yes, @System-Arch @waruqi

We have discussed this, I have changed the tag to "feature", it means that we will implement this as global generators in the Conan home extensions folder.

Thanks.

System-Arch · 2023-04-19T13:45:39Z

Hi @memsharded,
That's great! Thank you for the rapid turnaround on this. It will definitely help unblock our Conan 2 adoption efforts. Best wishes

memsharded · 2023-04-20T08:22:09Z

Implemented in #13718, will be released in next 2.0.5, thanks for the feedback!

waruqi · 2023-04-20T08:29:51Z

Implemented in #13718, will be released in next 2.0.5, thanks for the feedback!

Thanks! 👍

waruqi · 2023-05-09T23:18:54Z

Implemented in #13718, will be released in next 2.0.5, thanks for the feedback!

Can I ask when 2.0.5 will probably be released? We have made support for it and are looking forward to the next release. xmake-io/xmake#3703

memsharded · 2023-05-10T07:49:05Z

Very soon, maybe later this week or early next week

waruqi mentioned this issue Apr 17, 2023

Integrate with Conan package manager xmake-io/xmake#327

Closed

memsharded self-assigned this Apr 17, 2023

memsharded added the type: question label Apr 17, 2023

memsharded added the type: look into label Apr 18, 2023

memsharded added this to the 2.0.5 milestone Apr 18, 2023

memsharded added type: feature and removed type: question type: look into labels Apr 18, 2023

waruqi mentioned this issue Apr 18, 2023

conan更新到2之后，报错提示无效参数 xmake-io/xmake#3633

Closed

memsharded mentioned this issue Apr 19, 2023

custom global generators in cache extensions/generators folder #13718

Merged

memsharded closed this as completed in #13718 Apr 20, 2023

waruqi mentioned this issue May 1, 2023

Support conan 2.0 xmake-io/xmake#3703

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[question] How to custom generator for conan 2.x #13709

[question] How to custom generator for conan 2.x #13709

waruqi commented Apr 17, 2023 •

edited

Loading

memsharded commented Apr 17, 2023

waruqi commented Apr 17, 2023

waruqi commented Apr 18, 2023 •

edited

Loading

memsharded commented Apr 18, 2023

waruqi commented Apr 18, 2023 •

edited

Loading

System-Arch commented Apr 18, 2023 •

edited

Loading

memsharded commented Apr 18, 2023

waruqi commented Apr 18, 2023

System-Arch commented Apr 19, 2023

memsharded commented Apr 20, 2023

waruqi commented Apr 20, 2023

waruqi commented May 9, 2023

memsharded commented May 10, 2023

[question] How to custom generator for conan 2.x #13709

[question] How to custom generator for conan 2.x #13709

Comments

waruqi commented Apr 17, 2023 • edited Loading

What is your question?

Have you read the CONTRIBUTING guide?

memsharded commented Apr 17, 2023

waruqi commented Apr 17, 2023

waruqi commented Apr 18, 2023 • edited Loading

memsharded commented Apr 18, 2023

waruqi commented Apr 18, 2023 • edited Loading

System-Arch commented Apr 18, 2023 • edited Loading

memsharded commented Apr 18, 2023

waruqi commented Apr 18, 2023

System-Arch commented Apr 19, 2023

memsharded commented Apr 20, 2023

waruqi commented Apr 20, 2023

waruqi commented May 9, 2023

memsharded commented May 10, 2023

waruqi commented Apr 17, 2023 •

edited

Loading

waruqi commented Apr 18, 2023 •

edited

Loading

waruqi commented Apr 18, 2023 •

edited

Loading

System-Arch commented Apr 18, 2023 •

edited

Loading