[service] Auto publish new packages

[qchateau]

The title is somewhat of a clickbait, but a ton a packages are hosted on GitHub, and many new releases don't require a brain to be released to CCI, it's just about modifying the right files and opening a PR.

A wrote this little script to help with some of these operations: https://github.com/qchateau/conan-center-bot

In the current state it's already useful for package developers to make the release process to CCI a little faster, but the idea behind this could go further. It would not be very hard to automatically create a branch, commit and open a PR.

Right now the main limitations to make this script fully automated are:

• YAML files format is not well-defined. The process of editing it automatically reformats it, creating useless diffs
• Versions can be any string, making it hard to process automatically for some projects
• Github tags need to be translated to CCI versions, sometimes it's trivial, sometimes it is not

I wrote the script to work around these limitations (best effort), which means a human needs to read the diff before committing and opening a PR. For a project with a well written recipe and Github tags that make sense, the human has nothing to modify.

Just my 2 cents: with a little more constraints, a lot of recipes could receive automatic updates from GitHub. Not all recipe need to be compatible with automatic updates straight away, but if we could have multiple level of "recipe conformity", the highest one could provide that kind of features to the library devs.

[SpaceIm]

I don't know if updating a version should be automatized. Usually, even for the simplest libraries, I try to check source code diff between previous and new version to see if there was modifications in the build system, dependencies added or removed etc.

It could be useful for header-only lib without dependencies.

[gocarlos]

One could do the same as the npm dependency bot, it only opens a MR, which can be changed or merged right in

[qchateau]

I agree with both comments, a human supervision is obviously still required to keep a good quality standard concerning recipe and packages. But the CCI maintainers already do this job. A bot could automatically open PRs when a new version is released on github, maybe it needs recipe modification, in which case, the PR must be modified, but maybe it does not and the CCI maintainers can just merge it.

I think it's win-win in both cases:

• If we can merge straight away, simple library updates don't require the developer to spend time releasing it to CCI
• If we can't, a PR is still opened, and anyone can pick it up and complete it. Part of the job has already been done automatically.

I updated my script. It's a bit less dirty and for each targeted recipe, it creates a branch, adds the latest github version to it, commits the change and pushes it.
It's then easy to create a PR on github, do your part by verifying that it's what you expect, and the maintainers will do the rest.

[madebr]

Nice work @qchateau
I would split your python project in 2 parts though:

• check for new releases of upstream projects:
  By checking once every day whether a new release is available, you can create a kind of dashboard/status page/pinned issue where for each recipe is said whether a new release is available vs the current available version on cci.
• automatically create a new commit+branch for a new release + test whether it works

I created conan-recipe-generator which is able to automatically generate a recipe using heuristics. It's far from perfect because it does not prioritize the build systems correctly or does not allow user intervention (using some cli/gui).

[jgsogo]

Hi! We really love to hear about all these initiatives. Thanks! We can consider opening a section in the docs to list all these tools that can help recipe creators/maintainers. 😊

About the script facilitated by @qchateau. From the point of view of CCI, the strategy of polling for new releases can be very time consuming and, IMHO, it wouldn't be as satisfactory as it could be: a bot would open a PR, but no one (except the maintainers) would be able to push to the new branch and, being honest, there are too many things to have a look to and we try to avoid adding more burden.

BUT, it would be amazing to have a Github Action that authors can add to their repositories, the action would create a new PR to conan-center-index when a new release is created in Github. It should be easy to implement, no need for polling, the author can modify the PR if needed,... wdyt?

[qchateau]

I don't quite agree, let me explain:

the strategy of polling for new releases can be very time consuming

It takes 25s of CPU time on my shitty laptop to get the status of all github-based recipes. And that is without even using apis.github.com (I use git ls-remote to avoid the need for a github auth token).

a bot would open a PR, but no one (except the maintainers) would be able to push to the new branch

This is more a workflow problem than a technical one, anyone can use the new branch in his own fork, fix it and open another PR. I agree it creates a lot of PRs, but the tool can target another workflow. As @madebr said, the tool could be split in 2, one half to get the status of recipes, and another half to update the recipes. It would then be easy to have a dashboard of "updateable" recipes, then anyone could use this tool to do an auto-update a recipe that's updateable (human or bot).

it would be amazing to have a Github Action that authors can add to their repositories

I don't agree with this, I mean it would not be a bad thing, but the strength of conan resides in the fact that the library author does not need to know of care about conan. And this is utterly important as conan is not standard.

Then again, a bot cannot auto-update a complex library that release breaking changes. But many releases (patch-level) of many libraries can be auto-updated by a bot, and this is what my tool is about. What I mean is that there are step in the scale of automation, maybe we can help automate some tasks, without pushing it all the way to a fully automated system.

If you dislike bots because you don't want to have auto-generated PRs (which I understand), then the tool can at least perform two useful tasks:

• list the updateable libraries
• help conan contributors update recipes for simple updates

[jgsogo]

We can run and keep updated (daily) an issue like this one: #2232 with updatable recipes, but opening PRs can add too much noise to the repository.

OTH the GH action would help a lot in turning Conan into the standard package manager </marketing-mode-off> 😅

[qchateau]

I've updated my conan-center-bot and added a script called once a day by a github action that updates an issue similarly to #2232

For now it targets qchateau/conan-center-bot#1

[mathbunnyru]

@qchateau funny thing is that I had the same thought about having an utility like this today and I've just found out that you already created such a tool.

For now, I have several questions / suggestions:

0. I do personally think, that the bot is really cool. Great job and an idea! :) I do also think, that it will be a bit difficult in the beginning to merge all the ~100 updates in the beginning. But it will be much easier in the future. I do like the approach with automatic PR creation, but I have a few ideas to make it maintainable.

1. It would be nice to be able to specify version of the recipe. For example, one might want to add the old version of the existing library. I'm not sure there's an utility for that.

2. Can your bot create PR with several versions? For example, there might be several versions released since the last bot call.

3. I think it would be helpful to know, if there already exists the PR trying to update the version of the package. To be able to run the bot twice in a row - the first time it will create many PRs and the second run will tell that PRs are already created.

4. About processing the versions and how to know if the version is newer than the last one in CCI. First, I think there are several heuristics that might make the life much easier (I took a look here: Updatable recipes qchateau/conan-center-bot#1) :

• check the release date (for example, coin-cgl will say, that the "new" release is actually really old). I do think after implementing this the update list might be much shorter and clear.
• check the last commit date for the existing version and for the "new" one (might be time-heavy)
• check that the release is not marked as pre-release (because having all the beta versions and releases is a nightmare, an example is catch2). It will also save some time and make the list shorter, I suppose.
• add some blacklist for recipes (an example is libpcap-1.10.0-bp, it's a branch point, it might be not the beast idea to create a PR for this)

5. All these blacklisting should be avoided when I specify the exact version myself (see point 1)

6. All the filtering should happen with the logging (probably, when verbose is enabled)

7. I would also be glad, if you added the links to the github projects here (Updatable recipes qchateau/conan-center-bot#1) as an additional column to make it easier to browse the releases of the packages.

[qchateau]

I do also think, that it will be a bit difficult in the beginning to merge all the ~100 updates in the beginning.

It's just about updating what needs to be. If CCI can't merge PR as the libraries it hosts are updated, this will become more and more of a problem as the amount of libraries grows. But that's a problem for the JFrog guys

It would be nice to be able to specify version of the recipe.

True, it's probably not much work to add, though there always is the issue of finding the right recipe folder if the recipe does not use "all"

Can your bot create PR with several versions?

Does CCI even allow PRs containing multiple versions ? I think I had a PR rejected by the CI because I was updating multiple versions in a single PR

I think it would be helpful to know, if there already exists the PR trying to update the version of the package.

That's if we want to integrate the bot all the way to opening a PR. @jgsogo argued that it is not the direction they want to go, which I understand.

I admit auto-detection "pending updates" in currently opened PR could be useful in the same way detecting upstream releases is, but it's probably much harder: it requires interfacing with Github (as opposed to git), and probably parsing all PR content to find the diffs to conandata.yml files.

Not impossible, but not easy either.

About processing the versions and how to know if the version is newer than the last one in CCI.

Yes the heuristic is quite basic atm, but it's somewhat of a POC

I would also be glad, if you added the links to the github projects here

Good point

[jgsogo]

I think there are two different approaches here:

• We can update an issue with everything found by the bot, it is easy to do, it doesn't add noise to the repository but it requires a contributor to open the PR manually. IMHO, this should be the first approach.
• The bot can open PRs, it requires more logistics: we should keep track of PRs that were already opened (avoid duplicates and stop submitting PRs that have already failed), probably a sidecar repository to create PR from there (we don't want to create so many branches in this one) and other considerations that were mentioned above.

I really like the idea of opening PRs, if the checks are good enough these PRs will be labeled as bump version and we can merge them right away and automate all the process... but it will take time as there are other priorities right now.

[mathbunnyru]

Does CCI even allow PRs containing multiple versions ?

Yes, it does, the most recent example: https://github.com/conan-io/conan-center-index/pull/3352/files

[qchateau]

I've added options to update a recipe with the version of your choice, improved the heuristic to reduce false positive and added links to the project homepage on the issue.

I've created issues on https://github.com/qchateau/conan-center-bot for the rest

[mathbunnyru]

Great, now the list is much more readable, thanks!

[qchateau]

I've added pending PR detection to the issue reporting the status of the recipes.

I've also opened a mirror issue here #3470