[question] Backing up an offline Build/Sources of a recipe/package #6876
Comments
|
This is a use case that has been going around, and we have been implementing different strategies according to what the users were requesting.
There are users of ConanCenter that wouldn't like the idea that those packages are being built from sources that do not come from the "official" library git repository or homepage, and they would be unhappy that suddenly the packages are built from a copy hosted by ConanCenter. And yes, on the other hand, there is this request that ConanCenter packages should provide a backup mechanism in case the original source code is removed from the internet. IMHO, it is a bit out of scope for a package manager 😅 It is very difficult, not to say impossible to satisfy sometimes completely opposite requirements: some want robustness at any price, while others prefer to use the official sources and be able to have their forks. But I understand this question keeps coming from time to time. We need to improve the way in recipes from ConanCenter are adopted for more enterprise cases. So my suggestion would be:
Wouldn't this be enough for your use case? Please tell me if this helps. We will be discussing this idea of a sources backup when we find some time to study the enterprise adoption of recipes from ConanCenter, but this will take some time. Thanks for the feedback! |
|
I greatly appreciate you taking the time to answer. Conan has become a corner stone for us to start questioning the way we are working, and over coming issues like "we are 100% covered in all scenarios" no matter how (un)reasonable the ask are crucial for us to more forward.
I could not agree more, perhaps a commercial JFrog product could fill this gap? An Artifactory extension to backup sources of sorts? This would certainly have a market.
👍 Never crossed my mind but that would certainly fill part of the demands. The challenge with this is knowing how to get the sources. To my knowledge there's no conan command to download the A: 💯 Yes!
Sadly there is not an easy way to tie a recipe revision to a commit in that fork. It would be very daunting to re-calculate the revision over every commit or at least very time consumming. We absolutely require the rrev, sometimes the packages from a updated recipe are not available right away, other times the updates require a newer version of conan which has not been deployed.
It would be brilliant if this was a command If you could point me to an example where this was done in a recipe, I would love to try out this idea. Just the repo will be enough, I'll dig through it's history. 🚧 I see a few calls to conan get, conan source, conan export... I'll have everything locally. |
I am thinking that the |
|
Thank you very much for taking this onto your plate. Being stuck at home, I have some extra cycles to spare, more then willing to help 😄 |
Nice to know. This wouldn't be a very straightforward issue, but if you are willing to give it a try, that would be very welcome! The heavy part should probably be in conans/client/cmd/export.py, but feel free to ask for guidance any time if you finally try 😄 |
|
Hi @prince-chrismc , I would like to work with you on this feature. Please let me know if you've started anything already. |
|
Alright, so Ive been playing with this a bit and I've come up with this so far. I start with a simple recipe package to do my testing against. I added an option to
I did however hardcode the 'dst' folder since I was not able to figure out how to add a new folder to the your leads were excellent =) |
|
@memsharded , It would be nice if you could take a few minutes when you have the time and share your thoughts on "editing recipes" with the Conan's API. Thanks in advance. |
|
I have had a look to the diff, and I have some feedback:
@solvingj has been also working on this, I suggest to get in touch and share efforts. Thanks very much for your effort! |
|
Hi @prince-chrismc I started looking into this a few days ago and I reviewed your implementation. After speaking with Conan team, it seemed that adding a new function was being considered as an option, so that obviously led me to a very different implementation from what you had. Thanks very much for exploring the approach that you did. It was very resourceful. As @memsharded said, some of the tactics are unsupportable, but it's good to think outside the box and try things when you are new. So, in summary, the new approach just lets you do Let us know if this new implementation will probably work for you based on that info. |
|
Thank you very much for taking the time to answer. I am glad I broke so many rules. 🤣 I have been trying to figure out a quick solution for my team to use. Which is manually editing the recipes and keeping those in a local repo. Playing in the source also helps with writing recipes for CCI, very worth while. I certainly learnt a lot (considering how tiny this one section is) hopefully I will be more helpful in future work. Checking out the references PR I much better understand
I appreciate the positive feedback! I will take the time to try out the new features to see if I can obtain my goal. Hopefully it an provide some feedback 👍 |
|
if you are not in slack already, i suggest joining. it's a good place to ask a lot of fundamental questions when you're trying to bootstrap your knowledge about a tool and engineer a real-world solution with it at the same time. https://cpplang.now.sh #conan channel |
|
I was starting my work on an external tool, and i discovered Do you think extending a recipe would be a possible solution? |
It is possible to extend a Conanfile class contained in a python_requires with the |
|
Ahhh, thank you for the insight! |
|
We have merged #6943, which provide support for methods. but please read: #6474 (comment) As the changes might be more part of the infrastructure of ConanCenter at the moment (the complexity of the feature, that involves changing server-side and protocols, would be excessive for the value), it is impossible to do more for 1.26, so removing milestone atm. |
|
I agree, meeting this use case is very complex. I believe the work from #6943 is a great improvement. I am sure in the future this issue will be closed but for now removing the 1.26 milestone is good.
I think the only impediment, for many of the related issues, is the absence of settings/options. Without knowing the version there is little option for downloading sources. However with the increased functionality, with the I suspect this might be a V2.0 idea, but a "reusable" recipe would be the continuation of the work done for 1.26. Having the option to "re-use" the source that have already been downloaded or had previously been exported would make an "installed/created" package usable offline. |
|
I have a potential idea. I had forgot about the Conan download cache. If we could build a remote backend for this cache, this could possibly work amazingly. Conan 1.29 is too close, lets try to look into it a bit in next iteration. |
|
I don't see how it can work... Usually build servers don't have internet access it needs to go through artifactory anyways. Another thing with sources it's straight-forward to audit the code. If it's hidden behind an obscure hash it's trickier... |
Well, if it is a backup of the original sources (lets say a github release .zip), it should be backuped somewhere else. If the build servers don't have internet access, how will they download the original sources in the first place? The download cache has all the information, including the URLs, the hashes, etc. Of course that information needs to be part of the remote backend of the download cache. Lets clarify what I understood we are trying to provide here:
|
|
Hi! We are doing some progress related to this feature request. I wanted to share it with you to see if our proposal makes sense or we need to pivot and use a different approach. We think that creating a backup of the sources used by a recipe is probably an enterprise feature, for the same reason you want to build all the binaries from sources, you want to store the sources themselves. In a company, this backup is probably performed only by one privileged user (or the CI) itself: only one upload packages to the server, only one backup the files,... all the developers consume these uploaded packages and these stored files. Our first proposal takes advantage of the download cache (#8211). From the developer point of view:
Usage is totally transparent for the developers. We could add some safety-check to raise if any file cannot be found in the internal remote (not implemented right now). Privileged user (maybe the CI), the one that can upload packages and files to the internal server:
No need to modify recipes at all, sources will be stored in the internal server. We think that requiring this extra step is ok, we think that the backup process can be something explicit, and it is something done only by one person (or a centralized server) with enough privileges and then every user/consumer takes advantage of it. If source files and/or contents need to be audited, then it is ok if the upload is not a built-in feature in the Conan client, and using JFrog CLI is a very convenient way to implement it (handling credentials, other Artifactory related features,...). This should work out-of-the-box with recipes from ConanCenter as they are only downloading tarballs, but this process is not suitable for recipes that are cloning a repository (this will wait for a second iteration). What do you think about this approach? does it satisfy what you have in mind? Is it far away from your scenarios? We are still designing the feature and any feedback is welcome. Thanks! |
|
I really appreciate the community focused approach and I enjoy discussion solutions. ✔️ on the extra step, I'd argue it's better. By breaking up the local development flow from the release/tag workflow (where backups are required). Backups are certainly not something that are required on every single commit. This conceptually matches with what's been deployed, however there's a few technical challenges which we encountered.
I assume Conan can handle the graph since it creates it. From our perspective we are trying to leverage the base lockfile where ever possible but we have not revisited the backups. Does the download cache preserve the file path of the remote sources? |
|
Actually, the download cache does most of the job and it is already working. This download-cache uses a hash algorithm that computes sha256 of url+checksum, the resulting hash is the filename used to store the files in the download-cache-folder. Here, Conan caches everything downloaded from internet: sources, recipe files, package files,... This new internal-server backup/cache will use that same_ hashed_ filenames to look for the artifacts in the internal server. This internal-server-cache will be used only for the #8211 adds one file ( |
|
@jgsogo I think my major disagreement is with your statement:
Even though CI can backup files it's not practical:
Also download cache doesn't address (nicely) case if sources need to be patched. Another thoughts on the subject (based on how we use Conan):
|
This is the detail that I was missing, Recovery procedure would be as follows:
Correct me if I am wrong!
That's very true. It's far more likely we will need to add compiler support to a project in 5-7 years that was not originally compiled and/or supported. I think this is a first step where Conan can actually gather everything locally and reuse it at a later time. How does a consumer "unpack" the |
|
First, answering @prince-chrismc This is not a recovery procedure. Every developer can have Patching sources: Conan workflow doesn't change, Conan will get the sources (from the internal-server or from the internet) and then will apply the patches (patches exported together with the recipe or patches retrieved from the internal-server or from the internet). The implementation of this internal-server-cache is pretty straightforward: Conan will try with the internal-server before going to look for those files on the internet. So, after someone has uploaded the files to the internal-servers and the recipes to your internal Conan remote, you can switch-off the internet and run |
I see that many recipes in cci just apply all patches from the source folder. So in "patching conandata.yml" case it's a matter of dropping more patches. Of course you need to rebuild, but it's fairly straightforward and the recipe itself is untouched (as I understand is the main purpose is to avoid modifying conanfile.py's source section). And can be easily automated (updating yaml doesn't compare to patching python code). |
|
@ytimenkov here there are different topics, I will answer some of them inline:
My main point was that only one person needs to run this backup upload process and everyone else will benefit from it (only download). For sure, there is a different scenario at every different home. If there is some validation process, the backup should be a final stage of it: after we know these sources can be used, we backup them, everyone (CI included) will use them from our internal server and we know that they won't be changed.
If a package is rebuilt and all the sources/artifacts are not downloaded from the internal server (and the recipe hasn't changed), then it's been a problem on the validation+backup process done before: maybe they didn't review all the sources or they forgot to upload them. We might consider a feature in Conan to fail if sources are not found in the internal-server, it is something easy, but not core to this feature. If you feel like it would be useful, say it and we will add it to the backlog (probably implement it right away if approved).
There is nothing different in this scenario: if the patches are retrieved from the internet, they will be cached locally and in the internal-server the same way sources are. This feature is not making a backup of the sources after running the
Nothing changes, Conan won't find the files in the internal-server and will get them from the internet. Probably it is what happens right now when some developer tries the recipe locally.
As said before, this will backup the sources as they are downloaded from the internet, same tarball, same checksum. It doesn't require to modify the recipe or the
When using the feature as a developer, Conan is doing a GET request to whatever is written in the In order to show the full feature, we needed to write something to upload files somewhere. We choose to write it as a JFrog CLI command and use Artifactory (for obvious reasons 😅 ). We can help with other scripts, but it should be easy enough to write something in python/bash/ps1 that iterates local files and does a PUT request to a server (filenames don't change).
Right now (current implementation), during the upload, JFrog CLI command associates each file to the original URL (because Artifactory supports assigning properties to uploaded files), other servers may use other approaches depending on what they support. So, IMHO, your request depends on the capabilities of the server. How to associate a file in the server to a recipe, or to the original URL will be very different if you are using Artifactory or an FTP (or a shared drive), Conan client only need access to these shared resource. I will think about how to do it with Artifactory and JFrog CLI command, but it won't satisfy other scenarios... and it won't be needed to use these feature (it is just something convenient to run a purge). Hope this adds some light to my previous comments, please, keep asking if something is not clear or you think some more details can be added around this feature. It is in a very early stage even if it works, still we need feedback to match expectations. Thanks @ytimenkov ! |
|
I think you meant to say... - This is not a recovery procedure.
+ This is not an intended recovery procedure.🤣 but that's how it's going to be used! Summary of the technical challenge⚡ The legal/contractual obligation to provide support for an LTS is the driving force here, we need the Conan recipes (including patches and sources) in a state where we can edit them in the far future. When consuming from CCI (or any third party remote), the binaries are precompiled and so developers do not need to download the sources (development workflow). Everything is through Artifactory so it's always cached on-site and re-using it is a breeze. What happens when we need to add platform support? What if we need to support a different compiler? What happens if there's a bug?
TL;DRThe linked PR fixes the first of the two bullet points above. How can we guarantee our ability to rebuild package on demand?The ideal scenario would be the outputs of the conan client would be used create new packages. As a fairly new consumer, I hear you guys with the pre-conception
When I originally posted this issue that was not the case. Let's take a step back and explain how we are handling this with non-Conan OSS. Today what we have is cloned/mirrored repository, we have the source code, it's integrated into our legacy build procedure already, we just commit to that repository and push. This is a very simple and elegant solution, we have everything in our hands and it's very easy bug fix third party dependencies. Now with the introduction of Conan, but most specifically CCI, we now only have the binaries. We can not easily edit machine code to apply a bug fix. Moreover, we need to create a new binary, upload that new revision, and specify that one to install and build against. So what we really need is a backup of everything CCI used to create the original package. How can we obtain a copy of everything needed to build a package offline?Just purely with the Conan client. No external workflow. What we currently have:
What's missing:
↪️ I think the PR you pointed to and the
I am looking at this from a "higher" perspective, that all still applies to how Conan works, however for the enterprise to fix a bug in a dependency that is consumed from Conan. There's a separate intentional manual action that needs to be taken. How do we add patches to an existing Conan recipe?We are using Can I edit the files in my Conan cache
The cmake wrapper and patches are missing locally but they can be recovered via the download cache? I suspect the layout of the cache is not inline with the recipe which is why it fails.
This again fails from missing exports Can I clone and reverse lookup the commit from the reciepe revision?Yes. but I regret doing it. That was a while ago... now with the activity we have it's near impossible. Than I can simply do my Can I rebuild a package without needing a copy of CCI?To date. no. 🤷♂️ Separate issue? probably. |
|
I read many times ago about the desire lines and I love them. We can build a new path or we can move the one we were planning to build. 😃 I get two takeaways from your comment:
|
|
@jgsogo I think we did some implicit assumptions: it seems you suggest using cci as remote while I thought to continue using own repo for recipes (we have one similar to cci github) so main goal was to avoid modifying the recipe when adding it to that repo (which we do now). Therefore patching 💡 I also didn't think about sha sums of source tarballs which eliminate tampering. Now I'd like to build on your idea of using conan-center remote.
I think the major benefit of using cci remote directly is that a separate git repo simply not needed (well, maybe as you mentioned recipe can contain a reference to it's original location and revision, but it doesn't play a significant role). So then my workflow would look like:
when promoting (by a tool):
If a patching needed...
🎅🎄 to everyone. |
|
More comments
Happy Christmas! 🎄 |
|
Implemented in #13461 for 2.0.3, closing, feedback welcome! |
|
Hi, sorry for the late notification, but I am evaluating Conan for enterprise usage and I've been thinking a lot about reproducibility, which brought me to this family of issues. I'm thinking about the approach I will recommend for introducing Conan to our enterprise workflow. By the way, Conan is an awesome tool and very powerful, kudos to the team. In general it seems like the approach of using
I'm curious in your professional opinion if this sounds like a valid approach. I see the method implemented using the download cache, but it is not yet documented and I can't quite wrap my mind around how it works from the discussion here. Also, on this page we have:
Which indicates to me that I probably shouldn't use this download cache method yet. In general I'm happy that the Devops guide page exists, and I'm wondering if it would be possible to extend that page with some discussion of this particular issue and the possible approaches to solving it? |
|
Hi @samkearney
Thanks for your kind words! We are about to launch the "backup sources" feature, to automatically store in your own server all the sources downloaded from the internet, I think this is the missing piece you might be looking for. It is already running in ConanCenter in production. I think it will be launched in the following weeks. Maybe the best would be to create a new ticket when this happens, and discuss the details there? Thanks! |
|
@memsharded Thanks for the quick reply! Sounds good, so I'll look for this feature in the Conan release notes and will open a new issue if I still need clarification. |
How could I go about creating an entire backup of the `conan install .. --build` process?
To be more precise, The entirety of the conan cache, including the short_paths location, and the build dependencies.My requirement: In 8 years, be able to bug fix the source code an open-source project we are leveraging, re-compile for a short list of platforms, then use the result of what was generated with conan.
I would like to be able to download the backup, edit the source code run conan commands like build, package, export, upload and end up with the bug fixed package I am able to use.
I am installing almost uniquely packages from conan-center-index, with a few provided from the bincrafters remote (until my PRs get merged 😄 : ).
What I've seen in the docs, there's are a large number of issues with the same message, run a local Artifactory or conan server to store/host the packages. Which is perfect for any real world use case and everyday requirements. It meets my needs as I am able to build and deliver my software even if the internet would be shutoff or more realistically there are service disruptions.
After several weeks, we have not been able to develop an internal tool for our needs. Hopefully someone from the Conan.io team is able to provide some guidance.
The text was updated successfully, but these errors were encountered: