[question] Non-relocatable packages don't work in 2.0. Is this a supported use case?

[samuel-emrys]

What is your question?

Is there a way to force packages to be built directly into their final cache location in conan 2.0? I have a number of non-relocatable packages, and they have ~/.conan2/p/t/<id> rather than ~/.conan2/p/<id> embedded into the files which makes everything non functional because the path with t in it doesn't exist when it comes time to consume the packages.

My assumption is that the t in the path here is shorthand for temporary. What would resolve my problem would be to be able to force all operations in the package(self) method to operate on the ~/.conan2/p/<id> directory rather than ~/.conan2/p/t/<id>.

Have you read the CONTRIBUTING guide?

• I've read the CONTRIBUTING guide

[samuel-emrys]

Relates to #11601

[samuel-emrys]

Looking into this a bit more, I'm not filled with a great deal of confidence that non-relocatable packages will be easily supported without modification to the source/new features to support it. The temporary path is defined here:

conan/conan/internal/cache/cache.py

Lines 56 to 60 in dc6eb51

	@staticmethod
	def _get_tmp_path_pref(pref):
	# The reference will not have revision, but it will be always constant
	h = pref.ref.name[:5] + DataCache._short_hash_path(pref.repr_notime())
	return os.path.join("t", h)

Before being used here

conan/conan/internal/cache/cache.py

Lines 70 to 77 in dc6eb51

	def create_export_recipe_layout(self, ref: RecipeReference):
	# This is a temporary layout while exporting a new recipe, because the revision is not
	# computed yet, until it is. The entry is not added to DB, just a temp folder is created
	assert ref.revision is None, "Recipe revision should be None"
	assert ref.timestamp is None
	reference_path = self._get_tmp_path(ref)
	self._create_path(reference_path)
	return RecipeLayout(ref, os.path.join(self.base_folder, reference_path))

And then ultimately finds its way to be used here

conan/conans/client/cmd/export.py

Line 31 in dc6eb51

recipe_layout = cache.create_export_recipe_layout(ref)

@memsharded is this something that you see could be supported going forward? This is a reasonably critical function for python virtualenvs, if they are to be installed into the cache. From memory this also applies to a number of gnu packages that embed their installation directory into the compiled binary, which isn't easily circumvented.

Any other alternatives you might suggest would also be appreciated.

[samuel-emrys]

From a UX perspective, perhaps this would be useful to define as a new package_type - non-relocatable, which sets build_policy="missing" and upload_policy="skip"?

Is it currently possible to have a tuple of package types?

class Pkg(ConanFile):
    settings = "os"
    package_type = ["executable", "non-relocatable"]

[memsharded]

Hi @samuel-emrys

Thanks for your question:
This is how it works in the cache in 2.0:

• things are built and packaged first in a "temporary" t folder, because the necessary revision is not known yet.
• The revision is necessary to be able to compute the final, permanent and stable (always the same folder for the same revision) location.
• Packages are immutable, once the revision has been computed, and we know the final location, it is not possible to change anymore the package, as it would break that immutability assumption

The thing is that Conan packages must be relocatable by definition otherwise it is not possible to upload and download them in different machines, or even the same machine, different cache.

For the case of custom relocation of things, the 2 recipes approach in #13169 would probably be the way to go. Were you aware of this?

Regarding the package_type, that is intended for other cases, basically how traits are propagated down the graph, while your use case seems a high level abstraction over the policies, so it would be a different use case.

[koeleck]

Hi @memsharded,

I apologize, but I don't think I understand your proposed solution.
Looking at PR #13169 and issue #5059 , wouldn't both these solutions run into the same problem: The that final package folder path needs to be available in the package method, but the path can only be known after the package method has returned?

As an example:
In your test_relocatable_toolchain.py line 49: Let's say I have to store the final package folder in sdk.txt:

save(self, os.path.join(self.package_folder, "sdk.txt"), f"CUSTOM PATH: {self.package_folder}\n{sdk}")

As I understood it, this would not work, since package_folder will at this point refer to the temporary package folder.
But this is exactly what e.g. the yocto relocation script would do for a toolchain package. So splitting the toolchain into two packages would not solve the issue. Or do I have some general misunderstanding here?

I can see why you chose the new immutable package approach and do see it's benefits, but from my point of view this specific problem is not solvable with the way conan currently works (without resorting to undefined behavior, e.g. performing the relocation in the package_info method).
But maybe the immutable package restrictions could be loosened a bit for packages which have upload_policy = skip set? Could those maybe package directly into the final destination instead of a temporary folder?

[samuel-emrys]

Thanks for the explanation @memsharded, that makes the reasoning behind the new method a bit clearer. To address some of your comments:

The thing is that Conan packages must be relocatable by definition otherwise it is not possible to upload and download them in different machines, or even the same machine, different cache.

I'm not sure that this must be the case though. I think that there might be a perfectly valid use case where a package recipe is provided, but it always must be built from source on the machine it's going to be used on. This is not as desirable as being able to store/share prebuilt binaries, but in lieu of being able to do that, at least some things would be package-able.

For the case of custom relocation of things, the 2 recipes approach in #13169 would probably be the way to go. Were you aware of this?

Thanks for that - I wasn't aware of this. I'm not certain it solves my issue though. Perhaps I'll provide you with a concrete example of my use case, and you can point out what I might be missing.

The immediate use case I have is essentially to support creation of a python virtual environment in the conan cache (the utility of which relates closely to #11601). Python virtual environments, unfortunately, are non-relocatable. The way this manifests is that, for example, if sphinx is installed into the virtual environment, the final result includes a sphinx-build in the bin directory of the package folder, containing something approximating:

#!/home/user/.conan2/p/t/pytho3f5a78aab9a55/p/bin/python
# -*- coding: utf-8 -*-
import re
import sys
from sphinx.cmd.build import main
if __name__ == '__main__':
    sys.argv[0] = re.sub(r'(-script\.pyw|\.exe)?$', '', sys.argv[0])
    sys.exit(main())

Unfortunately, as you can see, the shebang line contains the temporary directory as this is where the virtualenv was created. When attempting to execute sphinx-build, the following can be observed:

/home/user/.conan2/p/pytho78374f2951a04/p/bin/sphinx-build --help
zsh: /home/user/.conan2/p/pytho78374f2951a04/p/bin/sphinx-build: bad interpreter: /home/user/.conan2/p/t/pytho3f5a78aab9a55/p/bin/python: no such file or directory

You can see this in action in this failed CI run for the conan 2 pipeline.

Given the nature of the failure, it's seems unlikely to me that using a second package to copy the contents of the first package to a second package folder would resolve the issue. Even if this operation of moving the contents from base to sdk in your provided solution were to patch the paths inside files in the virtualenv, this operation would copy to a temporary folder, and the paths would be invalidated when moved to the final package_folder, would they not?

It seems to me this would be the case since the whole problem here is that the final package_folder path can't be known until package() has finished executing.

[memsharded]

Thanks for the feedback @koeleck and @samuel-emrys

Now I understand better what you mean, and this would certainly be complicated. It would have been great to have this discussion pre-2.0 launch in one of the alphas or betas, now it will be more challenging to address this, because the way this works in the Conan 2.0 cache is very structural and might need some important changes.

I am marking this for investigation and discussion for 2.0.2 (just the investigation), and hopefully we will be able to provide better insights about this then.

[jandoerntlein]

+1 (voting for this change, since yocto toolchains are not usable this way)

[memsharded]

New proposal in #13833, hopefully 2.0.5

[memsharded]

Fixed in #13833, from Conan 2.0.5, packages built in the local cache will not be relocated to a different folder. Any package built locally, and that includes packages with upload_policy="skip" and build_policy="missing"