add virtual-libjpeg recipe #14814

SpaceIm · 2022-12-19T11:51:49Z

The recipe aims to take responsibility of libjpeg implementation selection in a dependency graph, so that we can avoid to pollute recipes with inconsistent with_jpeg/jpeg options in conan-center recipes, recipes which also try to force these options in upstream recipes to avoid libjpeg implementation discrepancies, quite ugly.
For example see current mess in opencv 3.x recipe:

conan-center-index/recipes/opencv/3.x/conanfile.py

Line 24 in ac2b0df

"with_jpeg": [False, "libjpeg", "libjpeg-turbo"],

conan-center-index/recipes/opencv/3.x/conanfile.py

Lines 80 to 82 in ac2b0df

    
           self.options["*"].jpeg = self.options.with_jpeg 
        
           self.options["*"].with_libjpeg = self.options.with_jpeg 
        
           self.options["*"].with_jpeg = self.options.with_jpeg

Therefore the next step will be to replace libjpeg/libjpeg-turbo requirement in all conan-center recipes by this one.
Once all recipes updated, it will guarantee consistency of libjpeg implementation in any dependency graph based on CCI recipes.

I've decided to select libjpeg-turbo as default implementation, it's the one used in most package managers.

It's worth noting that find_package(JPEG) or pkg_check_modules(... libjpeg) will work out of the box in downstream recipes, regardless of selected implementation, since libjpeg-turbo generates proper files with CMakeDeps/cmake_find_package & PkgConfigDeps/pkg_config (since #13745). We just need to also fix mozjpeg-turbo recipe (EDIT: fixed in #14825).

TODO:

white list this recipe in several CCI hooks (KB-H010, KB-H012, KB-H014).
test how it behaves in downstream recipes of CCI (some concerns about package id: [question] How to create a virtual like package of libjpeg & libjpeg-turbo? conan#12374)

/cc @memsharded @jcar87 @czoido @danimtb @uilianries @prince-chrismc @SSE4

I've read the contributing guidelines.
I've followed the PEP8 style guides for Python code in the recipes.
I've used the latest Conan client version.
I've tried at least one configuration locally with the conan-center hook activated.

SpaceIm · 2022-12-19T12:18:53Z

So I've tested to update libtiff recipe in order to depend on this virtual-libjpeg recipe, instead of libjpeg or libjpeg-turbo (and also replace option "jpeg": [False, "libjpeg", "libjpeg-turbo"] by "jpeg": [True, False])

These 2 commands lead to the same libtiff package id, it's a serious blocker (it was my main concern explained in conan-io/conan#12374):

conan create . libtiff/4.4.0@ -o virtual-libjpeg:implementation="libjpeg-turbo"
conan create . libtiff/4.4.0@ -o virtual-libjpeg:implementation="libjpeg"

madebr · 2022-12-19T13:59:04Z

recipes/virtual-libjpeg/all/conanfile.py

+        self.requires(f"{self.options.implementation}/{self._impl_version}", transitive_headers=True, transitive_libs=True)
+
+    def package_id(self):
+        self.info.clear()


I think you need to clear everything but options.

You mean as a workaround for #14814 (comment)?

Yes, but after reading it again. I'm not sure anymore.
This virtual recipe must "push" its options to its users.

I see parallels with INTERFACE from cmake.

The main issue is what I've mentioned in conan-io/conan#12374:

I see one main issue with such virtual recipe and current conan behavior: if a recipeA depends on this virtual recipe, the underlying recipe implementation is not anymore a direct dependency of recipeA, so I suspect package id issues.
How to design this virtual recipe so that any recipe having it as a direct dependency would behave as if either libjpeg, libjpeg-turbo or mozjpeg would be considered a direct dependency?

jcar87 · 2022-12-19T14:20:04Z

Hi @SpaceIm - thanks for taking the time to look into this. We have added this to our team backlog for further discussion and will provide feedback here once we determine a way forward.

May I ask: other than the inconsistencies at the recipe level (which are, indeed, an issue) - have we observed concrete issues where a resolved dependency graph ends up pulling both libjpeg and jpeg-turbo, potentially causing issues due to the same symbols being provided by multiple libraries? This would be my priority - ensure that consumers are not exposed to obscure linker errors. If that is indeed addressed, I'd err on the side of providing the fastest jpeg implementation they can get without causing experiencing issues.

Therefore the next step will be to replace libjpeg/libjpeg-turbo requirement in all conan-center recipes by this one.

On the surface of it, I think this would work well if the jpeg library implementations are compatible and replacements of each other - this would potentially ensure that for any resolved dependency graph, there is only one jpeg implementation, avoiding those potential symbol collections I mention above.

However, for the particular case of the jpeg libraries, I'm not sure this cover all cases:

jpeg-turbo provides emulation for libjpeg v8, but not for v9, as explained here. Granted, the one ABI difference in v9 does not seem to be used in the wild - however we would need to be certain that all dependees of libjpeg v9 today are actually compatible with the v8 API, in order for this to be seamless
jpeg-turbo provides a different API as well, the turbojpeg library - which presumably can be explicitly used by consumers as well.

So at the moment I'm not entirely sure, because there are scenarios in which they're not interchangeable. In Ubuntu, I can see that when libjpeg8 is requested, libjpeg turbo is installed, but libjpeg9 is its own, thus limiting this to libjpeg v8 and jpeg turbo. This proposal would go further than that.

As mentioned in my previous reply, we will study this case and decide a way forward with the team! Very interesting case at hand.

conan-center-bot · 2022-12-27T23:06:23Z

Conan v1 pipeline

Failure in build 3 (a2f591f3d413675993c0d9b4acde8f813da7a6c4):

virtual-libjpeg/virtual@:
CI failed to create some packages (All logs)

Logs for packageID 5ab84d6acfe1f23c4fae0ab88f26e3a396351ac9:

[settings]
arch=x86_64
build_type=Release
compiler=gcc
compiler.libcxx=libstdc++11
compiler.version=5
os=Linux

[...]
	arch=x86_64
	arch_build=x86_64
	compiler=gcc
	compiler.version=5
	compiler.libcxx=libstdc++
	build_type=Release
*** You can change them in /home/conan/w/prod/BuildSingleReference/.conan/profiles/default ***
*** Or override with -s compiler='other' -s ...s***


Configuration:
[settings]
arch=x86_64
build_type=Release
compiler=gcc
compiler.libcxx=libstdc++11
compiler.version=5
os=Linux
[options]
[build_requires]
[env]
[conf]
tools.system.package_manager:mode=install
tools.system.package_manager:sudo=True

libjpeg-turbo/2.1.4: Not found in local cache, looking in remotes...
libjpeg-turbo/2.1.4: Trying with 'conan-center'...
Downloading conanmanifest.txt
Downloading conanfile.py
Downloading conan_export.tgz
libjpeg-turbo/2.1.4: Downloaded recipe revision ef733a318868459226ed87029fdbc801
virtual-libjpeg/virtual: Forced build from source
Installing package: virtual-libjpeg/virtual
Requirements
    libjpeg-turbo/2.1.4 from 'conan-center' - Downloaded
    virtual-libjpeg/virtual from local cache - Cache
Packages
    libjpeg-turbo/2.1.4:6c10bf6b14dfe16c17e7dfa730c79932f9505f68 - Download
    virtual-libjpeg/virtual:5ab84d6acfe1f23c4fae0ab88f26e3a396351ac9 - Build

Installing (downloading, building) binaries...
libjpeg-turbo/2.1.4: Retrieving package 6c10bf6b14dfe16c17e7dfa730c79932f9505f68 from remote 'conan-center' 
Downloading conanmanifest.txt
Downloading conaninfo.txt
Downloading conan_package.tgz
libjpeg-turbo/2.1.4: Package installed 6c10bf6b14dfe16c17e7dfa730c79932f9505f68
libjpeg-turbo/2.1.4: Downloaded package revision e014c89ff8b3ca4caa083ab820528cde
[HOOK - conan-center.py] pre_source(): ERROR: [IMMUTABLE SOURCES (KB-H010)] Create a file 'conandata.yml' file with the sources to be downloaded. (https://github.com/conan-io/conan-center-index/blob/master/docs/error_knowledge_base.md#KB-H010) 
[HOOK - conan-center.py] pre_source(): ERROR: [IMMUTABLE SOURCES (KB-H010)] Use 'tools.get(**self.conan_data["sources"]["XXXXX"])' in the source() method to get the sources. (https://github.com/conan-io/conan-center-index/blob/master/docs/error_knowledge_base.md#KB-H010) 
ERROR: [HOOK - conan-center.py] pre_source(): Some checks failed running the hook, check the output

^{Note: To save resources, CI tries to finish as soon as an error is found. For this reason you might find that not all the references have been launched or not all the configurations for a given reference. Also, take into account that we cannot guarantee the order of execution as it depends on CI workload and workers availability.}

Conan v2 pipeline (informative, not required for merge)

Failure in build 3 (a2f591f3d413675993c0d9b4acde8f813da7a6c4):

virtual-libjpeg/virtual@:
CI failed to create some packages (All logs)

Logs for packageID null:

[settings]
arch=x86_64
build_type=Release
compiler=gcc
compiler.libcxx=libstdc++11
compiler.version=11
os=Linux

********************************************************************************
conan install --require=virtual-libjpeg/virtual@#831a4e80846a833b072856a90b497ad8 --build=virtual-libjpeg/virtual -pr:h /home/conan/w/prod-v2/BuildSingleReference/6578/bb11355b-d65a-46d4-b375-55398deabb21/profile_linux_11_libstdcpp11_gcc_release_64..txt -c:h tools.system.package_manager:mode=install -c:h tools.system.package_manager:sudo=True -pr:b /home/conan/w/prod-v2/BuildSingleReference/6578/bb11355b-d65a-46d4-b375-55398deabb21/profile_linux_11_libstdcpp11_gcc_release_64..txt -c:b tools.system.package_manager:mode=install -c:b tools.system.package_manager:sudo=True
********************************************************************************

-------- Input profiles --------
Profile host:
[settings]
arch=x86_64
build_type=Release
compiler=gcc
compiler.libcxx=libstdc++11
compiler.version=11
os=Linux
[conf]
tools.system.package_manager:mode=install
tools.system.package_manager:sudo=True

Profile build:
[settings]
arch=x86_64
build_type=Release
compiler=gcc
compiler.libcxx=libstdc++11
compiler.version=11
os=Linux
[conf]
tools.system.package_manager:mode=install
tools.system.package_manager:sudo=True


-------- Computing dependency graph --------
libjpeg-turbo/2.1.4: Not found in local cache, looking in remotes...
libjpeg-turbo/2.1.4: Checking remote: conan-center-v2
libjpeg-turbo/2.1.4: Checking remote: c3i_PR-v2-14814
Graph root
    virtual
Requirements
    virtual-libjpeg/virtual#831a4e80846a833b072856a90b497ad8 - Cache
Graph error
    Package 'libjpeg-turbo/2.1.4' not resolved: Unable to find 'libjpeg-turbo/2.1.4' in remotes

-------- Computing necessary packages --------
ERROR: Package 'libjpeg-turbo/2.1.4' not resolved: Unable to find 'libjpeg-turbo/2.1.4' in remotes

^{Note: To save resources, CI tries to finish as soon as an error is found. For this reason you might find that not all the references have been launched or not all the configurations for a given reference. Also, take into account that we cannot guarantee the order of execution as it depends on CI workload and workers availability.}

SpaceIm · 2022-12-27T23:09:09Z

May I ask: other than the inconsistencies at the recipe level (which are, indeed, an issue) - have we observed concrete issues where a resolved dependency graph ends up pulling both libjpeg and jpeg-turbo, potentially causing issues due to the same symbols being provided by multiple libraries? This would be my priority - ensure that consumers are not exposed to obscure linker errors. If that is indeed addressed, I'd err on the side of providing the fastest jpeg implementation they can get without causing experiencing issues.

It can't happen, this kind of dependency graph raises, because libjpeg-turbo & mozjpeg have provides = libjpeg. So currently, when users want to use for example libjpeg-turbo instead of libjpeg, they have to select proper option value in all recipes having libjpeg as a direct dependency (assuming all recipes provide such option... otherwise they are stuck). But it's not straightforward, they have to inspect all recipes to find which ones have libjpeg as a direct dependency.

jpeg-turbo provides emulation for libjpeg v8, but not for v9, as explained here. Granted, the one ABI difference in v9 does not seem to be used in the wild - however we would need to be certain that all dependees of libjpeg v9 today are actually compatible with the v8 API, in order for this to be seamless

So at the moment I'm not entirely sure, because there are scenarios in which they're not interchangeable. In Ubuntu, I can see that when libjpeg8 is requested, libjpeg turbo is installed, but libjpeg9 is its own, thus limiting this to libjpeg v8 and jpeg turbo. This proposal would go further than that.

Yes it might be an issue, it needs more investigation. AFAIK it works fine in vcpkg.

jpeg-turbo provides a different API as well, the turbojpeg library - which presumably can be explicitly used by consumers as well.

Not a problem, if someone wants to use turbojpeg library, it can still explicitly add libjpeg-turbo (or mozjpeg) to requirements, it's not mutually exclusive with this virtual recipe, which is only about libjpeg.

stale · 2023-02-02T01:30:45Z

This pull request has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

stale · 2023-03-12T15:31:39Z

This pull request has been automatically closed because it has not had recent activity. Thank you for your contributions.

Nekto89 · 2023-03-31T10:36:20Z

@jcar87 is there already some decision how to proceed? I'm interested in it to be able to use new ITK. It uses zlib-ng instead of zlib, but depends on libraries that use zlib. I want to be able to replace zlib with zlib-ng in compatibility mode.

conan-center-bot added the Failed label Dec 19, 2022

This comment has been minimized.

Sign in to view

SpaceIm mentioned this pull request Dec 19, 2022

White list recipes with virtual version in KB-H010/KB-H012/KB-H014 conan-io/hooks#463

Open

SpaceIm marked this pull request as draft December 19, 2022 12:21

add virtual-libjpeg recipe

a2f591f

SpaceIm force-pushed the new/virtual-libjpeg branch from 0d5e43f to a2f591f Compare December 19, 2022 12:29

This comment has been minimized.

Sign in to view

jcar87 self-assigned this Dec 19, 2022

madebr reviewed Dec 19, 2022

View reviewed changes

SpaceIm mentioned this pull request Dec 27, 2022

[question] Correct way to conditionally depend on libjpeg-turbo or libjpeg #14951

Closed

stale bot added the stale label Feb 2, 2023

stale bot closed this Mar 12, 2023

samuel-emrys mentioned this pull request Apr 23, 2023

[question] Overriding user and channel of a dependency in 2.0 conan-io/conan#13742

Closed

1 task

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

add virtual-libjpeg recipe #14814

add virtual-libjpeg recipe #14814

SpaceIm commented Dec 19, 2022 •

edited

Loading

This comment has been minimized.

SpaceIm commented Dec 19, 2022 •

edited

Loading

This comment has been minimized.

madebr Dec 19, 2022

SpaceIm Dec 19, 2022

madebr Dec 19, 2022

SpaceIm Dec 19, 2022

jcar87 commented Dec 19, 2022 •

edited

Loading

conan-center-bot commented Dec 27, 2022

SpaceIm commented Dec 27, 2022

stale bot commented Feb 2, 2023

stale bot commented Mar 12, 2023

Nekto89 commented Mar 31, 2023

	self.options["*"].jpeg = self.options.with_jpeg
	self.options["*"].with_libjpeg = self.options.with_jpeg
	self.options["*"].with_jpeg = self.options.with_jpeg

add virtual-libjpeg recipe #14814

add virtual-libjpeg recipe #14814

Conversation

SpaceIm commented Dec 19, 2022 • edited Loading

This comment has been minimized.

SpaceIm commented Dec 19, 2022 • edited Loading

This comment has been minimized.

madebr Dec 19, 2022

Choose a reason for hiding this comment

SpaceIm Dec 19, 2022

Choose a reason for hiding this comment

madebr Dec 19, 2022

Choose a reason for hiding this comment

SpaceIm Dec 19, 2022

Choose a reason for hiding this comment

jcar87 commented Dec 19, 2022 • edited Loading

conan-center-bot commented Dec 27, 2022

Conan v1 pipeline

Conan v2 pipeline (informative, not required for merge)

SpaceIm commented Dec 27, 2022

stale bot commented Feb 2, 2023

stale bot commented Mar 12, 2023

Nekto89 commented Mar 31, 2023

SpaceIm commented Dec 19, 2022 •

edited

Loading

SpaceIm commented Dec 19, 2022 •

edited

Loading

jcar87 commented Dec 19, 2022 •

edited

Loading