Improve binary re-use to fasten compilation and reduce package size (Conan)

[ymartin59]

Current situation:

• many package depends on same libraries, and for example, even if "borgbackup" depends on "python3" which already includes openssl, package generation for "borgbackup" builds and embeds "openssl" again
• some packages like python provides busybox command line utilities, to avoid any python-based application to rebuild same tools.

Context:

• DSM packages cannot be used the same way as Linux distribution packages, because of Package Center user interaction, many small dependencies would be painful to install/update for users
• binary re-use has been proven possible with large ffmpeg package and its libraries required for "consumer" application like tvheadend, chromaprint, comskip (proper install prefix, RPATH generation)

Problem:

• current binary re-use "concept" (symbolic link to pkgconfig files) requires to keep in same workspace all generated "work-directories" for arch-DSM so that subsequent application update can be compiled and linked on "libraries package".

Proposals/ideas:

• add build time stats in build chain to identify large libraries which impacts the most package compilation time - and investigate direct library linking at runtime. To go back to my first example, borgbackup wheels which require "openssl" should re-use python3 openssl include+lib
• store generated "install/" work directories - ideally online as Conan archive in bintray for instance - so that it can be downloaded back when required later as a dependency (another application or same application update)

[chickenandpork]

Some "build avoidance" strategies from long ago look at trying to determine a "signature" of a built object ("Derived Object", or "DO"). For example: busy box's signature might be a hash of a set of text config files that define the added parts of the build plus the version/arch of the toolchain -- or even the toolchain's signature if available.

If that resulting build exists in the object cache, pull it into place (Clearmake called this "winking in a DO"). This worked fairly well, but the lesson in Clearcase/Clearmake was that a single whitespace change can result in a different signature that ripples down the line, destroying the build-avoidance strategy.

This sort of signature algorithm would require very consistent and repeatable signature logic, even to the point of "beautifying" config files to remove nonessential/whitespace fluctuations. In toolchains in an unrelated project, I remember also having to set the current notion of a date (to granularity of a day) to set internal strings in toolchain and generated kernel identical.

[ymartin59]

As an admin and user of Artifactory, I find Conan package manager suitable for our needs: https://docs.conan.io/en/latest/getting_started.html

[mtonnie]

@ymartin59 and @chickenandpork are you already workin on this topic?
Would be really nice to have such a feature, because we are more flexible with package dependencies.
It would be possible to have more than just one version of an (cross) package.

Would be really nice to see if there is someone working on it.

[mtonnie]

I hade a look to conan and Ig guess it should be possible to use conan to preserve the content of install folder.

But I faced the following question:

• What's the goal, just a local cache or a remote one as well?
• How packages are pushed to remote (automatically or manually)?
• How to include synology architecture to conan?
  • Replace the arch settings from conan (monaco, braswell instead of armv7, ...)?
  • Use the user from repository as architecture, e.g. zlib/1.2.11@monaco/stable?
  • Extend conan with custom property eg. synology?

I guess it's possible to generate a conanfile.txt for each package, something like this:
`[requires]
zlib/1.2.11@some user name/stable

[generators]
deploy

[options]

[imports]
./, * -> ./target`

Then we should be able to restore the content of $(INSTALL_DIR)/$(INSTALL_PREFIX)/$(PKG_NAME)/target

I guess a quick and dirty proof of concept would be nice, at which place would you suggest to insert such a behavior?

[mtonnie]

What are your opinion about including version in dependencies?
Maybe we just reuse the scheme from conan with prefix for directories e.g. cross/zlib/1.2.11 or cross/zlib/1.2.11@myuser/testing.
And if there is no version, current case for all dependencies, we will use the version from Makefile?

For conan we will just remove the directory in front of and for build we will remove everything after he package.
What do you think about this?

For the beginning I would prefer to restrictive caching to cross packages only.
What do you think about this?

How to publish the prebuilt packages to remote?

Who will register a remote on bintray?

Should we crate a branch this change?

[ymartin59]

With Python3 delivery I am desperate to consume time/cpu/energy to endlessly rebuilt same files (openssl, python...) for each arch, for each application package... So it is time to achieve this idea to implement Conan support.
CC @th0ma7 @hgy59 @Safihre @smaarn