Make installing -dev package "just work" and pull in all other dev libraries it needs

[xnox]

Developer impact:

I want to compile and link against curl, i installed curl-dev and nothing works, i even installed openssl-dev and everything still does not work:

diff --git a/scans/curl.yaml.scan b/scans/curl.yaml.scan
index 98c235989..5260fd010 100644
--- a/scans/curl.yaml.scan
+++ b/scans/curl.yaml.scan
@@ -13,6 +13,11 @@ depend = brotli-dev
 depend = libpsl-dev
 depend = nghttp2-dev
 depend = openssl-dev
+depend = pc:libbrotlidec
+depend = pc:libnghttp2
+depend = pc:libpsl
+depend = pc:openssl
+depend = pc:zlib
 depend = so:libcurl.so.4
 provides = cmd:curl-config=8.10.1-r1
 provides = pc:libcurl=8.10.1-r1

With this change curl-dev automatically gains depends on openssl zlib libpsl libnghttp2 libbrotlidec dev packages to ensure that building and linking against curl-dev "just works".

• generateRuntimePkgConfigDeps: only do so for public .pc, not vendored
  The existing neom test case of vendored .pc files verifies this, and
  fails, when this feature is turned on.

  The diff is ugly, but it moves and indents the if generateRuntimePkgConfigDeps{} to be nested in the if isInDir(){}
  first condition for public files.

• Enable pc file dependencies
  Downloaded and extracted 1,030 .pc files from wolfi (base on 24th
  September file listing of all packages from @dustin).

  Executing pkgconf --exist on them revieled that 57 of them declare
  dependencies that do not, in fact exist.

  It is safe to turn pc: depends, as for vast majority of packages they
  will work correctly.

  For the 57 broken ones, there is now test/pkgconf pipeline, and one
  genuinely has to fix those. By either packaging libraries that are
  declared as required, ensuring those dependant libraries ship
  pkgconfig file, or patch/remove said broken pkgconfig file.

  Out of the sample of 1,030 .pc files, the 788 of them do have
  requirements declared. Thus they will need rebuilds to gain the
  correct dependencies. These however are in just 469 binary packages,
  some of which come from the same origin. Thus roughly at most 400 or
  so origins to rebuild.

  Furthermore wolfi pre-submit, install apk check will catch the broken
  -dev packages that cannot be installed.

  With this feature enabled, build log has:

  2024/09/27 23:44:24 INFO scanning for pkg-config data...
  2024/09/27 23:44:24 INFO   found pkg-config libarchive for usr/lib/pkgconfig/libarchive.pc
  ...
  2024/09/27 23:44:24 INFO   found pkg-config dependency (requires private) libcrypto for usr/lib/pkgconfig/libarchive.pc
  ...
  2024/09/27 23:44:24 INFO   runtime:
  2024/09/27 23:44:24 INFO     bzip2-dev
  2024/09/27 23:44:24 INFO     lz4-dev
  ...
  2024/09/27 23:44:24 INFO     pc:libcrypto
  ...
  2024/09/27 23:44:24 INFO     so:libarchive.so.13
  2024/09/27 23:44:24 INFO     xz-dev
  2024/09/27 23:44:24 INFO     zlib-dev
  2024/09/27 23:44:24 INFO     zstd-dev
  2024/09/27 23:44:24 INFO   provides:
  2024/09/27 23:44:24 INFO     pc:libarchive=3.7.6-r0
  

  These extra new dependency generated pc:libcrypto which upon
  installing libarchive-dev allows one to use it straight away for both
  dynamic and static linking.

  Note all of our public pc files already generated provides:

  $ apk info --provides openssl-dev
  openssl-dev-3.3.2-r0 provides:
  pc:libcrypto=3.3.2
  pc:libssl=3.3.2
  pc:openssl=3.3.2
  

[mattmoor]

👨‍🍳 💋

[xnox]

A full rescan (using internal-only mount https://chainguard.engineering/docs/squads/tools/mount-wolfi/ ) shows that these dependencies will be added:

xnox/os@1fad499

All of them exists and are available.

It is not entirely deterministic depends, because there are multiple providers of the same .pc files:

$ git grep -h provides scans/ | cut -d= -f1-2 | grep pc: | sort | uniq -c | grep -v '1 provides'
      2 provides = pc:egl
      2 provides = pc:fontsproto
      2 provides = pc:gdal
      2 provides = pc:gi-docgen
      2 provides = pc:gl
      2 provides = pc:glesv1_cm
      2 provides = pc:glesv2
      2 provides = pc:kadm-client
      2 provides = pc:kadm-server
      2 provides = pc:krb5
      2 provides = pc:krb5-gssapi
      2 provides = pc:libecpg
      2 provides = pc:libecpg_compat
      2 provides = pc:libisoburn-1
      2 provides = pc:libpgtypes
      2 provides = pc:libpq
      2 provides = pc:libtirpc
      2 provides = pc:libudev
      2 provides = pc:mimalloc
      4 provides = pc:pybind11
      2 provides = pc:simdjson
      2 provides = pc:SPIRV-Tools
      2 provides = pc:SPIRV-Tools-shared
      2 provides = pc:tbb
      2 provides = pc:udev
      2 provides = pc:xkbfile
      2 provides = pc:xproto
      2 provides = pc:xscrnsaver

Some are intentional (udev & eudev) but some may need a closer look in case we have naming clashes here. This may need upstream bug reports.

[xnox]

Given analysis in #1522 (comment) I believe this is ready to land, and is safe to do so, and it will simplify "apk add foo-dev" which will pull in all the right "bar-dev baz-dev" that are also required.

[dustinkirkland]

That would be amazing :-)

[dustinkirkland]

I support and agree with the idea, design, and end results. I defer to @jonjohnsonjr and @joshrwolf on the code review of sca.go.

[kaniini]

This design is basically the same as we are doing in Alpine, so it's certainly a valid approach. One thing to be aware of is that you probably want to also add virtual dependencies for CMake modules, since CMake decided to NIH the pkg-config concept...

[smoser]

This LGTM.

[xnox]

This design is basically the same as we are doing in Alpine, so it's certainly a valid approach. One thing to be aware of is that you probably want to also add virtual dependencies for CMake modules, since CMake decided to NIH the pkg-config concept...

i was going to do cmake modules separately and later; we still have cmake bootstrap cycles and not enough things that can provide cmake modules actually do yet. Also not sure about which way the NIH - i somehow suspect Cmake modules were there first =) however CMake is a programming language, not just declarative files so it definitely is not the most straight forward one.

[xnox]

@kaniini thank you for your code implementation, it was and still is absolutely correct and didn't need any changes, so happy to finally getting around to rolling this out.

[xnox]

There is small fallout from this change, but which is also expected. if generated "depend = pc:foo" has more than one package that provides such pc file, an explicit choice needs to be made as to which one it is desired for a given -dev package to depend on. So far it has been obvious, but it might be a case when any might want to be desired to be supported. In such cases we might need to suppress such dependency.