ARROW-17692: [R] Add support for building with system AWS SDK C++

[thisisnic]

This PR uses "pkg-config --static ... arrow" to collect build flags. "pkg-config --static ... arrow" reports suitable build flags that depend on build options and used libraries for Apache Arrow C++. This works with the system AWS SDK C++.

[github-actions]

https://issues.apache.org/jira/browse/ARROW-17692

[wjones127]

This seems fine, but I honestly am very confused what BUNDLED_LIBS is supposed to represent (or, why aren't we just adding them to PKG_LIBS?). Maybe it's that these are the dependencies of what is in the bundled libraries, so they need to after when linking?

[kou]

1. We need to add AWS SDK C++ related -lXXXs only when Apache Arrow C++ is built with system AWS SDK C++ (If we can require arrow.pc, we can implement it simply...)
2. We need to add AWS SDK C++ related -lXXXs before -lssl -lcrypto -lcurl because AWS SDK C++ depend on -lssl -lcrypto -lcurl

[nealrichardson]

I think we can use pkg-config to detect what libs are required from the system. In fact, we already do on L187. So is the right fix in the arrow.pc file? (Or probably in the pkg-config file that aws-sdk-cpp produces?)

[kou]

OK. Then I can work on the arrow.pc fix. We need to improve Apache Arrow C++'s CMake codes for it.

(Or probably in the pkg-config file that aws-sdk-cpp produces?)

Unfortunately, AWS SDK C++ doesn't provide .pc files...

[kou]

@github-actions crossbow submit -g r

[kou]

@github-actions crossbow submit -g r

[kou]

@github-actions crossbow submit -g r

[kou]

@github-actions crossbow submit -g r

[kou]

@github-actions crossbow submit -g r

[kou]

@github-actions crossbow submit -g r

[kou]

@github-actions crossbow submit -g r

[kou]

@github-actions crossbow submit -g r

[kou]

@github-actions crossbow submit -g r

[kou]

@github-actions crossbow submit -g r

[kou]

@github-actions crossbow submit -g r

[kou]

@github-actions crossbow submit -g r

[github-actions]

Revision: 2ee02b6

Submitted crossbow builds: ursacomputing/crossbow @ actions-52774bb977

Task	Status
conda-linux-aarch64-cpu-r41
conda-linux-aarch64-cpu-r42
conda-linux-x64-cpu-r41
conda-linux-x64-cpu-r42
conda-osx-arm64-cpu-r41
conda-osx-arm64-cpu-r42
conda-osx-x64-cpu-r41
conda-osx-x64-cpu-r42
conda-win-x64-cpu-r41
homebrew-r-autobrew
homebrew-r-brew
r-binary-packages
test-fedora-r-clang-sanitizer
test-r-arrow-backwards-compatibility
test-r-depsource-bundled
test-r-depsource-system
test-r-dev-duckdb
test-r-devdocs
test-r-gcc-11
test-r-gcc-12
test-r-install-local
test-r-install-local-minsizerel
test-r-library-r-base-latest
test-r-linux-as-cran
test-r-linux-rchk
test-r-linux-valgrind
test-r-minimal-build
test-r-offline-maximal
test-r-offline-minimal
test-r-rhub-debian-gcc-devel-lto-latest
test-r-rhub-debian-gcc-release-custom-ccache
test-r-rhub-ubuntu-gcc-release-latest
test-r-rstudio-r-base-4.1-opensuse153
test-r-rstudio-r-base-4.2-centos7-devtoolset-8
test-r-rstudio-r-base-4.2-focal
test-r-ubuntu-22.04
test-r-versions
test-ubuntu-18.04-r-sanitizer

[kou]

Failures are unrelated because they are happen in nightly CI too.

I merge this.

[ursabot]

Benchmark runs are scheduled for baseline = 89d9fa3 and contender = dca8c07. dca8c07 is a master commit associated with this PR. Results will be available as each benchmark for each run completes.
Conbench compare runs links:
[Finished ⬇️0.0% ⬆️0.0%] ec2-t3-xlarge-us-east-2
[Finished ⬇️0.46% ⬆️0.0%] test-mac-arm
[Finished ⬇️0.0% ⬆️0.0%] ursa-i9-9960x
[Finished ⬇️0.21% ⬆️0.03%] ursa-thinkcentre-m75q
Buildkite builds:
[Finished] dca8c07d ec2-t3-xlarge-us-east-2
[Finished] dca8c07d test-mac-arm
[Finished] dca8c07d ursa-i9-9960x
[Finished] dca8c07d ursa-thinkcentre-m75q
[Finished] 89d9fa3d ec2-t3-xlarge-us-east-2
[Finished] 89d9fa3d test-mac-arm
[Finished] 89d9fa3d ursa-i9-9960x
[Finished] 89d9fa3d ursa-thinkcentre-m75q
Supported benchmarks:
ec2-t3-xlarge-us-east-2: Supported benchmark langs: Python, R. Runs only benchmarks with cloud = True
test-mac-arm: Supported benchmark langs: C++, Python, R
ursa-i9-9960x: Supported benchmark langs: Python, R, JavaScript
ursa-thinkcentre-m75q: Supported benchmark langs: C++, Java

[raulcd]

Are there new dependencies required to build due to this PR? I can see the job for our cookbooks fail if I use the nightlies:
https://github.com/raulcd/arrow-cookbook/actions/runs/3730980375/jobs/6328817362

 /usr/bin/ld: cannot find -lnghttp2: No such file or directory
/usr/bin/ld: cannot find -lidn2: No such file or directory
/usr/bin/ld: cannot find -lrtmp: No such file or directory
/usr/bin/ld: cannot find -lssh: No such file or directory
/usr/bin/ld: cannot find -lpsl: No such file or directory
/usr/bin/ld: cannot find -lgssapi_krb5: No such file or directory
/usr/bin/ld: cannot find -llber: No such file or directory
/usr/bin/ld: cannot find -lldap: No such file or directory
/usr/bin/ld: cannot find -llber: No such file or directory
collect2: error: ld returned 1 exit status

This PR could be related right? This is the workflow definition: https://github.com/apache/arrow-cookbook/blob/main/.github/workflows/test_r_cookbook.yml#L47

[raulcd]

I'll fix it on the cookbooks, I've tested adding the dependencies on a PR I am already working on.

[assignUser]

@raulcd afaik it should also work to add aws_SOURCE=BUNDLED (or what ever the var is called) to force building it.

[kou]

Oh... I didn't notice that these libraries are required with pre-built Apache Arrow C++ binaries...

It seems that I should implement the following:

#14235 (comment)

If we use shared libraries, we only need libcurl-dev. We don't need to install additional packages such as libssh-dev because libcurl-dev depends on needed packages for shared linking.

Could you install https://github.com/apache/arrow/pull/14235/files#diff-8a7b064de580a41c3d6212eeeef79930236caf01b58c73407c84fec929b39260R62-R74 for now?