Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Update github.com/libgit2/git2go to v31.6.1 #222

Merged
merged 7 commits into from
Oct 8, 2021
Merged

Update github.com/libgit2/git2go to v31.6.1 #222

merged 7 commits into from
Oct 8, 2021

Conversation

hiddeco
Copy link
Member

@hiddeco hiddeco commented Sep 28, 2021

This PR updates github.com/libgit2/git2go to v31.6.1 (with libgit2 1.1.1), and changes the container image build process so that it makes use of ghcr.io/fluxcd/golang-with-libgit2.

This image provides a Makefile with specific build instructions to compile libgit2 in such a way that it is linked against OpenSSL and LibSSH2 (without gcrypt) (see rationale and usage for more detailed information).

The linked set of dependencies should solve most known issues around unsupported private key types, but does not resolve the issues with ECDSA* and ED25519 hostkeys yet. Solving this requires a newer version of libgit2 (>=1.2.0), which currently does not seem to work properly with git2go/v32.

In addition, to support the new image and improve the contributor experience, a couple of changes have been made to the various make targets:

  • Detection of the system version of libgit2 is attempted using pkg-config. If this fails, or does not match the version as defined in the Makefile (or configured using LIBGIT2_VERSION=1.x.y), the library is compiled using the instructions from the LIBGIT2_IMG, and installed to REPOSITORY_ROOT/hack/libgit2.
  • Where libgit2 is required as a dependency, LD_LIBRARY_PATH and/or PKG_CONFIG_PATH instructions are added as a prefix to the command. This works for system libraries as well, because the paths are ignored if they do not exist.
  • docker-build is now always making use of docker buildx. It allows configuration over the target platform(s) and additional build arguments using BUILD_PLATFORMS and BUILD_ARGS.

💡 using xx, the build time for source-controller was improved from around 40 minutes to 10 minutes in total.

@hiddeco hiddeco force-pushed the update-libgit2 branch 6 times, most recently from 40f0679 to 84facf8 Compare September 30, 2021 12:28
To help listing the available commands, using the same formatting as
already used in `fluxcd/flux2`.

Signed-off-by: Hidde Beydals <hello@hidde.co>
- Detection of the system version of `libgit2` is attempted using
  `pkg-config`. If this fails, or does not match the version as defined
  in the `Makefile` (or configured using `LIBGIT2_VER=1.x.y`), the
  library is compiled using the instructions from the `BASE_IMG`, and
  installed to `REPOSITORY_ROOT/hack/libgit2`.
- Where `libgit2` is required as a dependency, `LD_LIBRARY_PATH` and/or
  `PKG_CONFIG_PATH` instructions are added as a prefix to the command.
  This works for system libraries as well, because the paths are
  ignored if they do not exist.

Signed-off-by: Hidde Beydals <hello@hidde.co>
@hiddeco hiddeco force-pushed the update-libgit2 branch 2 times, most recently from 601fa92 to cde8c3e Compare October 8, 2021 12:49
@hiddeco hiddeco marked this pull request as ready for review October 8, 2021 12:50
@hiddeco
Copy link
Member Author

hiddeco commented Oct 8, 2021

@squaremo same as with fluxcd/source-controller#437 (comment). Do not like to overrule, but we want to get this out (with the aim to fix at least 1/2 of the transport related issues).

@hiddeco hiddeco changed the title Update github.com/libgit2/git2go to v32 Update github.com/libgit2/git2go to v31.6.1 Oct 8, 2021
This commit updates `github.com/libgit2/git2go` to `v31.6.1` (with
`libgit2` `1.1.1`), and changes the container image build process so
that it makes use of `ghcr.io/hiddeco/golang-with-libgit2`.

This image provides a pre-build dynamic `libgit2` dependency linked
against OpenSSL and LibSSH2 (without gcrypt), and a set of cross-compile
build tools (see
[rationale](https://github.com/hiddeco/golang-with-libgit2#rationale) and
[usage](https://github.com/hiddeco/golang-with-libgit2#usage) for more
detailed information).

The linked set of dependency should solve most known issues around
unsupport private key types, but does not resolve the issues with ECDSA*
and ED25519 hostkeys yet. Solving this requires a newer version of
`libgit2` (`>=1.2.0`), which currently does not seem to work properly
with `git2go/v32`.

Signed-off-by: Hidde Beydals <hello@hidde.co>
This moves the `libgit2` compilation to the image, to ensure it
can be build on builders that aren't backed by AMD64.

The image is structured in such a way that e.g. running nightly
builds targeting a different Go version, or targeting a different
OS vendor would be possible in the future via build arguments.

Signed-off-by: Hidde Beydals <hello@hidde.co>
This ensures the Dockerfile used for testing is making use of the
same scratch image to compile `libgit2` as the actual application
image.

In a future iteration we should restructure our GitHub Action
workflows to re-use the application image, saving us an additional
Dockerfile and a duplicate build. Inspiration for this (which makes
use of a local registry for the duration of the build) can be found
at: https://github.com/fluxcd/golang-with-libgit2/blob/main/.github/workflows/build.yaml

Signed-off-by: Hidde Beydals <hello@hidde.co>
Signed-off-by: Hidde Beydals <hello@hidde.co>
This includes a tiny fix for Darwin to ensure the generated `.pc`
file includes the right paths.

Signed-off-by: Hidde Beydals <hello@hidde.co>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants