Add gitlab pipeline for host injection #2941

TonyCTHsu · 2023-06-29T18:44:32Z

What does this PR do?

Add gitlab pipeline for host injection

.gitlab/build-deb-rpm.sh

.gitlab/install_ddtrace_deps.rb

lib-injection/host_inject.rb

ivoanjo · 2023-07-05T14:10:10Z

lib-injection/host_inject.rb

@@ -0,0 +1,97 @@
+return if ENV["DD_TRACE_SKIP_LIB_INJECTION"] == "true"


This file seems like it copies a lot over from auto_inject.rb. Which in this case... may be easier than shipping multiple files.

Nevertheless, I suggest trying to refactor the common bits so that we could keep them in-sync by a simple copy/paste, as right now they're in a similar-but-not-the-same situation, which is a slightly dangerous middle ground to be in, because we can easily introduce bugs when trying to keep them in-sync, or forget to update one of them.

ivoanjo · 2023-07-05T14:12:05Z

.gitlab/build-deb-rpm.sh

@@ -0,0 +1,17 @@
+#!/bin/bash


Is the intention here to build a rpm (or deb) with the gems already installed and the dependencies pre-compiled?

Because that usually means that we need to have variants for every Ruby supported, it's usually problematic to pick a native extension that was compiled for some Ruby version and then use it a different Ruby version.

.gitlab-ci.yml

randomanderson

As far as .gitlab-ci.yml and the bash scripts are concerned, LGTM. I can't comment on the ruby code

.gitlab-ci.yml

.gitlab/Dockerfile-2.7.8

Co-authored-by: Marco Costa <marco.costa@datadoghq.com>

.gitlab/build-deb-rpm.sh

ivoanjo · 2023-07-20T10:28:57Z

lib-injection/README.md

+| Environment| version |
+|---|---|
+| Ruby  | `2.7`, `3.0`, `3.1`, `3.2`|
+| OS    | Debian 10+ |


Are we still building .rpm? Because Debian doesn't use rpm, so it looks like this list is out-of-sync with what we're actually generating?

.gitlab-ci.yml

ivoanjo

Left a few notes!

ivoanjo · 2023-07-21T09:22:13Z

.gitlab-ci.yml

+  image: $RUBY_CUSTOM_IMAGE_BASE/$RUBY_VERSION:current
+  parallel:
+    matrix:
+      - RUBY_VERSION: ["2.7.8", "3.0.6", "3.1.4"]


🤔 Are we missing 3.2.2 here?

(If not, may be worth leaving a comment explaining why?)

ivoanjo · 2023-07-21T09:24:42Z

.gitlab/Dockerfile-2.7.8

@@ -0,0 +1,95 @@
+#
+# NOTE: THIS DOCKERFILE IS GENERATED VIA "apply-templates.sh"


Hey, I don't see this apply-templates.sh anywhere -- mind adding a note/link to where it can be found?

ivoanjo · 2023-07-21T09:41:56Z

.gitlab/install_ddtrace_deps.rb

+FileUtils.cd(versioned_path.join("extensions/#{Gem::Platform.local.to_s}"), verbose: true) do
+  FileUtils.ln_sf Gem.extension_api_version, ruby_api_version
+end


Maybe leave here a comment explaining why this is needed? It's very subtle :)

ivoanjo · 2023-07-21T09:42:32Z

lib-injection/README.md

+
+In order to ship `ddtrace` and its dependencies as a pre-install package, we need a few tweaks in our build pipeline.
+
+* Use multiple custom built Ruby images to build native extensions. Those images are based on Debian `buster` to support older distribution and Ruby is compiled as a static library with `--disbale-shared` option which disables the creation of shared libraries (also known as dynamic libraries or DLLs).


Minor:

Suggested change

* Use multiple custom built Ruby images to build native extensions. Those images are based on Debian `buster` to support older distribution and Ruby is compiled as a static library with `--disbale-shared` option which disables the creation of shared libraries (also known as dynamic libraries or DLLs).

* Use multiple custom built Ruby images to build native extensions. Those images are based on Debian `buster` to support older distribution and Ruby is compiled as a static library with `--disable-shared` option which disables the creation of shared libraries (also known as dynamic libraries or DLLs).

ivoanjo · 2023-07-21T09:44:28Z

lib-injection/host_inject.rb

+  unless Bundler::CLI.commands["add"] && Bundler::CLI.commands["add"].options.key?("require")
+    debug_log "[ddtrace] You are currently using Bundler version #{Bundler::VERSION} which is not supported by host injection. Please upgrade >= 2.3, check https://github.com/rubygems/rubygems/blob/master/bundler/CHANGELOG.md#enhancements-31"
+    return
+  end


I'm not sure the link to the changelog adds a lot -- e.g. what do we want customers to learn from that link?

ivoanjo

I gave a final after-merging pass, and added a couple of comments :)

ivoanjo · 2023-07-26T07:55:53Z

.gitlab-ci.yml

+  image: $RUBY_CUSTOM_IMAGE_BASE/$RUBY_VERSION:current
+  parallel:
+    matrix:
+      - RUBY_VERSION: ["2.7.8", "3.0.6", "3.1.4"]


(If not, may be worth leaving a comment explaining why?)

ivoanjo · 2023-07-26T07:56:19Z

.gitlab/Dockerfile-2.7.8

@@ -0,0 +1,91 @@
+# This is copied from official Docker image, but compile Ruby with `--disable-shared` instead.


May be worth leaving a link here to the upstream file, so we can pick up changes in the future?

TonyCTHsu and others added 12 commits June 29, 2023 20:34

Add gitlab pipeline for host injection

2b29cb5

Multi-ruby job sketch

02237e7

fix dependencies

d14c680

add runner tags

59a584e

fix host_inject copy

54520f4

Merge branch 'landerson/multi-ruby' into tonycthsu/gitlab-packaging

26ad459

Install gem dependencies

2caa84a

Remove slim image

11adbde

Fix path for ruby script

faf2c66

Fix ruby invocation

7edbcbd

Remove pry

0ddad29

Use Ruby 3.1 image

7d83048

TonyCTHsu marked this pull request as ready for review July 3, 2023 15:31

TonyCTHsu requested a review from a team July 3, 2023 15:31

marcotc reviewed Jul 4, 2023

View reviewed changes

ivoanjo reviewed Jul 5, 2023

View reviewed changes

randomanderson and others added 3 commits July 5, 2023 16:09

Hack to skip signing

2cc2d82

require rubygems

22979ff

Remove nested vendor

71961bc

TonyCTHsu force-pushed the tonycthsu/gitlab-packaging branch from 6862684 to 71961bc Compare July 6, 2023 11:11

TonyCTHsu added 8 commits July 6, 2023 13:18

Fix lock file path

acf28ed

Add ruby 3.2.2 image and manual image step

98f2578

Fix directory .gitlab

4e00f2f

Add ruby 3.0 and 3.1 images

4cc3c73

Use manual images

5ed201e

Disable system libffi

1e2d17e

Skip profiling extension

559c1e7

Rename directories by removing -static suffix

37503a9

randomanderson reviewed Jul 11, 2023

View reviewed changes

.gitlab-ci.yml Outdated Show resolved Hide resolved

Ensure read permissions

f41c4bb

randomanderson approved these changes Jul 18, 2023

View reviewed changes

chouquette reviewed Jul 20, 2023

View reviewed changes

.gitlab-ci.yml Show resolved Hide resolved

.gitlab-ci.yml Show resolved Hide resolved

.gitlab/Dockerfile-2.7.8 Show resolved Hide resolved

TonyCTHsu and others added 3 commits July 20, 2023 11:39

Update lib-injection/README.md

99c1818

Co-authored-by: Marco Costa <marco.costa@datadoghq.com>

Fix gitlab rules for building manual images

0cbeb7e

Better message

c20f314

ivoanjo reviewed Jul 20, 2023

View reviewed changes

.gitlab/build-deb-rpm.sh Outdated Show resolved Hide resolved

ivoanjo reviewed Jul 20, 2023

View reviewed changes

.gitlab-ci.yml Show resolved Hide resolved

Improve with Rb::Config["ruby_version"]

bc544f0

TonyCTHsu force-pushed the tonycthsu/gitlab-packaging branch 8 times, most recently from ae05c87 to b93d630 Compare July 20, 2023 13:09

Symlink

be71f96

TonyCTHsu force-pushed the tonycthsu/gitlab-packaging branch from b93d630 to be71f96 Compare July 20, 2023 13:10

marcotc approved these changes Jul 20, 2023

View reviewed changes

TonyCTHsu added 2 commits July 21, 2023 11:16

Cleanup ruby script

2719fbc

Use pkg folder

d0e9bd9

ivoanjo reviewed Jul 21, 2023

View reviewed changes

TonyCTHsu added 2 commits July 21, 2023 12:15

More documentation

8dd90f2

Clean up pipeline script

06670f6

TonyCTHsu force-pushed the tonycthsu/gitlab-packaging branch from bcbf3fc to 06670f6 Compare July 21, 2023 10:27

TonyCTHsu merged commit 37d556d into master Jul 24, 2023

TonyCTHsu deleted the tonycthsu/gitlab-packaging branch July 24, 2023 10:40

ivoanjo reviewed Jul 26, 2023

View reviewed changes

TonyCTHsu mentioned this pull request Jul 30, 2023

Bump to version 1.13.0 #3009

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add gitlab pipeline for host injection #2941

Add gitlab pipeline for host injection #2941

TonyCTHsu commented Jun 29, 2023

ivoanjo Jul 5, 2023

ivoanjo Jul 5, 2023

randomanderson left a comment

ivoanjo Jul 20, 2023

ivoanjo left a comment

ivoanjo Jul 21, 2023

ivoanjo Jul 26, 2023

ivoanjo Jul 21, 2023

ivoanjo Jul 21, 2023

ivoanjo Jul 21, 2023

ivoanjo Jul 21, 2023

ivoanjo left a comment

ivoanjo Jul 26, 2023

ivoanjo Jul 26, 2023

		@@ -0,0 +1,97 @@
		return if ENV["DD_TRACE_SKIP_LIB_INJECTION"] == "true"

		@@ -0,0 +1,95 @@
		#
		# NOTE: THIS DOCKERFILE IS GENERATED VIA "apply-templates.sh"


		In order to ship `ddtrace` and its dependencies as a pre-install package, we need a few tweaks in our build pipeline.

		* Use multiple custom built Ruby images to build native extensions. Those images are based on Debian `buster` to support older distribution and Ruby is compiled as a static library with `--disbale-shared` option which disables the creation of shared libraries (also known as dynamic libraries or DLLs).

		@@ -0,0 +1,91 @@
		# This is copied from official Docker image, but compile Ruby with `--disable-shared` instead.

Add gitlab pipeline for host injection #2941

Add gitlab pipeline for host injection #2941

Conversation

TonyCTHsu commented Jun 29, 2023

Choose a reason for hiding this comment

Choose a reason for hiding this comment

randomanderson left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ivoanjo left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ivoanjo left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment