magefile: fix and clean up packaging #7576

axw · 2022-03-18T06:24:28Z

Motivation/summary

On beats main, mage.UseElasticBeatPackaging has been removed. For now we call the lower-level mage.MustUsePackaging directly.

Later when we look at disabling cgo, we should take more control over how and where we build our binaries. Then we can either use mage.UseElasticBeatXPackPackaging, which expects binaries in build rather than x-pack/apm-server/build, or else we should stop using the common Beats package templates.

Some cleanup while we're there:

remove Jaeger standalone config from apm-server.yml rendering
remove unused Mage targets
split Package target instead of using environment variables
stop building/distributing go-daemon, and System V init script

Checklist

Update CHANGELOG.asciidoc
~~- [ ] Update package changelog.yml (only if changes to apmpackage have been made)~~
Documentation has been updated

How to test these changes

Smoke test on RPM and Debian systems (install Deb/RPM, check that APM Server starts via systemd)

Related issues

Closes #7575

On main, mage.UseElasticBeatPackaging has been removed. For now we call the lower-level mage.MustUsePackaging directly. Later when we look at disabling cgo, we should take more control over how and where we build our binaries. Then we can either use mage.UseElasticBeatXPackPackaging, which expects binaries in `build` rather than `x-pack/apm-server/build`, or else we should stop using the common Beats package templates. Some cleanup while we're there: - remove Jaeger standalone config from apm-server.yml rendering - remove unused Mage targets - split Package target instead of using environment variables - stop building/distributing go-daemon, and System V init script

apmmachine · 2022-03-18T06:28:46Z

💚 Build Succeeded

the below badges are clickable and redirect to their specific view in the CI or DOCS

Expand to view the summary

Build stats

Start Time: 2022-03-21T01:43:37.676+0000
Duration: 32 min 16 sec

Test stats 🧪

Test	Results
Failed	0
Passed	3926
Skipped	13
Total	3939

🤖 GitHub comments

To re-run your PR in the CI, just comment with:

/test : Re-trigger the build.
/hey-apm : Run the hey-apm benchmark.
/package : Generate and publish the docker images.
/test windows : Build & tests on Windows.
run elasticsearch-ci/docs : Re-trigger the docs validation. (use unformatted text in the comment!)

stuartnelson3 · 2022-03-18T13:29:20Z

magefile.go

@@ -104,8 +85,6 @@ func shortConfigFileParams() mage.ConfigFileParams {
 		ExtraVars: map[string]interface{}{
 			"elasticsearch_hostport": "localhost:9200",
 			"listen_hostport":        "localhost:" + config.DefaultPort,
-			"jaeger_grpc_hostport":   "localhost:14250",
-			"jaeger_http_hostport":   "localhost:14268",


apmmachine · 2022-03-18T13:31:31Z

💔 Build Failed

the below badges are clickable and redirect to their specific view in the CI or DOCS

Expand to view the summary

Build stats

Start Time: 2022-03-21T01:43:37.676+0000
Duration: 0 min 6 sec

Pipeline error

This error is likely related to the pipeline itself. Please go to the traditional console output here" You will see the error (either a wrong syntax or configuration)

bmorelli25

Does this need to be removed as well? https://github.com/elastic/apm-server/blame/034b72f95ae637f18c31db50f4195abbea55a816/docs/legacy/copied-from-beats/docs/repositories.asciidoc#L171-L177

axw · 2022-03-21T01:34:57Z

@bmorelli25 yes, thanks. Updated.

marclop · 2022-04-04T07:46:58Z

Tested on amazon linux 2 (RPM):

$ curl -sLO https://staging.elastic.co/8.2.0-8ea0c40f/downloads/apm-server/apm-server-8.2.0-x86_64.rpm
$ sudo rpm -i apm-server-8.2.0-x86_64.rpm
$ sudo systemctl start apm-server
$ curl localhost:8200
{
  "build_date": "2022-03-31T16:27:37Z",
  "build_sha": "b500c941ef80634de4eed7ea5c343f69fa12d383",
  "publish_ready": false,
  "version": "8.2.0"
}
$ systemctl status -l apm-server
● apm-server.service - Elastic APM Server
   Loaded: loaded (/usr/lib/systemd/system/apm-server.service; enabled; vendor preset: disabled)
   Active: active (running) since Mon 2022-04-04 07:40:00 UTC; 1min 25s ago
     Docs: https://www.elastic.co/apm
 Main PID: 3553 (apm-server)
   CGroup: /system.slice/apm-server.service
           └─3553 /usr/share/apm-server/bin/apm-server --environment systemd -c /etc/apm-server/apm-server.yml --path.home /usr/share/apm-server --path.config /etc/apm-server --path.data /var/lib/apm-server --path.logs /var/log/apm-server

Apr 04 07:40:00 ip-172-31-18-216.eu-central-1.compute.internal apm-server[3553]: {"log.level":"info","@timestamp":"2022-04-04T07:40:00.558Z","log.logger":"handler","log.origin":{"file.name":"api/mux.go","file.line":128},"message":"Path /firehose added to request handler","service.name":"apm-server","ecs.version":"1.6.0"}
Apr 04 07:40:00 ip-172-31-18-216.eu-central-1.compute.internal apm-server[3553]: {"log.level":"info","@timestamp":"2022-04-04T07:40:00.558Z","log.logger":"beater","log.origin":{"file.name":"beater/server.go","file.line":233},"message":"Starting apm-server [b500c941ef80634de4eed7ea5c343f69fa12d383 built 2022-03-31 16:27:37 +0000 UTC]. Hit CTRL-C to stop it.","service.name":"apm-server","ecs.version":"1.6.0"}
Apr 04 07:40:00 ip-172-31-18-216.eu-central-1.compute.internal apm-server[3553]: {"log.level":"info","@timestamp":"2022-04-04T07:40:00.558Z","log.logger":"beater","log.origin":{"file.name":"beater/http.go","file.line":96},"message":"RUM endpoints disabled.","service.name":"apm-server","ecs.version":"1.6.0"}
Apr 04 07:40:00 ip-172-31-18-216.eu-central-1.compute.internal apm-server[3553]: {"log.level":"info","@timestamp":"2022-04-04T07:40:00.558Z","log.logger":"beater","log.origin":{"file.name":"beater/http.go","file.line":106},"message":"SSL disabled.","service.name":"apm-server","ecs.version":"1.6.0"}
Apr 04 07:40:06 ip-172-31-18-216.eu-central-1.compute.internal apm-server[3553]: {"log.level":"info","@timestamp":"2022-04-04T07:40:06.397Z","log.logger":"request","log.origin":{"file.name":"middleware/log_middleware.go","file.line":61},"message":"request ok","service.name":"apm-server","url.original":"/","http.request.method":"GET","user_agent.original":"curl/7.79.1","source.address":"127.0.0.1","http.request.body.bytes":0,"http.request.id":"feb0033f-cd76-4b63-bcd3-94dd5223093f","event.duration":104862,"http.response.status_code":200,"ecs.version":"1.6.0"}
Apr 04 07:40:15 ip-172-31-18-216.eu-central-1.compute.internal apm-server[3553]: {"log.level":"error","@timestamp":"2022-04-04T07:40:15.562Z","log.logger":"beater","log.origin":{"file.name":"beater/waitready.go","file.line":64},"message":"precondition failed: dial tcp 127.0.0.1:9200: connect: connection refused","service.name":"apm-server","ecs.version":"1.6.0"}
Apr 04 07:40:30 ip-172-31-18-216.eu-central-1.compute.internal apm-server[3553]: {"log.level":"error","@timestamp":"2022-04-04T07:40:30.565Z","log.logger":"beater","log.origin":{"file.name":"beater/waitready.go","file.line":64},"message":"precondition failed: dial tcp 127.0.0.1:9200: connect: connection refused","service.name":"apm-server","ecs.version":"1.6.0"}
Apr 04 07:40:45 ip-172-31-18-216.eu-central-1.compute.internal apm-server[3553]: {"log.level":"error","@timestamp":"2022-04-04T07:40:45.569Z","log.logger":"beater","log.origin":{"file.name":"beater/waitready.go","file.line":64},"message":"precondition failed: dial tcp 127.0.0.1:9200: connect: connection refused","service.name":"apm-server","ecs.version":"1.6.0"}
Apr 04 07:41:00 ip-172-31-18-216.eu-central-1.compute.internal apm-server[3553]: {"log.level":"error","@timestamp":"2022-04-04T07:41:00.574Z","log.logger":"beater","log.origin":{"file.name":"beater/waitready.go","file.line":64},"message":"precondition failed: dial tcp 127.0.0.1:9200: connect: connection refused","service.name":"apm-server","ecs.version":"1.6.0"}
Apr 04 07:41:15 ip-172-31-18-216.eu-central-1.compute.internal apm-server[3553]: {"log.level":"error","@timestamp":"2022-04-04T07:41:15.578Z","log.logger":"beater","log.origin":{"file.name":"beater/waitready.go","file.line":64},"message":"precondition failed: dial tcp 127.0.0.1:9200: connect: connection refused","service.name":"apm-server","ecs.version":"1.6.0"}

Tested on Ubuntu (deb):

$ curl -sLO https://staging.elastic.co/8.2.0-8ea0c40f/downloads/apm-server/apm-server-8.2.0-amd64.deb
$ sudo dpkg -i apm-server-8.2.0-amd64.deb
Selecting previously unselected package apm-server.
(Reading database ... 63895 files and directories currently installed.)
Preparing to unpack apm-server-8.2.0-amd64.deb ...
Unpacking apm-server (8.2.0) ...
Setting up apm-server (8.2.0) ...
$ sudo systemctl start apm-server
$ curl localhost:8200
{
  "build_date": "2022-03-31T16:27:37Z",
  "build_sha": "b500c941ef80634de4eed7ea5c343f69fa12d383",
  "publish_ready": false,
  "version": "8.2.0"
}
$ systemctl status -l apm-server
● apm-server.service - Elastic APM Server
     Loaded: loaded (/lib/systemd/system/apm-server.service; disabled; vendor preset: enabled)
     Active: active (running) since Mon 2022-04-04 07:45:27 UTC; 56s ago
       Docs: https://www.elastic.co/apm
   Main PID: 1451 (apm-server)
      Tasks: 7 (limit: 4693)
     Memory: 27.9M
     CGroup: /system.slice/apm-server.service
             └─1451 /usr/share/apm-server/bin/apm-server --environment systemd -c /etc/apm-server/apm-server.yml --path.home /usr/share/apm-server --path.config /etc/apm-server --path.data /var/lib/apm-server --path.logs /var/log/apm-server

Apr 04 07:45:28 ip-172-31-24-128 apm-server[1451]: {"log.level":"info","@timestamp":"2022-04-04T07:45:28.051Z","log.logger":"handler","log.origin":{"file.name":"api/mux.go","file.line":128},"message":"Path /intake/v2/events added to request handler","service.name":"apm-server","ecs.version":"1.6.0"}
Apr 04 07:45:28 ip-172-31-24-128 apm-server[1451]: {"log.level":"info","@timestamp":"2022-04-04T07:45:28.051Z","log.logger":"handler","log.origin":{"file.name":"api/mux.go","file.line":128},"message":"Path /intake/v2/profile added to request handler","service.name":"apm-server","ecs.version":"1.6.0"}
Apr 04 07:45:28 ip-172-31-24-128 apm-server[1451]: {"log.level":"info","@timestamp":"2022-04-04T07:45:28.051Z","log.logger":"handler","log.origin":{"file.name":"api/mux.go","file.line":128},"message":"Path /firehose added to request handler","service.name":"apm-server","ecs.version":"1.6.0"}
Apr 04 07:45:28 ip-172-31-24-128 apm-server[1451]: {"log.level":"info","@timestamp":"2022-04-04T07:45:28.051Z","log.logger":"beater","log.origin":{"file.name":"beater/server.go","file.line":233},"message":"Starting apm-server [b500c941ef80634de4eed7ea5c343f69fa12d383 built 2022-03-31 16:27:37 +0000 UTC]. Hit CTRL-C to stop it.","service.name":"apm-server","ecs.version":"1.6.0"}
Apr 04 07:45:28 ip-172-31-24-128 apm-server[1451]: {"log.level":"info","@timestamp":"2022-04-04T07:45:28.051Z","log.logger":"beater","log.origin":{"file.name":"beater/http.go","file.line":96},"message":"RUM endpoints disabled.","service.name":"apm-server","ecs.version":"1.6.0"}
Apr 04 07:45:28 ip-172-31-24-128 apm-server[1451]: {"log.level":"info","@timestamp":"2022-04-04T07:45:28.051Z","log.logger":"beater","log.origin":{"file.name":"beater/http.go","file.line":106},"message":"SSL disabled.","service.name":"apm-server","ecs.version":"1.6.0"}
Apr 04 07:45:39 ip-172-31-24-128 apm-server[1451]: {"log.level":"info","@timestamp":"2022-04-04T07:45:39.264Z","log.logger":"request","log.origin":{"file.name":"middleware/log_middleware.go","file.line":61},"message":"request ok","service.name":"apm-server","url.original":"/","http.request.method":"GET","user_agent.original":"curl/7.68.0","source.address":"127.0.0.1","http.request.body.bytes":0,"http.request.id":"3818be97-4b17-46da-9a16-44e9e1f67f90","event.duration":103212,"http.response.status_code":200,"ecs.version":"1.6.0"}
Apr 04 07:45:43 ip-172-31-24-128 apm-server[1451]: {"log.level":"error","@timestamp":"2022-04-04T07:45:43.053Z","log.logger":"beater","log.origin":{"file.name":"beater/waitready.go","file.line":64},"message":"precondition failed: dial tcp 127.0.0.1:9200: connect: connection refused","service.name":"apm-server","ecs.version":"1.6.0"}
Apr 04 07:45:58 ip-172-31-24-128 apm-server[1451]: {"log.level":"error","@timestamp":"2022-04-04T07:45:58.056Z","log.logger":"beater","log.origin":{"file.name":"beater/waitready.go","file.line":64},"message":"precondition failed: dial tcp 127.0.0.1:9200: connect: connection refused","service.name":"apm-server","ecs.version":"1.6.0"}
Apr 04 07:46:13 ip-172-31-24-128 apm-server[1451]: {"log.level":"error","@timestamp":"2022-04-04T07:46:13.061Z","log.logger":"beater","log.origin":{"file.name":"beater/waitready.go","file.line":64},"message":"precondition failed: dial tcp 127.0.0.1:9200: connect: connection refused","service.name":"apm-server","ecs.version":"1.6.0"}

axw added backport-skip Skip notification from the automated backport with mergify v8.2.0 labels Mar 18, 2022

axw force-pushed the mage-fix-packaging branch from 02d039a to ab7c4d4 Compare March 18, 2022 06:27

axw force-pushed the mage-fix-packaging branch from ab7c4d4 to 956f1a3 Compare March 18, 2022 06:27

axw mentioned this pull request Mar 18, 2022

docs: remove use of System V init #7577

Closed

axw marked this pull request as ready for review March 18, 2022 07:40

axw requested a review from a team March 18, 2022 07:41

stuartnelson3 approved these changes Mar 18, 2022

View reviewed changes

Merge branch 'main' into mage-fix-packaging

034b72f

bmorelli25 reviewed Mar 18, 2022

View reviewed changes

bmorelli25 mentioned this pull request Mar 18, 2022

Update repositories.asciidoc elastic/beats#30918

Merged

axw added 2 commits March 21, 2022 09:35

Drop another mention of non-systemd

8e00c2d

Merge branch 'main' into mage-fix-packaging

8d8441d

axw enabled auto-merge (squash) March 21, 2022 01:43

axw merged commit 74facf5 into elastic:main Mar 21, 2022

axw deleted the mage-fix-packaging branch March 21, 2022 02:36

marclop added test-plan-skip test-plan and removed test-plan-skip labels Mar 30, 2022

marclop self-assigned this Apr 1, 2022

marclop added the test-plan-ok label Apr 4, 2022

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

magefile: fix and clean up packaging #7576

magefile: fix and clean up packaging #7576

axw commented Mar 18, 2022 •

edited

Loading

apmmachine commented Mar 18, 2022 •

edited

Loading

Build stats

Test stats 🧪

stuartnelson3 Mar 18, 2022

apmmachine commented Mar 18, 2022 •

edited

Loading

Build stats

bmorelli25 left a comment •

edited

Loading

axw commented Mar 21, 2022

marclop commented Apr 4, 2022

magefile: fix and clean up packaging #7576

magefile: fix and clean up packaging #7576

Conversation

axw commented Mar 18, 2022 • edited Loading

Motivation/summary

Checklist

How to test these changes

Related issues

apmmachine commented Mar 18, 2022 • edited Loading

💚 Build Succeeded

Build stats

Test stats 🧪

🤖 GitHub comments

stuartnelson3 Mar 18, 2022

Choose a reason for hiding this comment

apmmachine commented Mar 18, 2022 • edited Loading

💔 Build Failed

Build stats

Pipeline error

bmorelli25 left a comment • edited Loading

Choose a reason for hiding this comment

axw commented Mar 21, 2022

marclop commented Apr 4, 2022

axw commented Mar 18, 2022 •

edited

Loading

apmmachine commented Mar 18, 2022 •

edited

Loading

apmmachine commented Mar 18, 2022 •

edited

Loading

bmorelli25 left a comment •

edited

Loading