[Build Chromium] Improve git checkout #83225

tsullivan · 2020-11-11T22:50:37Z

Summary

This PR attempts to expand on the guide to building Chromium for Kibana Reporting. It emphasizes the steps to building locally and guides the reader more to understanding the Google build tools for Chromium.

This PR updates the Python scripts for building. The goal is to require the user to run init.py once, and then allow them to run build.py as many times as they want. Running build.py multiple times will not repeat expensive work that happened when running init.py

Some shortcuts were added to the scripts so that same work doesn't get repeated, and avoid syncing any code the working directory repo more than is necessary. For example, instead of fetching the entire git repo when initializing the Chromium code, the script only fetches the code for the commit hash that is already known.

elasticmachine · 2020-11-11T22:51:19Z

Pinging @elastic/kibana-app-arch (Team:AppArch)

elasticmachine · 2020-11-11T22:51:20Z

Pinging @elastic/kibana-reporting-services (Team:Reporting Services)

tsullivan · 2020-12-14T16:20:05Z

@elasticmachine merge upstream

commit e817796d107d554504870cdb8e6dd10db1079a1e Merge: 19e6b300260 61b4e052fdd Author: Timothy Sullivan <tsullivan@elastic.co> Date: Thu Nov 12 13:25:09 2020 -0700 Merge branch 'chore/build_chromium/improvements' of github.com:tsullivan/kibana into chore/build_chromium/improvements commit 19e6b300260822418cea4dc594bdfdf621a331e0 Author: Timothy Sullivan <tsullivan@elastic.co> Date: Thu Nov 12 13:23:01 2020 -0700 fixes commit 9de24ec298792a46e88207f41c9f6aff1997dbf1 Author: Timothy Sullivan <tsullivan@elastic.co> Date: Thu Nov 12 13:22:52 2020 -0700 add instructions to build local commit 872c0f68e0077ad50c541b605a87253befd1a891 Author: Timothy Sullivan <tsullivan@elastic.co> Date: Thu Nov 12 11:58:25 2020 -0700 simplify commit 8dae9484efcdc483ad20ceab32cae4e97735ab1c Author: Timothy Sullivan <tsullivan@elastic.co> Date: Thu Nov 12 10:28:18 2020 -0700 fixes commit 492f5cfe25e2c7ccffce55a8733383a30aae1cb6 Author: Timothy Sullivan <tsullivan@elastic.co> Date: Wed Nov 11 20:15:43 2020 -0700 --wip-- [skip ci] commit acba359b121f7be8e6c353f90e938bc5d88658a1 Author: Timothy Sullivan <tsullivan@elastic.co> Date: Wed Nov 11 15:47:50 2020 -0700 [Build Chromium] Improve git checkout commit 61b4e052fdd04456fe23956ad8ce99c8e771c01e Author: Timothy Sullivan <tsullivan@elastic.co> Date: Wed Nov 11 20:15:43 2020 -0700 --wip-- [skip ci] commit 58300c9deeca15cce838b3956c55c2994f99f530 Author: Timothy Sullivan <tsullivan@elastic.co> Date: Wed Nov 11 15:47:50 2020 -0700 [Build Chromium] Improve git checkout

elasticmachine · 2020-12-14T16:40:12Z

Pinging @elastic/kibana-app-services (Team:AppServices)

tsullivan · 2020-12-14T16:43:33Z

x-pack/build_chromium/README.md


-As of this writing, there is an officially supported headless Chromium build args file for Linux: `build/args/headless.gn`. This does not work on Windows or Mac, so we have taken that as our starting point, and modified it until the Windows / Mac builds succeeded.
+To get a list of the build arguments that are enabled, install `depot_tools` and run


This is a big improvement to the README: how to get the list of build arguments

tsullivan · 2020-12-14T16:44:36Z

x-pack/build_chromium/build.py

+checked_out = runcmdsilent('git checkout build-' + base_version)
+if checked_out != 0:
+  print('Syncing remote version')
+  runcmd('git fetch origin ' + source_version)


This is a big improvement to initializing the working directory: only fetch the source version of the commit ID that we passed into the script

x-pack/build_chromium/README.md

joelgriffith

Minor MD changes, code LGTM. Full-disclaimer: not a python expert.

Co-authored-by: Joel Griffith <joel@joelgriffith.net>

tsullivan · 2020-12-19T00:02:46Z

@elasticmachine merge upstream

tsullivan · 2020-12-22T00:45:19Z

@elasticmachine merge upstream

joelgriffith

LGTM thanks for the python changes!

tsullivan · 2020-12-28T21:48:01Z

@elasticmachine merge upstream

kibanamachine · 2020-12-29T00:00:00Z

💚 Build Succeeded

continuous-integration/kibana-ci/pull-request
Commit: efc012f

Metrics [docs]

Distributable file count

id	before	after	diff
`default`	47267	48027	+760

History

💚 Build #95895 succeeded 8156392
💚 Build #95592 succeeded 935e9e5
💚 Build #94082 succeeded d3d16c4
💚 Build #87782 succeeded 5286b7863af842c3291bd5a122f7a67f5ceb5890
💚 Build #87744 succeeded f3655cec89c9dc0ab782e7f6dda4831fd7a43774

To update your PR or re-run it, just comment with:
@elasticmachine merge upstream

jbudz

py and readme changes LGTM. I haven't run a full build, if you want me to do another loop can do.

* [Build Chromium] Document steps for local build, improve checkout commit e817796d107d554504870cdb8e6dd10db1079a1e Merge: 19e6b300260 61b4e052fdd Author: Timothy Sullivan <tsullivan@elastic.co> Date: Thu Nov 12 13:25:09 2020 -0700 Merge branch 'chore/build_chromium/improvements' of github.com:tsullivan/kibana into chore/build_chromium/improvements commit 19e6b300260822418cea4dc594bdfdf621a331e0 Author: Timothy Sullivan <tsullivan@elastic.co> Date: Thu Nov 12 13:23:01 2020 -0700 fixes commit 9de24ec298792a46e88207f41c9f6aff1997dbf1 Author: Timothy Sullivan <tsullivan@elastic.co> Date: Thu Nov 12 13:22:52 2020 -0700 add instructions to build local commit 872c0f68e0077ad50c541b605a87253befd1a891 Author: Timothy Sullivan <tsullivan@elastic.co> Date: Thu Nov 12 11:58:25 2020 -0700 simplify commit 8dae9484efcdc483ad20ceab32cae4e97735ab1c Author: Timothy Sullivan <tsullivan@elastic.co> Date: Thu Nov 12 10:28:18 2020 -0700 fixes commit 492f5cfe25e2c7ccffce55a8733383a30aae1cb6 Author: Timothy Sullivan <tsullivan@elastic.co> Date: Wed Nov 11 20:15:43 2020 -0700 --wip-- [skip ci] commit acba359b121f7be8e6c353f90e938bc5d88658a1 Author: Timothy Sullivan <tsullivan@elastic.co> Date: Wed Nov 11 15:47:50 2020 -0700 [Build Chromium] Improve git checkout commit 61b4e052fdd04456fe23956ad8ce99c8e771c01e Author: Timothy Sullivan <tsullivan@elastic.co> Date: Wed Nov 11 20:15:43 2020 -0700 --wip-- [skip ci] commit 58300c9deeca15cce838b3956c55c2994f99f530 Author: Timothy Sullivan <tsullivan@elastic.co> Date: Wed Nov 11 15:47:50 2020 -0700 [Build Chromium] Improve git checkout * Apply suggestions from code review Co-authored-by: Joel Griffith <joel@joelgriffith.net> Co-authored-by: Joel Griffith <joel@joelgriffith.net> Co-authored-by: Kibana Machine <42973632+kibanamachine@users.noreply.github.com>

* [Build Chromium] Document steps for local build, improve checkout commit e817796d107d554504870cdb8e6dd10db1079a1e Merge: 19e6b300260 61b4e052fdd Author: Timothy Sullivan <tsullivan@elastic.co> Date: Thu Nov 12 13:25:09 2020 -0700 Merge branch 'chore/build_chromium/improvements' of github.com:tsullivan/kibana into chore/build_chromium/improvements commit 19e6b300260822418cea4dc594bdfdf621a331e0 Author: Timothy Sullivan <tsullivan@elastic.co> Date: Thu Nov 12 13:23:01 2020 -0700 fixes commit 9de24ec298792a46e88207f41c9f6aff1997dbf1 Author: Timothy Sullivan <tsullivan@elastic.co> Date: Thu Nov 12 13:22:52 2020 -0700 add instructions to build local commit 872c0f68e0077ad50c541b605a87253befd1a891 Author: Timothy Sullivan <tsullivan@elastic.co> Date: Thu Nov 12 11:58:25 2020 -0700 simplify commit 8dae9484efcdc483ad20ceab32cae4e97735ab1c Author: Timothy Sullivan <tsullivan@elastic.co> Date: Thu Nov 12 10:28:18 2020 -0700 fixes commit 492f5cfe25e2c7ccffce55a8733383a30aae1cb6 Author: Timothy Sullivan <tsullivan@elastic.co> Date: Wed Nov 11 20:15:43 2020 -0700 --wip-- [skip ci] commit acba359b121f7be8e6c353f90e938bc5d88658a1 Author: Timothy Sullivan <tsullivan@elastic.co> Date: Wed Nov 11 15:47:50 2020 -0700 [Build Chromium] Improve git checkout commit 61b4e052fdd04456fe23956ad8ce99c8e771c01e Author: Timothy Sullivan <tsullivan@elastic.co> Date: Wed Nov 11 20:15:43 2020 -0700 --wip-- [skip ci] commit 58300c9deeca15cce838b3956c55c2994f99f530 Author: Timothy Sullivan <tsullivan@elastic.co> Date: Wed Nov 11 15:47:50 2020 -0700 [Build Chromium] Improve git checkout * Apply suggestions from code review Co-authored-by: Joel Griffith <joel@joelgriffith.net> Co-authored-by: Joel Griffith <joel@joelgriffith.net> Co-authored-by: Kibana Machine <42973632+kibanamachine@users.noreply.github.com> Co-authored-by: Joel Griffith <joel@joelgriffith.net> Co-authored-by: Kibana Machine <42973632+kibanamachine@users.noreply.github.com>

* master: (108 commits) [DOCS] Refreshes Data Visualizer screenshot (elastic#87017) [Security Solution] Change 'anti-virus' text to 'antivirus' (elastic#87001) [securitySolution/cypress] temporarily limit to PRs [AppServices/Examples] Add the example for Reporting integration (elastic#82091) [Build Chromium] Improve git checkout (elastic#83225) Deprecate `services.callCluster` in alerts and actions executors (elastic#86474) [Security Solution] Use system node version for Cypress and increase exec command timeout (elastic#86985) [Lens] Add more chart editor tests based on the debug state (elastic#86750) [Lens] Integrate searchSessionId into Lens app (elastic#86297) skip "pagination updates results and page number" elastic#86975 skip "Custom detection rules" elastic#83772 [logging/json] use merge from kbn/std (elastic#86330) skip network and timeline inspection. elastic#85677, elastic#85678 skip "adds correctly a filter to the global search bar" elastic#86552 [ftr/flags] improve help text (elastic#86971) skip "Fields Browser rendering. elastic#60209 skip "Closes and opens alerts" elastic#83773 [Security Solution] Skip failing Cypress tests (elastic#86967) Removes archives (elastic#86537) [ML] Fix charts grid on the Anomaly Explorer page (elastic#86904) ...

* master: (72 commits) [DOCS] Refreshes Data Visualizer screenshot (elastic#87017) [Security Solution] Change 'anti-virus' text to 'antivirus' (elastic#87001) [securitySolution/cypress] temporarily limit to PRs [AppServices/Examples] Add the example for Reporting integration (elastic#82091) [Build Chromium] Improve git checkout (elastic#83225) Deprecate `services.callCluster` in alerts and actions executors (elastic#86474) [Security Solution] Use system node version for Cypress and increase exec command timeout (elastic#86985) [Lens] Add more chart editor tests based on the debug state (elastic#86750) [Lens] Integrate searchSessionId into Lens app (elastic#86297) skip "pagination updates results and page number" elastic#86975 skip "Custom detection rules" elastic#83772 [logging/json] use merge from kbn/std (elastic#86330) skip network and timeline inspection. elastic#85677, elastic#85678 skip "adds correctly a filter to the global search bar" elastic#86552 [ftr/flags] improve help text (elastic#86971) skip "Fields Browser rendering. elastic#60209 skip "Closes and opens alerts" elastic#83773 [Security Solution] Skip failing Cypress tests (elastic#86967) Removes archives (elastic#86537) [ML] Fix charts grid on the Anomaly Explorer page (elastic#86904) ...

tsullivan added (Deprecated) Feature:Reporting Use Reporting:Screenshot, Reporting:CSV, or Reporting:Framework instead release_note:skip Skip the PR/issue when compiling release notes Team:AppArch v8.0.0 labels Nov 11, 2020

tsullivan force-pushed the chore/build_chromium/improvements branch 3 times, most recently from 1f1bc2c to 58300c9 Compare November 11, 2020 23:48

tsullivan marked this pull request as draft November 11, 2020 23:48

spalger added v7.11.0 and removed v7.11 labels Nov 12, 2020

tsullivan force-pushed the chore/build_chromium/improvements branch 3 times, most recently from acaff5b to 4885f9d Compare November 12, 2020 22:56

tsullivan force-pushed the chore/build_chromium/improvements branch from db6c869 to d3d16c4 Compare December 14, 2020 16:31

tsullivan requested review from joelgriffith, tylersmalley and a team December 14, 2020 16:39

tsullivan marked this pull request as ready for review December 14, 2020 16:40

tsullivan commented Dec 14, 2020

View reviewed changes

joelgriffith reviewed Dec 14, 2020

View reviewed changes

x-pack/build_chromium/README.md Outdated Show resolved Hide resolved

joelgriffith reviewed Dec 14, 2020

View reviewed changes

x-pack/build_chromium/README.md Show resolved Hide resolved

joelgriffith suggested changes Dec 14, 2020

View reviewed changes

Apply suggestions from code review

0f8994a

Co-authored-by: Joel Griffith <joel@joelgriffith.net>

Merge branch 'master' into chore/build_chromium/improvements

935e9e5

tsullivan requested a review from joelgriffith December 22, 2020 00:45

Merge branch 'master' into chore/build_chromium/improvements

8156392

joelgriffith approved these changes Dec 22, 2020

View reviewed changes

Merge branch 'master' into chore/build_chromium/improvements

efc012f

jbudz approved these changes Dec 29, 2020

View reviewed changes

tsullivan merged commit e699d91 into elastic:master Dec 29, 2020

tsullivan mentioned this pull request Dec 29, 2020

[7.11] [Build Chromium] Improve git checkout (#83225) #86998

Merged

tsullivan mentioned this pull request Dec 29, 2020

[7.x] [Build Chromium] Improve git checkout (#83225) #87009

Merged

tsullivan deleted the chore/build_chromium/improvements branch December 29, 2020 19:19

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Build Chromium] Improve git checkout #83225

[Build Chromium] Improve git checkout #83225

tsullivan commented Nov 11, 2020 •

edited

Loading

elasticmachine commented Nov 11, 2020

elasticmachine commented Nov 11, 2020

tsullivan commented Dec 14, 2020

elasticmachine commented Dec 14, 2020

tsullivan Dec 14, 2020

tsullivan Dec 14, 2020

joelgriffith left a comment

tsullivan commented Dec 19, 2020

tsullivan commented Dec 22, 2020

joelgriffith left a comment

tsullivan commented Dec 28, 2020

kibanamachine commented Dec 29, 2020

jbudz left a comment


		As of this writing, there is an officially supported headless Chromium build args file for Linux: `build/args/headless.gn`. This does not work on Windows or Mac, so we have taken that as our starting point, and modified it until the Windows / Mac builds succeeded.
		To get a list of the build arguments that are enabled, install `depot_tools` and run

[Build Chromium] Improve git checkout #83225

[Build Chromium] Improve git checkout #83225

Conversation

tsullivan commented Nov 11, 2020 • edited Loading

Summary

elasticmachine commented Nov 11, 2020

elasticmachine commented Nov 11, 2020

tsullivan commented Dec 14, 2020

elasticmachine commented Dec 14, 2020

tsullivan Dec 14, 2020

Choose a reason for hiding this comment

tsullivan Dec 14, 2020

Choose a reason for hiding this comment

joelgriffith left a comment

Choose a reason for hiding this comment

tsullivan commented Dec 19, 2020

tsullivan commented Dec 22, 2020

joelgriffith left a comment

Choose a reason for hiding this comment

tsullivan commented Dec 28, 2020

kibanamachine commented Dec 29, 2020

💚 Build Succeeded

Metrics [docs]

Distributable file count

History

jbudz left a comment

Choose a reason for hiding this comment

tsullivan commented Nov 11, 2020 •

edited

Loading