Chromium update and Apple Silicon build #71

jonmmease · 2021-02-09T12:14:32Z

WIP PR to work on updating the chromium build and adding an apple silicon arm64 build

jonmmease · 2021-02-09T13:11:19Z

NOTE: Windows CI failure is during fetch: https://app.circleci.com/pipelines/github/plotly/Kaleido/177/workflows/ac087259-73eb-4bcb-8319-4ede09a70d0c/jobs/896

#!powershell.exe
.\repos\win_scripts\fetch_chromium.ps1

Cloning into 'depot_tools'...
C:\Users\circleci\project\repos\depot_tools;C:\Users\circleci\project\repos\depot_tools\bootstrap-3_8_0_chromium_8_bin\python\bin;C:\Program Files (x86)\Windows Application Driver;C:\Program Files (x86)\Microsoft Visual Studio\2019\Community\MSBuild\Current\Bin;C:\Program Files (x86)\Microsoft Visual Studio\Installer\;C:\tools\ruby26;C:\tools\ruby26\bin;C:\ProgramData\nvm;C:\tools\miniconda3;C:\tools\miniconda3\Library\mingw-w64\bin;C:\tools\miniconda3\Library\usr\bin;C:\tools\miniconda3\Library\bin;C:\tools\miniconda3\Scripts;C:\miniconda3\miniconda3\condabin;C:\Program Files (x86)\Microsoft SDKs\Azure\CLI2\wbin;C:\Windows\system32;C:\Windows;C:\Windows\System32\Wbem;C:\Windows\System32\WindowsPowerShell\v1.0\;C:\Windows\System32\OpenSSH\;C:\ProgramData\GooGet;C:\Program Files\Google\Compute Engine\metadata_scripts;C:\Program Files (x86)\Google\Cloud SDK\google-cloud-sdk\bin;C:\Program Files\PowerShell\7\;C:\Program Files\Google\Compute Engine\sysprep;C:\Program Files\Docker;C:\ProgramData\chocolatey\bin;C:\Program Files\Git\cmd;C:\Program Files\Git\mingw64\bin;C:\Program Files\Git\usr\bin;C:\Program Files\Git LFS;C:\Program Files\Amazon\AWSCLI\bin\;C:\Program Files\Microsoft Service Fabric\bin\Fabric\Fabric.Code;C:\Program Files\Microsoft SDKs\Service Fabric\Tools\ServiceFabricLocalClusterManager;C:\Program Files (x86)\vim\vim80;C:\Go\bin;C:\Program Files\OpenJDK\jdk-12.0.2\bin;C:\Program Files\nodejs;C:\Program Files\dotnet\;C:\Program Files (x86)\Windows Kits\10\Windows Performance Toolkit\;C:\Program Files\Microsoft SQL Server\130\Tools\Binn\;C:\Program Files\Microsoft SQL Server\Client SDK\ODBC\170\Tools\Binn\;C:\Program Files (x86)\IncrediBuild;C:\Users\circleci\AppData\Local\Microsoft\WindowsApps;
HEAD is now at e342fb16 Don't suggest change author as reviewer.
commit e342fb16f98530bc8efd0abfedc5986a2c58478c
Author: Edward Lesmes <ehmaldonado@google.com>
Date:   Fri Feb 5 00:35:54 2021 +0000

    Don't suggest change author as reviewer.
    
    Change-Id: Ie34fc34883b2c86fc134f10ded24b26a631886ec
    Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/2676835
    Reviewed-by: Josip Sokcevic <sokcevic@google.com>
    Commit-Queue: Edward Lesmes <ehmaldonado@chromium.org>

commit 680a6c37a0fbb29292576f142431f162cb3d814e
Author: John Chen <johnchen@chromium.org>
Date:   Thu Feb 4 05:28:12 2021 +0000

    Improve upload_to_google_storage.py missing file handling
    
    When there are missing input files, upload_to_google_storage.py
    should not display 'Success'.
    
    Bug: 920654
    Change-Id: I1bc7ba6e8eb48ea28b634da01e1be9a80af7b719
    Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/2647870
    Reviewed-by: Dirk Pranke <dpranke@google.com>
    Commit-Queue: John Chen <johnchen@chromium.org>
[E2021-02-09T12:32:44.446096Z 9060 0 annotate.go:241] original error: file does not exist
[E2021-02-09T12:32:44.446096Z 9060 0 annotate.go:241] 
[E2021-02-09T12:32:44.446096Z 9060 0 annotate.go:241] goroutine 1:
[E2021-02-09T12:32:44.446096Z 9060 0 annotate.go:241] #0 go.chromium.org/luci/vpython/python/interpreter.go:120 - python.(*Interpreter).GetVersion()
[E2021-02-09T12:32:44.446096Z 9060 0 annotate.go:241] #1 go.chromium.org/luci/vpython/venv/config.go:286 - venv.(*Config).resolvePythonInterpreter()
[E2021-02-09T12:32:44.446096Z 9060 0 annotate.go:241]   reason: failed to determine Python version for: C:\Users\circleci\project\repos\depot_tools\\python.bat
[E2021-02-09T12:32:44.446096Z 9060 0 annotate.go:241] 
[E2021-02-09T12:32:44.446096Z 9060 0 annotate.go:241] #2 go.chromium.org/luci/vpython/venv/config.go:186 - venv.(*Config).makeEnv()
[E2021-02-09T12:32:44.446096Z 9060 0 annotate.go:241]   reason: failed to resolve system Python interpreter
[E2021-02-09T12:32:44.446096Z 9060 0 annotate.go:241] 
[E2021-02-09T12:32:44.446096Z 9060 0 annotate.go:241] #3 go.chromium.org/luci/vpython/venv/venv.go:150 - venv.With()
[E2021-02-09T12:32:44.446096Z 9060 0 annotate.go:241]   reason: failed to initialize empty probe environment
[E2021-02-09T12:32:44.447112Z 9060 0 annotate.go:241] 
[E2021-02-09T12:32:44.447112Z 9060 0 annotate.go:241] #4 go.chromium.org/luci/vpython/run.go:62 - vpython.Run()
[E2021-02-09T12:32:44.447112Z 9060 0 annotate.go:241] #5 go.chromium.org/luci/vpython/application/application.go:320 - application.(*application).mainImpl()
[E2021-02-09T12:32:44.447112Z 9060 0 annotate.go:241] #6 go.chromium.org/luci/vpython/application/application.go:408 - application.(*Config).Main.func1()
[E2021-02-09T12:32:44.447112Z 9060 0 annotate.go:241] #7 go.chromium.org/luci/vpython/application/support.go:46 - application.run()
[E2021-02-09T12:32:44.447112Z 9060 0 annotate.go:241] #8 go.chromium.org/luci/vpython/application/application.go:407 - application.(*Config).Main()
[E2021-02-09T12:32:44.447112Z 9060 0 annotate.go:241] #9 vpython/main.go:109 - main.mainImpl()
[E2021-02-09T12:32:44.447112Z 9060 0 annotate.go:241] #10 vpython/main.go:115 - main.main()
[E2021-02-09T12:32:44.447112Z 9060 0 annotate.go:241] #11 runtime/proc.go:204 - runtime.main()
[E2021-02-09T12:32:44.447112Z 9060 0 annotate.go:241] #12 runtime/asm_amd64.s:1374 - runtime.goexit()

jonmmease · 2021-02-10T14:30:33Z

Note: The macos arm64 build seems to be working, but it runs out of disk space on circleci after compiling [14928/19359]: https://app.circleci.com/pipelines/github/plotly/Kaleido/178/workflows/8dee67c6-06c1-4226-a7b4-08a2ab8acdae/jobs/926

The x64 build doesn't run into this problem.

Was causing depot_tools build errors for updated version

Try to see if there's anything else we can delete to save disk space

jonmmease · 2021-02-11T10:52:59Z

Note: Windows builds fixed by 41dde9d. MacOS ARM CI build still running out of disk space

…hromium_update

Downgrade to version used by latest stable version of electron 11. This was the first major version that Electron started shipping Apple Silicon support.

jonmmease · 2021-02-13T10:13:49Z

Problems:

Windows builds are failing with:

Mode                LastWriteTime         Length Name                                                                  
----                -------------         ------ ----                                                                  
d-----        2/13/2021  12:40 AM                Kaleido_win_x64                                                       
Traceback (most recent call last):
  File "C:/Users/circleci/project/repos/src/build/vs_toolchain.py", line 573, in <module>
    sys.exit(main())
  File "C:/Users/circleci/project/repos/src/build/vs_toolchain.py", line 569, in main
    return commands[sys.argv[1]](*sys.argv[2:])
  File "C:/Users/circleci/project/repos/src/build/vs_toolchain.py", line 400, in CopyDlls
    _CopyDebugger(target_dir, target_cpu)
  File "C:/Users/circleci/project/repos/src/build/vs_toolchain.py", line 436, in _CopyDebugger
    (debug_file, full_path))
Exception: api-ms-win-downlevel-kernel32-l2-1-0.dll not found in "C:\Program Files (x86)\Windows Kits\10\Debuggers\x64\api-ms-win-downlevel-kernel32-l2-1-0.dll"
You must installWindows 10 SDK version 10.0.19041.0 including the "Debugging Tools for Windows" feature.
ERROR at //build/toolchain/win/BUILD.gn:49:3: Script returned non-zero exit code.
  exec_script("../../vs_toolchain.py",
  ^----------

Mac intel build is failing with

[30988/31038] CXX obj/headless/headless_shared_sources/memory.o
[30989/31038] CXX obj/headless/headless_shared_sources/types_memory.o
[30990/31038] CXX obj/headless/headless_renderer/headless_content_main_delegate.o
FAILED: obj/headless/headless_renderer/headless_content_main_delegate.o 
../../third_party/llvm-build/Release+Asserts/bin/clang++ -MMD -MF obj/headless/headless_renderer/headless_content_main_delegate.o.d -DHEADLESS_USE_EMBEDDED_RESOURCES -D_LIBCPP_HAS_NO_ALIGNED_ALLOCATION -DCR_XCODE_VERSION=1220 -DCR_CLANG_REVISION=\"llvmorg-12-init-5035-gd0abc757-3\" -D__STDC_CONSTANT_MACROS -D__STDC_FORMAT_MACROS -D_FORTIFY_SOURCE=2 -D_LIBCPP_ABI_UNSTABLE -D_LIBCPP_DISABLE_VISIBILITY_ANNOTATIONS -D_LIBCXXABI_DISABLE_VISIBILITY_ANNOTATIONS -D_LIBCPP_ENABLE_NODISCARD -DCR_LIBCXX_REVISION=375504 -D__ASSERT_MACROS_DEFINE_VERSIONS_WITHOUT_UNDERSCORES=0 -DNDEBUG -DNVALGRIND -DDYNAMIC_ANNOTATIONS_ENABLED=0 -DENABLE_IPC_FUZZER -DWEBP_EXTERN=extern -DUSE_EGL -DTOOLKIT_VIEWS=1 -DV8_COMPRESS_POINTERS -DV8_31BIT_SMIS_ON_64BIT_ARCH -DV8_DEPRECATION_WARNINGS -DV8_REVERSE_JSARGS -DU_USING_ICU_NAMESPACE=0 -DU_ENABLE_DYLOAD=0 -DUSE_CHROMIUM_ICU=1 -DU_ENABLE_TRACING=1 -DU_ENABLE_RESOURCE_TRACING=0 -DU_STATIC_IMPLEMENTATION -DICU_UTIL_DATA_IMPL=ICU_UTIL_DATA_FILE -DUCHAR_TYPE=uint16_t -DGOOGLE_PROTOBUF_NO_RTTI -DGOOGLE_PROTOBUF_NO_STATIC_INITIALIZER -DHAVE_PTHREAD -DSK_CODEC_DECODES_PNG -DSK_CODEC_DECODES_WEBP -DSK_ENCODE_PNG -DSK_ENCODE_WEBP -DSK_USER_CONFIG_HEADER=\"../../skia/config/SkUserConfig.h\" -DSK_GL -DSK_CODEC_DECODES_JPEG -DSK_ENCODE_JPEG -DSK_USE_LIBGIFCODEC -DSK_SUPPORT_GPU=1 -DSK_GPU_WORKAROUNDS_HEADER=\"gpu/config/gpu_driver_bug_workaround_autogen.h\" -DSK_BUILD_FOR_MAC -DSK_METAL -DWEBRTC_ENABLE_AVX2 -DWEBRTC_NON_STATIC_TRACE_EVENT_HANDLERS=0 -DWEBRTC_CHROMIUM_BUILD -DWEBRTC_POSIX -DWEBRTC_MAC -DABSL_ALLOCATOR_NOTHROW=1 -DWEBRTC_USE_BUILTIN_ISAC_FIX=0 -DWEBRTC_USE_BUILTIN_ISAC_FLOAT=1 -DHAVE_SCTP -DNO_MAIN_THREAD_WRAPPING -DLEVELDB_PLATFORM_CHROMIUM=1 -DLEVELDB_PLATFORM_CHROMIUM=1 -DV8_COMPRESS_POINTERS -DV8_31BIT_SMIS_ON_64BIT_ARCH -DV8_DEPRECATION_WARNINGS -DV8_REVERSE_JSARGS -I../.. -Igen -I../../third_party/perfetto/include -Igen/third_party/perfetto/build_config -Igen/third_party/perfetto -I../../third_party/libyuv/include -I../../third_party/jsoncpp/source/include -I../../third_party/libwebp/src -I../../third_party/khronos -I../../gpu -Igen/third_party/dawn/src/include -I../../third_party/dawn/src/include -I../../v8/include -Igen/v8/include -I../../third_party/icu/source/common -I../../third_party/icu/source/i18n -I../../third_party/protobuf/src -Igen/protoc_out -I../../third_party/protobuf/src -I../../third_party/abseil-cpp -I../../third_party/boringssl/src/include -I../../third_party/ced/src -I../../third_party/skia -I../../third_party/libgifcodec -I../../third_party/webrtc_overrides -I../../third_party/webrtc -Igen/third_party/webrtc -I../../third_party/mesa_headers -I../../third_party/libwebm/source -I../../third_party/leveldatabase -I../../third_party/leveldatabase/src -I../../third_party/leveldatabase/src/include -I../../v8/include -Igen/v8/include -fno-strict-aliasing -fstack-protector -fcolor-diagnostics -fmerge-all-constants -fcrash-diagnostics-dir=../../tools/clang/crashreports -mllvm -instcombine-lower-dbg-declare=0 -mllvm -enable-dse-memoryssa=false -fcomplete-member-pointers -arch x86_64 -Wno-builtin-macro-redefined -D__DATE__= -D__TIME__= -D__TIMESTAMP__= -Xclang -fdebug-compilation-dir -Xclang . -no-canonical-prefixes -Wall -Werror -Wextra -Wimplicit-fallthrough -Wunreachable-code -Wthread-safety -Wextra-semi -Wunguarded-availability -Wno-missing-field-initializers -Wno-unused-parameter -Wno-c++11-narrowing -Wno-unneeded-internal-declaration -Wno-undefined-var-template -Wno-psabi -Wno-ignored-pragma-optimize -Wno-implicit-int-float-conversion -Wno-final-dtor-non-final-class -Wno-builtin-assume-aligned-alignment -Wno-deprecated-copy -Wno-non-c-typedef-for-linkage -Wno-max-tokens -O2 -fno-omit-frame-pointer -g0 -isysroot ../../../../../../../Applications/Xcode-12.2.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX11.0.sdk -mmacosx-version-min=10.10.0 -ftrivial-auto-var-init=pattern -fvisibility=hidden -Xclang -add-plugin -Xclang find-bad-constructs -Wheader-hygiene -Wstring-conversion -Wtautological-overlap-compare -Wno-shorten-64-to-32 -std=c++14 -fno-trigraphs -Wno-trigraphs -stdlib=libc++ -fno-exceptions -fno-rtti -nostdinc++ -isystem../../buildtools/third_party/libc++/trunk/include -isystem../../buildtools/third_party/libc++abi/trunk/include -fvisibility-inlines-hidden -Wno-deprecated-declarations -c ../../headless/lib/headless_content_main_delegate.cc -o obj/headless/headless_renderer/headless_content_main_delegate.o
../../headless/lib/headless_content_main_delegate.cc:47:10: fatal error: 'headless/embedded_resource_pak.h' file not found
#include "headless/embedded_resource_pak.h"
         ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
1 error generated.
[30991/31038] CXX obj/headless/headless_shared_sources/layer_tree.o
[30992/31038] CXX obj/headless/headless_renderer/headless_print_render_frame_helper_delegate.o
[30993/31038] CXX obj/headless/headless_shared_sources/log.o
[30994/31038] CXX obj/headless/headless_shared_sources/types_layer_tree.o
[30995/31038] CXX obj/headless/headless_shared_sources/types_log.o
[30996/31038] CXX obj/headless/headless_shared_sources/network.o
[30997/31038] CXX obj/headless/headless_renderer/headless_web_contents_impl.o

And it is much larger than prior version

Mac ARM build looks like it is succeeding, but it produces a 200MB binary! (past versions were ~55MB).

See if this works around ../../headless/lib/headless_content_main_delegate.cc:47:10: fatal error: 'headless/embedded_resource_pak.h' file not found

Matches Linux docker image

Also up min chromium version to 10.11 to match current google chrome minimum.

Build fails when embedding resources on arm (but not intel).

jonmmease added 4 commits February 9, 2021 06:13

WIP enable Apple Silicon (arm64) build

92dc48f

WIP update chromium and depot tools versions

2c32b2f

Pin setuptools version

d6df8fd

Bump chromium-builder docker image tag to 0.9

e0ca8df

Delete unused platforms to save space on mac build jobs

c2cd466

jonmmease added 6 commits February 10, 2021 18:31

Remove DEPOT_TOOLS_UPDATE env on windows

41dde9d

Was causing depot_tools build errors for updated version

Display circleci mac application sizes

6171400

Try to see if there's anything else we can delete to save disk space

Delete more application files

b71c2b5

Can't delete app files

4941859

Remove CREDITS.html file, autogenerate it

bc2882f

Write license file during fetch on Mac

dbaa857

jonmmease added 11 commits February 11, 2021 12:16

Build CREDITS.html on fetch

54f2461

Remove CREDITS.html

44c799e

ignore CREDITS.html

723ef86

Mac build fixes

1908d3a

Merge branch 'chromium_update' of github.com:jonmmease/Kaleido into c…

c5cd0b3

…hromium_update

CI fixes for generating CREDITS.html on fetch

f2bbf74

fix mac artifacts

2a1ae69

Back to mac "official" build to see if it reduces size

b2a6d60

Downgrade chromium version

6aed2a7

Downgrade to version used by latest stable version of electron 11. This was the first major version that Electron started shipping Apple Silicon support.

Fix powershell environment variable

812045d

again

f10b1a0

jonmmease added 4 commits February 13, 2021 06:17

Install windows debugging tools on circleci with chocolatey

16a903b

Try disabling headless_use_embedded_resource on mac

5ac079e

See if this works around ../../headless/lib/headless_content_main_delegate.cc:47:10: fatal error: 'headless/embedded_resource_pak.h' file not found

Back to current stable chromium

7ec5952

Matches Linux docker image

Copy resources directory on mac

2e763a1

jonmmease mentioned this pull request Feb 14, 2021

Plans to support Apple Silicon? #70

Closed

jonmmease added 2 commits February 14, 2021 18:32

Try embedding mac resources again

ecb1d64

Copy only select mac dynamic libraries

e1982cf

jonmmease changed the title ~~WIP: Chromium update and Apple Silicon build~~ Chromium update and Apple Silicon build Feb 15, 2021

jonmmease added 6 commits February 15, 2021 18:27

Use arm64 pip platform tag for Apple Silicon

672b9fa

Also up min chromium version to 10.11 to match current google chrome minimum.

Don't embed resources in mac arm build

c528ad2

Build fails when embedding resources on arm (but not intel).

Don't embed resources on x86 mac build either

00d3b3e

Embed resources, fix build_blink

81d6122

Update apple silicon platform tag to macosx-11-arm64

c9604e9

Add minor version to platform tag

5e23274

jonmmease merged commit 37e6ec4 into master Feb 26, 2021

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Chromium update and Apple Silicon build #71

Chromium update and Apple Silicon build #71

jonmmease commented Feb 9, 2021

jonmmease commented Feb 9, 2021

jonmmease commented Feb 10, 2021

jonmmease commented Feb 11, 2021

jonmmease commented Feb 13, 2021 •

edited

Loading

Chromium update and Apple Silicon build #71

Chromium update and Apple Silicon build #71

Conversation

jonmmease commented Feb 9, 2021

jonmmease commented Feb 9, 2021

jonmmease commented Feb 10, 2021

jonmmease commented Feb 11, 2021

jonmmease commented Feb 13, 2021 • edited Loading

Problems:

jonmmease commented Feb 13, 2021 •

edited

Loading