Emulate x86 so local dev is possible on arm64 machines #1142

longrunningprocess · 2021-09-08T17:37:28Z

Description

I have a new M1 machine and was running into docker build failures for app. This was the result of lfmerge not having an arm64 pkg. Creating one would be too much effort so this PR is the effort to emulate an x86 during the image build where lfmerge is used.

Type of Change

configuration

Screenshots

N/A

How Has This Been Tested?

app builds without error
make creates a running app locally (with basic functionality testing)
make unit-tests should run all tests successfully
make e2e-tests should run all tests successfully
test latest changes on my old machine
test on a win machine

Checklist:

I have performed a self-review of my own code
I have reviewed the title/description of this PR which will be used as the squashed PR commit message
I have commented my code, particularly in hard-to-understand areas
I have added (manual) tests that prove my fix is effective or that my feature works

longrunningprocess · 2021-09-08T20:21:20Z

Notes

https://docs.docker.com/engine/reference/commandline/build
docker build -f app/Dockerfile --platform linux/amd64 .. is building fine on my new machine...slow, but it is building.

docker/for-mac#5364
Apparently docker-compose.yml has a platform option, this might be useful on the app service…need to test it, if it works, make sure it doesn't have any side-effects on my old machine

Additionally, there’s an env var which might be worth looking at: DOCKER_DEFAULT_PLATFORM

megahirt · 2021-09-08T23:32:59Z

@longrunningprocess it appears that this branch includes the sass branch but doesn't have any pushed commits related to the --platform linux/amd64 change. Looking forward to seeing that when it's pushed.

longrunningprocess · 2021-09-08T23:48:06Z

@longrunningprocess it appears that this branch includes the sass branch but doesn't have any pushed commits related to the --platform linux/amd64 change. Looking forward to seeing that when it's pushed.

I haven't made any yet, this PR is still in draft. Those commits are from my other PR that isn't merged into develop yet. I intend to rebase this one at the right time and those will disappear.

megahirt · 2021-09-09T05:36:10Z

@longrunningprocess I think I get it now - you wanted a place to keep notes on your research that would potentially result in code changes, but it turned out you didn't have any changes to propose at this time in a PR.

I think you would be wanting an issue to keep notes on your research, not a draft PR. PRs only make sense for code changes, and any PR, draft or not, is an invitation for others to review your code. A draft PR is intended to publish work-in-progress where you invite others to comment on your code or see the direction you're going, but it is explicit that changes are not yet ready for review/merge.

So in the future I would expect to see notes made in issues, and then PRs which summarize work done as code changes, linked to relevant issues. Thanks!

longrunningprocess · 2021-09-09T10:29:43Z

Gotcha, that makes sense. Once I rebased and this thing closed without my wanting it to do that I was imagining you were going to be cracking up! I guess once I have changes to push into this branch I'll have to reopen it.

longrunningprocess · 2021-09-09T12:49:25Z

got through the lfmerge problem, now finding this one:

 ⠙ ld-db Pulling                                                                                                                                                                                             4.1s
no matching manifest for linux/arm64/v8 in the manifest list entries
make: *** [start] Error 18

longrunningprocess · 2021-09-09T13:00:32Z

failing on make unit-tests

docker-compose build test-php
[+] Building 1.4s (3/3) FINISHED                                                                                                                                                                                  
 => [internal] load build definition from Dockerfile                                                                                                                                                         0.3s
 => => transferring dockerfile: 220B                                                                                                                                                                         0.0s
 => [internal] load .dockerignore                                                                                                                                                                            0.5s
 => => transferring context: 2B                                                                                                                                                                              0.0s
 => ERROR [internal] load metadata for docker.io/library/lf-app:latest                                                                                                                                       0.7s
------
 > [internal] load metadata for docker.io/library/lf-app:latest:
------
failed to solve: rpc error: code = Unknown desc = failed to solve with frontend dockerfile.v0: failed to create LLB definition: pull access denied, repository does not exist or may require authorization: server message: insufficient_scope: authorization failed
make: *** [unit-tests] Error 17

longrunningprocess · 2021-09-09T13:11:46Z

failing make e2e-tests

docker-compose build app-for-e2e test-e2e
[+] Building 2.4s (6/6) FINISHED                                                                                                                                                                                  
 => [test-e2e internal] load build definition from Dockerfile                                                                                                                                                0.6s
 => => transferring dockerfile: 1.35kB                                                                                                                                                                       0.0s
 => [app-for-e2e internal] load build definition from Dockerfile                                                                                                                                             1.2s
 => => transferring dockerfile: 298B                                                                                                                                                                         0.0s
 => [test-e2e internal] load .dockerignore                                                                                                                                                                   0.9s
 => => transferring context: 2B                                                                                                                                                                              0.0s
 => [app-for-e2e internal] load .dockerignore                                                                                                                                                                1.4s
 => => transferring context: 2B                                                                                                                                                                              0.0s
 => CANCELED [test-e2e internal] load metadata for docker.io/library/node:14.16.1-alpine3.11                                                                                                                 1.2s
 => ERROR [app-for-e2e internal] load metadata for docker.io/library/lf-app:latest                                                                                                                           0.7s
------
 > [app-for-e2e internal] load metadata for docker.io/library/lf-app:latest:
------
failed to solve: rpc error: code = Unknown desc = failed to solve with frontend dockerfile.v0: failed to create LLB definition: pull access denied, repository does not exist or may require authorization: server message: insufficient_scope: authorization failed
make: *** [e2e-tests] Error 17

longrunningprocess · 2021-09-09T13:19:38Z

failing make e2e-tests

docker-compose-wait - Everything's fine, the application can now start!
--------------------------------------------------------
Waiting for a Protractor process to appear so the debugger can be attached...

> x-forge@1.0.0 test-e2e
> protractor test/app/protractorConf.js

[13:14:45] I/launcher - Running 1 instances of WebDriver
[13:14:45] I/hosted - Using the selenium server at http://selenium:4444/wd/hub
[13:14:52] E/launcher - unknown error: Chrome failed to start: crashed.
  (chrome not reachable)
  (The process started from chrome location /usr/bin/google-chrome is no longer running, so ChromeDriver is assuming that Chrome has crashed.)
Build info: version: '3.141.59', revision: 'e82be7d358', time: '2018-11-14T08:25:53'
System info: host: 'e47f7750179c', ip: '172.20.0.8', os.name: 'Linux', os.arch: 'amd64', os.version: '5.10.47-linuxkit', java.version: '1.8.0_292'
Driver info: driver.version: unknown
remote stacktrace: #0 0x00400068e949 <unknown>

[13:14:52] E/launcher - WebDriverError: unknown error: Chrome failed to start: crashed.
  (chrome not reachable)
  (The process started from chrome location /usr/bin/google-chrome is no longer running, so ChromeDriver is assuming that Chrome has crashed.)
Build info: version: '3.141.59', revision: 'e82be7d358', time: '2018-11-14T08:25:53'
System info: host: 'e47f7750179c', ip: '172.20.0.8', os.name: 'Linux', os.arch: 'amd64', os.version: '5.10.47-linuxkit', java.version: '1.8.0_292'
Driver info: driver.version: unknown
remote stacktrace: #0 0x00400068e949 <unknown>

    at Object.checkLegacyResponse (/data/node_modules/selenium-webdriver/lib/error.js:546:15)
    at parseHttpResponse (/data/node_modules/selenium-webdriver/lib/http.js:509:13)
    at /data/node_modules/selenium-webdriver/lib/http.js:441:30
    at processTicksAndRejections (internal/process/task_queues.js:93:5)
From: Task: WebDriver.createSession()
    at Function.createSession (/data/node_modules/selenium-webdriver/lib/webdriver.js:769:24)
    at Function.createSession (/data/node_modules/selenium-webdriver/chrome.js:761:15)
    at createDriver (/data/node_modules/selenium-webdriver/index.js:170:33)
    at Builder.build (/data/node_modules/selenium-webdriver/index.js:626:16)
    at Hosted.getNewDriver (/data/node_modules/protractor/built/driverProviders/driverProvider.js:53:33)
    at Runner.createBrowser (/data/node_modules/protractor/built/runner.js:195:43)
    at /data/node_modules/protractor/built/runner.js:339:29
    at _fulfilled (/data/node_modules/q/q.js:834:54)
    at /data/node_modules/q/q.js:863:30
    at Promise.promise.promiseDispatch (/data/node_modules/q/q.js:796:13)
[13:14:52] E/launcher - Process exited with error code 199
npm ERR! code 199
npm ERR! path /data
npm ERR! command failed
npm ERR! command sh -c protractor test/app/protractorConf.js

npm ERR! A complete log of this run can be found in:
npm ERR!     /root/.npm/_logs/2021-09-09T13_14_52_262Z-debug.log
make: *** [e2e-tests] Error 199

longrunningprocess · 2021-09-09T20:12:56Z

this seems like the most relevant info on the selenium issue: SeleniumHQ/docker-selenium#1076

I tried a couple of possibilities, none of which worked seamlessly like our original image:

  selenium:
    # image: selenium/standalone-chrome-debug:3.141.59-20210713
    # platform: linux/amd64
    # image: seleniarm/node-chromium:4.0.0-beta-1-20210215
    # image: seleniarm/hub:4.0.0-beta-1-20210215

I was able to see some output from selenium that it was ready (using the hub image) but no test output (not running or not outputting results as before) so based on SeleniumHQ/docker-selenium#1076 (comment), maybe we need to reconfigure things to have a service for the hub and another service for the headless browser portion, not sure. It will need some more investigation and testing. Maybe this is not something we need to necessarily get working on Apple Silicon at the moment if the E2E tests are still passing on TC.

longrunningprocess · 2021-09-09T22:57:15Z

At this point, there are a couple of options as I see it:

keeping working through the right combination of configurations and setup to get things working again
declare this a very narrow use case and move on without it

I'm inclined to move on knowing I will not be able to run e2e tests on my m1 for now. I'm going to take this one out of draft under the assumption that will be ok, if not, this is where we can hash it out more.

longrunningprocess · 2021-09-09T23:01:03Z

@megahirt mind testing make (and exercising some basic functionality locally) and make unit-tests on your machine and checking that win test off above if it goes well? Thanks!

megahirt

LGTM

These changes will make our project more compatible with Apple Silicon

longrunningprocess changed the title ~~Chore/emulate x86 so app will build~~ Emulate x86 so app will build on arm64 machines Sep 8, 2021

longrunningprocess closed this Sep 9, 2021

longrunningprocess force-pushed the chore/emulate-x86-so-app-will-build branch from 275b1da to 1123840 Compare September 9, 2021 00:03

fixed failing app build

8297a38

longrunningprocess reopened this Sep 9, 2021

fixed failing ld_db startup

9b60f5d

fixed failing unit tests

e863d7a

fixed failing app-for-e2e build

78ebc79

megahirt assigned longrunningprocess Sep 9, 2021

longrunningprocess marked this pull request as ready for review September 9, 2021 22:57

longrunningprocess requested a review from megahirt September 9, 2021 23:01

longrunningprocess changed the title ~~Emulate x86 so app will build on arm64 machines~~ Emulate x86 so local dev is possible on arm64 machines Sep 9, 2021

updated docs

1ae0c8d

longrunningprocess mentioned this pull request Sep 13, 2021

Added ability to run security scans of our images #1154

Merged

6 tasks

megahirt approved these changes Sep 14, 2021

View reviewed changes

longrunningprocess merged commit 20ad980 into develop Sep 14, 2021

longrunningprocess deleted the chore/emulate-x86-so-app-will-build branch September 14, 2021 12:54

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Emulate x86 so local dev is possible on arm64 machines #1142

Emulate x86 so local dev is possible on arm64 machines #1142

longrunningprocess commented Sep 8, 2021 •

edited

Loading

longrunningprocess commented Sep 8, 2021

megahirt commented Sep 8, 2021

longrunningprocess commented Sep 8, 2021

megahirt commented Sep 9, 2021

longrunningprocess commented Sep 9, 2021 •

edited

Loading

longrunningprocess commented Sep 9, 2021

longrunningprocess commented Sep 9, 2021 •

edited

Loading

longrunningprocess commented Sep 9, 2021

longrunningprocess commented Sep 9, 2021

longrunningprocess commented Sep 9, 2021 •

edited

Loading

longrunningprocess commented Sep 9, 2021

longrunningprocess commented Sep 9, 2021 •

edited

Loading

megahirt left a comment

Emulate x86 so local dev is possible on arm64 machines #1142

Emulate x86 so local dev is possible on arm64 machines #1142

Conversation

longrunningprocess commented Sep 8, 2021 • edited Loading

Description

Type of Change

Screenshots

How Has This Been Tested?

Checklist:

longrunningprocess commented Sep 8, 2021

Notes

megahirt commented Sep 8, 2021

longrunningprocess commented Sep 8, 2021

megahirt commented Sep 9, 2021

longrunningprocess commented Sep 9, 2021 • edited Loading

longrunningprocess commented Sep 9, 2021

longrunningprocess commented Sep 9, 2021 • edited Loading

longrunningprocess commented Sep 9, 2021

longrunningprocess commented Sep 9, 2021

longrunningprocess commented Sep 9, 2021 • edited Loading

longrunningprocess commented Sep 9, 2021

longrunningprocess commented Sep 9, 2021 • edited Loading

megahirt left a comment

Choose a reason for hiding this comment

longrunningprocess commented Sep 8, 2021 •

edited

Loading

longrunningprocess commented Sep 9, 2021 •

edited

Loading

longrunningprocess commented Sep 9, 2021 •

edited

Loading

longrunningprocess commented Sep 9, 2021 •

edited

Loading

longrunningprocess commented Sep 9, 2021 •

edited

Loading