WIP: arm builds (for mac to start) #237

dpastoor · 2022-01-01T15:28:19Z

This PR attempts to track getting quarto going for arm platforms, per needs outlined in
#190 which may ultimately help resolve rstudio/rstudio#10159

I've just started poking around the repo so this could be quite naive 🙃

summary of changes:

disables the deno canary builds as not compat with ARM arch

TODO:

dev-related:

arm build for esbuild
arm build for pandoc

release-related:

everything 🤪

outputs:

The build output is as such with the initial changes:

❯ ./configure-macos.sh 
Bootstrapping Deno...
~/repos/dpastoor/quarto-cli/package ~/repos/dpastoor/quarto-cli
~/repos/dpastoor/quarto-cli/package/dist ~/repos/dpastoor/quarto-cli/package ~/repos/dpastoor/quarto-cli
~/repos/dpastoor/quarto-cli/package/dist/bin ~/repos/dpastoor/quarto-cli/package/dist ~/repos/dpastoor/quarto-cli/package ~/repos/dpastoor/quarto-cli
unsetting canary commit from  as not built on ARM
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100   668  100   668    0     0   3091      0 --:--:-- --:--:-- --:--:--  3165
100 28.6M  100 28.6M    0     0  13.9M      0  0:00:02  0:00:02 --:--:-- 15.6M
Archive:  deno-aarch64-apple-darwin.zip
  inflating: deno                    
Download https://deno.land/x/cliffy@v0.19.3/command/mod.ts
Download https://deno.land/std@0.119.0/flags/mod.ts
.... TRUNCATED other downloads
Download https://deno.land/x/cliffy@v0.19.3/table/utils.ts
Check file:///Users/devin/repos/dpastoor/quarto-cli/src/quarto.ts
~/repos/dpastoor/quarto-cli/package/dist ~/repos/dpastoor/quarto-cli/package ~/repos/dpastoor/quarto-cli
~/repos/dpastoor/quarto-cli/package ~/repos/dpastoor/quarto-cli
~/repos/dpastoor/quarto-cli
~/repos/dpastoor/quarto-cli/package/src ~/repos/dpastoor/quarto-cli
Check file:///Users/devin/repos/dpastoor/quarto-cli/package/src/bld.ts
Using configuration:
{"productName":"Quarto","version":"0.2","importmap":"/Users/devin/repos/dpastoor/quarto-cli/src/import_map.json","directoryInfo":{"root":"/Users/devin/repos/dpastoor/quarto-cli","pkg":"/Users/devin/repos/dpastoor/quarto-cli/package","dist":"/Users/devin/repos/dpastoor/quarto-cli/package/dist","share":"/Users/devin/repos/dpastoor/quarto-cli/package/dist/share","src":"/Users/devin/repos/dpastoor/quarto-cli/src","out":"/Users/devin/repos/dpastoor/quarto-cli/package/out","bin":"/Users/devin/repos/dpastoor/quarto-cli/package/dist/bin"}}
Configuring local machine for development
Downloading dependencies
Preparing deno_dom
Downloading deno_dom
Downloading https://github.com/b-fuze/deno-dom/releases/download/v0.1.17-alpha/libplugin.dylib
to /Users/devin/repos/dpastoor/quarto-cli/package/dist/bin/libplugin.dylib
Configuring deno_dom
Cleaning up
deno_dom complete.

Preparing Pandoc
Downloading Pandoc
Downloading https://github.com/jgm/pandoc/releases/download/2.16.2/pandoc-2.16.2-macOS.zip
to /Users/devin/repos/dpastoor/quarto-cli/package/dist/bin/pandoc-2.16.2-macOS.zip
Configuring Pandoc
Untar
Input: /Users/devin/repos/dpastoor/quarto-cli/package/dist/bin/pandoc-2.16.2-macOS.zip
Cwd: /Users/devin/repos/dpastoor/quarto-cli/package/dist/bin
Cleaning up
Pandoc complete.

Preparing Dart Sass Compiler
Downloading Dart Sass Compiler
Downloading https://github.com/sass/dart-sass/releases/download/1.32.8/dart-sass-1.32.8-macos-x64.tar.gz
to /Users/devin/repos/dpastoor/quarto-cli/package/dist/bin/dart-sass-1.32.8-macos-x64.tar.gz
Configuring Dart Sass Compiler
Untar
Input: /Users/devin/repos/dpastoor/quarto-cli/package/dist/bin/dart-sass-1.32.8-macos-x64.tar.gz
Cwd: /Users/devin/repos/dpastoor/quarto-cli/package/dist/bin
Cleaning up
Dart Sass Compiler complete.

Preparing esbuild javscript bundler
Downloading esbuild javscript bundler
Downloading https://registry.npmjs.org/esbuild-darwin-64/-/esbuild-darwin-64-0.12.10.tgz
to /Users/devin/repos/dpastoor/quarto-cli/package/dist/bin/esbuild-darwin-64
Configuring esbuild javscript bundler
Untar
Input: /Users/devin/repos/dpastoor/quarto-cli/package/dist/bin/esbuild-darwin-64
Cwd: /Users/devin/repos/dpastoor/quarto-cli/package/dist/bin
Cleaning up
esbuild javscript bundler complete.

Creating Quarto script
Creating Quarto Symlink
> Trying /usr/local/bin/quarto
> Failed
> Trying /Users/devin/bin/quarto
> Success
~/repos/dpastoor/quarto-cli
Downloading Deno Stdlib
99.9.9

note: if the ~/bin dir was not previously added to the users path this will instead state the message:

...
> Success
~/repos/dpastoor/quarto-cli
Quarto symlink doesn't appear to be configured.

checking resulting build

❯ quarto check

[✓] Checking Quarto installation......OK
      Version: 99.9.9
      Path: /Users/devin/repos/dpastoor/quarto-cli/package/dist/bin

[✓] Checking basic markdown render....OK

[✓] Checking Python 3 installation....OK
      Version: 3.9.9
      Path: /opt/homebrew/opt/python@3.9/bin/python3.9
      Jupyter: (None)

      Jupyter is not available in this Python installation.
      Install with python3 -m pip install jupyter

[✓] Checking R installation...........OK
      Version: 4.1.2
      Path: /Library/Frameworks/R.framework/Resources
      LibPaths:
        - /Library/Frameworks/R.framework/Versions/4.1-arm64/Resources/library
      rmarkdown: 2.11

[✓] Checking Knitr engine render......OK

some additional questions:

do different architectural DL's need to be adjusted for deps?

TODO:

pandoc - Please build a universal binary for macOS jgm/pandoc#6960 in digging into the homebrew manifest found different builds for osx 11/12 + different arm/x86 builds.
esbuild -
dart-sass
deno_dom?

more in-depth evaluation of cli functionality:

capabilities

❯ quarto capabilities
{
  "formats": [
    "html",
    "pdf",
    "docx",
    "odt",
    "pptx",
    "beamer",
    "revealjs",
    "gfm",
    "hugo",
    "epub",
    "asciidoc",
    "asciidoctor",
    "commonmark",
    "commonmark_x",
    "context",
    "docbook",
    "docbook4",
    "docbook5",
    "dokuwiki",
    "dzslides",
    "epub2",
    "epub3",
    "fb2",
    "haddock",
    "html4",
    "html5",
    "icml",
    "ipynb",
    "jats",
    "jats_archiving",
    "jats_articleauthoring",
    "jats_publishing",
    "jira",
    "json",
    "latex",
    "man",
    "markdown",
    "markdown_github",
    "markdown_mmd",
    "markdown_phpextra",
    "markdown_strict",
    "mediawiki",
    "ms",
    "muse",
    "native",
    "opendocument",
    "opml",
    "org",
    "plain",
    "rst",
    "rtf",
    "s5",
    "slideous",
    "slidy",
    "tei",
    "texinfo",
    "textile",
    "xwiki",
    "zimwiki"
  ],
  "themes": [
    "default",
    "zephyr",
    "vapor",
    "simplex",
    "cerulean",
    "cyborg",
    "quartz",
    "materia",
    "flatly",
    "litera",
    "slate",
    "lumen",
    "journal",
    "sandstone",
    "yeti",
    "morph",
    "united",
    "pulse",
    "lux",
    "superhero",
    "cosmo",
    "darkly",
    "solar",
    "spacelab",
    "sketchy",
    "minty"
  ],
  "python": {
    "versionMajor": 3,
    "versionMinor": 9,
    "versionPatch": 9,
    "versionStr": "3.9.9 (main, Nov 21 2021, 03:16:13)  [Clang 13.0.0 (clang-1300.0.29.3)]",
    "conda": false,
    "execPrefix": "/opt/homebrew/opt/python@3.9/Frameworks/Python.framework/Versions/3.9",
    "executable": "/opt/homebrew/opt/python@3.9/bin/python3.9",
    "jupyter_core": null,
    "nbformat": null,
    "nbclient": null,
    "ipykernel": null,
    "pyLauncher": false,
    "venv": true
  }
}

serve

TODO

render

dpastoor · 2022-01-01T15:33:14Z

@jjallaire if there are specific areas that might be high risk or specific things to test to kick this a bit more please point me any places relevant.

My next thought was to grab a couple of the example repos you've converted and run them through to see if everything renders properly

jjallaire · 2022-01-01T16:16:09Z

Hi @dpastoor thanks for starting to work on this!

The main thing here is just getting ARM builds of the binary dependencies (pandoc, esbuild, and dart-sass). We actually do not need an ARM build of deno_dom as it can fall back to a WASM version (which we can trigger by not defining DENO_DOM_PLUGIN here

quarto-cli/package/scripts/common/quarto

Line 84 in a2885bd

export DENO_DOM_PLUGIN=$QUARTO_BIN_PATH/deno_dom/libplugin.dylib

when on ARM).

It looks like this should already exist for esbuild (evanw/esbuild#820) and dart-saas (google/dart_cli_pkg#89).

For Pandoc we'd need to use the MacPorts build which I'm not 100% sure would work w/o including other dependencies from MacPorts? Perhaps better to wait for Pandoc proper to create a universal binary.

dpastoor · 2022-01-01T16:54:46Z

For Pandoc we'd need to use the MacPorts build which I'm not 100% sure would work w/o including other dependencies from MacPorts? Perhaps better to wait for Pandoc proper to create a universal binary.

Is there a reason we need macports? I think pandoc proper now has universal binaries (see linked issue in top PR comment). at least brew install pandoc now seems to seemlessly bring in the 2.16 arm build. I'll chase that upwards and see where brew is pulling it from.

I'll see if i can confirm.

For the other builds thanks for the links - I will also check if the build process is properly finding those or if need to put some additional conditional logic. It seems vague to see like https://registry.npmjs.org/esbuild-darwin-64/-/esbuild-darwin-64-0.12.10.tgz as the target being pulled from if that is the proper one.

I did manually invoke the pulled esbuild binary in dist/bin and cursorily that seemed to work fine so clearly something is coming in that doesn't explode immediately.

jjallaire · 2022-01-01T17:13:09Z

Sorry I didn't mean MacPorts I meant brew install. If the brew version works w/o dragging along a bunch of extra stuff then that's fine.

Note that if you have Rosetta installed then all of the x86 binaries are still going to work (so you may need to put your system in some sort of "don't use rosetta" state to truly test).

I think the ideal state is that we bundle both set of binaries into quarto and then have it automatically choose the right binaries at runtime (that way users don't need to distinguish processor architecture -- I'm sure some % will get this right but many, many others won't). You may need to sync with @dragonstyle on how to do this correctly.

If you could send us a contributor agreement (https://quarto.org/about.html#contribute) that would be much appreciated.

Thanks again for working on this!

dpastoor · 2022-01-01T20:53:02Z

cool deal - CLA sent.

Regarding pandoc - some googling led me to the fact you can find out the appropriate arch via asking file

so looking from the global pandoc:

❯ file (which pandoc)
/opt/homebrew/bin/pandoc: Mach-O 64-bit executable arm64

however currently in the bin dir, looks like its still pulling in the x86 version:

package/dist/bin on  main [?] 
❯ file ./pandoc
./pandoc: Mach-O 64-bit executable x86_64

same thing for esbuild:

package/dist/bin on  main [?] 
❯ file esbuild
esbuild: Mach-O 64-bit executable x86_64

so next step is update the logic for pulling these to get the arm versions as well (created a TODO section in the top level comment)

dpastoor · 2022-01-03T15:18:49Z

in digging around there are some non-trivial adjustments to make in the arm builds, as such I opened a new ticket (worth activating discussions in this repo?) that lays out potential options. In the meantime, regardless of the new interface, will need to investigate how resolveable arm build are to consider how robust the existing script will be in arm environments. It seems the two definite ones could be linux + mac - I'm not sure if windows arm is that popular/used, but seems like at least linux/mac would help with the objectives around bundling with rstudio and ancillary projects like rocker that have the dual builds for linux at this point rstudio/rstudio#8809

can be used as a building block for more specific abstractions or used directly. The response validator will help with enforcement of such as by validating headers/etc

…tion

dpastoor · 2022-01-05T15:05:24Z

note - this is on "hold" until build out a little more from discussion in #240 and some additional plumbing implementation such as #242

…refactor/platform-deps

dpastoor · 2022-01-06T21:24:33Z

this is now a superset of #242 as it leverages the downloadFile implementation + test setup, so the diff is currently larger in the PR since it incorporates all the changes from that branch. Once that gets merged in, the changeset will simplify a bit.

cscheid · 2022-09-02T17:49:29Z

(I'm going to close this one in favor of the feature/deno-1.25.1 branch which is up to date and more or less ready to merge.)

build: update mac to support arm build for m1 macs

de0a988

dpastoor marked this pull request as draft January 1, 2022 15:28

dpastoor mentioned this pull request Jan 3, 2022

platform arch refactor #240

Open

dpastoor added 5 commits January 3, 2022 14:42

build: switch to arch like other scripts such as the rstudio IDE config

ebe03f8

Merge branch 'patch-1' of github.com:dpastoor/quarto-cli

441e30b

test: add configuration for vscode + source map for testing

6959217

feat: new generalized download function

2280475

can be used as a building block for more specific abstractions or used directly. The response validator will help with enforcement of such as by validating headers/etc

feat: implement downloadFile abstraction close to existing implementa…

febca24

…tion

dpastoor added 4 commits January 6, 2022 09:23

refactor: update download invocations to downloadFile

270e265

Merge branch 'main' of https://github.com/quarto-dev/quarto-cli into …

e3463e2

…refactor/platform-deps

refactor: use existing downloadFile

80d6591

feat: extend esbuild for arm binaries

17fb23d

vnijs mentioned this pull request May 1, 2022

Quarto in docker aarch64 #781

Closed

cscheid closed this Sep 2, 2022

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

WIP: arm builds (for mac to start) #237

WIP: arm builds (for mac to start) #237

dpastoor commented Jan 1, 2022 •

edited

Loading

dpastoor commented Jan 1, 2022

jjallaire commented Jan 1, 2022

dpastoor commented Jan 1, 2022

jjallaire commented Jan 1, 2022

dpastoor commented Jan 1, 2022 •

edited

Loading

dpastoor commented Jan 3, 2022

dpastoor commented Jan 5, 2022

dpastoor commented Jan 6, 2022

cscheid commented Sep 2, 2022

WIP: arm builds (for mac to start) #237

WIP: arm builds (for mac to start) #237

Conversation

dpastoor commented Jan 1, 2022 • edited Loading

summary of changes:

TODO:

outputs:

checking resulting build

some additional questions:

do different architectural DL's need to be adjusted for deps?

more in-depth evaluation of cli functionality:

capabilities

serve

render

dpastoor commented Jan 1, 2022

jjallaire commented Jan 1, 2022

dpastoor commented Jan 1, 2022

jjallaire commented Jan 1, 2022

dpastoor commented Jan 1, 2022 • edited Loading

dpastoor commented Jan 3, 2022

dpastoor commented Jan 5, 2022

dpastoor commented Jan 6, 2022

cscheid commented Sep 2, 2022

dpastoor commented Jan 1, 2022 •

edited

Loading

dpastoor commented Jan 1, 2022 •

edited

Loading