node: Add yarn2 support #252

catsout · 2021-12-28T02:05:49Z

todo:

unit test

need to manually add $FLATPAK_BUILDER_BUILDDIR/flatpak-node/yarn2-setup.sh to build-commands

gasinvein · 2022-07-12T11:26:30Z

Isn't lockfile v2 format just normal YAML?

proletarius101 · 2022-07-20T09:00:28Z

I think you've managed to use this PR in https://sourcegraph.com/github.com/flathub/net.cozic.joplin_desktop@5420d1b2d6c0debe599222af99d048f10de2ac1f/-/blob/net.cozic.joplin_desktop.yml?L62. But just wondering how you get yarn2-setup.sh? Is that file introduced by this PR? Can I use it right now?

catsout · 2022-07-20T09:34:54Z

@proletarius101
yarn2-setup.sh is introduced by 'generated-sources.json'
Just run it before yarn install
You can use it right now, I have used it for a long time, also please me let know the result.

proletarius101 · 2022-07-20T12:56:16Z

@catsout I found there might be some errors: https://buildbot.flathub.org/#/builders/10/builds/12201

catsout · 2022-07-22T14:39:43Z

@proletarius101
I fixed error above, but when I try to build the standardnotes, I meet an error with react-native-keychain
It's a yarn 1.x bug yarnpkg/yarn#685, which cause yarn pack include .yarn/.yarnrc.yml.
So cache file isn't reproducible.
We need to patch lock file to solve this problem, like npm's git patch.

proletarius101 · 2022-07-22T15:10:15Z

@catsout Could you please push your fix so that I can check, or will you fix it yourself?

catsout · 2022-07-23T02:29:19Z

@proletarius101
You only need the yarn.lock on the top folder of project.
flatpak-node-generator -r isn't needed.

proletarius101 · 2022-07-23T08:58:11Z

flatpak-node-generator -r isn't needed.

True. But do you know why? I thought -r was designed for that usage.

I confirm that this PR works for standard notes.

proletarius101 · 2022-09-05T17:26:57Z

Hi @gasinvein @catsout any updates? :)

proletarius101 · 2022-09-06T12:39:12Z

f0d317a causes the following error: https://buildbot.flathub.org/#/builders/10/builds/13033

➤ YN0000: ┌ Link step
➤ YN0007: │ husky@npm:3.1.0 must be built because it never has been before or the last one failed
➤ YN0007: │ styled-components@npm:5.3.5 [76447] must be built because it never has been before or the last one failed
➤ YN0007: │ electron@npm:17.4.2 must be built because it never has been before or the last one failed
➤ YN0007: │ keytar@npm:7.9.0 must be built because it never has been before or the last one failed
➤ YN0007: │ styled-components@npm:5.3.5 [2efff] must be built because it never has been before or the last one failed
➤ YN0007: │ core-js@npm:3.23.3 must be built because it never has been before or the last one failed
➤ YN0007: │ node-sass@npm:6.0.1 must be built because it never has been before or the last one failed
➤ YN0007: │ core-js-pure@npm:3.23.3 must be built because it never has been before or the last one failed
➤ YN0007: │ detox@npm:19.7.1 [7a3f4] must be built because it never has been before or the last one failed
➤ YN0007: │ react-native-sodium-jsi@npm:1.2.0 [7a3f4] must be built because it never has been before or the last one failed
➤ YN0007: │ styled-components@npm:5.3.5 [7a3f4] must be built because it never has been before or the last one failed
➤ YN0007: │ electron@npm:11.5.0 must be built because it never has been before or the last one failed
➤ YN0007: │ dtrace-provider@npm:0.8.8 must be built because it never has been before or the last one failed
➤ YN0007: │ node-sass@npm:7.0.1 must be built because it never has been before or the last one failed
➤ YN0007: │ core-js@npm:2.6.12 must be built because it never has been before or the last one failed
➤ YN0009: │ electron@npm:11.5.0 couldn't be built successfully (exit code 1, logs can be found here: /tmp/xfs-cdf77bf5/build.log)
➤ YN0009: │ electron@npm:17.4.2 couldn't be built successfully (exit code 1, logs can be found here: /tmp/xfs-8a1b771e/build.log)
➤ YN0000: └ Completed in 4m 43s
➤ YN0000: Failed with errors in 5m 41s
FB: host_command_exited_cb 1777080 256

afc6911 doesn't

catsout · 2022-10-25T07:43:50Z

Seems I forgot name, _ = name.rsplit('@', 1) at afc6911

muelli · 2023-02-28T13:15:18Z

I'm trying to run this on https://github.com/mifi/lossless-cut/blob/37e49d5682a4dada498f3fe7ec3d2d2cf530a7df/yarn.lock but it fails:

python3 -m flatpak_node_generator yarn ~/vcs/lossless-cut/package.json --output ~/vcs/no.mifi.losslesscut/generated-sources.json
Reading packages from lockfiles...
Traceback (most recent call last):
  File "/usr/lib/python3.8/runpy.py", line 194, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "/usr/lib/python3.8/runpy.py", line 87, in _run_code
    exec(code, run_globals)
  File "dev/flatpak-builder-tools/node/flatpak_node_generator/__main__.py", line 3, in <module>
    main()
  File "dev/flatpak-builder-tools/node/flatpak_node_generator/main.py", line 277, in main
    asyncio.run(_async_main())
  File "/usr/lib/python3.8/asyncio/runners.py", line 44, in run
    return loop.run_until_complete(main)
  File "/usr/lib/python3.8/asyncio/base_events.py", line 616, in run_until_complete
    return future.result()
  File "dev/flatpak-builder-tools/node/flatpak_node_generator/main.py", line 198, in _async_main
    packages.update(lockfile_provider.process_lockfile(lockfile))
  File "dev/flatpak-builder-tools/node/flatpak_node_generator/providers/yarn.py", line 145, in process_lockfile
    lock_dict: Dict[str, Any] = self.parse_lockfile(lockfile)
  File "dev/flatpak-builder-tools/node/flatpak_node_generator/providers/yarn.py", line 64, in parse_lockfile
    assert level <= len(parent_entries) - 1
AssertionError

catsout · 2023-02-28T13:50:03Z

@muelli
You need https://github.com/mifi/lossless-cut/raw/37e49d5682a4dada498f3fe7ec3d2d2cf530a7df/yarn.lock.
I rebased to new master to fix esbuild version.

catsout · 2023-02-28T14:03:36Z

example: https://github.com/flathub/net.cozic.joplin_desktop/blob/master/net.cozic.joplin_desktop.yml

muelli · 2023-03-01T14:29:34Z

I'm still experiencing the error.

I'm on c278045 now. And I'm calling like this: python3 -m flatpak_node_generator yarn ~/vcs/lossless-cut/package.json --output ~/vcs/no.mifi.losslesscut/generated-sources.json. Does it actually work for you?

catsout · 2023-03-01T14:35:08Z

Yes, it works.

wget https://github.com/mifi/lossless-cut/raw/37e49d5682a4dada498f3fe7ec3d2d2cf530a7df/yarn.lock

❯ flatpak-node-generator yarn yarn.lock                                                  
Reading packages from lockfiles...
1036 packages read.
Generating packages [1036/1036] esbuild @ 0.16.17                                             
Wrote 1064 source(s).

❯ git show --summary 
commit c278045d7cc73437945b3dfb919fc7aea5d43cb5 (HEAD -> yarn2, origin/yarn2)

muelli · 2023-03-02T14:04:02Z

ah, silly me. I have indeed taken the wrong file.

I has generated the sources file. But it doesn't seem to working. I wonder whether the upstream file is broken, though. I'm trying to make it work here: flathub/no.mifi.losslesscut#21

Running: yarn install
FB: Running 'flatpak build --die-with-parent --env=FLATPAK_BUILDER_BUILDDIR=/run/build/lossless-cut --nofilesystem=host:reset [...]  /bin/sh -c 'yarn install'' on host
➤ YN0000: ┌ Resolution step
➤ YN0060: │ i18next-scanner@npm:4.1.0 provides acorn (p26271) with version 8.7.1, which doesn't satisfy what acorn-dynamic-import requests
➤ YN0002: │ lossless-cut@workspace:. doesn't provide react-is (p8fea6), requested by evergreen-ui
➤ YN0000: │ Some peer dependencies are incorrectly met; run yarn explain peer-requirements <hash> for details, where <hash> is the six-letter p-prefixed code
➤ YN0000: └ Completed
➤ YN0000: ┌ Fetch step
➤ YN0013: │ acorn-class-fields@npm:0.3.7 can't be found in the cache and will be fetched from the remote registry
➤ YN0013: │ acorn-dynamic-import@npm:4.0.0 can't be found in the cache and will be fetched from the remote registry
➤ YN0013: │ acorn-private-class-elements@npm:0.2.7 can't be found in the cache and will be fetched from the remote registry
➤ YN0013: │ acorn-private-methods@npm:0.3.3 can't be found in the cache and will be fetched from the remote registry
➤ YN0001: │ Error: acorn-class-fields@npm:0.3.7: Request to 'https://registry.yarnpkg.com/acorn-class-fields/-/acorn-class-fields-0.3.7.tgz' has been blocked because of your configuration settings
    at kRe (/run/build/lossless-cut/.yarn/releases/yarn-3.3.0.cjs:392:9810)

I cannot find acorn-class-fields in the yarn.lock file. I wonder why it thinks it's necessary.

catsout · 2023-03-02T15:01:10Z

You need to use https://raw.githubusercontent.com/mifi/lossless-cut/v3.52.0/yarn.lock.

catsout · 2023-03-02T15:07:22Z

converting cache to zip
converting finished
Running: yarn install
➤ YN0000: ┌ Resolution step
➤ YN0060: │ i18next-scanner@npm:4.1.0 provides acorn (p26271) with version 8.7.1, which doesn't satisfy what acorn-dynamic-import requests
➤ YN0002: │ lossless-cut@workspace:. doesn't provide react-is (p8fea6), requested by evergreen-ui
➤ YN0000: │ Some peer dependencies are incorrectly met; run yarn explain peer-requirements <hash> for details, where <hash> is the six-letter p-prefixed code
➤ YN0000: └ Completed
➤ YN0000: ┌ Fetch step
➤ YN0013: │ resolve@patch:resolve@npm%3A1.21.0#~builtin<compat/resolve>::version=1.21.0&hash=
➤ YN0013: │ resolve@patch:resolve@npm%3A1.22.1#~builtin<compat/resolve>::version=1.22.1&hash=
➤ YN0013: │ resolve@patch:resolve@npm%3A2.0.0-next.3#~builtin<compat/resolve>::version=2.0.0-
➤ YN0019: │ locator appears to be unused - removing
➤ YN0000: └ Completed in 0s 423ms
➤ YN0000: ┌ Link step
➤ YN0007: │ electron@npm:23.1.0 must be built because it never has been before or the last one failed
..........

muelli · 2023-03-02T16:05:46Z

k. works for me. Thanks!

So, what is needed to merge this?

Would it be helpful to detect whether the setup script was run? It could then fail (or warn on stderr) to indicate the lack of setup. Maybe the setup script can set an environment variable or something and the generated-source can embed a script to check for that.

muelli · 2023-04-07T14:54:40Z

@gasinvein do you have any doubt about merging this? It provides the functionality I need for https://github.com/flathub/no.mifi.losslesscut/ and seems to work well.
Unless anybody objects, I would merge the next time I need this.

gasinvein · 2023-04-07T17:31:12Z

I would like to see at least some explanation of the changes proposed here. Massive code dumps like this, especially without comments, are nigh impossible to review.

refi64 · 2023-04-07T21:56:10Z

Okay so, I will admit, I did not realize this was undrafted until a few weeks ago, and then promptly forgot about it again.

Ditto on the above comment, but also...do we really need a 250 LoC yarn plugin? Would very much like to know what it does in the very least...seems to be related to converting something to zip files, triggered by some resolution entry?

catsout · 2023-04-08T02:30:55Z

lock file

__metadata:
  version: 6
  cacheKey: 8

"@babel/code-frame@npm:^7.0.0":
  version: 7.16.7
  resolution: "@babel/code-frame@npm:7.16.7"
  dependencies:
    "@babel/highlight": ^7.16.7
  checksum: db2f7faa31bc2c9cf63197b481b30ea57147a5fc1a6fab60e5d6c02cdfbf6de8e17b5121f99917b3dabb5eeb572da078312e70697415940383efc140d4e0808b
  languageName: node
  linkType: hard

...

From: https://github.com/yarnpkg/berry/blob/master/packages/yarnpkg-core/sources/Resolver.ts


/**
 * Resolvers are the components that do all the lifting needed in order to
 * produce a lockfile. In clear, they transfom the following:
 *
 *   webpack@^4.0.0
 *
 * into this:
 *
 *   webpack@4.28.0 | dependencies: ajv@^6.1.0, ajv-keyword@^3.1.0, ...
 *
 * In order to do this, they have three different data structures used to
 * represents the various states of the package resolution:
 *
 *   - **Descriptors** contain a package name and a range (for example, using
 *     the previous example, "^4.0.0" would be the range). This range might
 *     point to multiple possible resolutions, so a descriptor alone isn't
 *     enough to fetch the package data from its remote location.
 *
 *   - **Locators** contain a package name and a reference that is used to
 *     both uniquely identify a package and fetch it from its remote location.
 *     To keep using the same example, "4.28.0" would be the reference. Note
 *     that locators have a funny property: they also are valid descriptors!
 *
 *   - **Packages** are locators that made it big. While locators are quite
 *     small, package definitions are relatively fat and contain much more
 *     information than their cousins - for example the dependency list of the
 *     package.
 */

More type detail: https://github.com/yarnpkg/berry/blob/%40yarnpkg/shell/3.2.5/packages/yarnpkg-core/sources/types.ts#L81

locator {
  locatorHash: string;
  scope: string | null;
  name: string;
  reference: string;
}

So:

resolution  <->  locator  <->  package registry url

checksum -> sha512 of zip cache file

cache file

filename:

@babel-code-frame-npm-7.16.0-13dafb7fe1-8961d0302e.zip

https://github.com/yarnpkg/berry/blob/%40yarnpkg/shell/3.2.5/packages/yarnpkg-core/sources/Cache.ts#L99
https://github.com/yarnpkg/berry/blob/%40yarnpkg/shell/3.2.5/packages/yarnpkg-core/sources/structUtils.ts#L625

`@${locator.scope}-${locator.name}-${humanProtocol}-${humanVersion}-${locator.locatorHash.slice(0, 10)}-${zip_cache_checksum.slice(0, 10)}`

humanProtocol, humanVersion come from locator.reference

cache file is valid only when:

zip cache checksum == checksum in lockfile == checksum in zip cache filename

fetcher

https://github.com/yarnpkg/berry/blob/%40yarnpkg/shell/3.2.5/packages/plugin-npm/sources/NpmSemverFetcher.ts#L49
The tgz will be converted to zip

the js plugin

The py generate the flathub source of tgz file.
And I use the filename to save the locator and checksum info.
filename = f'{self.name_base64_locator(locator, resolution)}-{source.integrity.digest[:10]}.tgz'

Then in js plugin, loop the file, extract the locator info to generate the zip cache filename.
And convert the tgz to zip, using the same tgzUtils of yarn to get same checksum.

catsout · 2023-04-08T02:37:28Z

git source

yarn uses yarn_v1 pack, npm pack to get zip from git repo.
https://github.com/yarnpkg/berry/blob/%40yarnpkg/shell/3.2.3/packages/yarnpkg-core/sources/scriptUtils.ts#L215

catsout · 2023-04-08T02:44:12Z

patch lockfile

There is a yarn 1.x bug yarnpkg/yarn#685, which cause yarn pack include .yarn/.yarnrc.yml.
So the cache file isn't reproducible.
We need to patch lock file to solve this problem, like npm's git patch.

Lunarequest · 2023-04-30T17:15:05Z

Hi, I've been using this for a bit and have discovered this doesn't work with yarn canary/yarn 4 the plugin used always crashes with this error,

yarn cacheFolder: /run/build/NightPDF/flatpak-node/yarn-mirror/yarn-berry
converting cache to zip
Type Error: ZipFS is not a constructor
    at convertToZip (/run/build/NightPDF/.yarn/plugins/flatpak-builder.cjs:162:25)
    at async /run/build/NightPDF/.yarn/plugins/flatpak-builder.cjs:223:13
    at async Promise.all (index 0)
    at async convertToZipCommand.execute (/run/build/NightPDF/.yarn/plugins/flatpak-builder.cjs:248:11)
    at async convertToZipCommand.validateAndExecute (/run/build/NightPDF/.yarn/releases/yarn-4.0.0-rc.42.cjs:91:12775)
    at async vo.run (/run/build/NightPDF/.yarn/releases/yarn-4.0.0-rc.42.cjs:95:3074)
    at async vo.runExit (/run/build/NightPDF/.yarn/releases/yarn-4.0.0-rc.42.cjs:95:3258)
    at async o (/run/build/NightPDF/.yarn/releases/yarn-4.0.0-rc.42.cjs:386:4014)
    at async r (/run/build/NightPDF/.yarn/releases/yarn-4.0.0-rc.42.cjs:386:2118)
Error: module NightPDF: Child process exited with code 1

Lunarequest · 2023-04-30T17:22:31Z

zipfs seems to have been moved to @yarnpkg/libzip

Note that updating the generated sources is more complicated at the moment until flatpak-node-generator starts supporting Yarn version 2+. I had to use the code in pull request flatpak/flatpak-builder-tools#252 which adds support for Yarn version 2+. The exact commit that I used is: flatpak/flatpak-builder-tools@eb43ec2. A newer update of this pull request also removes the need to execute "$FLATPAK_BUILDER_BUILDDIR/flatpak-node/yarn2-setup.sh" before running "yarn install" so I removed that line from the build-commands. Commands that I used to update the generated sources are: wget https://raw.githubusercontent.com/mifi/lossless-cut/refs/tags/v3.64.0/yarn.lock pipx run "flatpak_node_generator @ git+https://github.com/flatpak/flatpak-builder-tools.git@refs/pull/252/head#subdirectory=node" yarn yarn.lock --output generated-sources.json rm yarn.lock

tjanez · 2024-11-11T10:53:50Z

@catsout , I've used the current latest commit in this PR (eb43ec2) to successfully generate generated-sources.json for LosslessCut version 3.64.0. See flathub/no.mifi.losslesscut#73.

However, the Flatpak build process now fails at yarn install step with countless errors of the form:

➤ YN0001: │ RequestError: getaddrinfo EAI_AGAIN registry.yarnpkg.com
    at ClientRequest.<anonymous> (/run/build/lossless-cut/.yarn/releases/yarn-4.4.0.cjs:147:14258)
    at Object.onceWrapper (node:events:632:26)
    at ClientRequest.emit (node:events:529:35)
    at u.emit (/run/build/lossless-cut/.yarn/releases/yarn-4.4.0.cjs:142:14855)
    at TLSSocket.socketErrorListener (node:_http_client:501:9)
    at TLSSocket.emit (node:events:517:28)
    at emitErrorNT (node:internal/streams/destroy:151:8)
    at emitErrorCloseNT (node:internal/streams/destroy:116:3)
    at process.processTicksAndRejections (node:internal/process/task_queues:82:21)
    at GetAddrInfoReqWrap.onlookup [as oncomplete] (node:dns:107:26)

The LosslessCut project currently uses Yarn 4.4.0.

Do you have any suggestion on which commands to use in build-commands step to make this work?

catsout marked this pull request as draft December 28, 2021 14:35

catsout force-pushed the yarn2 branch 3 times, most recently from 159ca9f to ff204fd Compare December 30, 2021 04:53

catsout force-pushed the yarn2 branch 2 times, most recently from d22a790 to 8aaa053 Compare July 12, 2022 08:37

catsout marked this pull request as ready for review July 12, 2022 08:44

catsout force-pushed the yarn2 branch from 8aaa053 to afc6911 Compare July 22, 2022 22:53

catsout force-pushed the yarn2 branch from afc6911 to f0d317a Compare August 12, 2022 23:14

catsout force-pushed the yarn2 branch from f0d317a to a596720 Compare October 24, 2022 23:46

catsout force-pushed the yarn2 branch 3 times, most recently from 799eb2a to 0894c91 Compare October 25, 2022 07:58

catsout force-pushed the yarn2 branch from 0894c91 to c278045 Compare February 28, 2023 13:49

node: Add yarn2 support

ff88d3a

catsout force-pushed the yarn2 branch from c278045 to ff88d3a Compare July 17, 2023 15:24

ProjectSynchro mentioned this pull request Feb 8, 2024

New upstream version (2.0.0) flathub/app.ytmdesktop.ytmdesktop#3

Closed

yarn: update js plugin and remove setup.sh

f55e1b6

hfiguiere added the yarn label Apr 5, 2024

update firefox-ubuntu to 20.04

eb43ec2

benni347 mentioned this pull request Jul 15, 2024

flatpak-node-generator #388

Open

tjanez mentioned this pull request Nov 11, 2024

Update to version 3.64.0 flathub/no.mifi.losslesscut#73

Open

tjanez mentioned this pull request Nov 25, 2024

Update to 3.61.1 flathub/no.mifi.losslesscut#62

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

node: Add yarn2 support #252

node: Add yarn2 support #252

catsout commented Dec 28, 2021 •

edited

Loading

gasinvein commented Jul 12, 2022

proletarius101 commented Jul 20, 2022

catsout commented Jul 20, 2022

proletarius101 commented Jul 20, 2022

catsout commented Jul 22, 2022 •

edited

Loading

proletarius101 commented Jul 22, 2022

catsout commented Jul 23, 2022

proletarius101 commented Jul 23, 2022

proletarius101 commented Sep 5, 2022

proletarius101 commented Sep 6, 2022

catsout commented Oct 25, 2022

muelli commented Feb 28, 2023

catsout commented Feb 28, 2023

catsout commented Feb 28, 2023

muelli commented Mar 1, 2023

catsout commented Mar 1, 2023 •

edited

Loading

muelli commented Mar 2, 2023

catsout commented Mar 2, 2023

catsout commented Mar 2, 2023

muelli commented Mar 2, 2023

muelli commented Apr 7, 2023

gasinvein commented Apr 7, 2023

refi64 commented Apr 7, 2023

catsout commented Apr 8, 2023 •

edited

Loading

catsout commented Apr 8, 2023

catsout commented Apr 8, 2023

Lunarequest commented Apr 30, 2023

Lunarequest commented Apr 30, 2023

tjanez commented Nov 11, 2024

node: Add yarn2 support #252

Are you sure you want to change the base?

node: Add yarn2 support #252

Conversation

catsout commented Dec 28, 2021 • edited Loading

gasinvein commented Jul 12, 2022

proletarius101 commented Jul 20, 2022

catsout commented Jul 20, 2022

proletarius101 commented Jul 20, 2022

catsout commented Jul 22, 2022 • edited Loading

proletarius101 commented Jul 22, 2022

catsout commented Jul 23, 2022

proletarius101 commented Jul 23, 2022

proletarius101 commented Sep 5, 2022

proletarius101 commented Sep 6, 2022

catsout commented Oct 25, 2022

muelli commented Feb 28, 2023

catsout commented Feb 28, 2023

catsout commented Feb 28, 2023

muelli commented Mar 1, 2023

catsout commented Mar 1, 2023 • edited Loading

muelli commented Mar 2, 2023

catsout commented Mar 2, 2023

catsout commented Mar 2, 2023

muelli commented Mar 2, 2023

muelli commented Apr 7, 2023

gasinvein commented Apr 7, 2023

refi64 commented Apr 7, 2023

catsout commented Apr 8, 2023 • edited Loading

lock file

cache file

fetcher

the js plugin

catsout commented Apr 8, 2023

git source

catsout commented Apr 8, 2023

patch lockfile

Lunarequest commented Apr 30, 2023

Lunarequest commented Apr 30, 2023

tjanez commented Nov 11, 2024

catsout commented Dec 28, 2021 •

edited

Loading

catsout commented Jul 22, 2022 •

edited

Loading

catsout commented Mar 1, 2023 •

edited

Loading

catsout commented Apr 8, 2023 •

edited

Loading