Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

yarn create @agoric/dapp demo fails with yarn 4 (nodeLinker pnp) #9852

Open
0xpatrickdev opened this issue Aug 6, 2024 · 8 comments
Open
Assignees
Labels
bug Something isn't working

Comments

@0xpatrickdev
Copy link
Member

Describe the bug

yarn create @agoric/dapp demo fails with yarn 4. The error message suggests it might be due to Plug'n'Play (the default nodeLinker setting).

To Reproduce

Steps to reproduce the behavior:

  1. Ensure using node 18.18.x and yarn 4.x
node -v
v18.18.2

yarn -v
4.3.1
  1. Run
yarn create @agoric/dapp demo
  1. See error
yarn create @agoric/dapp demo
➤ YN0000: · Yarn 4.3.1
➤ YN0000: ┌ Resolution step
➤ YN0085: │ + @agoric/create-dapp@npm:0.1.1-u16.2, @agoric/access-token@npm:0.4.22-upgrade-16a-dev-fb592e4.0, @agoric/assert@npm:0.6.1-upgrade-16a-dev-fb592e4.0, and 423 more.
➤ YN0000: └ Completed in 1s 361ms
➤ YN0000: ┌ Post-resolution validation
➤ YN0086: │ Some peer dependencies are incorrectly met by dependencies; run yarn explain peer-requirements for details.
➤ YN0000: └ Completed
➤ YN0000: ┌ Fetch step
➤ YN0013: │ 426 packages were added to the project (+ 121.3 MiB).
➤ YN0000: └ Completed in 0s 279ms
➤ YN0000: ┌ Link step
➤ YN0000: │ ESM support for PnP uses the experimental loader API and is therefore experimental
➤ YN0007: │ protobufjs@npm:6.11.4 must be built because it never has been before or the last one failed
➤ YN0007: │ better-sqlite3@npm:9.6.0 must be built because it never has been before or the last one failed
➤ YN0007: │ @agoric/xsnap@npm:0.14.3-upgrade-16a-dev-fb592e4.0 must be built because it never has been before or the last one failed
➤ YN0007: │ microtime@npm:3.1.1 must be built because it never has been before or the last one failed
➤ YN0000: └ Completed in 18s 786ms
➤ YN0000: · Done with warnings in 20s 477ms

/private/var/folders/tz/jycbd2h54t39vpjnypzwktc80000gn/T/xfs-8f80eeba/dlx-25296/.pnp.cjs:10434
      Error.captureStackTrace(firstError);
            ^

Error: @agoric/create-dapp tried to access .bin, but it isn't declared in its dependencies; this makes the require call ambiguous and unsound.

Required package: .bin (via ".bin/agoric")
Required by: @agoric/create-dapp@npm:0.1.1-u16.2 (via /private/var/folders/tz/jycbd2h54t39vpjnypzwktc80000gn/T/xfs-8f80eeba/dlx-25296/.yarn/cache/@agoric-create-dapp-npm-0.1.1-u16.2-4cdbbfdecb-a5578b9b12.zip/node_modules/@agoric/create-dapp/src/)

Require stack:
- /private/var/folders/tz/jycbd2h54t39vpjnypzwktc80000gn/T/xfs-8f80eeba/dlx-25296/.yarn/cache/@agoric-create-dapp-npm-0.1.1-u16.2-4cdbbfdecb-a5578b9b12.zip/node_modules/@agoric/create-dapp/src/create-dapp.js
    at require$$0.Module._resolveFilename (/private/var/folders/tz/jycbd2h54t39vpjnypzwktc80000gn/T/xfs-8f80eeba/dlx-25296/.pnp.cjs:10434:13)
    at Function.resolve (node:internal/modules/cjs/helpers:125:19)
    at file:///private/var/folders/tz/jycbd2h54t39vpjnypzwktc80000gn/T/xfs-8f80eeba/dlx-25296/.yarn/cache/@agoric-create-dapp-npm-0.1.1-u16.2-4cdbbfdecb-a5578b9b12.zip/node_modules/@agoric/create-dapp/src/create-dapp.js:8:27
    at ModuleJob.run (node:internal/modules/esm/module_job:194:25)

Node.js v18.18.2

Expected behavior

Users should be able to use yarn create @agoric/dapp with yarn 4

Platform Environment

  • what OS are you using? what version of Node.js?
    OS X, Node 18.18. Also fails on 18.20

  • is there anything special/unusual about your platform?
    No

  • what version of the Agoric-SDK are you using? (run git describe --tags --always)
    create-dapp depends on "agoric": "^0.21.1"

@0xpatrickdev 0xpatrickdev added the bug Something isn't working label Aug 6, 2024
@0xpatrickdev
Copy link
Member Author

I suspect the solve here is adding a .yarnrc.yml with nodeLinker: node-modules, but am unsure how to test this locally. (I am familiar with yarn link but not for something like yarn create, where a project/package.json doesn't yet exist).

I also suspect we will need to tell CI (which currently uses yarn 1.22.x) to run this with yarn 4.x, and also add a test like:

diff --git a/packages/create-dapp/test/sanity.test.js b/packages/create-dapp/test/sanity.test.js
index f4424d587..4e6395a68 100644
--- a/packages/create-dapp/test/sanity.test.js
+++ b/packages/create-dapp/test/sanity.test.js
@@ -15,4 +15,5 @@ test('sanity', async t => {
   t.is(await myMain(['--help']), 0, '--help exits zero');
   t.is(await myMain(['--version']), 0, '--version exits zero');
   t.is(await myMain(['--zorgar']), 1, 'unknown flag fails');
+  t.is(await myMain(['demo']), 0, 'create-dapp demo exits 0');
 });

@0xpatrickdev
Copy link
Member Author

Note: #451 is a potential solution, but probably the one with the longest lead time. It seems like we have two interim solutions, the second of which might close this issue:

  1. Document that developers should use yarn 1.22.x in Getting Started. See Getting Started > Installing yarn does not specify a version documentation#1174

    • Note: dapp-agoric-basics uses yarn 1.22.x at the time of writing, while dapp-agoric-basics seems to be using yarn 4.x. I'm not sure how this will affect things, but something to keep in mind
  2. Add a .yarnrc.yml with nodeLinker: node-modules to /packages/create-dapp, as mentioned above

@mujahidkay
Copy link
Member

but am unsure how to test this locally.

I am currently using verdaccio to proxy a private locally-hosted registry and overriding the npmRegistryServer with it to actually pull packages from it.

@mujahidkay
Copy link
Member

mujahidkay commented Aug 22, 2024

So just adding a .yarnrc.yml with nodeLinker: node-modules didn't work. I believe yarn4 has pnp globally enabled - so I tried to switch it to use node-modules explicitly (can use either yarn config set nodeLinker node-modules or put it in a .yarnrc.yml in the directory you are gonna run yarn create @agoric/dapp demo.

It got rid of the pnp error but it threw the following:

node:internal/errors:496
    ErrorCaptureStackTrace(err);
    ^

Error [ERR_MODULE_NOT_FOUND]: Cannot find module '/private/var/folders/5n/lpny95rx2mz3jwpr6vbbsvg80000gn/T/xfs-93f97866/dlx-84923/node_modules/@agoric/vats/tools/board-utils.js' imported from /private/var/folders/5n/lpny95rx2mz3jwpr6vbbsvg80000gn/T/xfs-93f97866/dlx-84923/node_modules/agoric/src/lib/format.js
    at new NodeError (node:internal/errors:405:5)
    at finalizeResolution (node:internal/modules/esm/resolve:327:11)
    at moduleResolve (node:internal/modules/esm/resolve:980:10)
    at defaultResolve (node:internal/modules/esm/resolve:1193:11)
    at ModuleLoader.defaultResolve (node:internal/modules/esm/loader:404:12)
    at ModuleLoader.resolve (node:internal/modules/esm/loader:373:25)
    at ModuleLoader.getModuleJob (node:internal/modules/esm/loader:250:38)
    at ModuleWrap.<anonymous> (node:internal/modules/esm/module_job:76:39)
    at link (node:internal/modules/esm/module_job:75:36) {
  url: 'file:///private/var/folders/5n/lpny95rx2mz3jwpr6vbbsvg80000gn/T/xfs-93f97866/dlx-84923/node_modules/@agoric/vats/tools/board-utils.js',
  code: 'ERR_MODULE_NOT_FOUND'
}

This^ was resolved by bumping agoric dependency to ^0.22.0-u16.2 in packages/create-dapp/package.json.

I believe the user triggering yarn create has to set their nodeLinker to node-modules using either of the two ways i mentioned above, and then run the yarn create command to be able to install the package and run the dapp successfully.

What do you think/suggest @0xpatrickdev ? With this^, we would need to update the docs accordingly as well to mention those steps.

@0xpatrickdev
Copy link
Member Author

@mujahidkay thanks for the great analysis here. I responded in the PR

mergify bot added a commit that referenced this issue Aug 29, 2024
closes: #XXXX
refs: #9852

## Description
As described in the issue linked above, currently, if a user has yarn4 and they run `yarn create @agoric/dapp demo` the command fails because it uses pnp as the linker by default. 

The correct way to go about it imo is to properly document that if a user is using yarn4, they have to configure it to use `node-modules` as its `nodeLinker`.  This can either be done by creating a `.yarnrc.yml` in the directory they are supposedly running the `yarn create` command, or by globally configuring it via `yarn config set nodeLinker node-modules`

bumping the agoric package is needed for yarn4 otherwise it throws an [error](#9852 (comment)) 

## Steps to Test
### Publishing to a local, private registry
1. install and start verdaccio
2. set npm registry server to the localhost address exposed by verdaccio 
```
npm config set registry http://localhost:4873/
```
3. inside the create-dapp folder, run
```
npm publish 
``` 
you may have to auth a user. You can use `npm adduser` and provide test username, password (for verdaccio).
You can also explicitly publish to your private registry using `npm publish --registry=http://localhost:4873`
### Downloading the package from local registry
1. Set nodeLinker to node-modules:
```
yarn config set nodeLinker node-modules
```
2. point npmRegistryServer to localhost:
```
yarn config get npmRegistryServer
```
3. add localhost to unsafe http whitelist:
```
yarn config set unsafeHttpWhitelist 'localhost'
```
4. pull the package using 
```
yarn create @agoric/dapp demo
```
Ensure the package being pulled is the one you published by matching the package version. 

P.S. the above can be achieved by using a `.yarnrc.yml` with the same configuration as well. But for the above instructions, please don't forget to set them to their defaults after testing :) 
### Security Considerations


### Scaling Considerations


### Documentation Considerations

Add a section on docs site while setting up yarn to properly configure nodeLinker

### Testing Considerations

TODO: add CI step to test package download using different yarn versions via [verdaccio](https://verdaccio.org/)

### Upgrade Considerations
@mhofman
Copy link
Member

mhofman commented Aug 30, 2024

It got rid of the pnp error but it threw the following:

I cannot reproduce this error, for me after setting the linker, it inits the dapp correctly

➤ YN0085: │ + @agoric/create-dapp@npm:0.1.1-u16.2, @agoric/access-token@npm:0.4.22-upgrade-16a-dev-fb592e4.0, @agoric/assert@npm:0.6.1-upgrade-16a-dev-fb592e4.0, and 423 more.

The fact that we ended up resolving to upgrade-16a-dev packages is not good, we need to revisit how we generate pre-release tags for dev branches.

@mujahidkay
Copy link
Member

It got rid of the pnp error but it threw the following:

I cannot reproduce this error, for me after setting the linker, it inits the dapp correctly

➤ YN0085: │ + @agoric/create-dapp@npm:0.1.1-u16.2, @agoric/access-token@npm:0.4.22-upgrade-16a-dev-fb592e4.0, @agoric/assert@npm:0.6.1-upgrade-16a-dev-fb592e4.0, and 423 more.

The fact that we ended up resolving to upgrade-16a-dev packages is not good, we need to revisit how we generate pre-release tags for dev branches.

Thanks @mhofman for the insights! I think I understand what happened here. For me, it was pulling @agoric/create-dapp@npm:0.1.0 which resulted in the above error. Now, I'm assuming this shouldn't ever happen with npm registry and could have been due to a misconfigured localhost environment.

Do you know why the agoric-sdk/packages/create-dapp/package.json still points to 0.1.0 (different from what's on npm)?

@mhofman
Copy link
Member

mhofman commented Aug 30, 2024

Do you know why the agoric-sdk/packages/create-dapp/package.json still points to 0.1.0 (different from what's on npm)?

Our release process currently does not have a well defined way to merge back the changes of the release branch into master, which means that we've not been able to incorporate the version and generated changelog changes of releases back into trunk.

It is partially blocked technically on #9490, but we do need a process in place anyway.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working
Projects
None yet
Development

No branches or pull requests

3 participants