Skip to content
This repository has been archived by the owner on Feb 12, 2024. It is now read-only.

Package size is 300 Mb #2550

Closed
rumkin opened this issue Oct 18, 2019 · 21 comments
Closed

Package size is 300 Mb #2550

rumkin opened this issue Oct 18, 2019 · 21 comments
Labels
exp/wizard Extensive knowledge (implications, ramifications) required P2 Medium: Good to have, but can wait until someone steps up status/ready Ready to be worked topic/perf Performance

Comments

@rumkin
Copy link

rumkin commented Oct 18, 2019

  • Version: 0.38.0
  • Platform: Node.JS
  • Subsystem:

Type: Overweight

Severity: Medium

Description: Package weight is 336 Mb

Steps to reproduce the error:

npm i ipfs
du -sh node_modules

Current IPFS package weight too much, it takes too long time to install. It hits performance on my pipeline and increases my bundle size dramatically.

@rumkin rumkin changed the title 300 Mb Package size is 300 Mb Oct 18, 2019
@alanshaw
Copy link
Member

@rumkin can I check you're quoting the size of your node_modules folder on disk after unzipping all the modules and building all the native deps?

FWIW a browser build of ipfs weighs in at around 600kb currently (which is still big) but is nowhere near 300mb!

Looking at the node_modules directory a lot of this seems to be the published standalone browser builds for each module.

@rumkin
Copy link
Author

rumkin commented Oct 19, 2019

I've updated steps to reproduce and added size measure step, which is du -sh node_modules. I've also checked bundlephobia to get the size and received 1.5Mb and 465Kb minified, but this numbers are received by manipulation over package source. Not sure how do they get them.

I'm using IPFS as dependency to my electron app. It would be nice you provide instruction how to minify the size or (what's preferrable to me), provide a compiled bundle for node.js.

@daviddias
Copy link
Member

Some notes from the twitter thread: https://twitter.com/rumkin/status/1185009479101964288

Suggestion from @hugomrdias https://twitter.com/hugomrdias/status/1185825396551499776

@alanshaw alanshaw added exp/wizard Extensive knowledge (implications, ramifications) required P2 Medium: Good to have, but can wait until someone steps up topic/perf Performance status/ready Ready to be worked labels Oct 21, 2019
@mikeal
Copy link
Contributor

mikeal commented Oct 21, 2019

npm install —production should skip the devDependencies, which you don’t need if you’re just using it as a depedency. That’ll drop the size significantly since aegir accounts for most of the size since it has to pull down a build of headless Chrome.

To test this, go into your project and

rm -rf node_modules
npm install —production

@achingbrain
Copy link
Member

npm i in the ipfs/js-ipfs repo gives me a node_modules folder that's over 950MB(!).

npm i --production (or npm install ipfs which is what @rumkin is doing) gives me a node_modules that's 326M.

Of this 326MB, roughly:

  • 197MB is folders named dist - e.g browser bundles, minified js & source maps
  • 20MB is folders named docs or doc
  • 12MB is in folders named test or tests
  • 8MB is json - e.g. package.json, etc
  • 3.5MB is compiled native code
  • 3.5MB is typescript source

@mikeal
Copy link
Contributor

mikeal commented Oct 21, 2019

electron app

If you’re concerned about the size of the app you may want to use https://github.com/electron-userland/electron-webpack

If you compile the entire app down with webpack only the code you rely will be there and you can do an electron build without node_modules.

@rumkin
Copy link
Author

rumkin commented Oct 31, 2019

As the first step I'd suggest to split the package into several:

  • IPFS core: for build IPFS into another products.
  • IPFS node: for running standalone node as a micro/web-service as a daemon. This is what ipfs package should be, as I think.
  • IPFS UI: for node management and inspection. This package could consist of two others: manager and viewer.

In my project I don't need web interface or CLI. The only thing I need is IPFS networking and storage capabilities provided by the core package.

@achingbrain
Copy link
Member

achingbrain commented Oct 31, 2019

I'm not 100% convinced this would solve the size problem.

Just under half of the LOC in /src is CLI/web related and removing them would save about 200KB (unminified & uncompressed) and a few megs of supporting dependencies which granted is not insignificant.

The problem is there's almost 200MB of browser bundles in node_modules which is what the original objection was about and almost all of those are in libp2p-*/ipld-*/ipfs-* modules that IPFS requires to function.

@rumkin
Copy link
Author

rumkin commented Oct 31, 2019

I don't see how it conflicts with what I propose. CLI and Web pull a bunch of own dependencies like @hapi/hapi, yargs which has their own dependencies and so. Separation would make ipfs package less heavy and faster to install.

This should be moved into separate issue. But I think it should be done anyway. Currently the package is overloaded with responsibilities and tightly coupled.

@alanhoff
Copy link

on osx npm i ipfs uses 969MB of storage...

@ottodevs
Copy link

ottodevs commented Mar 5, 2020

npm install —production should skip the devDependencies, which you don’t need if you’re just using it as a depedency. That’ll drop the size significantly since aegir accounts for most of the size since it has to pull down a build of headless Chrome.

To test this, go into your project and

rm -rf node_modules
npm install —production

This seems not valid since it will also skip the devDependencies of my own project...

The point is, I want to install the devDependencies declared in my package because I use them to develop my project, but I don't want to install 1gb of garbage dependencies that I will not use and that ipfs does not depend on to work, so why are we enforced to install electron?

From a technical perspective this does not seem a great design, a "consumer" of the ipfs library wouldn't need to be enforced to install 1Gb of unrelated dependencies, while the library just weights a couple hundred Kilobytes

Is there any alternative to avoid this? I suggest moving the tests to a separate package and make this package depend on aegir and ipfs so ipfs stays "dependency clean"

Another alternative seems to only publish a bundle so users can consume the pre-built ipfs dependency without needing all the extra stuff

Maybe I am missing something, if anyone knows a way to avoid installing devDependencies from ipfs, while keeping my own direct devDependencies please tell me, it makes no sense for me, I am not even depending on ipfs directly, but on 3box that uses it!!!

@hugomrdias
Copy link
Member

@ottodevs npm does not install dev deps from your deps

what we can do is reduce the size of the dist folders.

@ottodevs
Copy link

ottodevs commented Mar 5, 2020

@ottodevs npm does not install dev deps from your deps

what we can do is reduce the size of the dist folders.

Thanks for the quick answer and clarification @hugomrdias

I might be doing something wrong but when a git clone myproject then cd myproject && npm i I get this folder:
ls node_modules/ipfs/node_modules/electron along with other dependencies that are supposed to be devDependencies of my dependencies, and not declared anywhere into my package.json

I am wondering if that's a npm problem and yarn does it right, in any case, if you think reducing dist folders size will help, that would be also nice, but that electron folder shouldn't be there in my opinion when a user just adds ipfs package, as electron is not a direct dependency and is unwanted

@hugomrdias
Copy link
Member

can you run yarn why electron and post here the output ?

@ottodevs
Copy link

ottodevs commented Mar 5, 2020

Weird output...

yarn why electron
yarn why v1.22.0

[1/4] 🤔  Why do we have the module "electron"...?
[2/4] 🚚  Initialising dependency graph...
(...lots of: [warning  > 3box > ipfs > multicodec@0.5.7: stable api reached] omitted)
[3/4] 🔍  Finding dependency...
error We couldn't find a match!
✨  Done in 116.89s.

... but the folder definitely exists...

ls node_modules/ipfs/node_modules/electron
LICENSE		cli.js		electron.d.ts	install.js	package.json
README.md	dist		index.js	node_modules	path.txt

Maybe is because my project does normally use npm so there is not a yarn.lock file

@hugomrdias
Copy link
Member

rm -rf node_modules && yarn && yarn why electron

@achingbrain
Copy link
Member

What version of ipfs do you end up with? There was a problem with the shrink wrap file for a while that mean the dev deps could sometimes get installed.

@ottodevs
Copy link

ottodevs commented Mar 7, 2020

rm -rf node_modules && rm package-lock.json && yarn && yarn why electron

Still does not finds a match; after removing the node_modules and do a clean install with yarn, the electron folder is gone.

then I try again:
rm -rf node_modules && rm yarn.lock && npm i && npx npm-why electron
This shows: No one requires electron.

But in this second case back to npm, the output of ls node_modules/ipfs/node_modules/electron shows that electron is present again, so the issue seems to be with npm installing devDepencencies somehow... in this case the node_modules/ipfs folder size grows over 1GB

Maybe is time to encourage my team to shift to yarn, pnpm or similar...

@ottodevs
Copy link

ottodevs commented Mar 7, 2020

What version of ipfs do you end up with? There was a problem with the shrink wrap file for a while that mean the dev deps could sometimes get installed.

npm ls ipfs
my-project@0.0.1
  └─┬ 3box@1.17.1
    └─┬ ipfs@0.40.0 
      └── ipfs@0.38.1  extraneous

npm ERR! extraneous: ipfs@0.38.1 /Users/otto/box/git/my-project/node_modules/ipfs/node_modules/ipfs

@achingbrain
Copy link
Member

0.40.0 had the npm-shrinkwrap.json file with the extra dependencies in it - this will be fixed when 3box upgrade to 0.41.1 or later.

@achingbrain
Copy link
Member

The situation here has changed quite a lot in the intervening months. We've pulled a bunch of uncommonly used deps out of ipfs, so right now doing npm i ipfs in an empty directory results in a node_modules folder that's 167MB in size, a significant reduction from the 336MB originally reported.

If you are only running an in-process node as part of an application and don't need the CLI or HTTP API server components you can reduce the install size by a further 30MB by using the ipfs-core module which is a drop-in replacement for ipfs in your application code.

We'll keep trying to slim the install down but as it stands pressure on CI and other space/bandwidth constrained environments should now be significantly lightened.

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
exp/wizard Extensive knowledge (implications, ramifications) required P2 Medium: Good to have, but can wait until someone steps up status/ready Ready to be worked topic/perf Performance
Projects
None yet
Development

No branches or pull requests

8 participants