chore: Dependency updates

[SgtPooki]

This issue is intended to track & discuss any and all issues related to old dependencies:

Issues tracking dependency work

Core dependency updates needed (priority)

• feat: update ipfs-http-client #1840
• Replace ipld-dag-pb with @ipld/dag-pb ipld-explorer-components#329
  • PR is out for this: feat!: replace ipld-dag-pb with @ipld/dag-pb ipld-explorer-components#330
• Remove dependency on ipld-git #2045
• Remove dependency on ipld-ethereum #2046
• Remove dependency on ipfs-block-service #2047
• Update to the latest multiformats package #2048

Other work

• chore: update ipld-explorer/i18n deps #1924
• chore(deps): update react-scripts #1969
• chore: install missing dependencies #1981
• chore: Update storybook #1982
• chore: Remove engines-strict and update package.json->engines #1987
• update ipfs and ipfsd-ctl #1988
• chore: remove unused dependencies #2015
• remove unused exports to clean up code #2036

Nice to have but not required

• Add check for missing dependencies to CI #2020

Items blocked by outdated deps in one way or another

• feat: import DAG CAR #1838
• Can't build with nodejs v17 and npm v8 #1940
• npm install error #1975
• Error: No resolver found for codec "dag-json" (and dag-cbor) #1923
• Enabling WebTransport in Kubo v0.16.0-rc1 breaks Status page #2033
• Failing Master Branch (error:0308010C:digital envelope routines::unsupported) ipld-explorer-components#342

Issues blocking dependency updates

• peerDependencies prevents react-upgrade  tableflip/react-dropdown#8

---

Investigating dependencies

npm outdated --all > npm-outdated.raw.log gives us this: npm-outdated.raw.log

Then we want to remove duplicates, because npm outdated doesn't do that for us for some reason (echo "$(head -n 1 npm-outdated.raw.log && tail -n +2 npm-outdated.raw.log | sort | uniq -d)" > npm-outdated.uniq.log): npm-outdated.uniq.log
cat npm-outdated.raw.log | awk '{print $6}' | sort | uniq -c | sort > most-impactful-dependencies.log

To see which dependencies are mentioned the most in the depended by column, we can run cat npm-outdated.uniq.log | awk '{print $6}' | sort | uniq -c | sort > most-impactful-dependencies.log to get: most-impactful-dependencies.log

and then to get the top 10 most impactful dependency updates, tail -n 10 most-impactful-dependencies.log:

  8 ipfs-http-client
  9 @babel/core
  9 @storybook/theming
  9 css-loader
 10 jest-cli
 11 ipfs-core-utils
 11 ipfs-utils
 11 react-dev-utils
 15 @storybook/components
 51 @babel/preset-env

So which of those versions do we 'want'?

> tail -n 10 most-impactful-dependencies.log | awk '{print $2}' | xargs -n1 -I% sh -c 'cat npm-outdated.uniq.log | grep "^%"'
@babel/core                                                            7.15.5               7.18.6               7.18.6  node_modules/@babel/core                                                                                                              @babel/helper-compilation-targets
@babel/core                                                            7.15.5               7.18.6               7.18.6  node_modules/@babel/core                                                                                                              @babel/helper-create-class-features-plugin
@babel/core                                                            7.15.5               7.18.6               7.18.6  node_modules/@babel/core                                                                                                              @babel/plugin-proposal-class-properties
@babel/core                                                            7.15.5               7.18.6               7.18.6  node_modules/@babel/core                                                                                                              @babel/plugin-proposal-dynamic-import
@babel/core                                                            7.15.5               7.18.6               7.18.6  node_modules/@babel/core                                                                                                              @babel/plugin-proposal-export-namespace-from
@babel/core                                                            7.15.5               7.18.6               7.18.6  node_modules/@babel/core                                                                                                              @babel/plugin-proposal-logical-assignment-operators
@babel/core                                                            7.15.5               7.18.6               7.18.6  node_modules/@babel/core                                                                                                              @babel/plugin-proposal-nullish-coalescing-operator
@babel/core                                                            7.15.5               7.18.6               7.18.6  node_modules/@babel/core                                                                                                              @babel/plugin-proposal-numeric-separator
@babel/core                                                            7.15.5               7.18.6               7.18.6  node_modules/@babel/core                                                                                                              @babel/plugin-proposal-optional-chaining
@babel/core                                                            7.15.5               7.18.6               7.18.6  node_modules/@babel/core                                                                                                              @babel/plugin-proposal-private-methods
@babel/core                                                            7.15.5               7.18.6               7.18.6  node_modules/@babel/core                                                                                                              @babel/plugin-syntax-flow
@babel/core                                                            7.15.5               7.18.6               7.18.6  node_modules/@babel/core                                                                                                              @babel/plugin-syntax-typescript
@babel/core                                                            7.15.5               7.18.6               7.18.6  node_modules/@babel/core                                                                                                              @babel/plugin-transform-flow-strip-types
@babel/core                                                            7.15.5               7.18.6               7.18.6  node_modules/@babel/core                                                                                                              @babel/plugin-transform-modules-commonjs
@babel/core                                                            7.15.5               7.18.6               7.18.6  node_modules/@babel/core                                                                                                              @babel/plugin-transform-typescript
@babel/core                                                            7.15.5               7.18.6               7.18.6  node_modules/@babel/core                                                                                                              @babel/preset-typescript
@babel/core                                                            7.15.5               7.18.6               7.18.6  node_modules/@babel/core                                                                                                              @jest/transform
@babel/core                                                            7.15.5               7.18.6               7.18.6  node_modules/@babel/core                                                                                                              @svgr/plugin-jsx
@babel/core                                                            7.15.5               7.18.6               7.18.6  node_modules/@babel/core                                                                                                              istanbul-lib-instrument
@babel/core                                                            7.15.5               7.18.6               7.18.6  node_modules/@babel/core                                                                                                              jest-snapshot
@storybook/theming                                                     5.3.21               5.3.21                6.5.9  node_modules/@storybook/theming                                                                                                       @storybook/components
ipfs-core-utils                                                        0.10.5               0.10.5               0.15.1  node_modules/ipfs-core-utils                                                                                                          ipfs-http-client
@babel/preset-env                                                     7.12.11               7.18.6               7.18.6  node_modules/@babel/preset-env                                                                                                        @svgr/webpack

And why are they there? tail -n 10 most-impactful-dependencies.log | awk '{print $2}' | xargs -n1 -I% sh -c 'npm explain %' > explain.txt: explain.txt

That's super hard to understand, so let's aggregate again... cat explain.txt | grep 'from the root project' | awk '{$1=$1};1' | sort | uniq -c | sort:

   1 dev @babel/core@"^7.15.5" from the root project
   1 dev @storybook/addon-links@"^5.3.19" from the root project
   1 dev @storybook/addons@"^5.3.19" from the root project
   1 ipld-explorer-components@"^2.4.1" from the root project
   2 dev @svgr/cli@"^5.4.0" from the root project
   4 ipfs-http-client@"49.0.2" from the root project
   5 dev @storybook/addon-a11y@"^5.3.19" from the root project
   6 dev @storybook/addon-actions@"^5.3.19" from the root project
   6 dev @storybook/addon-knobs@"^5.3.19" from the root project
  29 dev @playwright/test@"^1.12.1" from the root project
 106 dev ipfs@"0.58.3" from the root project
 117 dev ipfsd-ctl@"^7.2.0" from the root project
 190 dev jest@"^26.6.3" from the root project
 213 dev @storybook/react@"^5.3.19" from the root project
 301 dev jest-playwright-preset@"^1.7.0" from the root project
 754 react-scripts@"^4.0.3" from the root project

Breakdown of the above:

1. List all outdated npm modules
2. find the outdated modules that are mentioned the most
3. find and count the npm modules that depend on the most-mentioned out-of-date modules.
   • These are the "most impactful dependencies"
4. get npm to tell us why these most impactful dependencies are in our repo.
5. filter through step4 above to tell us which module we actually depend on is at fault for all the above
6. find the most-mentioned module from step5 that we depend on
   • The module with the most entries should be the first target for our upgrade plan.

So what does this mean?

Using react-scripts as an example: there are 754 listings of react-scripts being the ROOT CAUSE. react-scripts is the package we depend on, that is listed in the npm-explain output, when trying to determine why out-of-date modules are listed in our dependency graph.

Why did you do all this?

I attempted to upgrade some (see main...SgtPooki:ipfs-webui:fix/node-14-windows) to resolve the build errors showing at https://github.com/SgtPooki/ipfs-webui/actions/runs/2605014389, (see #1961) and was consistently blocked by npm because of semver conflicts of wanted packages. In order to move forward.. since there is no npm-dependency-graph-resolver upgrade tool, I figured I would need all of this data.

[SgtPooki]

@lidel Can we chat about what this story looks like by tying ourselves specifically to Kubo? e.g. Pulling out ipfs-http-client entirely from webui, and replacing with https://github.com/ipfs/js-kubo-rpc-client.

[SgtPooki]

on the latest main:

> cat npm-install.log | sort | uniq | grep 'npm WARN deprecated' | wc -l
      52

> npm i &> npm-install.log

> cat npm-install.log | perl -pe 's/npm WARN deprecated //g' | perl -pe 's/(^[^:]+)@[^:]+/$1/g' | sort | uniq | perl -ne 'print if /ipfs|cid|multi|ipld|libp2p/'
cids: This module has been superseded by the multiformats module
interface-ipld-format: This module has been superseded by the multiformats module
ipfs-block-service: This module has been merged into ipfs
ipld-dag-cbor: This module has been superseded by @ipld/dag-cbor and multiformats
ipld-dag-pb: This module has been superseded by @ipld/dag-pb and multiformats
ipld-ethereum: This module is no longer maintained
ipld-git: This module is no longer maintained
ipld-raw: This module has been superseded by the multiformats module
ipld: Please use the multiformats module instead
multibase: This module has been superseded by the multiformats module
multicodec: This module has been superseded by the multiformats module
multihashing-async: This module has been superseded by the multiformats module

All of these packages listed need to be removed/updated prior to us being able to safely&sanely keep up with new kubo feature development.

cat npm-install.log | perl -pe 's/npm WARN deprecated //g' | perl -pe 's/(^[^:]+)@[^:]+/$1/g' | sort | uniq | perl -ne 'print if /ipfs|cid|multi|ipld|libp2p/' | awk -F: '{ print $1 }' | xargs -tn1 npm explain &> npm-explain.log

npm-explain.log

Callouts

Of those 52 deprecated packages, 12 are core packages.
Of those 12 core packages, 7 require replacing fully with the multiformats package.
Of those 12 core packages, 1 was merged into ipfs.
Of those 12 core packages, 2 are no longer maintained
Of those 12 core packages, 2 require replacing with a mixture of @ipld/pkg + multiformats.