Start moving third party libraries to use npm #9706

ebogo1 · 2021-08-01T17:09:28Z

This is a chunk of progress towards #9473.

Comparing the generated .zip files with the one in main after #9705 gets merged:

64.1 MB in this branch vs 63.6 MB in main
In Build/Cesium/:
- Cesium.js is 3.9 MB in this branch vs 3.7 MB in main (uncompressed)
- ThirdParty/Workers/ now only has basis and draco files (we removed deflate/inflate.js and ktx-parse-modern.js)
In Source/ThirdParty/:
- This folder is 3.6 MB up from 3.1 MB in main (uncompressed)
- The new files generated from npm modules seem correct; they share the same folder as the third party files currently in main

Comparing the .tgz with main:

22.0 MB here vs 21.4 MB in main
decodeGoogleEarthEnterprisePacket is about 1.5x as big in this branch; probably because we're importing more than we need from pako..

Remove `sprintf` third party library

Rather than submit libraries to Source/ThirdParty, which always end up being modified for our build system and quickly go out of date, this change starts to use libraries via npm instead. Currently Autolinker, earcut, when, tween.js, rbush, kdbush, quickselect, and topojson are ported. The main hurdle that prevented us from doing this sooner was the fact that Cesium has a long history of not requiring a build step after every code change and has a goal of writing valid JS code, unlike many libraries today that mandate a bundler to turn invalid JS code into valid code. Rather than mandate a bundler during development, this initial step adds a "buildThirdyParty" function to the current "build" step. This function runs third party libraries (defined in the ThirdParty/npm/ folder) through RollUp and creates an equivalent file in `Source/ThirdPartyNpm`. The change to end users will be non-existent, especially since the combined Cesium.js will still re-export any third party modules as part of the private API just like it used to. This doesn't prevent code duplication for users using some of the same third party libraries as Cesium. This is just an improvement as to how depend on third party libraries internally. I think Cesium's days of being "bundler free" are probably limited long term, performance is the main hurdle and newer tools like esbuild may make that no longer a problem. But that's outside the scope of this initial goal.

Start of replacing submitted third party libraries with npm modules

…ging

Use npm for third party libraries

zip.js with npm

Move Gltf-Pipeline to Source/Scene

…into npm-third-party-staging

Move FXAA3_11 shader into Source/Shaders

…ve-in-ThirdParty

cesium-concierge · 2021-08-01T17:09:32Z

Thanks for the pull request @ebogo1!

✔️ Signed CLA found.
❔ Unit tests were not updated.
- Make sure you've updated tests to reflect your changes, added tests for any new code, and ran the code coverage tool.

Reviewers, don't forget to make sure that:

Cesium Viewer works.
Works in 2D/CV.
Works (or fails gracefully) in IE11.

ebogo1 · 2021-08-01T17:10:41Z

CHANGES.md

@@ -6,6 +6,7 @@

 - Added `ImageryLayerCollection.pickImageryLayers` which determines the imagery layers that are intersected by a pick ray. [#9651](https://github.com/CesiumGS/cesium/pull/9651)
 - Added a `polylinePositions` getter to `Cesium3DTileFeature` that gets the decoded positions of a polyline vector feature. [#9684](https://github.com/CesiumGS/cesium/pull/9684)
+- Started moving over third party libraries to use npm modules.



Is this the right place to mention the change? I think there should be a note of this somewhere in CHANGES.md.

This has zero user facing implications at this time, while we are moving to npm, there are still build time dependencies, not runtime dependencies (i.e. users who use Cesium don't actually have a direct dependencies on these libraries via npm)

Move ThirdPartyNpm files into ThirdParty

mramato · 2021-08-02T14:09:30Z

ThirdParty/npm/pako.js

@@ -0,0 +1,2 @@
+import pako from 'pako';
+export { pako as default };


If all we use is pako-inflate, would it be better to only include inflate here? Then make sure roll-up treeshakes and it should create a smaller file.

treeshaking may not work, but pako also includes an inflate only dist.. looks like we can save over a hundred KB by using that instead.

This should be as simple as going back to inflate only like:

import pako_inflate from 'pako/lib/inflate.js'; export { pako_inflate as default };

and then update the calling code to just call pako_inflate instead of pako.inflate.

Actually, I'm wrong.. you literally don't need to change the calling code.. just do

import pako from 'pako/lib/inflate.js'; export { pako as default };

and you end up with a pako object that only has inflate related stuff.

mramato · 2021-08-02T14:18:35Z

@ebogo1 it would be better to look at the size of the actual combined Cesium.js files and the files in workers.. the size of the zips, while indicative of size differences, don't really tell us what users are pulling down to use Cesium over the network.

mramato · 2021-08-02T14:57:38Z

Looks like we added ~512KB to the size to the combined/unminified Cesium.js and then a similar amount to the Workers, this is most likely caused by including way more of some third party libraries than we used to (like pako). It's probably easy to fix,but do we have any idea as to which libraries are causing the issue?

lilleyse · 2021-08-03T01:12:17Z

I updated gltf-pipeline in dc2e925. gltf-pipeline now uses prettier, though it uses a different version of prettier than CesiumJS so I couldn't remove it from .prettierignore.

@ebogo1 I tagged you to review CesiumGS/gltf-pipeline#600 which has some fixes for generating these files for CesiumJS.

ebogo1 · 2021-08-04T17:37:50Z

After importing less from pako and zip the unminified Cesium.js size went down to by about 300 kB to 11.1 MB and the Workers folder is now the same size as the one in the 1.84 release. Cesium.js.map is still about 500 kB bigger than in the release but is down 600 kB after b962003.

ebogo1 · 2021-08-16T17:41:53Z

I looked into the file size discrepancy more and don't think there's an easy solution:

Almost all of the bloat is coming from @zip.js/zip.js. Even though version 2.0.0 on npm is significantly smaller (unpacked size 744 kB vs 2.4 MB in the version here), downgrading the version had no effect on the unminified Cesium.js or Cesium.js.map sizes..
Per the README in the package's dist/ folder, I don't think there's a smaller import we can use. The npm package's lib/core/ folder is ~300 kB, so I'd guess it's responsible for most of the file size.

@mramato I could be wrong but is the only alternative here to try a different zip npm library?

As of the last commit:

Unminified Cesium.js is up to 11.1 MB from 10.9 MB in the 1.84 release (or 11.0 MB without npm zip.js)
Unminified Cesium.js.map is up to 19.8 MB from 19.3 MB in the 1.84 release (or 19.4 MB without npm zip.js)

mramato · 2021-08-22T18:05:44Z

All in all, this is okay with me as an incremental step. But it's made it more clear than ever that we need to bite the bullet and migrate to "live" compilation where Cesium dev doesn't function without tooling building valid JS in the background. The industry and moved on and we aren't doing our service by trying to hold on to the old ways. In particular esbuild would probably be a great fit for our needs and simplify this entire process.

I diffed main and this branch zip output and went through the combined Cesium.js diff by diff and nothing major jumped out. Everything appears to work locally as well.

I do have two questions:

Did we revisit LICENSE.md to make sure it is up to date with any changed/updated libraries?
Is there a plan to update the dependencies? A few of them are quite out of date.

Both can be handled in follow-up PRs. @lilleyse if you are okay with this feel free to merge.

lilleyse · 2021-08-23T18:00:37Z

Thanks @ebogo1 and @mramato. The changes look good to me as well. I ran unit tests locally and spot checked some sandcastle examples.

ebogo1 and others added 30 commits April 8, 2021 16:41

remove sprintf and use padStart

4a4c564

Merge pull request #9476 from CesiumGS/remove-sprintf

267b98e

Remove `sprintf` third party library

Merge pull request #9466 from CesiumGS/npm-third-party

52f690c

Start of replacing submitted third party libraries with npm modules

Merge remote-tracking branch 'origin/master' into npm-third-party-sta…

73e00e1

…ging

fix when imports

8391bfe

fix multiple holes test in PolygonPipelineSpec

a0d67f8

add npm packages pt 1

e122859

add back zip.js

709f206

ktx-parse to npm

4012e2e

add ThirdParty/npm/ktx-parse.js

8165fd9

Merge pull request #9688 from CesiumGS/npm-second-pass

f8c9a23

Use npm for third party libraries

zip.js in exportKml

65f5924

Merge branch 'npm-second-pass' into npm-zip-js

2dd0ca7

fix exportKml and update KmlDataSource

aa276f3

remove inflate and deflate workers

7b40a60

Merge pull request #9696 from CesiumGS/npm-zip-js

038f732

zip.js with npm

move GltfPipeline to Source/Scene and add it to .eslintignore

86de8af

Mark functions as @Private

c0be363

mark GltfPipeline in .prettierignore

661f9ba

first pass

2f8be0c

revert prettier changes

3764981

Merge pull request #9699 from CesiumGS/move-gltf-pipeline

c8f576b

Move Gltf-Pipeline to Source/Scene

move google-earth-dbroot-parser back to ThirdParty

be34f56

Fix gulpfile

63d5860

update prettier and eslint ignore files

824b774

update CHANGES.md

4ec47be

Merge branch 'npm-third-party-staging' of github.com:CesiumGS/cesium …

d382be0

…into npm-third-party-staging

Merge pull request #9704 from CesiumGS/move-into-source

110af2a

Move FXAA3_11 shader into Source/Shaders

first pass

c8592cb

ebogo1 added 4 commits July 31, 2021 17:47

ignore new ThirdParty files from source

f73cbd4

fix gulpfile again

690edff

Merge remote-tracking branch 'origin/main' into npm-third-party-staging

74b874a

Merge remote-tracking branch 'origin/npm-third-party-staging' into sa…

138566d

…ve-in-ThirdParty

ebogo1 requested a review from mramato August 1, 2021 17:09

ebogo1 commented Aug 1, 2021

View reviewed changes

Merge pull request #9705 from CesiumGS/save-in-ThirdParty

f1da729

Move ThirdPartyNpm files into ThirdParty

mramato reviewed Aug 2, 2021

View reviewed changes

Update gltf-pipeline files, now using prettier

dc2e925

lilleyse added 2 commits August 2, 2021 21:13

Merge branch 'main' into npm-third-party-staging

2ccf0fd

Update CHANGES.md

c17e60b

lilleyse mentioned this pull request Aug 3, 2021

Update build-cesium task to account for prettier formatting CesiumGS/gltf-pipeline#600

Merged

reduce import sizes for pako and zip

b962003

Merge branch 'main' into npm-third-party-staging

fa50855

ebogo1 mentioned this pull request Aug 20, 2021

Replace Uri.js with urijs npm module #9740

Merged

Merge branch 'main' into npm-third-party-staging

0346a5d

lilleyse merged commit 94e5646 into main Aug 23, 2021

lilleyse deleted the npm-third-party-staging branch August 23, 2021 18:00

ebogo1 mentioned this pull request Aug 23, 2021

Update LICENSE.md #9752

Merged

This was referenced Sep 7, 2021

"/node_modules/cesium/Source/ThirdParty/zip.js" this file can't not be packed #9790

Closed

1.85.0 experimental syntax error #9801

Closed

lilleyse mentioned this pull request Sep 15, 2021

Blob is not defined at configureWebWorker CesiumGS/obj2gltf#266

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Start moving third party libraries to use npm #9706

Start moving third party libraries to use npm #9706

ebogo1 commented Aug 1, 2021

cesium-concierge commented Aug 1, 2021

ebogo1 Aug 1, 2021

mramato Aug 1, 2021

mramato Aug 2, 2021

mramato Aug 2, 2021

mramato Aug 2, 2021

mramato Aug 2, 2021

mramato commented Aug 2, 2021

mramato commented Aug 2, 2021

lilleyse commented Aug 3, 2021 •

edited

Loading

ebogo1 commented Aug 4, 2021 •

edited

Loading

ebogo1 commented Aug 16, 2021 •

edited

Loading

mramato commented Aug 22, 2021

lilleyse commented Aug 23, 2021

		@@ -0,0 +1,2 @@
		import pako from 'pako';
		export { pako as default };

Start moving third party libraries to use npm #9706

Start moving third party libraries to use npm #9706

Conversation

ebogo1 commented Aug 1, 2021

cesium-concierge commented Aug 1, 2021

ebogo1 Aug 1, 2021

Choose a reason for hiding this comment

mramato Aug 1, 2021

Choose a reason for hiding this comment

mramato Aug 2, 2021

Choose a reason for hiding this comment

mramato Aug 2, 2021

Choose a reason for hiding this comment

mramato Aug 2, 2021

Choose a reason for hiding this comment

mramato Aug 2, 2021

Choose a reason for hiding this comment

mramato commented Aug 2, 2021

mramato commented Aug 2, 2021

lilleyse commented Aug 3, 2021 • edited Loading

ebogo1 commented Aug 4, 2021 • edited Loading

ebogo1 commented Aug 16, 2021 • edited Loading

mramato commented Aug 22, 2021

lilleyse commented Aug 23, 2021

lilleyse commented Aug 3, 2021 •

edited

Loading

ebogo1 commented Aug 4, 2021 •

edited

Loading

ebogo1 commented Aug 16, 2021 •

edited

Loading