Migrate Cesium to ES6 Modules

See #8224 for details.

eslint
There are a handful of new .eslintrc.json files, mostly to identify the files that are still AMD modules (Sandcastle/Workers). These are needed because you can't change the parser type with a comment directive (since the parser is the thing reading the file). We can finally detect unusued modules! So those have all been cleaned up as well.

requirejs -> rollup & clean-css
requirejs, almond, and karma-requirejs have all been removed. We now use rollup for building and minifying (via uglify) JS code and clean-css for css. These changes are fairly straight-forward and just involve calling rollup instead of requirejs in the build process.

Overall build time is significantly faster. CI is ~11 minutes compared to ~17 in master. Running makeZipFile on my machine takes 69 seconds compared to 112 seconds in master. There's probably plenty of room for additional optimization here too.

We wrote an published a small npm module, rollup-plugin-strip-pragma, for stripping the requirejs pragmas we use out of the release builds. This is maintained in the Tools/rollup-plugin-strip-pragma directory.

As for what we produce. The built version of Cesium is now a UMD module. So it should work anywhere that hasn't made the jump to ES6 yet. For users that were already using the "legacy" combined/minified approach, nothing changes.

One awesome thing about roll-up is that it compiles all of the workers at once and automatically detects shared codes and generates separate bundles under the hood. This means the size of our worker modules shrink dramatically and Cesium itself will load them much faster. The total minified/gzipped size of all workers in master is 2.6 MB compared to 225 KB in this branch! This should be most noticeable on demos like Geometry & Appearances which load lots of workers for the various geometry typs.

roll-up is also used to build Cesium Viewer, which is now an ES6 app.

We use clean-css via gulp and it is also a straightforward change from requirejs that requires no special mention.

Workers
While the spec allows for ES6 Web Workers, no browser actually supports them yet. That means we needed a way to get our workers into non-ES6 form. Thankfully, roll-up can generate AMD modules, which means we now have a build step to compile our Worker source code back into AMD and use the existing TaskProcessor to load and execute them. This build step is part of the standard build task and is called createWorkers. During development, these "built" workers are un-optimized so you can still debug them and read the code.

Since there is a build step, that means if you are changing code that affects a worker, you need to re-run build, or you can use the build-watch task to do it automatically.

The ES6 versions of Worker code has moved into Source/WorkersES6 and we build the workers into their "old home" of Source/Workers. cesiumWorkerBootstrapper and transferTypedArrayTest which were already non-AMD ES5 scripts remain living in the Workers directory.

Surprisingly little was changed about TaskProcessor or the worker system in general, especially considering that I thought this would be one of the major hurdles.

ThirdParty
A lot of our ThirdParty either already had a hand-written wrapper for AMD (which I updated to ES6) or had UMD which created problems when importing the same code in both Node and the browser. I basically had to update the wrapper of every third-party library to fix these problems. In some cases I updated the library version itself (Autolinker, topojson). Nothing to be too concerned about, but future clean-up would be using npm versions of these libraries and auto-generating the wrappers as needed so we don't hand-edit things.

Sandcastle
Sandcastle is eternal and manages to live another day in it's ancient requirejs/dojo 1.x form. Sandcastle now automatically uses the ES6 version of Cesium if it is available and fallsback to the ES5 unminified version if it is now. The built version of Sandcastle always uses CesiumUnminified, just like master. This means Sandcastle still works in IE11 if you run the combine step first (or use the relase zip)

Removed Cesium usage from Sandcastle proper, since it wasn't really needed
Generate a VERSION propertyin the gallery index since Cesium is no longer being included.
Remove requirejs from Sandcastle bucket
Update bucket to use the built version of Cesium if it is available by fallbackto the ES6 version during development.
Standalone.html was also updated
There's a bit of room for further clean-up here, but I think this gets us into master. I did not rename bucket-requirejs.html because I'm pretty sure it would break previously shared demos. We can put in some backwards compatible code later on if we want. (But I'd rather just see a full Sandcastle rewrite).

Specs
Specs are now all ES6, except for TestWorkers, which remain standard JS worker modules. This means you can no longer run the unbuilt unit tests in IE11. No changes for Chrome and Firefox.

Since the specs use ES6 modules and built Cesium is an ES5 UMD, I added a build-specs build step which generates a combined ES5 version of the specs which rely on Cesium as a global variable. We then inject these files into jasmine instead of the standard specs and everything works exactly as it did before. SpecRunner.html has been updated to inject the correct version of the script depending on the build/release query parameters.

The Specs must always use Cesium by importing Source/Cesium.js, this is so we can replace it with the built Cesium as describe above.

There's a bunch of room for clean-up here, such as unifying our two copies of jasmine into a single helper file, but I didn't want to start doing that clean-up as part of this already overly big PR. The important thing is that we can still test the built version and still test on IE/Edge as needed.

I also found and fixed two bugs that were causing failing unit tests, one in BingMapsImageryProviderSpec.js (which was overwriting createImage andnot setting it back) and ShadowVolumeAppearance.js (which had a module level caching bug). I think these may have been the cause of random CI failures in master as well, but only time will tell.

For coverage, we had to switch to karma-coverage-istanbul-instrumenter for native ES6 support, but that's it.

Finally, I updated appveryor to build Cesium and run the built tests under IE. We still don't fail the build for IE, but we should probably fix that if we want to keep it going.

NodeJS
When NODE_ENV is production, we now require in the minified CesiumJS directly, which works great because it's now a UMD module. Otherwise, we use the excellant esmpackage to load individual modules, it was a fairly straightforward swap from our old requirejs usage. We could probably drop esm too if we don't care about debugging or if we provie source maps at some point.

Author: mramato

.appveyor.yml

@@ -13,8 +13,9 @@ install:
   - npm install
 
 test_script:
-  - npm --silent run build
-  - npm --silent run test -- --browsers IE --webgl-stub --suppressPassed
+  - npm --silent run minifyRelease
+  - npm --silent run build-specs
+  - npm --silent run test -- --browsers IE --webgl-stub --release --suppressPassed
 
 # Don't actually build.
 build: off

.eslintignore

@@ -4,8 +4,11 @@ Build/**
 Documentation/**
 Source/Shaders/**
 Source/ThirdParty/**
-Source/Workers/cesiumWorkerBootstrapper.js
+Source/Workers/**
+!Source/Workers/transferTypedArrayTest.js
 ThirdParty/**
 Tools/**
 Apps/Sandcastle/jsHintOptions.js
 Apps/Sandcastle/gallery/gallery-index.js
+Source/Core/buildModuleUrl.js
+Specs/spec-main.js

.eslintrc.json

@@ -5,5 +5,18 @@
     ],
     "rules": {
         "no-unused-vars": ["error", {"vars": "all", "args": "none"}]
-    }
+    },
+    "overrides": [
+        {
+            "files": [
+                "index.js",
+                "server.js",
+                "gulpfile.js",
+                "Source/Workers/transferTypedArrayTest.js"
+            ],
+            "parserOptions": {
+                "sourceType": "script"
+            }
+        }
+    ]
 }

.gitignore

@@ -14,12 +14,12 @@ Thumbs.db
 
 /Source/Cesium.js
 
-/Source/Shaders/*.js
-/Source/Shaders/*/*.js
-/Source/Shaders/*/*/*.js
-/Source/ThirdParty/Shaders/*.js
-
 /Specs/SpecList.js
+/Source/Shaders/**/*.js
+/Source/ThirdParty/Shaders/**/*.js
+/Source/Workers/**
+!/Source/Workers/cesiumWorkerBootstrapper.js
+!/Source/Workers/transferTypedArrayTest.js
 
 /node_modules
 npm-debug.log

.npmignore

@@ -11,10 +11,11 @@
 /.travis.yml
 /.vscode
 /Apps
+/Build/Apps
 /Build/Coverage
 /Build/minifyShaders.state
-/Build/Stubs
 /Build/Documentation
+/Build/Specs
 /Cesium-*.zip
 /Documentation
 /favicon.ico
@@ -24,7 +25,6 @@
 /launches
 /server.js
 /Source/copyrightHeader.js
-/Source/main.js
 /Specs
 /ThirdParty
 /Tools

.travis.yml

@@ -26,6 +26,7 @@ script:
   - npm --silent run deploy-s3 -- -b cesium-dev -d cesium/$TRAVIS_BRANCH --confirm -c 'no-cache'
   - npm --silent run deploy-status -- --status success --message Deployed
 
+  - npm --silent run build-specs
   - npm --silent run test -- --browsers ChromeCI --failTaskOnError --webgl-stub --release --suppressPassed
 
 # Various Node.js smoke-screen tests

Apps/CesiumViewer/CesiumViewer.js

@@ -1,36 +1,21 @@
-define([
-        'Cesium/Core/Cartesian3',
-        'Cesium/Core/createWorldTerrain',
-        'Cesium/Core/defined',
-        'Cesium/Core/formatError',
-        'Cesium/Core/Math',
-        'Cesium/Core/objectToQuery',
-        'Cesium/Core/queryToObject',
-        'Cesium/DataSources/CzmlDataSource',
-        'Cesium/DataSources/GeoJsonDataSource',
-        'Cesium/DataSources/KmlDataSource',
-        'Cesium/Scene/TileMapServiceImageryProvider',
-        'Cesium/Widgets/Viewer/Viewer',
-        'Cesium/Widgets/Viewer/viewerCesiumInspectorMixin',
-        'Cesium/Widgets/Viewer/viewerDragDropMixin',
-        'domReady!'
-    ], function(
-        Cartesian3,
-        createWorldTerrain,
-        defined,
-        formatError,
-        CesiumMath,
-        objectToQuery,
-        queryToObject,
-        CzmlDataSource,
-        GeoJsonDataSource,
-        KmlDataSource,
-        TileMapServiceImageryProvider,
-        Viewer,
-        viewerCesiumInspectorMixin,
-        viewerDragDropMixin) {
-    'use strict';
-
+import {
+    Cartesian3,
+    createWorldTerrain,
+    defined,
+    formatError,
+    Math as CesiumMath,
+    objectToQuery,
+    queryToObject,
+    CzmlDataSource,
+    GeoJsonDataSource,
+    KmlDataSource,
+    TileMapServiceImageryProvider,
+    Viewer,
+    viewerCesiumInspectorMixin,
+    viewerDragDropMixin
+} from '../../Source/Cesium.js';
+
+function main() {
     /*
      Options parsed from query string:
        source=url          The URL of a CZML/GeoJSON/KML data source to load at startup.
@@ -218,4 +203,6 @@ define([
     }
 
     loadingIndicator.style.display = 'none';
-});
+}
+
+main();

Apps/CesiumViewer/CesiumViewerStartup.js

@@ -10,10 +10,10 @@
     <title>Cesium Viewer</title>
     <link rel="shortcut icon" href="favicon.ico" type="image/x-icon" />
     <link rel="stylesheet" href="CesiumViewer.css" media="screen">
-    <script data-main="CesiumViewerStartup" src="../../ThirdParty/requirejs-2.1.20/require.js"></script>
 </head>
 <body style="background: #000;">
     <div id="cesiumContainer" class="fullWindow"></div>
     <div id="loadingIndicator" class="loadingIndicator"></div>
+    <script src="CesiumViewer.js" type="module"></script>
 </body>
 </html>

Apps/CesiumViewer/index.html

@@ -1,16 +1,34 @@
 {
     "extends": "../../.eslintrc.json",
+    "env": {
+        "amd": true
+    },
     "globals": {
         "JSON": true,
         "require": true,
         "console": true,
         "Sandcastle": true,
         "Cesium": true
     },
+    "parserOptions": {
+        "ecmaVersion": 5,
+        "sourceType": "script"
+    },
     "rules": {
         "no-alert": ["off"],
         "no-implicit-globals": "off",
         "no-unused-vars": ["off"],
         "quotes": "off"
-    }
+    },
+    "overrides": [
+        {
+            "files": [
+                "load-cesium-es6.js"
+            ],
+            "parserOptions": {
+                "ecmaVersion": 2015,
+                "sourceType": "module"
+            }
+        }
+    ]
 }