Use a build tool to improve performance

[robyngit]

MetacatUI currently relies on RequireJS for asset loading. We aim to improve performance by integrating a modern bundler such as Webpack or Parcel. The goal is to improve MetacatUI's performance and load time through minification, uglification, and bundling of JavaScript files and other assets (e.g. CSS files), as well as code-splitting and tree-shaking techniques.

Although we would eventually like to replace RequireJS with ES6 modules, as a first step, we want RequireJS to co-exist with the new bundler. This way, we preserve full backward compatibility, allowing repositories to choose whether or not to adopt the new build system. Likely, the bundled code will be organized in a new /dist/ directory, distinct from the current /src/ directory.

Considerations:

• Config and Themes: The app config should ideally be part of the bundled code to speed up loading. However, this means that any config change would require a new build. Theme files also pose a similar challenge; they should be kept separate for easier edits unless repositories run the build process themselves.
• Script and App Initialization: Our current setup dynamically loads scripts and starts the app. The new bundler needs to support this functionality without conflicts.
• Loading Order: Shifting to a new bundler could change the order in which files and modules are loaded, affecting the app's behavior. Care must be taken to avoid unexpected issues.
• Bundle Size: The new bundler could produce a large initial bundle, offsetting some performance gains. Techniques like code-splitting can mitigate this.
• Testing: Given that we don't have complete test coverage, a meticulous testing phase is essential to ensure that existing features continue to work as expected.
• Polyfills: Our app uses a polyfill file to support older browsers. Investigate if the new bundler can do feature detection to only serve polyfills when needed. Since we've dropped Internet Explorer support, the polyfills necessity should be reconsidered
• HTML Templates: Make sure the bundler can handle HTML templates in a manner compatible with our JavaScript.
• Images and Fonts: Check how the bundler will handle non-code assets like images, SVGs, and fonts. Asset inlining or other optimization techniques may be available.
• URLs and File Paths: The new bundler's output should be compatible with the current system for constructing URLs to various resources.
• Environment Variables: Consider using the bundler's features to manage things like API keys.
• Cache Management: Instead of using the existing version property for cache-busting, explore using the bundler's built-in cache management features.

Metrics for Evaluating App Performance:

To measure the success of performance improvements, we can focus on:

1. Load Time: Measure how quickly the app is ready for use.
2. Interactivity: Assess the time it takes for all app features to be fully functional.
3. First Content Display: Track the time until the first content appears on the screen.

Criteria for Choosing a Bundler

Must-Have:

1. Works with RequireJS: The bundler should operate alongside RequireJS to maintain backward compatibility.
2. Optimizes Performance: Should excel in reducing code size and improving loading through techniques like minification and code-splitting.
3. Supports Dynamic Loading: Must be able to handle dynamic script loading similar to our existing setup.
4. HTML Template Handling: Should allow for the bundling of HTML templates in a manner compatible with our JavaScript.
5. CSS Optimization: At a minimum, the bundler should offer CSS minification.

Nice-to-Have:

1. Quick Builds: Faster compilation times are a bonus.
2. Polyfill Options: The capability to manage or even replace our current polyfill file.
3. Cache Control: Built-in cache management features would be beneficial.
4. Environment Variables: If it can manage API keys securely, that's a plus.
5. Community and Longevity: Signs of strong community support and future maintainability are advantageous.

Bundler Options

Two bundlers seem to be the most promising candidates:

1. Webpack: Given that we need backwards compatibility with RequireJS, Webpack seems to be the most suitable option. Webpack is mature, highly configurable, and has a rich plugin ecosystem. The downside is that it can be challenging to configure and has a steep learning curve.
2. Parcel: Parcel could also be a strong candidate if we're willing to compromise on some level of control for ease of use and speed of setup, since it aims to be zero-config. We need to evaluate if it is suitable for AMD setups with RequireJS.

Bundlers ruled out:

• Rollup: Best for smaller projects.
• Snowpack: No longer actively maintained and is not recommended for new projects!
• Vite and esbuild: Not suitable for AMD setups with RequireJS as far as I can tell.

Next Steps

1. Investigate Other Bundlers: Explore other bundlers that could be suitable for our use case.
2. Shortlist Bundlers: Compare bundlers using our criteria to create a shortlist.
3. Test Compatibility: Run initial tests with shortlisted bundlers for compatibility with our setup.
4. Evaluate Performance: Use the metrics above to test the performance of compatible bundlers.
5. Document Findings: Summarize performance data, challenges, and a potential implementation plan.
6. Pick a Bundler: Choose the most suitable option.
7. Implement: Set up the new build process.
8. Guide: Write a how-to guide for adopting the new build process in repositories.

[robyngit]

Note that RequireJS also has an optimization tool that we have considered using in the past.

[robyngit]

Initial experiments with webpack

• Webpack was able to crawl through the code and bundle it if the entry point was set to app.js. The real entry point is index.html which loads loader.js which loads app.js.
• Without specifying anything, webpack split the bundled code into many files (presumably to prevent creating one huge file).
• The webpack config basically had to re-create the requireJS config.
• Require JS's require-text can be replaced with text-loader to load templates.
• I set some dependencies as external to avoid bundling them because they were triggering errors ("cesium"), did not exist?? ("views/RegistryView"), are actually loaded exernally ("gmaps"), or need to be loaded in a specific way ("require").
• I didn't get far enough to get the bundled code to actually run. The page doesn't load beyond the initial MetacatUI spinner, with some errors in the console to debug.
• We might consider keeping all of the dependencies in component external, and just bundle the MetacatUI code.
• I need to figure out how to ensure that the config and theme files can still be loaded dynamically.
• If the goal right now is just to minify and uglify the code, then perhaps webpack is not the right tool. We could use some tool like grunt or gulp to run a minifier and uglifier on the code. Webpack by definition bundles the code, which might add more complexity than we need right now.
• @helbashandy shared his current experimental set up. He's using a much simpler webpack config that serves to transpile his react code. In terms of MetacatUI files, only the loader.js file is bundled at the moment. He triggers webpack to build from the server.js file using webpack-hot-middleware and webpack-dev-middleware.
• More experiments to follow.

My experimental `webpack.config.js`

const path = require("path");
const HtmlWebpackPlugin = require("html-webpack-plugin");
const CopyPlugin = require("copy-webpack-plugin");

const baseUrlMetacatUI = path.resolve(__dirname, "src", "js");
const baseUrl = path.resolve(__dirname, "src");
const recaptchaURL = "https://www.google.com/recaptcha/api/js/recaptcha_ajax";

module.exports = {
  entry: "./src/js/app.js",
  output: {
    path: path.resolve(__dirname, "dist"),
    filename: "js/app.js",
  },
  resolveLoader: { alias: { text: "text-loader" } },
  resolve: {
    preferRelative: true,
    extensions: [".js"],
    alias: {
      collections: path.resolve(baseUrlMetacatUI, "collections"),
      common: path.resolve(baseUrlMetacatUI, "common"),
      models: path.resolve(baseUrlMetacatUI, "models"),
      routers: path.resolve(baseUrlMetacatUI, "routers"),
      templates: path.resolve(baseUrlMetacatUI, "templates"),
      views: path.resolve(baseUrlMetacatUI, "views"),
      themes: path.resolve(baseUrlMetacatUI, "themes"),
      img: path.resolve(baseUrl, "img"),
      jquery: path.resolve(baseUrl, "components/jquery-1.9.1.min"),
      jqueryui: path.resolve(baseUrl, "components/jquery-ui.min"),
      jqueryform: path.resolve(baseUrl, "components/jquery.form"),
      underscore: path.resolve(baseUrl, "components/underscore-min"),
      backbone: path.resolve(baseUrl, "components/backbone-min"),
      localforage: path.resolve(baseUrl, "components/localforage.min"),
      bootstrap: path.resolve(baseUrl, "components/bootstrap.min"),
      text: path.resolve(baseUrl, "components/require-text"), // <-- ?
      jws: path.resolve(baseUrl, "components/jws-3.2.min"),
      jsrasign: path.resolve(baseUrl, "components/jsrsasign-4.9.0.min"),
      async: path.resolve(baseUrl, "components/async"),
      recaptcha: [recaptchaURL, "scripts/placeholder"],
      nGeohash: path.resolve(baseUrl, "components/geohash/main"),
      fancybox: path.resolve(
        baseUrl,
        "components/fancybox/jquery.fancybox.pack"
      ), //v. 2.1.5
      annotator: path.resolve(
        baseUrl,
        "components/annotator/v1.2.10/annotator-full"
      ),
      bioportal: path.resolve(
        baseUrl,
        "components/bioportal/jquery.ncbo.tree-2.0.2"
      ),
      clipboard: path.resolve(baseUrl, "components/clipboard.min"),
      uuid: path.resolve(baseUrl, "components/uuid"),
      md5: path.resolve(baseUrl, "components/md5"),
      rdflib: path.resolve(baseUrl, "components/rdflib.min"),
      x2js: path.resolve(baseUrl, "components/xml2json"),
      he: path.resolve(baseUrl, "components/he"),
      citation: path.resolve(baseUrl, "components/citation.min"),
      promise: path.resolve(baseUrl, "components/es6-promise.min"),
      metacatuiConnectors: path.resolve(
        baseUrlMetacatUI,
        "/js/connectors/Filters-Search"
      ),
      // showdown + extensions (used in the MarkdownView to convert markdown to html)
      showdown: path.resolve(baseUrl, "components/showdown/showdown.min"),
      showdownHighlight: path.resolve(
        baseUrl,
        "components/showdown/extensions/showdown-highlight/showdown-highlight"
      ),
      highlight: path.resolve(
        baseUrl,
        "components/showdown/extensions/showdown-highlight/highlight.pack"
      ),
      showdownFootnotes: path.resolve(
        baseUrl,
        "components/showdown/extensions/showdown-footnotes"
      ),
      showdownBootstrap: path.resolve(
        baseUrl,
        "components/showdown/extensions/showdown-bootstrap"
      ),
      showdownDocbook: path.resolve(
        baseUrl,
        "components/showdown/extensions/showdown-docbook"
      ),
      showdownKatex: path.resolve(
        baseUrl,
        "components/showdown/extensions/showdown-katex/showdown-katex.min"
      ),
      showdownCitation: path.resolve(
        baseUrl,
        "components/showdown/extensions/showdown-citation/showdown-citation"
      ),
      showdownImages: path.resolve(
        baseUrl,
        "components/showdown/extensions/showdown-images"
      ),
      showdownXssFilter: path.resolve(
        baseUrl,
        "components/showdown/extensions/showdown-xss-filter/showdown-xss-filter"
      ),
      xss: path.resolve(
        baseUrl,
        "components/showdown/extensions/showdown-xss-filter/xss.min"
      ),
      showdownHtags: path.resolve(
        baseUrl,
        "components/showdown/extensions/showdown-htags"
      ),
      // woofmark - markdown editor
      woofmark: path.resolve(baseUrl, "components/woofmark.min"),
      // drop zone creates drag and drop areas
      Dropzone: path.resolve(baseUrl, "components/dropzone-amd-module"),
      // Packages that convert between json data to markdown table
      markdownTableFromJson: path.resolve(
      baseUrl,
        "components/markdown-table-from-json.min"
      ),
      markdownTableToJson: path.resolve(
        baseUrl,
        "components/markdown-table-to-json"
      ),
      // Polyfill required for using dropzone with older browsers
      corejs: path.resolve(baseUrl, "components/core-js"),
      // Searchable multi-select dropdown component
      semanticUItransition: path.resolve(
        baseUrl,
        "components/semanticUI/transition.min"
      ),
      semanticUIdropdown: path.resolve(
        baseUrl,
        "components/semanticUI/dropdown.min"
      ),
      // To make elements drag and drop, sortable
      sortable: path.resolve(baseUrl, "components/sortable.min"),
      //Cesium
      cesium: path.resolve(baseUrl, "components/cesium/Cesium"),
      //Have a null fallback for our d3 components for browsers that don't support SVG
      d3: path.resolve(baseUrl, "components/d3.v3.min"),
      LineChart: path.resolve(baseUrlMetacatUI, "views/LineChartView"),
      BarChart: path.resolve(baseUrlMetacatUI, "views/BarChartView"),
      CircleBadge: path.resolve(baseUrlMetacatUI, "views/CircleBadgeView"),
      DonutChart: path.resolve(baseUrlMetacatUI, "views/DonutChartView"),
      MetricsChart: path.resolve(baseUrlMetacatUI, "views/MetricsChartView"),
    },
  },
  module: {
    rules: [
      {
        test: /\.js$/,
        exclude: /node_modules/,
      },
      // text-loader
      {
        test: /\.txt$/,
        use: "text-loader",
      },
      {
        test: /\.svg$/i,
        use: "raw-loader",
      },
    ],
  },
  plugins: [
    new HtmlWebpackPlugin({
      template: "./src/index.html",
    }),
    new CopyPlugin({
      patterns: [
        {
          from: "./src/components/require.min.js",
          to: "./components/require.min.js",
        },
        {
          from: "./src/js/themes",
          to: "./js/themes",
        },
        {
          from: "./src/loader.js",
          to: "./loader.js",
        },
        {
          from: "./src/config/config.js",
          to: "./config/config.js",
        },
        {
          from: "./src/js/polyfill.js",
          to: "./js/polyfill.js",
        }
      ],
    }),
  ],
  externals: ["views/RegistryView", "gmaps", "cesium", "require"],
};

[robyngit]

TL;DR: Good news: Minifying with Gulp is very easy. Bad news: Minifying the JS files has little impact on performance.

Steps to set up Gulp

Setting up Gulp to minify the JS was very straight-forward compared to attempting to bundle and minify with Webpack.

1. Install Gulp Globally (optional, but makes it easier to run gulp from the command line):

   npm install -g gulp-cli

2. Install Gulp Locally:

   npm install --save-dev gulp

3. Install Required Gulp Plugins:
   Install the gulp-terser and gulp-tap plugins using npm.

   npm install --save-dev gulp-terser gulp-tap

4. Create The Gulpfile (gulpfile.js):

gulpfile.js

const gulp = require('gulp');
const terser = require('gulp-terser');
const tap = require('gulp-tap');

gulp.task('copy-all', function() {
    // Copy all files from src to dist
    return gulp.src('src/**/*')
        .pipe(gulp.dest('dist'));
});

gulp.task('minify-js', function() {
  return gulp.src('dist/**/*.js')
      .pipe(tap(function(file) {
          return gulp.src(file.path, { base: 'dist' })
              .pipe(terser())
              .on('error', function(err) {
                  console.warn(`Error in file ${file.path}: ${err.toString()}`);
                  this.emit('end');
              })
              .pipe(gulp.dest('dist')); // Save it back to the dist directory
      }));
});

gulp.task('build', gulp.series('copy-all', 'minify-js'));

5. Run the Build Task:
   Execute the build task to copy all files from src/ to dist/ and then minify all JavaScript files.

   gulp build

6. Switch the dev server to run from dist rather than src:
   In server.js switch const src_dir = "src"; to const src_dir = "dist";

Performance differences

I ran Chrome's lighthouse test twice: once with files served from src (unminified JS), and once files saved from dist (minified JS). The difference in performance was disappointing:

Un-minified files

Minified files

Conclusions

I think the recommendations detailed by Lauren are the tasks we should focus on in order to improve performance.

[ianguerin]

In considering Issue#224:

I've spent some time trying to see what it would take to migrate to a tool like Webpack. I think it would required migrating from require.js style modules to ES6 modules first. In the meantime, the r.js tool for optimizing and bundling does seem to be a significantly easier effort. I have a branch in my fork of this repo where I've used r.js to create one single bundle file that could be loaded in production instead (the commit). On my local machine this takes me from loading 186 JavaScript files (186 http requests) to loading only 17. I haven't been able to get minification to work (the single JS file is almost 8MB!!) but I think with some more effort it should be possible to figure out the remaining issues that are blocking that.

[robyngit]

Nice! Thank you for looking more into this @ianguerin. Did you measure or notice any differences in load time? I'd guess even with minification, those 17 files might be too large

[ianguerin]

I did not see a significant overall "score" using the Chrome DevTools Lighthouse extension, though it did get better.
I did see an increase in time to first contentful load (bad)
I've attached my two lighthouse runs below, before my changes and after my changes.

Before.pdf
After.pdf

I wouldn't recommend going through this risky of a change for such a minor improvement, but I think playing around with some of the configuration rules, maybe [modules configuration for r.js] (https://github.com/requirejs/r.js/blob/acec5366eb9094e670b6d1a87457634e74d6384e/build/example.build.js#L355) could be beneficial, and this commit has at least a PoC to getting that to work.