Skip to content

Production instrumentation

Jump to bottom
Laura Harker edited this page Nov 3, 2020 · 2 revisions

Experimental & available as of v20200830

Production Instrumentation is a feature which will instrument JavaScript source code that can be executed client-side and generate a report reflecting code coverage details.

Instrumentation is done in a way that it is properly obfuscated from the user so as to not leak any information about the source code. It is also minified significantly so that the performance impact to the user is minimal, with most of the work being done server-side.

General Overview

To get Production instrumentation working, below are the general steps which you need to complete:

  1. Create an extern which contains the definition for a global array. In addition to the extern, you must also create a JS file which will parse the global array and send the information to an end point (A working example is provided below).
  2. Compile the JavaScript source code as well as the file and extern created from step 1 with the proper production instrumentation flags
  3. Create a server that exposes an endpoint which can be called from the instrumented source code in order to save the reports generated while executing.
  4. Run the reporter to consume the reports generated by the instrumented client code and the mapping produced by the compiler to get final coverage data.

To get started with basic usability which provides a detailed report of coverage results, follow below:

How to enable

Run the closure compiler with the following arguments:

--instrument_for_coverage_option PRODUCTION \ # Enables the production instrumentation pass
--instrument_mapping_report instrumentationMapping.txt \  # The file name for the mapping generated by the instrumentation pass
--production_instrumentation_array_name ist_arr  # The name of the array used for instrumentation

If you wish to compile with ADVANCED Optimizations enabled, also pass the flag --compilation_level ADVANCED.

In addition to passing with these command line arguments, you must also include the following extern (closure compiler externs):

/**
 * @externs
 */

// Must match the name provided with --production_instrumentation_array_name flag
/** @const {!Array<String>} */
var ist_arr;

Along with the following JavaScript file:

goog.module('instrument.code');

/** @type {!Map<String,number>} */
const report = new Map();

/**
 * The instrumentCode function which is the instrumentation functon that will
 * be called during program execution. It gathers all the data in the
 * report variable defined above.
 */
function instrumentCode() {
  if (ist_arr.length > 10000) {
    for (const param of ist_arr) {
      report.has(param) ? report.set(param, report.get(param) + 1) :
                          report.set(param, 1);
    }
    ist_arr.length = 0;
    const reportJson = {};
    report.forEach((value, key) => {
      reportJson[key] = {frequency: value};
    })
    console.log(JSON.stringify(reportJson));
  }
}

window.setInterval(() => {instrumentCode()}, 500);

This file contains a simple and easy function that will periodically check if 10,000 (This should be changed to meet expectations) instrumentation points have been reached. If it has, it will iterative over the array and send a JSON report to a server also specified in this file. In the above file, this is a console.log(), however this should be changed when enabling instrumentation. It is important to note, that after the array has been parsed and the report generated, that the array must be emptied. This is done so as to avoid any memory issues while running on a browser.

The output of this compilation is the compiled JavaScript source code with all the added instrumentation points. This code is ready to run client-side.

While running client-side, the source code will periodically send reports to the server specified in the above file. Once enough reports have been generated (at the developers discretion), the developer can now run the reporting tool.

Using the reporting tool:

The reporting tool will eventually be merged into the Closure Compiler repo. At the moment it lives here: https://github.com/patrick-sko/instrumentation-reporter/pull/2 and must be cloned locally to run.

The reporting tool is a standalone Java program that consumes the mapping generated by the Production Instrumentation pass (as defined by --instrument_mapping_report flag) and all the reports generated by the Instrumented client code which should be all be stored in a single directory.

To first compile the standalone java program from the command line:
javac -cp "jarFilesLocation" location.of.java.files (To change when reporter has been added)

And then run the reporter as follows:

java -cp ".;.jarFilesLocation" 
     com.google.javascript.jscomp.instrumentation.ProductionInstrumentationReporter
     --mapping_file instrument_mapping_report // Location of mapping generated by compiler
     --reports_directory location_of_reports // The location of the directory which contains all the generated reports
     --result_output finalResult.json // The name of the JSON which contains the detailed instrumentation data breakdown

(Notice: This is standard compilation and running with the javac and java command. For reference, on Mac the command to run with java requires that .;. is replaced with .:. when passing to the -cp flag)

The external libraries required are a subset of those needed by the closure compiler.

JSON details

The JSON generated by this reporting tool provides a detailed summary of all instrumentation points. It provides a percentage of total instrumentation points that were executed, broken down by function and branch. It goes into further detail at a per function scope, of whether the function itself was executed and if any of the branches within the function were executed. For every instrumentation point, it calculates an average of the total percent of times this point was executed across all the reports parsed. A value of 0 signifies that it was never executed. Each instrumentation point also has a frequency which is the number of times that point was executed, averaged across all reports parsed. Please see the unit test for an example of what the inputs and output would look like.

Implementation details

This feature uses intermediary language instrumentation, where it modifies the Abstract Syntax Tree generated by the compiler and adds instrumentation calls at various part. At the beginning of function blocks, and after every branching condition (loops, switch, binary operators, coalesce, ternary operator). The instrumentation calls are of the following form ist_arr.push(id) where ist_arr is a global array named by the --production_instrumentation_array_name flag. Encoded id's, which can be decoded using the mapping, represent the location details of each instrumentation point. If an instrumentation point is reached, it will push the encoded id onto the global array.

Each instrumentation point also has encoded information about what it represents. Presently, the types are:
FUNCTION - This instrumentation point is added to the first line of a function call.
BRANCH - This instrumentation point is added to the first line of an if statement, switch case, loop structure, binary operator (||, &&), ternary operator and coalesce.
DEFAULT_BRANCH - This instrumentation point is added to the default case of a branch condition (else statement, default case). This type is unique as it has several edge cases. If it is an 'else' statement being instrumented, then the location will relate to the else-if statement immediately prior. Also, if a default branch is not presented, production instrumentation will add one. This will help determine if a branch condition is actually needed.

For more information about the implementation details of instrumentation, please see this doc: https://docs.google.com/document/d/1rsbzzZjz513Qaio7nZT3jLgSxZrJ58dL1carrS8cPbk/edit

Example:

Consider the following piece of code:

function fib(n) {
  if (n <= 1) {
    return 1;
  }
  return fib(n - 1) + fib(n - 2);
}

function main() {
  const n = 4;
  console.log(fib(4));
}
main();

When compiled with production instrumentation enabled, and the above function will yield the following mapping:

 FileNames:["fib.js","InstrumentCode.js"]
 FunctionNames:["fib","main","module$contents$instrument$code_instrumentCode","<Anonymous>"]
 Types:["BRANCH","BRANCH_DEFAULT","FUNCTION"]
C:AAAEC
E:AACEC
G:AAECA
I:ACESA
K:CEAoBI
M:CECoBI
O:CEAkBE
Q:CGE8BiB
S:CEAeC
U:CECeC
W:CEEaE
Y:CGEuCmB

Taking the mapping above along with a report generated by the instrumented code and feeding it to the reporter will produce the following (partial) JSON report.

{
  "totalReportsParsed": 1,
  "percentOfFunctionsExecuted": 100.0,
  "percentOfBranchesExecuted": 42.86,
  "result": [
    {
      "fileName": "fib.js",
      "percentOfFunctionsExecuted": 100.0,
      "percentOfBranchesExecuted": 100.0,
      "profilingResultPerFunction": {
        "main": [
          {
            "param": "I",
            "type": "FUNCTION",
            "lineNo": 9,
            "colNo": 0,
            "executed": 100.0,
            "data": {
              "frequency": 1
            }
          }
        ],
        "fib": [
          {
            "param": "C",
            "type": "BRANCH",
            "lineNo": 2,
            "colNo": 1,
            "executed": 100.0,
            "data": {
              "frequency": 5
            }
          },
...
Clone this wiki locally