JavaScript 2021 chapter and query updates (#2567)

* add requests, async, module queries * Added ajax request per page percentile based query. * Added most requested content-types. * Formatted query. * Added query for iframe per page. * Percentage usage of different ajax APIs. * Added h1 vs h2 vs h3 script. * Added web_components_sepc sql. * linting * consistent syntax to count number of rows * frameworks, sourcemaps queries * 1p vs 3p * adds unminified, preload/prefetch queries * PR feedback accomodate & fixes * link fix * Feedback II update * Fixed review comment related to web_component_specs. * update content_type and webcomponent query * Javascript sql 2021 (#2386) * Bump prettier from 2.4.0 to 2.4.1 in /src (#2334) Bumps [prettier](https://github.com/prettier/prettier) from 2.4.0 to 2.4.1. - [Release notes](https://github.com/prettier/prettier/releases) - [Changelog](https://github.com/prettier/prettier/blob/main/CHANGELOG.md) - [Commits](prettier/prettier@2.4.0...2.4.1) --- updated-dependencies: - dependency-name: prettier dependency-type: direct:development update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * Turn off uses-passive-event-listeners Lighthouse audit (#2335) * Bump node-watch from 0.7.1 to 0.7.2 in /src (#2336) Bumps [node-watch](https://github.com/yuanchuan/node-watch) from 0.7.1 to 0.7.2. - [Release notes](https://github.com/yuanchuan/node-watch/releases) - [Changelog](https://github.com/yuanchuan/node-watch/blob/master/Changelog.md) - [Commits](yuanchuan/node-watch@0.7.1...v0.7.2) --- updated-dependencies: - dependency-name: node-watch dependency-type: direct:development update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * PWA 2021 queries (#2272) * Initial conversion of queries to custom metrics * Better Lighthouse queries * Remove comma * Add Service Worker over time for top 1k sites query * Change adoption ranking to include all ranking categories * Formatting * Linting fixes * Spacing improvements * Missing queries * Query updates * Missed some * Remove s * Space * Misc updates * More Tweaks * Spacing fixes * Fix assestlinks query * Missing DESC * Group rankings properly * Linting fixes * Remove unnecessary extra join * Spacing * Additional queries * Linting fixes * Update Events to look at onEvents * Better array merge * Misc fixes * Better JS * Filtering false positives for install query * Fixes to events query * Adding comment for install query * Expand on comment * Review feedback * Linting fixes * Correct comment * Apply suggestions from code review Co-authored-by: David Fox <david@davidjfox.com> * Review feedback * Review feedback * Change to COUNT(DISTINCT url) Co-authored-by: demianrenzulli <demian77@gmail.com> Co-authored-by: David Fox <david@davidjfox.com> * Bump sqlfluff from 0.6.5 to 0.6.6 in /src (#2339) Bumps [sqlfluff](https://github.com/sqlfluff/sqlfluff) from 0.6.5 to 0.6.6. - [Release notes](https://github.com/sqlfluff/sqlfluff/releases) - [Changelog](https://github.com/sqlfluff/sqlfluff/blob/main/CHANGELOG.md) - [Commits](sqlfluff/sqlfluff@0.6.5...0.6.6) --- updated-dependencies: - dependency-name: sqlfluff dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * Add more 2 more PWA queries (#2340) * Small SQL linting fixes to old media queries (#2341) * Caching 2021 queries (#2254) * Update README.md * Top CDNs * caching queries part 1 * LH queries * caching sql part 2 * changes from review * comment * Delete sql/2021/third-parties directory (#2342) * Delete sql/2021/third-parties directory * Delete third-parties.md * frameworks, sourcemaps queries * 1p vs 3p * adds unminified, preload/prefetch queries * Bump actions/github-script from 4.1 to 5 (#2346) Bumps [actions/github-script](https://github.com/actions/github-script) from 4.1 to 5. - [Release notes](https://github.com/actions/github-script/releases) - [Commits](actions/github-script@v4.1...v5) --- updated-dependencies: - dependency-name: actions/github-script dependency-type: direct:production update-type: version-update:semver-major ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * Bump actions/setup-node from 2.4.0 to 2.4.1 (#2347) Bumps [actions/setup-node](https://github.com/actions/setup-node) from 2.4.0 to 2.4.1. - [Release notes](https://github.com/actions/setup-node/releases) - [Commits](actions/setup-node@v2.4.0...v2.4.1) --- updated-dependencies: - dependency-name: actions/setup-node dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * Bump github/super-linter from 4.7.3 to 4.8.0 (#2350) * Bump github/super-linter from 4.7.3 to 4.8.0 Bumps [github/super-linter](https://github.com/github/super-linter) from 4.7.3 to 4.8.0. - [Release notes](https://github.com/github/super-linter/releases) - [Changelog](https://github.com/github/super-linter/blob/master/docs/release-process.md) - [Commits](github/super-linter@v4.7.3...v4.8.0) --- updated-dependencies: - dependency-name: github/super-linter dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com> * Fix queries failing linting Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Barry <barry@tunetheweb.com> * PR feedback accomodate & fixes * link fix * Feedback II update * Fixed review comment related to web_component_specs. * update content_type and webcomponent query Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Barry Pollard <barry_pollard@hotmail.com> Co-authored-by: demianrenzulli <demian77@gmail.com> Co-authored-by: David Fox <david@davidjfox.com> Co-authored-by: Rick Viscomi <rviscomi@users.noreply.github.com> Co-authored-by: Barry <barry@tunetheweb.com> Co-authored-by: Pankaj Parkar <parkarpankaj3@gmail.com> * lint fixes * lint fix -2 * add review updates * add trailing newline * update queries & markdown init * update SQL queries * lint fixes * lint fixes trailing whitespace * update SQL file names * Apply suggestions from code review * update module_and_nomodule * chapter markdown update * Apply suggestions from code review * Update src/content/en/2021/javascript.md * add NishuGoel_bio * add relevant links, edit * incorporate reviews/edits * add description for figures * fixes & edits * Fix tables * Fix spacing * Fix headings * Smart quotes * Remove unnecessary tables * Fix links * Fix headings and rearrange content * Further reordering * More clean up * Add images * More edits * update on TODO items from @tunetheweb Co-authored-by: Pankaj Parkar <parkarpankaj3@gmail.com> Co-authored-by: Max Ostapenko <bvz2001@gmail.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Barry Pollard <barry_pollard@hotmail.com> Co-authored-by: demianrenzulli <demian77@gmail.com> Co-authored-by: David Fox <david@davidjfox.com> Co-authored-by: Rick Viscomi <rviscomi@users.noreply.github.com> Co-authored-by: Barry <barry@tunetheweb.com>
HTTPArchive · Nov 27, 2021 · cbd0950 · cbd0950
1 parent fd61de5
commit cbd0950
Show file tree

Hide file tree

Showing 37 changed files with 1,074 additions and 11 deletions.
diff --git a/sql/2021/javascript/ajax_async.sql b/sql/2021/javascript/ajax_async.sql
@@ -0,0 +1,16 @@
+# standardSQL
+# usage of async XMLHttpRequest using blink features from usage counters
+SELECT
+  client,
+  pct_urls
+FROM
+  `httparchive.blink_features.usage`
+WHERE
+  yyyymmdd = '20210701' AND
+  feature = 'XMLHttpRequestAsynchronous'
+GROUP BY
+  pct_urls,
+  client
+ORDER BY
+  pct_urls,
+  client
diff --git a/sql/2021/javascript/ajax_request_per_page.sql b/sql/2021/javascript/ajax_request_per_page.sql
@@ -1,3 +1,5 @@
+# standardSQL
+# breadkdown of asynchronous JavaScript and XML requests per page
 SELECT
   client,
   percentile,

diff --git a/sql/2021/javascript/ajax_sync.sql b/sql/2021/javascript/ajax_sync.sql
@@ -0,0 +1,16 @@
+# standardSQL
+# usage of synchronous XMLHttpRequest using blink features from usage counters
+SELECT
+  client,
+  pct_urls
+FROM
+  `httparchive.blink_features.usage`
+WHERE
+  yyyymmdd = '20210701' AND
+  feature = 'XMLHttpRequestSynchronous'
+GROUP BY
+  pct_urls,
+  client
+ORDER BY
+  pct_urls,
+  client
diff --git a/sql/2021/javascript/breakdown_of_pages_using_async_defer.sql b/sql/2021/javascript/breakdown_of_pages_using_async_defer.sql
@@ -0,0 +1,20 @@
+#standardSQL
+# Breakdown of scripts using Async, Defer, Module or NoModule attributes.  Also breakdown of inline vs external scripts
+SELECT
+  _TABLE_SUFFIX AS client,
+  COUNTIF(CAST(JSON_EXTRACT(JSON_EXTRACT_SCALAR(payload, '$._javascript'), '$.script_tags.async') AS INT64) > 0) AS async,
+  COUNTIF(CAST(JSON_EXTRACT(JSON_EXTRACT_SCALAR(payload, '$._javascript'), '$.script_tags.defer') AS INT64) > 0) AS defer,
+  COUNTIF(CAST(JSON_EXTRACT(JSON_EXTRACT_SCALAR(payload, '$._javascript'), '$.script_tags.async_and_defer') AS INT64) > 0) AS async_and_defer,
+  COUNTIF(CAST(JSON_EXTRACT(JSON_EXTRACT_SCALAR(payload, '$._javascript'), '$.script_tags.type_module') AS INT64) > 0) AS module,
+  COUNTIF(CAST(JSON_EXTRACT(JSON_EXTRACT_SCALAR(payload, '$._javascript'), '$.script_tags.nomodule') AS INT64) > 0) AS nomodule,
+  COUNTIF(CAST(JSON_EXTRACT(JSON_EXTRACT_SCALAR(payload, '$._javascript'), '$.script_tags.async') AS INT64) > 0) / COUNT(0) AS async_pct,
+  COUNTIF(CAST(JSON_EXTRACT(JSON_EXTRACT_SCALAR(payload, '$._javascript'), '$.script_tags.defer') AS INT64) > 0) / COUNT(0) AS defer_pct,
+  COUNTIF(CAST(JSON_EXTRACT(JSON_EXTRACT_SCALAR(payload, '$._javascript'), '$.script_tags.async_and_defer') AS INT64) > 0) / COUNT(0) AS async_and_defer_pct,
+  COUNTIF(CAST(JSON_EXTRACT(JSON_EXTRACT_SCALAR(payload, '$._javascript'), '$.script_tags.type_module') AS INT64) > 0) / COUNT(0) AS module_pct,
+  COUNTIF(CAST(JSON_EXTRACT(JSON_EXTRACT_SCALAR(payload, '$._javascript'), '$.script_tags.nomodule') AS INT64) > 0) / COUNT(0) AS nomodule_pct,
+  COUNTIF(CAST(JSON_EXTRACT(JSON_EXTRACT_SCALAR(payload, '$._javascript'), '$.script_tags.async') AS INT64) = 0 AND CAST(JSON_EXTRACT(JSON_EXTRACT_SCALAR(payload, '$._javascript'), '$.script_tags.defer') AS INT64) = 0) AS neither,
+  COUNTIF(CAST(JSON_EXTRACT(JSON_EXTRACT_SCALAR(payload, '$._javascript'), '$.script_tags.async') AS INT64) = 0 AND CAST(JSON_EXTRACT(JSON_EXTRACT_SCALAR(payload, '$._javascript'), '$.script_tags.defer') AS INT64) = 0) / COUNT(0) AS neither_pct
+FROM
+  `httparchive.pages.2021_07_01_*`
+GROUP BY
+  client
diff --git a/sql/2021/javascript/breakdown_of_scripts_using_async_defer_module_nomodule.sql b/sql/2021/javascript/breakdown_of_scripts_using_async_defer_module_nomodule.sql
@@ -0,0 +1,39 @@
+#standardSQL
+# Breakdown of scripts using Async, Defer, Module or NoModule attributes.  Also breakdown of inline vs external scripts
+CREATE TEMPORARY FUNCTION getScripts(payload STRING)
+RETURNS STRUCT<total INT64, inline INT64, src INT64, async INT64, defer INT64, async_and_defer INT64, type_module INT64, nomodule INT64>
+LANGUAGE js AS '''
+try {
+  var $ = JSON.parse(payload);
+  var javascript = JSON.parse($._javascript);
+  return javascript.script_tags;
+} catch (e) {
+  return {};
+}
+''';
+
+SELECT
+  client,
+  SUM(script.total) AS total_scripts,
+  SUM(script.inline) AS inline_script,
+  SUM(script.src) AS external_script,
+  SUM(script.src) / SUM(script.total) AS pct_external_script,
+  SUM(script.inline) / SUM(script.total) AS pct_inline_script,
+  SUM(script.async) AS async,
+  SUM(script.defer) AS defer,
+  SUM(script.async_and_defer) AS async_and_defer,
+  SUM(script.type_module) AS module,
+  SUM(script.nomodule) AS nomodule,
+  SUM(script.async) / SUM(script.src) AS pct_external_async,
+  SUM(script.defer) / SUM(script.src) AS pct_external_defer,
+  SUM(script.async_and_defer) / SUM(script.src) AS pct_external_async_defer,
+  SUM(script.type_module) / SUM(script.src) AS pct_external_module,
+  SUM(script.nomodule) / SUM(script.src) AS pct_external_nomodule
+FROM (
+  SELECT
+    _TABLE_SUFFIX AS client,
+    getScripts(payload) AS script
+  FROM
+    `httparchive.pages.2021_07_01_*`)
+GROUP BY
+  client
diff --git a/sql/2021/javascript/lighthouse_vulnerabilities.sql b/sql/2021/javascript/lighthouse_vulnerabilities.sql
@@ -0,0 +1,18 @@
+#standardSQL
+# Pages with vulnerable libraries
+SELECT
+  score,
+  COUNT(0) AS pages,
+  SUM(COUNT(0)) OVER () AS total,
+  COUNT(0) / SUM(COUNT(0)) OVER () AS pct
+FROM (
+  SELECT
+    JSON_EXTRACT_SCALAR(report, "$.audits['no-vulnerable-libraries'].score") AS score
+  FROM
+    `httparchive.lighthouse.2021_07_01_mobile`)
+WHERE
+  score IS NOT NULL
+GROUP BY
+  score
+ORDER BY
+  score
diff --git a/sql/2021/javascript/lighthouse_vulnerable_libraries.sql b/sql/2021/javascript/lighthouse_vulnerable_libraries.sql
@@ -0,0 +1,29 @@
+#standardSQL
+# Most frequent vulnerable libraries
+CREATE TEMPORARY FUNCTION getVulnerabilities(audit STRING)
+RETURNS ARRAY<STRING> LANGUAGE js AS '''
+try {
+  var $ = JSON.parse(audit);
+  return $.details.items.map(i => i.detectedLib.text.split('@')[0]);
+} catch(e) {
+  return [];
+}
+''';
+
+SELECT
+  lib,
+  COUNT(0) AS freq,
+  total,
+  COUNT(0) / total AS pct
+FROM
+  `httparchive.lighthouse.2021_07_01_mobile`,
+  UNNEST(getVulnerabilities(JSON_EXTRACT(report, "$.audits['no-vulnerable-libraries']"))) AS lib, (
+    SELECT
+      COUNT(DISTINCT url) AS total
+    FROM
+      `httparchive.lighthouse.2021_07_01_mobile`)
+GROUP BY
+  lib,
+  total
+ORDER BY
+  freq DESC
diff --git a/sql/2021/javascript/module_and_nomodule.sql b/sql/2021/javascript/module_and_nomodule.sql
@@ -0,0 +1,25 @@
+SELECT
+  client,
+  COUNT(DISTINCT IF(module, page, NULL)) AS module,
+  COUNT(DISTINCT IF(nomodule, page, NULL)) AS nomodule,
+  COUNT(DISTINCT IF(nomodule AND module, page, NULL)) AS both,
+  COUNT(DISTINCT page) AS total,
+  COUNT(DISTINCT IF(module, page, NULL)) / COUNT(DISTINCT page) AS pct_module,
+  COUNT(DISTINCT IF(nomodule, page, NULL)) / COUNT(DISTINCT page) AS pct_nomodule,
+  COUNT(DISTINCT IF(module AND nomodule, page, NULL)) / COUNT(DISTINCT page) AS pct_both
+FROM (
+  SELECT
+    client,
+    page,
+    script,
+    REGEXP_CONTAINS(script, r'(?i)\bmodule\b') AS module,
+    REGEXP_CONTAINS(script, r'(?i)\bnomodule\b') AS nomodule
+  FROM
+    `httparchive.almanac.summary_response_bodies`
+  LEFT JOIN
+    UNNEST(REGEXP_EXTRACT_ALL(body, r'(?i)(<script[^>]*>)')) AS script
+  WHERE
+    date = '2021-07-01' AND
+    firstHtml)
+GROUP BY
+  client
diff --git a/sql/2021/javascript/requests_by_3p.sql b/sql/2021/javascript/requests_by_3p.sql
@@ -0,0 +1,33 @@
+#standardSQL
+# Distribution of 1P/3P JS requests
+SELECT
+  percentile,
+  client,
+  host,
+  APPROX_QUANTILES(requests, 1000)[OFFSET(percentile * 10)] AS num_requests
+FROM (
+  SELECT
+    client,
+    page,
+    IF(NET.HOST(url) IN (
+      SELECT domain FROM `httparchive.almanac.third_parties` WHERE date = '2021-07-01' AND category != 'hosting'
+    ), 'third party', 'first party') AS host,
+    COUNT(0) AS requests
+  FROM
+    `httparchive.almanac.requests`
+  WHERE
+    date = '2021-07-01' AND
+    type = 'script'
+  GROUP BY
+    client,
+    page,
+    host),
+  UNNEST([10, 25, 50, 75, 90, 100]) AS percentile
+GROUP BY
+  percentile,
+  client,
+  host
+ORDER BY
+  percentile,
+  client,
+  host
diff --git a/sql/2021/javascript/web_components_specs.sql b/sql/2021/javascript/web_components_specs.sql
@@ -11,6 +11,8 @@ FROM (
     ARRAY_LENGTH(JSON_EXTRACT_ARRAY(JSON_EXTRACT_SCALAR(payload, '$._javascript'), '$.web_component_specs.shadow_roots')) AS shadow_roots,
     ARRAY_LENGTH(JSON_EXTRACT_ARRAY(JSON_EXTRACT_SCALAR(payload, '$._javascript'), '$.web_component_specs.template')) AS template
   FROM
+    # Note: We're intentionally querying the September dataset here because of a bug in the custom metric.
+    # See https://github.com/HTTPArchive/legacy.httparchive.org/pull/231.
     `httparchive.pages.2021_09_01_*`),
   UNNEST([10, 25, 50, 75, 90, 100]) AS percentile
 GROUP BY

diff --git a/src/config/2021.json b/src/config/2021.json
@@ -22,8 +22,7 @@
           "part": "I",
           "chapter_number": "2",
           "title": "JavaScript",
-          "slug": "javascript",
-          "todo": "true"
+          "slug": "javascript"
         },
         {
           "part": "I",