New: Add a standalone `parser-html` #1277

antross · 2018-08-28T16:00:33Z

Pull request checklist

Make sure you:

Signed the Contributor License Agreement
Followed the commit message guidelines

For non-trivial changes, please make sure you also:

Added/Updated related documentation.
Added/Updated related tests.

Short description of the change(s)

For use by connector-local in support of #1254 so we can expand the
set of hints which run in the VSCode extension.

Uses jsdom (with scripts and resource loading disabled) to maximize
support for existing hints (e.g. hint-axe).

sarvaje · 2018-08-28T22:39:09Z

packages/connector-local/src/connector.ts

@@ -242,6 +245,17 @@ export default class LocalConnector implements IConnector {
        });
    }

+    private watchForWindow() {


This will work if there is more than one html file?

This will only populate the connector for the target HTML file as the connector APIs are tied to that singleton as far as I could tell.

To further clarify, if connector-local is used with a glob pattern matching multiple HTML files, only the first one parsed will be used to populate DOM-related connector APIs.

Note in the context of the VS Code extension, connector-local will be called separately for each file so this is not an issue there.

Yep, the problem will be with a glob pattern.

I think you need to touch a little bit how the connector works to support multiple HTMLs.

Right now, the fetch is in parallel, but maybe you can refactor the code to emit the fetch::end::html serially so you can set the properties in the connector for each html, this way, the connector will support the parse-html for all the HTML files. Do you know what I mean?

The idea is when the connector store all the HTMLs in an array, without emits the fetch::end::html. Then, iterate that array serially doing:

emit the fetch::end::html

in the event parse::html set this._window

wait until the analysis of the html finish.

clean this._window

go to the next html.

Makes sense and doesn't look too difficult from glancing at the code. I'll see if I can incorporate this tomorrow.

sarvaje · 2018-08-28T22:41:18Z

packages/hint/src/lib/types/jsdom-async-html.ts

@@ -1,4 +1,5 @@
-import { IAsyncHTMLDocument, IAsyncHTMLElement } from './async-html'; //eslint-disable-line
+import { IAsyncHTMLDocument, IAsyncHTMLElement, IAsyncWindow } from './async-html'; //eslint-disable-line


What does it happen if you remove //eslint-disable-line?

Nothing it seems. Want me to remove it? I was originally only making necessary changes (as this comment was already in the file), but it's an easy change so I can include the removal in this PR.

Yes please. I saw it wasn't your change, but it would be nice to remove it :)

alrra · 2018-08-28T22:50:48Z

packages/parser-html/package.json

+    "watch:test": "ava --watch",
+    "watch:ts": "npm run build:ts -- --watch"
+  },
+  "version": "0.0.1"


alrra · 2018-08-28T22:54:38Z

packages/connector-local/package.json

@@ -7,6 +7,7 @@
    "timeout": "1m"
  },
  "dependencies": {
+    "@hint/parser-html": "^0.0.1",


Note to self: Move this into a different commit.

Honestly that should be a dev dependency as it's only used for types (specifically the HTMLParse event). The only functional tie connector-local has to the parser-html is via the parse::html::end event. I'll fix it (though that probably doesn't change your need to add even the dev dependency in a separate commit).

I was referring a different commit, not anything about code changes.

That is for the release process.

antross · 2018-08-28T23:28:19Z

packages/connector-local/src/connector.ts

+        this._window = null;
+
+        const setWindow = (event: HTMLParse) => {
+            this._window = event.window;


This also needs to emit a can-evaluate::script event here or hint-axe and friends won't work. I'll update the PR for this in a bit (need to grab a bite to eat).

antross · 2018-08-29T23:03:34Z

FYI, I merged and tested this with the VSCode extension PR and it worked (although error positions will need a bit of work as expected). Even got hint-axe running after changing its HintScope to any.

molant · 2018-08-29T23:56:39Z

Even got hint-axe running after changing its HintScope to any.

🎉🎉🎉

molant · 2018-08-30T01:55:21Z

Everything is failing because nyc can't be found 🤔

Error: Cannot find module 'C:\projects\hint\packages\parser-html\node_modules\nyc\bin\nyc.js'
    at Function.Module._resolveFilename (module.js:547:15)
    at Function.Module._load (module.js:474:25)
    at Function.Module.runMain (module.js:693:10)

molant

This looks great. I just have a couple comments but otherwise great job. @sarvaje, @alrra can you take a look when you have time please?

molant · 2018-08-30T02:10:08Z

packages/parser-html/README.md

+
+Note: This parser is not needed if using a [connector][connectors]
+which connects to a browser as the browser's DOM will be used to
+generate events instead.


The jsdom connector doesn't need it neither. Maybe reformulate to say it's only required by local?

molant · 2018-08-30T02:11:36Z

packages/parser-html/src/parser.ts

+        const resource = this._url = fetchEnd.response.url;
+
+        const html = fetchEnd.response.body.content;
+        const dom = new JSDOM(html, { runScripts: 'outside-only' });


Maybe add a comment here explaining why we want outside-only for future us?

molant · 2018-08-30T02:12:54Z

packages/parser-html/src/parser.ts

+
+        await this.engine.emitAsync(`traverse::up`, traverseEvent);
+
+        return Promise.resolve();


You are already using await through this method so you shouldn't need to return a Promise. Is TS complaining?

sarvaje

LGTM

packages/connector-local/package.json

alrra · 2018-09-03T16:58:53Z

packages/parser-html/README.md

@@ -0,0 +1,60 @@
+# HTML parser (`@hint/parser-html`)
+
+The `HTML` parser is built on top of `[jsdom][jsdom]` so hints can


[`jsdom`][jsdom]

alrra · 2018-09-03T17:01:52Z

packages/parser-html/package.json

+  },
+  "description": "webhint parser needed to analyze HTML files",
+  "devDependencies": {
+    "@types/cheerio": "^0.22.9",


This is no longer needed?

alrra · 2018-09-03T17:02:15Z

packages/parser-html/package.json

+    "@types/cheerio": "^0.22.9",
+    "@types/debug": "0.0.30",
+    "@types/jsdom": "^11.0.6",
+    "@types/node": "8.0.14",


Remove types that are not necessary.

(we try to keep them to a minimum, because they often break our builds 😔)

Also, after you do these changes, can you run yarn in the root so that yarn.lock file gets updated. Thanks!

alrra · 2018-09-03T17:06:22Z

packages/parser-html/tests/parser.ts

+    });
+});
+
+test.serial('If fetch::end::html is received, then we should parse the code and emit a parse::html::end event', async (t) => {


If 'fetch::end::html' is received, then the code should be parsed and the `parse::html::end` event emitted

antross · 2018-09-04T19:59:21Z

@alrra do you want me to rebase and resolve the yarn.lock conflicts?

alrra · 2018-09-04T20:07:56Z

do you want me to rebase and resolve the yarn.lock conflicts?

Yes, thanks!

Just remove the yarn.lock and run yarn, don't do it manually.

For use by `connector-local`. Currently uses `cheerio`, but may need to be cut-over to `jsdom` (with scripts and resource loading disabled) to maximize support for existing hints (e.g. `hint-axe`).

Currently requires parser-html to be explicitly add to the config for the data to be populated.

Includes changes to `connector-local` to implement `evaluate` and `querySelectorAll` based upon the new backing.

I missed adding a property on the new `IAsyncWindow` interface which was causing the tests to fail. I thought I had re-run the tests after that change, but apparently not.

I started using `engine.on` to listen for `parse::html::end`. Since this wasn't in the mock for `t.context.engine` one of the early tests failed (and the rest generated the error `'Attempted to wrap default which is already wrapped'` since the test didn't clean up.

Accomplished by keeping the fetch in parallel, but emitting the `fetch::end::${type}` events serially so we can have the correct internal `_window` reference during `hint` execution for each file (per David's suggestion). Also emit the `can-evaluate::script` event so `hint-axe` and others which need to inject script will do so.

Also fix JSDOM options to include source locations for elements.

Turns out `test-only` wasn't defined. I'll be fixing this in the template for `create-parser` in a separate commit.

Conditional on `#if official` (per Anton's recommendation) since it's only needed in the CI for `hint` itself.

Includes updating `yarn.lock`.

packages/create-parser/src/shared-templates/package.hbs

alrra · 2018-09-05T22:56:53Z

Package has been published. 🎉

antross added the status:work-in-progress label Aug 28, 2018

antross requested review from alrra, molant and sarvaje as code owners August 28, 2018 16:00

antross mentioned this pull request Aug 28, 2018

New: Add extension for VSCode #1254

Closed

15 tasks

antross removed the status:work-in-progress label Aug 28, 2018

sarvaje reviewed Aug 28, 2018

View reviewed changes

alrra reviewed Aug 28, 2018

View reviewed changes

antross commented Aug 28, 2018

View reviewed changes

alrra force-pushed the master branch from f5ab526 to 8e00cfd Compare August 28, 2018 23:35

This comment has been minimized.

Sign in to view

antross force-pushed the parser-html branch from 7a0e6e1 to 02f1bb1 Compare August 29, 2018 17:05

molant reviewed Aug 30, 2018

View reviewed changes

sarvaje approved these changes Aug 30, 2018

View reviewed changes

molant approved these changes Aug 31, 2018

View reviewed changes

alrra reviewed Aug 31, 2018

View reviewed changes

packages/connector-local/package.json Show resolved Hide resolved

alrra reviewed Sep 3, 2018

View reviewed changes

antross added 7 commits September 4, 2018 16:03

New: Add a standalone parser-html

e808cc6

For use by `connector-local`. Currently uses `cheerio`, but may need to be cut-over to `jsdom` (with scripts and resource loading disabled) to maximize support for existing hints (e.g. `hint-axe`).

Chore: Use parser-html data from connector-local

71264c0

Currently requires parser-html to be explicitly add to the config for the data to be populated.

Chore: Switch html-parser to jsdom

9ec139e

Includes changes to `connector-local` to implement `evaluate` and `querySelectorAll` based upon the new backing.

Chore: Add README content for parser-html

933f89c

Chore: Address PR feedback and fix tests

98db16f

I missed adding a property on the new `IAsyncWindow` interface which was causing the tests to fail. I thought I had re-run the tests after that change, but apparently not.

antross added 5 commits September 4, 2018 16:03

Chore: Address PR feedback

a41c739

Also fix JSDOM options to include source locations for elements.

Chore: Fix test failures in PR

40909ef

Turns out `test-only` wasn't defined. I'll be fixing this in the template for `create-parser` in a separate commit.

Chore: Include test-only in create-parser template

31aa0fc

Conditional on `#if official` (per Anton's recommendation) since it's only needed in the CI for `hint` itself.

Chore: Minor text changes from PR

100779f

Chore: Remove obsolete dependencies

6519ebf

Includes updating `yarn.lock`.

antross force-pushed the parser-html branch from 484b1a2 to 6519ebf Compare September 4, 2018 21:21

antross mentioned this pull request Sep 5, 2018

Include node locations for HTML in connector-jsdom #1289

Closed

alrra closed this in a7ca732 Sep 5, 2018

alrra reviewed Sep 5, 2018

View reviewed changes

packages/create-parser/src/shared-templates/package.hbs Show resolved Hide resolved

antross deleted the parser-html branch October 14, 2018 14:32

dependabot-preview bot mentioned this pull request Apr 19, 2019

Upgrade: Bump eslint-plugin-import from 2.16.0 to 2.17.2 #2283

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

New: Add a standalone `parser-html` #1277

New: Add a standalone `parser-html` #1277

antross commented Aug 28, 2018 •

edited

Loading

sarvaje Aug 28, 2018

antross Aug 28, 2018

antross Aug 28, 2018

sarvaje Aug 28, 2018

antross Aug 29, 2018

sarvaje Aug 28, 2018

antross Aug 28, 2018

sarvaje Aug 28, 2018

alrra Aug 28, 2018

alrra Aug 28, 2018

antross Aug 28, 2018

alrra Aug 28, 2018 •

edited

Loading

antross Aug 28, 2018

This comment has been minimized.

This comment has been minimized.

antross commented Aug 29, 2018

molant commented Aug 29, 2018

molant commented Aug 30, 2018

molant left a comment

molant Aug 30, 2018

molant Aug 30, 2018

molant Aug 30, 2018

sarvaje left a comment

alrra Sep 3, 2018

alrra Sep 3, 2018

alrra Sep 3, 2018

alrra Sep 3, 2018

alrra Sep 3, 2018

antross commented Sep 4, 2018

alrra commented Sep 4, 2018

alrra commented Sep 5, 2018

		@@ -1,4 +1,5 @@
		import { IAsyncHTMLDocument, IAsyncHTMLElement } from './async-html'; //eslint-disable-line
		import { IAsyncHTMLDocument, IAsyncHTMLElement, IAsyncWindow } from './async-html'; //eslint-disable-line


		await this.engine.emitAsync(`traverse::up`, traverseEvent);

		return Promise.resolve();

		@@ -0,0 +1,60 @@
		# HTML parser (`@hint/parser-html`)

		The `HTML` parser is built on top of `[jsdom][jsdom]` so hints can

New: Add a standalone parser-html #1277

New: Add a standalone parser-html #1277

Conversation

antross commented Aug 28, 2018 • edited Loading

Pull request checklist

Short description of the change(s)

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

alrra Aug 28, 2018 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

This comment has been minimized.

This comment has been minimized.

antross commented Aug 29, 2018

molant commented Aug 29, 2018

molant commented Aug 30, 2018

molant left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

sarvaje left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

antross commented Sep 4, 2018

alrra commented Sep 4, 2018

alrra commented Sep 5, 2018

New: Add a standalone `parser-html` #1277

New: Add a standalone `parser-html` #1277

antross commented Aug 28, 2018 •

edited

Loading

alrra Aug 28, 2018 •

edited

Loading