Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

chore: switch to new xmldom and fast-xml-parser #3920

Merged
merged 11 commits into from
Sep 13, 2021

Conversation

vasicvuk
Copy link
Contributor

@vasicvuk vasicvuk commented Sep 3, 2021

Fixes #3916

Description

Since x2js did not respond to PR I have changed the library to fast-xml-parser which is far more maintained and then I have updated xmldom to latest @xmldom/xmldom

Specific Changes

  • In package.json I have changed xmldom to latest @xmldom/xmldom which has newer versions
  • In package.json I have changed x2js with fast-xml-parser which is more maintained project for conversion.
  • In xml.ts i have changed x2js to xml-js
  • In xpath.ts i have changed xmldom to @xmldom/xmldom

Testing

All existing tests are passing as no functionality is changed

@vasicvuk
Copy link
Contributor Author

vasicvuk commented Sep 3, 2021

I also have removed the xml2js as fast-xml-parser works both with Node and Browser. I appended the XML header and trimmed the newline so that tests work without modification as they did for NodeJS.

@joshgummersall
Copy link
Contributor

For posterity, fast-xml-parser gets roughly 90x the weekly downloads as x2js on NPM.

@vasicvuk can you confirm that this package works well in the browser? Running yarn build inside the adaptive-expressions package should produce a file lib/browser.js which should load and run in a browser.

@joshgummersall joshgummersall changed the title Switch to new xmldom and fast-xml-parser #3919 chore: switch to new xmldom and fast-xml-parser Sep 3, 2021
@joshgummersall
Copy link
Contributor

Looks like @types/xmldom can be removed (as per depcheck results).

@coveralls
Copy link

coveralls commented Sep 3, 2021

Pull Request Test Coverage Report for Build 1218600997

  • 9 of 11 (81.82%) changed or added relevant lines in 2 files are covered.
  • No unchanged relevant lines lost coverage.
  • Overall coverage increased (+0.03%) to 84.543%

Changes Missing Coverage Covered Lines Changed/Added Lines %
libraries/adaptive-expressions/src/builtinFunctions/xml.ts 8 10 80.0%
Totals Coverage Status
Change from base Build 1175654529: 0.03%
Covered Lines: 19635
Relevant Lines: 21998

💛 - Coveralls

@vasicvuk
Copy link
Contributor Author

vasicvuk commented Sep 3, 2021

Sure, i will remove the typings, and check in the browser ones i get to PC later

@vasicvuk
Copy link
Contributor Author

vasicvuk commented Sep 4, 2021

Hi @joshgummersall,

I removed typings and also tested if the changed worked in browser.

I used following HTML for testing:

<html>
<head>
	<script src="lib/browser.js"></script>
</head>
<body>
	<script>
		const scope = {};
		var expression = AEL.Expression.parse('xml(\'{"person": {"name": "Sophia Owen", "city": "Seattle"}}\')'.toString());
		const value = expression.tryEvaluate(scope);
		console.log(value);
	</script>
</body>

</html>

As a result i got this:
image

which confirms that it works in browser too.

@joshgummersall
Copy link
Contributor

@vasicvuk, please remove package-lock.json and re-run yarn within the repo to update yarn.lock.

@vasicvuk
Copy link
Contributor Author

vasicvuk commented Sep 7, 2021

@joshgummersall I can remove the package lock but there is no yarn.lock in libraries/adaptive-expressions. Root folder yarn.lock will remain same as nothing changes when I run the command.

D:\Projects\botbuilder-js>yarn
yarn install v1.22.11
[1/4] Resolving packages...
success Already up-to-date.
Done in 0.53s.

@joshgummersall
Copy link
Contributor

@vasicvuk, yep that's expected! One yarn.lock covers the whole repo.

@vasicvuk
Copy link
Contributor Author

vasicvuk commented Sep 8, 2021

@joshgummersall
It seems that build of whole project cannot pass without @types/xmldom. Should i return it back?. In general it makes sense to have it as xmldom is used in the project still. Previously I have only run build on adaptive-expression as that is the project I have changed.

@joshgummersall
Copy link
Contributor

It looks like @xmldom/xmldom provides types internally: https://github.com/xmldom/xmldom/blob/master/package.json#L21, so @types/xmldom should be unnecessary. There must be something else going on. I'll clone your fork later today and take a closer look.

@vasicvuk
Copy link
Contributor Author

vasicvuk commented Sep 8, 2021

Okey. I just noticed that build passes when I return it and I get same error when I don't have it. Here are logs:

botbuilder-applicationinsights
 | ../botframework-connector/lib/auth/botFrameworkAuthenticationFactory.d.ts(32,435): error TS2304: Cannot find name 'RequestInfo'.
 | ../botframework-connector/lib/auth/botFrameworkAuthenticationFactory.d.ts(32,455): error TS2304: Cannot find name 'RequestInit'.
 | ../botframework-connector/lib/auth/botFrameworkAuthenticationFactory.d.ts(32,479): error TS2304: Cannot find name 'Response'.
 | ../botbuilder-core/lib/configurationBotFrameworkAuthentication.d.ts(50,227): error TS2304: Cannot find name 'RequestInfo'.
 | ../botbuilder-core/lib/configurationBotFrameworkAuthentication.d.ts(50,247): error TS2304: Cannot find name 'RequestInit'.
 | ../botbuilder-core/lib/configurationBotFrameworkAuthentication.d.ts(50,271): error TS2304: Cannot find name 'Response'.
 | ../botbuilder-core/lib/configurationBotFrameworkAuthentication.d.ts(71,242): error TS2304: Cannot find name 'RequestInfo'.
 | ../botbuilder-core/lib/configurationBotFrameworkAuthentication.d.ts(71,262): error TS2304: Cannot find name 'RequestInit'.
 | ../botbuilder-core/lib/configurationBotFrameworkAuthentication.d.ts(71,286): error TS2304: Cannot find name 'Response'.
error Command failed with exit code 1.
info Visit https://yarnpkg.com/en/docs/cli/run for documentation about this command.
 | `yarn build` failed with exit code 1
Stopping 17 active children
Aborted execution due to previous error
error Command failed with exit code 1.
info Visit https://yarnpkg.com/en/docs/cli/run for documentation about this command.

D:\Projects\botbuilder-js>yarn
yarn install v1.22.11
[1/4] Resolving packages...
[2/4] Fetching packages...
info fsevents@2.3.2: The platform "win32" is incompatible with this module.
info "fsevents@2.3.2" is an optional dependency and failed compatibility check. Excluding it from installation.
info fsevents@2.1.3: The platform "win32" is incompatible with this module.
info "fsevents@2.1.3" is an optional dependency and failed compatibility check. Excluding it from installation.
info fsevents@1.2.13: The platform "win32" is incompatible with this module.
info "fsevents@1.2.13" is an optional dependency and failed compatibility check. Excluding it from installation.
[3/4] Linking dependencies...
warning "depcheck > @vue/compiler-sfc@3.1.1" has unmet peer dependency "vue@3.1.1".
warning " > eslint-plugin-lodash@7.1.0" has unmet peer dependency "lodash@>=4".
warning " > typedoc-plugin-external-module-name@4.0.3" has incorrect peer dependency "typedoc@>=0.7.0 <0.18.0".
warning " > botbuilder-test-utils@0.0.0" has unmet peer dependency "jsonwebtoken@*".
warning " > botbuilder-test-utils@0.0.0" has unmet peer dependency "nock@*".
[4/4] Building fresh packages...
success Saved lockfile.
Done in 4.41s.

D:\Projects\botbuilder-js>yarn build
yarn run v1.22.11
$ wsrun -e -m -t build
botbuilder-repo-utils has no build script, skipping missing
generateclient-via-swagger has no build script, skipping missing
browser-functional-tests has no build script, skipping missing
functional-tests has no build script, skipping missing
adaptive-expressions
$ npm-run-all build:src build:tests
botbuilder-dialogs-adaptive-runtime-core
$ tsc -b
botbuilder-stdlib
$ tsc -b
botbuilder-test-utils
$ tsc -b
botframework-config
$ tsc -b
botframework-schema
$ tsc -b
botframework-streaming
$ npm-run-all -p build:lib build:es5
botbuilder-dialogs-adaptive-runtime-core
$ downlevel-dts lib _ts3.4/lib --checksum
botbuilder-stdlib
$ downlevel-dts lib _ts3.4/lib
botframework-schema
$ downlevel-dts lib _ts3.4/lib --checksum
botframework-config
$ downlevel-dts lib _ts3.4/lib --checksum
adaptive-expressions
$ tsc -b
botframework-streaming
$ tsc -p tsconfig.json
$ npm-run-all build:es5:tsc build:es5:browserify
$ tsc -p tsconfig-es5.json
adaptive-expressions
$ tsc -p tests/tsconfig.json
botframework-connector
$ tsc -b
$ npm-run-all -p build:browserify build:downlevel-dts
$ npm-run-all build:browserify:clean build:browserify:init build:browserify:run
$ downlevel-dts lib _ts3.4/lib --checksum
$ rimraf lib/browser.*
$ shx cp lib/index.js lib/browser.js
$ browserify -s BFC --debug lib/browser.js | exorcist lib/browser.js.map | sponge lib/browser.js
adaptive-expressions
$ rimraf lib/browser.* && shx cp lib/index.js lib/browser.js && yarn browserify
$ browserify lib/browser.js -s AEL --debug -p [ tinyify --no-flat ]  | exorcist lib/browser.js.map | sponge lib/browser.js
botframework-streaming
$ browserify -s BFSE --debug es5/index-browser.js | exorcist lib/index-browser.js.map > lib/index-browser.js
$ downlevel-dts lib _ts3.4/lib --checksum
azure has no build script, skipping missing
botbuilder-core
$ tsc -b
$ downlevel-dts lib _ts3.4/lib --checksum
botbuilder-azure-blobs
$ tsc -b
botbuilder-dialogs
$ tsc -b
botbuilder-applicationinsights
$ tsc -b
botbuilder
$ tsc -b
botbuilder-azure-blobs
$ downlevel-dts lib _ts3.4/lib --checksum
botbuilder
$ downlevel-dts lib _ts3.4/lib --checksum
botbuilder-dialogs
$ downlevel-dts lib _ts3.4/lib --checksum
bot-integration-tests
$ tsc -b
botbuilder-azure
$ tsc -b
botbuilder-azure-queues
$ tsc -b
botbuilder-testing
$ tsc -b
botbuilder-azure
$ downlevel-dts lib _ts3.4/lib --checksum
botbuilder-azure-queues
$ downlevel-dts lib _ts3.4/lib
botbuilder-applicationinsights
$ downlevel-dts lib _ts3.4/lib --checksum
botbuilder-testing
$ downlevel-dts lib _ts3.4/lib --checksum
adaptive-expressions-ie11
$ webpack
botbuilder-dialogs-declarative
$ tsc -b
botbuilder-lg
$ tsc -b
botbuilder-dialogs-declarative
$ downlevel-dts lib _ts3.4/lib --checksum
botbuilder-ai
$ tsc -b
botbuilder-dialogs-adaptive
$ npm-run-all build:src build:tests
$ tsc -b
botbuilder-ai
$ downlevel-dts lib _ts3.4/lib --checksum
botbuilder-dialogs-adaptive
$ tsc -p tests/tsconfig.json
testbot has no build script, skipping missing
transcript-tests has no build script, skipping missing
botbuilder-ai-luis
$ tsc -b
botbuilder-ai-qna
$ tsc -b
botbuilder-dialogs-adaptive
$ downlevel-dts lib _ts3.4/lib --checksum
botbuilder-ai-luis
$ downlevel-dts lib _ts3.4/lib --checksum
botbuilder-ai-qna
$ downlevel-dts lib _ts3.4/lib --checksum
adaptive-expressions-ie11
 | Hash: 8aabfb2d7fa45ba021e3
 | Version: webpack 4.44.2
 | Time: 15000ms
 | Built at: 09/08/2021 5:15:41 PM
 |          Asset       Size  Chunks             Chunk Names
 |     index.d.ts   74 bytes          [emitted]
 | index.d.ts.map  125 bytes          [emitted]
 |       index.js   1.03 MiB       0  [emitted]  main
 | Entrypoint main = index.js
 | [0] ./src/index.ts 300 bytes {0} [built]
 | [1] ../adaptive-expressions/lib/browser.js 1.69 MiB {0} [built]
 | [2] (webpack)/buildin/global.js 472 bytes {0} [built]
 |     + 280 hidden modules
botbuilder-dialogs-adaptive-testing
$ tsc -b
botbuilder-ai-orchestrator
$ tsc -b
botbuilder-dialogs-adaptive-runtime
$ tsc -b
botbuilder-ai-orchestrator
$ downlevel-dts lib _ts3.4/lib --checksum
botbuilder-dialogs-adaptive-testing
$ downlevel-dts lib _ts3.4/lib --checksum
botbuilder-dialogs-adaptive-runtime
$ downlevel-dts lib _ts3.4/lib --checksum
botbuilder-dialogs-adaptive-runtime-integration-azure-functions
$ tsc -b
botbuilder-dialogs-adaptive-runtime-integration-express
$ tsc -b
botbuilder-dialogs-adaptive-runtime-integration-restify
$ tsc -b
botbuilder-dialogs-adaptive-runtime-integration-azure-functions
$ downlevel-dts lib _ts3.4/lib --checksum
botbuilder-dialogs-adaptive-runtime-integration-restify
$ downlevel-dts lib _ts3.4/lib --checksum
botbuilder-dialogs-adaptive-runtime-integration-express
$ downlevel-dts lib _ts3.4/lib --checksum
consumer-test has no build script, skipping missing

You can see a build passing after yarn command. This is when I added types.

@joshgummersall
Copy link
Contributor

Huh - well, I suppose you can go ahead and add the types back. You'll have to add @types/xmldom to the depcheck ignore list in adaptive-expressions (see example here).

.depcheckrc Outdated Show resolved Hide resolved
@vasicvuk
Copy link
Contributor Author

vasicvuk commented Sep 9, 2021

I have added it again and included it in .depcheckrc file since --ignores flag seems to be ignored if .depcheckrc file provided. Also I have run depcheck to verify it and run build on clean repository and everything passes.

@vasicvuk
Copy link
Contributor Author

vasicvuk commented Sep 9, 2021

Fixed it now, had to add also sinon as dependency as it is used within tests. I had this error:

npm run depcheck

> adaptive-expressions@4.1.6 depcheck D:\Projects\botbuilder-js\libraries\adaptive-expressions
> depcheck --config ../../.depcheckrc --ignores @types/xmldom

Missing dependencies
* sinon: .\tests\expressionParser.test.js

@joshgummersall
Copy link
Contributor

Please remove the added sinon dependency and add --ignores sinon,@types/xmldom for adaptive-expressions. It's inherited from the root package.json file in the repo. depcheck isn't perfect and doesn't understand hoisting very well. Thanks!

@vasicvuk
Copy link
Contributor Author

vasicvuk commented Sep 9, 2021

@joshgummersall done

@joshgummersall
Copy link
Contributor

@cosmicshuai can you take a look at this as well? Just want a second set of eyes and you've worked on adaptive-expressions a lot in the past.

@vasicvuk
Copy link
Contributor Author

@joshgummersall New version of x2js is also released. So I can also now provide a PR with less changes, but I think it is a better option to change it to fast-xml-parser as it has more Contributors, Stars on GitHub, Downloads, it is used in other Microsoft OSS and other large companies too.

@joshgummersall
Copy link
Contributor

Agreed, I was hoping to get another set of eyes on the change but I’m inclined to move forward either way. I’ll land this today either way.

@joshgummersall joshgummersall merged commit b1d137f into microsoft:main Sep 13, 2021
@mogadanez
Copy link

fast-xml-parser have issue now too
https://security.snyk.io/vuln/SNYK-JS-FASTXMLPARSER-3325616

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

CVE-2021-32796 Package: xmldom
4 participants