-
-
Notifications
You must be signed in to change notification settings - Fork 1.2k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
test: Check return code, stdout, stderr of (almost) every example #2447
base: develop
Are you sure you want to change the base?
Conversation
…mple Runs each of the files ending in '.js' in the examples directory or a subdirectory thereof via `node`. Verifies the return code is normal for each one. Compares stdout and stderr to reference copies.
d9c1937
to
3dff2de
Compare
Happy to update this to latest mathjs develop, but need guidance on the many questions above. |
I'm not sure how I missed this PR before, sorry for not looking into it before.
|
Sorry about the delay in getting back on this, I've been focused on Pocomath and/or the TypeScript experiments.
Thanks for further guidance. |
|
OK, I think we are clear on all of the points. I will try to move this back toward being merge-ready. |
Runs each of the files ending in '.js' in the examples directory or
a subdirectory thereof via
node
. Verifies the return code is normalfor each one. Compares stdout and stderr to reference copies.
As requested in #2264 (hopefully this can be a first step in clearing that and/or #2279 out of the PR queue).
Some caveats that should be considered in review of this PR:
examples/advanced/custom_argument_parsing.js
in the correct way (it's quite unclear to me how one is supposed to create a subscope in user code (as opposed to internally within mathjs, where it is easy to import utils/scope.js).) Do I even need to make a copy of the scope I get in the raw function, as the document says it's already a shallow copy of the ambient scope? (i.e. can I set the specified variable in scope and dispense with the creation of fnScope altogether? I verified that it produced the same values if I did it that way, but I wasn't sure it was truly "safe" to the point where it's OK for the example to show doing things that way.)node examples/advanced/custom_loading.js
didn't actually run at all, since by defaultnode
doesn't support import. So I renamed this file tocustom_loading.mjs
(as it seems to be a module anyway) and then it's ignored by the test harness I have added. Problem solved ;-) Seriously, if there's a way to add a custom test for this one in examples.test.js, please guide me. Thanks.examples/import.js
. I assumed that on the automated test machine, numbers and numeric would not be installed. So I used as the reference the output when these are not installed. That's all fine, but it means importing these external packages is not tested. If it should be, what should I do? I didn't think that having a test install a package would be a good idea, as it will then be too easy for these packages to end up checked in as dependencies of mathjs, which they are not. So should I just leave this be or should we try to engineer a specific test for importing external packages into mathjs?examples/advanced/web_server
but thats a sub-sub-directory and so ignored by my harness. Again problem solved ;-) and again if there is a suggestion on a custom test for this stuff, please let me know.examples/browser
. There are no.js
files so the harness doesn't care. If there is a reasonable way to test them, let me know and I will add another harness just for them.Anyhow, these comprise a really solid putting of mathjs through its paces, so hopefully what's there in this PR will be helpful in the future. (I am thinking also of writing a test in a separate PR which checks the output of every example in every embedded-docs help against a reference copy as well, as that should be pretty complete, too. There's a fair amount of overlap there with the unit tests, but also a fair amount of unique examples, and so it seems like we may as well use them.)