expectation tags #128
expectation tags #128
Conversation
* Created main.py and am doing a test commit * test commit * test commit * Parses a markdown file and pulls out headings, code blocks, and paragraphs * took away a few magic values & cleaned up code a bit * added the initial executor code complete with repl behavior, by no means complete and it will have bugs but it would be good to get some eyes on it before I go too much further * added a few more comments and deleted a few unused functions * Added testing directions of using python3 main.py to test * Added automated testing functionality with fuzzy matching * cleaned up code a bit and added test functionality * testing * Added github actions for automated testing * Fixing github actions * bug fix for actions * Added push to main as a trigger * bug fix * testing * changed file type to yml * testing * fixed bug with fuzzywuzzy * changing sensitivity so that actions should fail as there is a warning * Changing Sensitivity and adjust actions test * testing * test should pass now * Trying to enable manual workflow trigger * testing github actions * Changing default test * added the ability to set command line variables from both comments and .ini files which are key value pairs * updated README.md and added new test scripts * fixed test file and changed github action to point to test file not readme * Hardedned program to deal with broken markdown files, there is still no good way to deal with interactive bash commands though * added some exception handling so the program doesn't crash on a timeout * Changed Parser file class to MarkdownParser to avoid name clash with std parser, added test scripts folder for testing, and added some additional error handling for commands * removing exit(1) from test section as it terminates early if multiple files are being tested * removed exit(1) from main when file does not parse correctly * removed exit 1 to stop early failures with multiple files * adding random string to end of resource group in testing section so that there are not name conflicts. Catch is the document itself has to use MY_RESOURCE_GROUP_NAME as the variable for testing * cleaned up file structure and fixed a few bugs in Readme.md * added verion locks to requirements.txt, deleted hardcoded vm create document, added .vscode to .gitignore, and modified test script to work based on new location
* Improve getting started docs 1. fix capitlization error 2. use virtual environment 3. separate installation from running * Switch to interactive mode Co-authored-by: jasonmesser7 <91570951+jasonmesser7@users.noreply.github.com> --------- Co-authored-by: jasonmesser7 <91570951+jasonmesser7@users.noreply.github.com>
* first draft of AKS quick start with event grid notfications * Added tutorial with AKS create + Subscribing to event grid notifications * Add tests for all steps, plus a few docs improvements. --------- Co-authored-by: Ross Gardler <nope@no.no>
…lti tenancy, however we need this signal for an action to fail and block a PR (#15)
…from the markdown codeblocks.
…outputs from the top level.
…d INI initialization to ie execute.
This PR is a port & expansion of the innovation engine to support our desired use cases. Changes included are: * Rewrite innovation engine from python -> golang. * Better output formatting for executing scenarios. * Makefile for centralizing build, test, and deployment steps. * `ie to-bash` to convert markdown files into bash scripts via their codeblocks. * Expected similarity checking for code blocks with JSON & plaintext outputs. * New scenarios & an automated testing pipeline for them. * Adding or Overriding variable exports in executable documents using `--var` on either `ie execute` or .`ie to-bash`.
* [remove] unused login. * [update] implementation for switching directories to be abstracted into a fs function. * [update] error handling for UsingDirectory so the original directory always tries to be restored. Remove unused variables. * [refactor] cursor manipulations out of engine and add unit tests. * [add] curl to AKS scenario back into steps. * [add] command rendering for scenarios. * [update] names of errors and functions. * [refactor] the engine. * [format] all files. * [refactor] the azure environment code to maintain support for the legacy environment flag. * [update] formatting & echo the web server url at end of aks scenario.
* [update] sed statement to use single quotations to avoid expansion during variable rendering and format document. * [update] command rendering failures to be reported. * [fix] resource URIs not being found.
… running in cloud shell. (#101)
* Fixed errors on AKS doc * Fixed errors on AKS doc * Fixed errors on AKS doc * Fixed scenario bugs and cleaned text * Reverted dns name change * Fixed styling issues * Reverting moving around variable declarations and small text changes --------- Co-authored-by: Mitchell Bifeld <mbifeld@Mitchell-Surface>
Fix multi-line command rendering and disable inline rendering
Bumps [golang.org/x/net](https://github.com/golang/net) from 0.8.0 to 0.17.0. - [Commits](golang/net@v0.8.0...v0.17.0) --- updated-dependencies: - dependency-name: golang.org/x/net dependency-type: indirect ... Signed-off-by: dependabot[bot] <support@github.com>
…net-0.17.0 Bump golang.org/x/net from 0.8.0 to 0.17.0
* [fix] remove horizontal align * [update] command output to not be tabbed
| echo "foo" | ||
| ``` | ||
|
|
||
| <!--expectation type="similarity" value="0.7"--> |
There was a problem hiding this comment.
Will we make implementation backwards compatible with the current format? For clarity, I'm not saying this is required, we do not have so many docs that updating them is problematic. However, we will want to fail when encountering an old format if not backward compatible.
There was a problem hiding this comment.
I initially wanted to, but I think we should just deprecate the old syntax entirely to avoid confusion in the future. It should be easy to update the documents to the new syntax and we could easily add errors when an encountering the older syntax.
| ```` | ||
|
|
||
| Instead of computing a similarity score, IE converts the `value` provided by | ||
| the author into a regular expression that is matched against the |
There was a problem hiding this comment.
Could we also allow the `value1 to be an RE? That is, if it is an RE (perhaps indicated by the presence of ".*" or "^" or "$") we do not auto-convert, if it is a string then we autoconvert.
Alternatively, we could add a third type of "regex".
There was a problem hiding this comment.
Yup, I believe the current implementation of this already supports any valid golang regex expression as it's input.
|
|
||
| ``` | ||
|
|
||
| <!--expectation type="matches" value="foo" --> |
There was a problem hiding this comment.
How useful is an exact match? Is that the intent here? I would have though "contains" would be more useful.
There was a problem hiding this comment.
matches isn't for exact matching, but it is used for matching patterns in the codeblock (Which just so happens to be an exact match here). However, I can see the confusion that calling it matches may create so I will revert back to just naming it contains
There was a problem hiding this comment.
Hi!
I like the idea of having two different tags to each match similarity score and regex.
Regarding having either matches or contains, I would rather see contains to only test if output contains the given string while matches would handle regex.
Nonetheless, note that you can test string is contained with regex too without using regex lingua :p.
Best regards.
No description provided.