All notable changes to this project will be documented in this file.
The format is based on Keep a Changelog and this project adheres to Semantic Versioning.
- Fixes download Step unable to use a partial filename match. The Step needed the whole name, including the extension. See #725.
- Moves "expected" status error to only be visible to internal test errors. Closes #933.
- Detects failed sign in. Closes #918.
- Improved checks of log codes - more robust and flexible. See #920.
- Returns report text from
.addToReport()so it can be used again for error messages. - Update puppeteer to v22.15.0. Closes #930.
- Update npm vulnerabilities except pdfjs-dist. See docs/decisions/pdfjs_version_2024_08_08.md.
- Adds decision docs
- Updated CONTRIBUTING.md
- Added example.env, closes #374
- Updates pdfjs-dist for node v20 and v22. See #952.
- Authors can now make sure that ALKiln deletes the interviews it creates for an ALKiln testing account. They can add a new (optional) value, an API key, to the "sign in" Step and use the "start interview" Step to go to the interview. The API key must belong to the user who signs in. Example:
Given I sign in with the email "<username env var>", the password "<password env var>", and the API key "<user API key>. Admins can create an API key for a regular user with https://docassemble.org/docs/api.html#api_user_user_id_api_post. We will add this to the documentation for the "sign in" Step.
- Update links in logs to new Suffolk docs pages
- Now ALKiln deletes interviews upon interview completion for signed in users as long as the user has an API key. Also, the author must have used the "start interview" Step to get to the interview. Admins can give an API key to a regular user with https://docassemble.org/docs/api.html#api_user_user_id_api_post. See #831.
- Tests deleting interviews upon interview completion for signed in users.
- Switches to mostly using a lowest-access-level user account for actually running interviews.
- Switches to adding our own
users[0]question because of changes in Assembly Line. - Adds @temp_error to neutralize tests that are getting undesired errors right now, but which we anticipate being fixed soon. We want to ignore them for a short while. For example, an upstream change temporarily breaking some tests.
- Excludes docassemble/ALKiln files from log code checks.
- New file with just failure results, including cucumber results when possible. It accumulates Scenario info as the tests run. Even if a test run stops early, there will still be some results.
- New output from ALKiln action -
PATH_TO_ARTIFACTS. The value is""or the path to the folder that contains the output of the test run, if the test run actually happened. Sometimes this folder doesn't get created. For example, if there was a typo in one or more.featurefile. For example,"alkiln-2024-06-03 at 02h29m44sUTC". - New outputs from the ALKiln action -
PATH_TO_REPORT_FILE,PATH_TO_UNEXPECTED_RESULTS_FILE, andPATH_TO_DEBUG_LOG_FILE. For example,"alkiln-2024-06-03 at 02h29m44sUTC/report.txt". Authors can choose to use these as they wish. For example, they can put the contents of these files in issues they create from their workflow. Closes #701. - Added cucumber output to report.txt.
- Added more explanation for the logs that appear when test runs error in a way that prevents debug_log.txt and such files from being created. We will watch for feedback about the error data itself. It may still be too confusing.
- report.txt accumulates over time instead of being created all at once at the end. Even if a test run stops early, there will still be some results.
- Tag expression not being parsed correctly. Surrounded
ALKILN_TAG_EXPRESSIONwith".
- Download ALKiln's artifacts back into the running job.
- Removed our own workflow's pause between triggers.
- Improved our use of ALKIN_TAG_EXPRESSION env var.
- Exclude tests that we are aware fail. We will test those manually.
- Added new debug log/code to reflect difference between a expected status mismatch and a cucumber status mismatch.
- Changed text of error to make it more clear.
- Changed logic of error finding to use expected status for users as well as internal tests. That needs discussion.
- Refactored code in that section.
- Wait till the end to throw some errors so the user can get as much info as possible about the failures.
- Exclude another file from code checking (cucumber-report.txt)
- Typo in run_cucumber.js
- Add logs for "success" and also non-coded logs for logs that are simply cosmetic - adding a new log to space messages out or show separators.
- Error messages now contain error codes. See #730.
- Removed extra warning from sources path checking. We currently check a bunch of paths to be flexible and forgiving, so almost every test has been get this warning and it can mislead users. Users do still get an error message if we found no paths at all.
- Added: Added script to validate log codes (error, warning, info, success, debug logs). Check for duplicate codes, with some exceptions for actions. Check for missing codes - removed codes are stored in another file, allowing this test to work properly.
- Added: The above script also reports what the highest log code is (if tests pass), letting the developer see what log code they can add next when they need to add a new log code.
- Added: Added code validation tests to package.json
testscript. - Change: Added and edited documentation comments to the workflow files
- Change: Avoid running double tests when someone makes a pull request. See #869.
- Change: Made sure tests for testing failure all had the same failure tags. Same for warnings.
- Change: Added "Fail" to the start of the Scenario descriptions of each test that tests failure.
- Change: Removed "example" code from bottom of ./action.yml page since we have our own file as an example now.
- Change: Switch a11y tests to only run manually since docassemble changes sometimes cause our tests to fail (which isn't actually relevant to our tests). This may be a temporary fix
- Fix: Stopped using ALKiln version "url".
- Fix: Update dependency action versions. See #844 and #871.
- Fixes text fields did not get cleared before entering new text. Created problems with text fields that had pre-existing answers, like default answers.
- Fixes undefined var name when setting long text or ajax combobox value
- Removes test for text on error pages. That keeps changing every now and then. It would be nice to test that we're catching some text, which we can't do right now. Maybe we should be broader with where we get the text since authors can also change the DOM. At least there are screenshots when possible.
- Fixed dates only being part-way filled.
- Fixes the issue that typing is too fast for some fields, like Ajax comboboxes. See #857.
- New Step: 'I go to ""' to go to any arbitrary url, not just arbitrary interview urls.
- New navigation report messages.
- Allow non-local environments to get debug logs. For example, the GitHub console during a workflow run.
- Avoid waiting for "success" on sign in. Not everyone has it.
- Detect signature page variable names with the trigger variable name as a fallback to the
_save_asinput. If a server's config has itsrestrict input variablesvalue set toTrue, the #trigger value is the only way for ALKiln to know which variable the signature is setting. Add an error to the report if ALKiln is unable to find either of those.
- Handle non-existence of
elemmore robustly when checking forelem.disabled - Add a pause between pull request tests and push tests when the PR is coming from our own repo
- Make getting page field values more robust by doing more to ensure that undefined option
values don't cause errors. For example, ajax comboboxselectoptions don't always havevalues
- New feature: 2-column Story Tables that work without the trigger variable. Now column 1, "var", should only contain the actual variable name with no proxies (x, i, j, etc). For example, instead of
x[i].name.firstthe var column should now containusers[0].name.first. Column 2 is the same as it was before - the answer for the field. See #582. - Added console logs enhancing the "progress dots". You don't usually see those till after the tests run. Now more of them should show up during the test run and the name of the Scenario being tested should appear there as well. Unfortunately, the first Scenario behaves differently. We're consulting with the creators of cucumber to understand why that's happening.
- Internal tests: Fixed Scenarios that had failed but had correct reports were instead passing.
- Internal tests: Allow more types of "expected status" for when we test dealing with "undefined", "skipped", etc.
- Internal tests: Changed name suffix values in tests. Assembly Line changed its punctuation a while ago.
- Added doc explaining what internal test results we expect to see as people new to the project are surprised by all our red Fs and yet passing tests.
- Changed our workflow's
on: pull_request_targettoon: pull_request. When the former runs tests,checkoutchecks out the target branch's files. It was unhelpful to test the files of the old branch with the new version of ALKiln. The latter checks out the head branch's files. We should be good now.
- Restored previous behavior - Projects are now deleted even when a workflow fails. See #831.
- Fixed typo of comment explaining how to reference the isolated GitHub server action.
- Harden security for action input handling. That said, the source of the problem would come from author workflows, so there's a lot more authors can do to help this situation than we can. Workflows triggered by pull requests are a specific place to watch. GitHub already has some default safeguards to prevent strangers, specifically first-time contributors, from triggering workflows like that. They can also set their org or repo to be more strict about pull requests from outside collaborators.
- Freeze
docassemblecliversion at 0.0.17. See #827.
- Step to tap an element without navigating. See #834.
- Use puppeteer `.click()`` again
- Re-activate looking for terms on the page.
- Allow tests to run when a PR is made by switching to docassemblecli's
dainstallto upload packagese to the Playground. See #760.
- Workflows:
- Split unit tests into their own file. They have to use
npm installin our repo and we don't want to use that for Playground tests. It avoids trying to push all the node modules to the Playground which would otherwise cause error 413 "too large". - Use our own action in the tests that install on the Playground, to avoid duplicating edits to the action - see #822.
- Rename files
- Also see #822
- Split unit tests into their own file. They have to use
- For isolated GitHub server, only show GitHub action console logs when
SHOW_DOCKER_LOGSinput is set totrue. Same for creating the docker logs GitHub artifact. See #819.
- Added a github action that allows authors to run tests on an isolated server that ALKiln creates on GitHub. They will need to create a new GitHub workflow to run the action. This reduces flakiness that authors' servers can create during server reload caused by updating their config, pulling in a package with a module, and other such things. As with all tests, especially with end to end test, there will always be some flakiness.
- Continuing development of random input test: Deal with non-existent elements, add a default input
typeof 'text', only press 'Back' 10% of the time instead of potentially 50%, navigate back to the interview from an external link (that goes outside the server) if necessary. Also add more debug logs to help diagnose problems. #633
- Downloads are now faster! You won't have to wait for the full duration of a step (30 seconds by default) to download a file anymore. See this PR for more details.
- BREAKS ALKilnInThePlayground v1.2.0 and below (as we discussed) as that project is still exploratory. Can now take the location of the "sources" folder or folders as an argument in the command line. The formats are
npm run cucumber @tags -- --sources=./fooandalkiln-run @tags --sources=./foo.
- New step to compare PDFs!
- An example:
I expect the baseline PDF "something-in-sources.pdf" and the new PDF "downloaded.pdf" to be the same - See #774 for more details.
- An example:
- Screenshots now come with an HTML file of the same page. Pages can be examined for their full errors and for accessibility details. Note that styles are weird because we don't control the page with js like docassemble does. Otherwise hidden elements are visible, etc. See #763.
- Fixed and improved error for accessibility failures. See #744.
- Refatored report functions into their own file. See #770.
- Changed the format of the aXe file name to shorten it by avoiding repeating the scenario name.
- New warning message to the developer when their date string doesn't have a
/in it.
- Told the user the url of the interview ALKiln tried to load. See #696.
- Restored Assembly Line custom datatype three-parts dates functionality. See #764.
- Fixed "today" not being converted into a date for custom datatypes.
- Make
I tap to continuecase insensitive to allow the author to match the capitalization of the button in Docassemble.
- Comboboxes - can now fill in comboboxes on the latest Docassemble.
- Filling out a hidden field right after a date field triggered the date's calendar popup, which I believe ate the input for the next visible field.
- Accept
SERVER_RELOAD_TIMEOUT_SECONDSas an env var input in GitHub.
- Increment version of ALKiln that action.yml is using.
- added additional functionality to the sign method to allow developers to take a name argument to sign on canvas: issue #596
- A new script:
alkiln-run, which acts likenpm run cucumber, but can be run in any directory, not just in an npm package. - Additional environment variables and their validation to allow for tests that run on a developer's server/Playground instead of through GitHub. Also, other functionality for that purpose. Issue #661
- Tests for new session_vars behavior and improve previous tests.
- Adds
npm-shrinkwrap.json, so installs from npm will have fixed version dependencies. - Allow author to specify loops with only
.target_number. e.g. to leave out.there_are_any. See issue #706 and Story Table documentation in docs folder.
- BREAKING: the github action no longer runs
npm run XYZ; it directly calls scripts, e.g.alkiln-setup,alkiln-run,alkiln-takedown - BREAKING: Remove EXTRA_LANGUAGES env var as languages are being handled through cucumber
Examplesnow. - upgraded cucumber v8.6.0
- using cucumber's JS API to run tests. For more details on how it works, see the cucumber-js docs.
- don't print the "publish this cucumber report" message
- Adjusted validation of some environment variables to account for Playground vs. GitHub or local test runs. Issue #661
- Docassemble Project name prefix now includes ALKiln in it for clarity
- Change interface for testing languages. See #713.
- Freeze all npm package versions.
- DOCKER UPDATE (for ALKilnInThePlayground): Allows ALKilnInThePlayground to upload files. See SuffolkLITLab/docassemble-ALKilnInThePlayground#18. This may require some developers to update their system (updating their docker image) if they are below docassemble-os 1.0.8. You can tell if you are below docassemble-os 1.0.8 if you run
docker image inspect jhpyle/docassemble | grep Createdon your host server (Lightsail, etc.) and the created date is before 2023-04-09. You can update as described in the documentation. Updating to a new docker images is something that all docassemble users will have to do sometime anyway, sometimes for security. - Conform upload file behavior to that of other Story Table rows - avoid erroring when field isn't set properly unless using all variables is required.
- Projects created in da each have a unique name. #663
- Shorten path names to try to accommodate limitations of windows systems while still keeping enough useful information to help devs identify the test outputs. #618
- Updated field decoding to handle new object field encoding. See #711
- Allow multiple languages to be tested again. See #713.
- Fill in time fields correctly. See #726.
- Allow
.target_numberto be 0. See #706. - Use the right number of loops for
.target_number. See #706.
- Pass docassemble API keys through HTTP headers instead of as parameters.
- Parameters to certain HTTP requests are printed directly in docassemble's uWSGI log, leaking API keys to actors with log access on your docassemble server
- Update dependencies. See #727.
- added a test for multiline json values (only ones set by code not by user input)
- added an internal section to the change log
- Get error data from server errors
- Shorten Axios errors to make them more readable (#632)
- Contribution docs
- #511, couldn't take screenshots of signature pages #511
- Internal - ignore local test output files. Part 2 of the process out of 2.
- Internal - deleted unignored local files (since adding .npmignore). Part 1 of the process out of 2.
- Fix artifacts not being saved in GitHub. See #629.
- Make internal test folder names a bit simpler and more modular-izable.
- Don't mark
...there_in_another | False |in story tables as invalid, as it's necessary for some workflows (see #580 for a longer discussion).- Explicitly not documented, as we don't want to encourage people to use it if it's not necessary for their interviews.
- continuing between screens will no longer press button with the
btn-primaryclass. This means that it won't press "Exit", or "Restart" buttons, to avoid getting in an infinite loop. - Shorten test report and artifact filenames, attempting to have fewer files whose whole paths are longer than 260 characters. (See #626 for details).
- Corrects the month in the artifact folder timestamp; was printing things like
81for September instead of09, because of a+being interpreted as Javascript string concatenation and not math.
retryof failed tests not implemented for developers. See #601.
- Environment variables
SERVER_RELOAD_TIMEOUT_SECONDSandMAX_SECONDS_FOR_SERVER_RELOADwere being used as milliseconds instead of seconds. See #606.
- runtime_config.json gets created when needed when artifacts folder is created. See #600.
- Allow devs to test an arbitrary interview. See #600.
- Create artifacts folder even when setup has been skipped. See #600.
- Date for filenames. Format date in y-m-d format instead of d-m-y that it had by default. Include ms.
- Inform the developer when a test failure may have been caused by a server reload. See #392.
- Retry all failing tests once.
- Ensure unique folder names for random input screenshots in case developer uses multiple random steps in one Scenario.
- Ensure random input detects unexpected interview errors.
- Fix failure when signature page is the first screen.
- Internal tests failing even when failure status is as expected.
- Double printing in report of scenario failure, one of them even when Scenario was possibly non-failure, like skipped.
- Generate random input to fill out a simple form automatically. Does not yet ensure that random screenshot folder names are unique, or that error screens are detected as errors. 4.7.1 implements those.
- Add page id to the "Missing Variable or variables on page" error report
- Allows Story Tables to not include a header row, as long as they have 2 or 3 columns. This isn't the suggested way at the moment, since including headers is more readable, but can prevent confusing and unnecessary errors.
- Put cucumber tests' artifacts in one folder. See #552.
- Add separate artifacts dir for our own internal tests, like unit tests
- Abstract names of some of those artifact directories
- Single dropdown fields on pages are now recognized as fields properly
- Steps to test web accessibility either on individual page or for all subsequent pages.
- (internal) Expected in report is reset properly
- if a test used "the Scenario report should include ...", then all tests that were run after that would pass even if the test itself failed but wasn't expected to fail
- Corrected some schema errors in the
action.yml(therunkey was specified twice, should only have been once, andrequired: Falseis needed for optional inputs)
- Steps that tap tabs (made using the
tabbed_templates_htmlfunction from ALToolbox) and other arbitrary elements on the page.
- Appends the results of the cucumber summary formatter to the
debug_log.txt, which includes useful stack traces into the Kiln code when tests fail.
- Prevent login info from being saved in the report or screenshots being taken on error. #599.
- Prevent error screenshots of screens that used a secret.
- GitHub environment variable
MAX_SECONDS_FOR_SETUPto set a custom maximum time for setup and takedown for packages that take a long time to load.
- Pin versions of dependencies in action.yml and package.json
- Change log.txt file to debug_log.txt and upload as github artifact.
- Update cheerio version to remove vulnerabilities.
- Step to set vars to secret variables. That is, using environment variables while hiding the names of those variables in the report, errors, and console logs.
- BREAKING: Add package.json creation/overwriting to action.yml. Simplify package.json.
- BREAKING: Add action.yml that runs most of what users' workflows run now, along with notes for a new user workflow that will take less maintainance. See #420. We need to add documentation on how to write a workflow file as it currently is if they want to take back control. Setup interview has not yet been updated with this workflow.
- Step:
the text in the JSON variable "variable_with_text_value" should be. Compare JSON variable with a text value to given text. See #470. Does not accept nested values, e.g. "child.name.first". Downloads all the JSON vars to a .json file in the "downloads" artifacts. - Step to log the page's JSON variables and values in the report. Future goal: save to file. See #454.
- Step to log into the developer's docassemble server account using GitHub SECRETs. See #499.
- Create log.txt for report items and uploaded as github artifact so there's always some kind of output.
- Add control of alkiln npm version to the action.yml
- BREAKING: NPM package is now suffolklitlab scoped package. Refer to @suffolklitlab/alkiln from now on.
- BREAKING: Update to v13 of puppeteer
- BREAKING: Update to v7 of cucumber
- BREAKING: Update action.yml node to v17
- BREAKING: Use alkiln v4 package.json we create during the test run in
action.yml(as discussed in #489) - BREAKING: Use API key to access da server, create projects, pull code, delete projects, and check for server restart.
- BREAKING: Throw an error for uploading nonexistent file, changed to error instead of warning in reports.
- Update test setup interview to v4 of ALKiln.
- Remove and ignore package-lock.json so that our tests will behave more like our users' tests
- Add warning in steps.js for name input with too many parts. Add test in reports.
- fix typo in the report
- Allow a developer to wait as a first Step. See #387.
- Peer dependencies and dev dependencies. Now
cucumberis just a dependency. See #396 for discussion. Setup interview has not been updated to remove dependencies.
- Proxy var story table row matches for any page containing the given variable name when there's only one row for the relevant variable. #464.
- Fixed interview name not allowing url parameters. #449
- Freeze colors npm package to before bug in both package.json and action.yml
- Fix invalid project name allowed.
- fix typo in the report.
- session_vars branch path regex not removing start of branch path
- Typo in report that was messing up copy/paste of test table
- Add Step to log into server account using GitHub SECRETs (environment variables). See #499.
- Handle an Assembly Line name Step value that has too many names, including a warning for the developer. Include the first three names and the last name to try to catch suffixes.
- Step:
the text in the JSON variable "variable_with_text_value" should be. Compare JSON variable with a text value to given text. See #470. Does not accept nested values, e.g. "child.name.first". Downloads all the JSON vars to a .json file in the "downloads" artifacts.
- Step to log the page's JSON variables and values in the report. Future goal: save to file. See #454.
- Install the security fix in the action's package.json as well. Because dependents have to require cucumber itself, if they're not using our action (if they have their own package.json), they're going to have to implement this fix in their repo too. That's why our fix isn't working. We'll start the process of helping developers update. We also need to deprecate all previous versions of ALKiln. See #489.
- Fully publish the security fix in v3.0.7
- Pinned colors library to 1.4.0. See https://www.bleepingcomputer.com/news/security/dev-corrupts-npm-libs-colors-and-faker-breaking-thousands-of-apps/. We don't use faker, so we don't need to handle that.
- See #371
- Scenario heading tags are now on a second line
- Artifact filenames use dashes more and are created separately from Scenario headings
- Scenario ids are separated from filenames and Scenario headings
- Add action.yml that runs most of what users' workflows run now, along with notes for a new user workflow that will take less maintainance. See #420. We need to add documentation on how to write a workflow file as it currently is if they want to take back control. Setup interview has not yet been updated with this workflow.
- Add package.json creation/overwriting to action.yml. Simplify package.json.
- Peer dependencies and dev dependencies. Now
cucumberis just a dependency. See #396 for discussion. Setup interview has not been updated to remove dependencies.
- Fixed interview name not allowing url parameters. #449
- Use different criteria for testing that the package was saved.
- Implement a temporary fix for inability to pull private repos into empty da Projects. See #417 (comment)
- Improve some error messages and handling for the new private repo workflow.
- Allow authenticated GitHub user to pull from a private repository for which they have at least read permissions.
- v3 release to update to da Bootstrap 5. Breaks for servers that have not yet updated to that da version.
- Give test resetting, a common failure point, a couple of tries and add a warning and instructions about simply re-running the tests.
- Abandon puppeteer clicks and handle them all ourselves. This means people can't run a mobile device because they can't tap. It takes some implementation. See https://stackoverflow.com/a/56547605/14144258.
- Quick fix for frequent timeout errors. See #389 (comment), Option 1 - disable cucumber timeout and, instead, handle timeout errors to allow 3 attempts to load an interview. For future development, convert to API key to check if server is restarting.
- Update to Bootstrap 5 Docassemble DOM changes
- Remove test(s?) for guessing proxy vars as we don't do that anymore #221
- Added instructions for using tags when running cucumber tests
- Handle clicking in
tapElement()ourselves now instead of with puppeteer. See previous commit for the change.
- 'today' and 'today - 1' for setting dates
- In page load, total timeout needs to be checked in more places to give useful error message before Step fully times out.
- Not giving enough time for package to load - failing too soon on error of module not found.
- Not waiting for package page to appear on setup: #367 (comment). I believe we tried this in the past and it wasn't enough.
- Wrong error when server times out on page load.
- Tests for failed Scenarios and their reports
- Tweak report wording
- Internal report check logic
- Loading page did not catch system error and re-trying without timeout was not robust.
- Report line for error on load.
- Internal testing Step to test contents of reports include certain content.
- Internal testing Step to check passing tests should be passing and failing ones should be failing.
- Steps that set vars (other than the story table) should fail when they can't set their var, but they did not fail.
- Upload Step wasn't waiting for files to finish uploading
- Values can be set to comma-separated list of names of files to upload to upload multiple files. Files must be in the package's 'sources' folder or in the 'tests/features' folder.
- Increased
showif_timeoutbecause a hidden field missed getting set via a race condition. The only explanation we could come up with was that the field wasn't shown fast enough. - Allow duplicates of '[class="file-caption-name"]' fields as they should not be dangerous.
- Report: List failed scenarios first, other non-passing scenarios second (should not run into those much I think), and passed scenarios last.
- Two internal tests adjusted to be 10 secs shorter each
- One internal test tag updated
- Some report ids not being recorded because old id from previous test was not different from new id in new scenario.
- Updated package-lock.json with
npm fixbased onnpm audit
toBase64returns default valid base 64 value with clear name if needed
- Clarifying code comments
- Variable undefined in scope.js (message)
- Tests for reports for unexpected failures
- Bump minor version for added timeout behavior, etc.
- Step: Custom global timeout, #317, for when interviews know they'll need longer timeouts for such things as loading large documents.
- Timeout is now 30 sec instead of 2 min as individual devs can set a custom timeout for their scenarios. We've made an assumption that this will be enough for most interviews and will have to see if that bears out.
- More detailed reports that print 1. variables that were set on each page 2. all rows that set variables, and 3. all rows that weren't used
- Backwards compatibility with
#sought_variableelement. - Report tests
- Typo in report
- Not sure. Can't find this commit.
- Var name typos in
scope.ensureSpecialRows()
- Reports now also show data about which variables were used on each page, as well as a list of which variables from a story table were used and which weren't.
- Refactor
scope.processVar(), including rename
- Use new Buffer reference instead of old deprecated reference.
- Support interview filenames ending in
.ymlin the first step
- Support for
.there_is_anotherloops in story tables using.target_number - New step: 'I should see the link to' and its test.
- 'I get to' is available in the
And I get to the question "some id" with this data:instead of just 'the user gets to' - Add some warnings/feedback for the developer.
- Tests for some observational Steps.
- Tests for some interactive Steps.
- Tests for Assembly Line package-specific tests.
- Move debug logs for getMatchingRow and getPageData to appropriate places.
- Added
triggercolumn to table to define trigger var (| var | value | checked | trigger |). See #256. This allows devs to use index vars and generic objects (proxy vars) freely in their interview. - Tried to make element selectors more picky.
- Make screenshot names unique and line up alphabetically with timestamps.
- Allow more languages in scenario name.
- Different data structure for matches - check one field at a time - and replace
page_datawithfields. - Remove the need for the
checkedstory table column and prop by: 1) Moving thevalueof checkboxes into their var name column. 2) Moving thecheckedvalue of checkboxes into thevaluecolumn. - Combined checkbox encoding matching regex into one regex.
- Steps setting choices now work the same as the story tables - only two values given. No need for a 'trigger' column value.
- Updated tests to match new functionality.
- Previous formats of the table:
| var | choice | value |and| var | value | checked |
- Some Steps that won't work with translations and are not currently being used by anyone. TODO: Look up and ennumerate which steps.
checkedcolumn.scope.getField()and other now unused scope functionality.
- Assert found invalid input.
- 'I sign' was passing incorrect arguments.
- Regex testing for match to proxy var - both the regex itself being more strict and the test for a match being more predictable.
- #202 handles accidental mixup of single quotes for double quotes or visa versa in all columns.
- Handles setting AssemblyLine (ALToolbox)
BirthDatedatatype. See example of HTML here.
- Dropdowns created with objects always fail.
- This repo no longer has an action to generate test folders and scripts in the dependant repo. From now on, devs will use the interview in https://github.com/plocket/docassemble-ALAutomatedTestingTests to add it themselves. Note: That interview will put the tests in the
/sourcesdir. Previous repos' scripts will continue as they are.
- Each test run creates a unique da Project now. No need to wait for previous tests to finish.
- Message about duplicate selector matches.
- Bump minor version number appropriately.
- Newly found race condition on 'continue'. Can replicate inside of DBD very consistently, but not outside of it. See #190.
- Make action buttons more permissive. Some action buttons don't lead to an event, they just have a url. See #200.
- Fix all checkboxes getting checked no matter what.
- Account for
choices:specifier DOM difference - unlike other fields, it has no.form-groupcontainer. - Incorrect truthy test.
- New warning in report about variable appearing on a page in multiple places.
- A new table format is highly encouraged:
| var | value | checked |. var = variable name, value = words of the value to set, checked = whether to select or unselect a checkbox (only needed for checkboxes). - Basically, address much of #158, as noted below.
- Previous version of table columns format (
| var | choice | value |). The new version uses| var | value | checked |, though we do need a better name thanchecked. It will simplify the table for developers and for the code.
- Some proxy field var name finding
- Improved page var name finding in general
- Refactored to get all fields from the page first, then compare to a new format of table
- Automated integrated self-testing
- Unit tests for getField with cheerio
- Refactor getField to use data
- Infinite loop when specific button tap causes error #168
- Some items not getting selected at all (scroll into view problem)
- Simplest case of index var (i.e.
i) question.
- Race condition where puppeteer thinks the page has progressed but it really hasn't.
- Options of dropdown fields created with objects are not found #149
- Signature field rows (
/sign) are not used up all in one loop
- .env-example file to files that will be pushed to dependants
- #79, race condition breaks story table scenarios (ex: MADE
rental agreement) - #115, order of table items matter - continue button pressed too soon
- #139,
scope.getField()set_todoesn't find all the fields
- Reports are saved as
.txtfiles instead of.mdfiles. - Reports are prettier.
- Story table: If there are only two columns needed in a table row, the table doesn't care which columns the second item is in. This is to allow generated tests to work without a human editing them.
- Automatically generated language tags now handle spaces.
- Broken error message - showed the wrong error.
- Skipped this version somehow. Was sure it was in there somewhere...
- Steps for setting variables, interacting with elements, and observing (to be documented elsewhere TODO: add link)
- 'Story' table - a Step with a cucumber table that can set several variables until a specified
question idis reached - Translations support - many Steps that work with any language, including the 'story' table
- Reports about the order the pages appeared that are saved in github artifacts
- Screenshots from scenarios that created errors
- Ability to download documents
- Basic documentation for test writing
- Ability for this repo to push all needed files to other repositories on which it has push permissions. Those files make sure the tests will get run by github. See the files outlined below:
example.feature- an example test to get a project startedgithub/workflows/run_form_tests.yml- Automated github workflow to run tests when code is pushed and a manual trigger for testing without needing to push.gitignore- ignores files that are generated by the tests and shouldn't be added to the repopackage.json- keep up-to-date version of this test repository and hold scripts that the automated tests run