Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Random input: start to implement new directory structure #552

Closed
9 tasks done
plocket opened this issue Apr 11, 2022 · 7 comments
Closed
9 tasks done

Random input: start to implement new directory structure #552

plocket opened this issue Apr 11, 2022 · 7 comments
Assignees
@plocket
Milestone
Random input testing

Comments

@plocket
Copy link
Collaborator

plocket commented Apr 11, 2022 •
edited
Loading

From #531 (comment)

This issue

First step: Move current artifacts to new structure:

  • report
  • debug
  • [redundant error screenshots (that stay out in the main folder)]
  • scenario folder
    • [report for just the scenario]
    • downloaded files
    • error screenshots
    • manually taken screenshots
    • JSON

First, for a test run, everything would be in one artifact folder, so it can all be downloaded at the same time. Name options?

  • "alkiln_tests_output-" and timestamp (dashes or underscores?)
  • Somehow get the name of the test run itself. Probably possible. Plus timestamp? Plus alkiln?

This will be good in general even before starting with random input and is the first step.

Discussions on what to use in the names of artifacts: #429 (comment)

Long term plan and comments

Then the below structure for the folders eventually:

Maybe the error screenshots should be in both places. This is the idea proposed here, though the names are missing a bunch of pieces to avoid a mess and show the arrangement more clearly.

  • report.txt (whole report)
  • error-3pm-scenario 1 description-endingPgID.png
  • error-4pm-scenario 2 description-endingPgID.png
  • 3pm-endingPgID- scenario 1 description (folder)
    • report-endingPgID.txt (scenario only)
    • download-3pm-file-1-pgID.pdf
    • error-3pm-pgID.png (Same pic as "error-3pm-scenario 1 description.png". Same name seems dumb because it would have the scenario name in it too)
    • json-3pm-pgID.json
    • json-3pm-pgID.json
    • scenario 1 description.feature
    • screenshot-3pm-pgID.png
  • 4pm-endingPgID- scenario 2 description (folder)
    • report.txt (scenario only)
    • error-4pm-pgID.png (Same as scenario 1 error pic name rationale)
    • scenario 2 description.feature

"3pm" and "4pm" just indicates a timestamp. "PgID" could be question id or screen trigger var, whichever is available, or neither if none available. Not sure which is preferred.
@plocket plocket self-assigned this Apr 11, 2022
@plocket plocket changed the title Random input testing: implement new directory structure Random input: implement new directory structure Apr 11, 2022
@plocket plocket changed the title Random input: implement new directory structure Random input: start to implement new directory structure Apr 11, 2022
@plocket
Copy link
Collaborator Author

plocket commented Apr 14, 2022 •
edited
Loading

[This was a list of todo items, but it's been moved into the initial issue comment.]

plocket added a commit that referenced this issue Apr 15, 2022
@plocket
Create one artifacts dir, move some files in. #552. Read more below
07a9089 
- Put full report in dir
- Put debug_log.txt in dir
- Create scenario-name folder in dir
- Put downloaded files in scenario folder

For some reason, I haven't been successful in renaming the
downloaded file to add the timestamp so files with the same name
won't overwrite each other. Would love clues.
@plocket
Copy link
Collaborator Author

plocket commented Apr 20, 2022

We've decided to keep all scenario folders, even those that only have a report. Better that expected output is there.

@plocket
Copy link
Collaborator Author

plocket commented Apr 20, 2022 •
edited
Loading

Pretty timestamp has been requested. Edit: See https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/Date

@plocket
Copy link
Collaborator Author

plocket commented Apr 20, 2022

Part of the reason each scenario is in a different folder is for random input testing output - I'd like each random input scenario to have its own .feature file that the dev can copy/paste into their sources folder to replicate the test.

@plocket
Copy link
Collaborator Author

plocket commented Apr 20, 2022 •
edited
Loading

  • report.txt (whole report)
  • debug_log.txt
  • error_on-endingPgID-scenario 1 description-3pm.png
  • error_on-endingPgID-scenario 2 description-4pm.png
  • scenario 1 description-3pm (folder)
    • report.txt (scenario only)
    • error_on-pgID.png (Same pic as "error-scenario 1 description-endingPgID-3pm.png". Same name seems dumb because scenario name would be redundant) (excluding timestamp because there will only be max one error screenshot per scenario, so no need to make name unique.)
    • download-file-1-pgID-3pm.pdf downloaded-file-name.pdf (whatever its name is in the interview. We can't change it, but also shouldn't change it. Documentation should say that all files of an interview should have a unique name)
    • json_for-pgID-3pm.json
    • json_for-pgID-3pm.json
    • screenshot-custom-name-on-pgID-3pm.png
    • screenshot_on-pgID-3pm.png
    • scenario 1 description.feature (random input only. maybe stretch goal of exploring cucumber objects to see if they have the info to make this file for regular tests.)
  • scenario 2 description-4pm (folder)
    • report.txt (scenario only)
    • error_on-pgID-4pm.png (Same as scenario 1 error pic name rationale)
    • scenario 2 description.feature

[Note: I'm including the timestamps in scenario folder names and the error screenshot names in case devs copy/paste a scenario and accidentally forget to change the description and the test fails on the same page. That is, I can easily imagine a situation in which the name wouldn't be unique without the timestamp.]

Edit:
Artifacts folder itself:

  • alkiln_tests_output-3pm UTC

Folder/files for our own local unit tests:

  • _alkiln_test-misc_artifacts
    • unit_tests_logs-3pm UTC

@plocket
Copy link
Collaborator Author

plocket commented Jun 22, 2022

Have to replay these changes on top of the new branch because I can't track down a weird bug that cropped up in the Before() of at least one of the tests after my attempt to merge.

@plocket plocket mentioned this issue Jun 23, 2022
Merged
plocket added a commit that referenced this issue Jun 28, 2022
@plocket @BryceStevenWilley
Close #552, random input prep to rearrange artifact dirs (#568)
9b997f8 
* Ignore new dirs

* Update changelog with planned changes

* Add function to make human-readable UTC date

* Add file to just handle files (names etc), but only add a bit to it

Refactoring other stuff will be figured out later

* Prep action files to use new folder structure

Jumped the gun here and it'll need testing, sorry

* Implement setup of new artifacts folder on creation of project

* Store our own unit tests output

* Fix name of unit tests dir in our own action

* Store values of some necessary path names in `scope`

* Adjust download,error screenshot, scenario filepaths. See more
* change download filepath
* adjust format of scenario-based filename with page id
and date
* update unit tests to match
* save two copies of an error screenshot

* Add reports to artifacts folder and scenario folder

* Change json filepath

* Rename custom screenshot files

* Add notes about problems with download times

* Changes dir for accessibility tests. Untested from what I can tell

* Combine duplicate keys into one `run`

Co-authored-by: Bryce Willey <Bryce.Steven.Willey@gmail.com>

* Store persistent necessary names in one file

* Put cucumber debug into right debug file, use log's file name

more consistently

* Clean up comments now that decision is made

* Remove tags from scenario folder names

* Remove scenario tags from tests for creating base scenario name

Co-authored-by: Bryce Willey <Bryce.Steven.Willey@gmail.com>
@plocket
Copy link
Collaborator Author

plocket commented Jun 30, 2022 •
edited
Loading

Closed by #568 [Can't implement the feature file names because those files don't yet exist.]

@plocket plocket closed this as completed Jun 30, 2022
@plocket plocket added this to the Random input testing milestone Jul 1, 2022
@plocket plocket mentioned this issue Aug 28, 2022
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@plocket plocket

Labels
None yet
Projects
None yet
Milestone
Random input testing
Development

No branches or pull requests
1 participant
@plocket