Pass Errors over to the Create Issue Action?

[michaelhofrichter]

If there is an error, you can use the action step to create a Github Issue - https://suffolklitlab.org/docassemble-AssemblyLine-documentation/docs/alkiln/#make-a-github-issue-when-tests-fail

Is there any way to pass more of the details about which scenarios failed to the issue?

[plocket]

Hmm, maybe. What would you want to know about the Scenarios? Their descriptions? What if we provide the report (to put in the issue)?

Just some notes for implementation: Regular GitHub action steps can output data and subsequent steps can use that data: https://docs.github.com/en/actions/learn-github-actions/contexts#needs-context. I'd like to assume composite actions can then pass that to the actions that used them because it seems like an obvious affordance, but... yeah, so we should do some research.

[Update: It's totally possible to pass output from the action, but I think we can only output strings and I'm not sure we can output dynamic variable names. That is, all the scenario failures would be in one long (though multiline) string. The dev would have to then parse out individual scenarios to make individual errors, or just make an issue for one error for all the scenarios.]

[michaelhofrichter]

I'd take as much detail as I can get.

Ideally, I'd get the step by step elements to see what failed. I can find that now in the log, but it takes a while to click there and then scroll to the right place:

Failures:

1) Scenario: Testing Caller ID works # docassemble/CallerID/data/sources/caller_ID_testing.feature:9
   ✔ Before # node_modules/@suffolklitlab/alkiln/lib/steps.js:103
   ✔ Given the maximum seconds for each Step is 400 # node_modules/@suffolklitlab/alkiln/lib/steps.js:289
   ✔ Given I log in with the email "PLAYGROUND_EMAIL" and the password "PLAYGROUND_PASSWORD" # node_modules/@suffolklitlab/alkiln/lib/steps.js:254
   ✔ Then I start the interview at "CallerID" # node_modules/@suffolklitlab/alkiln/lib/steps.js:156
   ✔ Then I set the variable "phone_number" to "111-111-1111" # node_modules/@suffolklitlab/alkiln/lib/steps.js:930
   ✔ Then I tap to continue # node_modules/@suffolklitlab/alkiln/lib/steps.js:732
   ✔ Then I set the variable "site_abbreviation" to "michael" # node_modules/@suffolklitlab/alkiln/lib/steps.js:930
   ✖ Then I tap to continue # node_modules/@suffolklitlab/alkiln/lib/steps.js:732
       The test was unable to continue because the server was reloading. ALKiln will try this Scenario a total of 2 times. An error will still be printed for this attempt.
   - Then I take a screenshot # node_modules/@suffolklitlab/alkiln/lib/steps.js:446
   - Then I can't continue # node_modules/@suffolklitlab/alkiln/lib/steps.js:561
   ✔ After # node_modules/@suffolklitlab/alkiln/lib/steps.js:1096

2) Scenario: Testing Caller ID with URL Args # docassemble/CallerID/data/sources/caller_ID_testing.feature:20
   ✔ Before # node_modules/@suffolklitlab/alkiln/lib/steps.js:103
   ✔ Given the maximum seconds for each Step is 400 # node_modules/@suffolklitlab/alkiln/lib/steps.js:289
   ✔ Given I log in with the email "PLAYGROUND_EMAIL" and the password "PLAYGROUND_PASSWORD" # node_modules/@suffolklitlab/alkiln/lib/steps.js:254
   ✖ Then I start the interview at "CallerID.yml&reset=1&phone=111-111-1111&ls_site=michael" # node_modules/@suffolklitlab/alkiln/lib/steps.js:156
       AssertionError
           + expected - actual

           -Exception: 

           at Object.load (/home/runner/work/docassemble-CallerID/docassemble-CallerID/node_modules/@suffolklitlab/alkiln/lib/scope.js:790:38)
           at runMicrotasks (<anonymous>)
           at processTicksAndRejections (node:internal/process/task_queues:96:5)
           at async World.<anonymous> (/home/runner/work/docassemble-CallerID/docassemble-CallerID/node_modules/@suffolklitlab/alkiln/lib/steps.js:159:3)
   - Then I take a screenshot # node_modules/@suffolklitlab/alkiln/lib/steps.js:446
   - Then I can't continue # node_modules/@suffolklitlab/alkiln/lib/steps.js:561
   ✔ After # node_modules/@suffolklitlab/alkiln/lib/steps.js:1096

[plocket]

Note: Revisit in the future to try implementing a different syntax for outputting the actual text as a proper multiline string: https://github.com/orgs/community/discussions/26288