Setup: User Testing 1 & 2 & '3'

[ezaleznik]

Page numbers following scenario: Non-admin user/Admin has set up secrets/Creating new test

• P1. (Idea) Specify what the tool does for the user.
  • "This tool helps developers or organizations set up automated integrated testing for docassemble packages that have a GitHub repository. First, it will set up GitHub secrets, if necessary. Next, it will create a testing branch for your docassemble package on your Github account that you may merge with the main branch”
• P1. Change time on task (up to 1 hour)
• P1. Change text in "GitHub Secrets" alt-text at top of page from "bottom of the page" to "bottom of this page"
• [Doing this differently] (P1. Consider adding instruction: "Before you start, contact your administrator to see if you have GitHub Secrets set up"?)
• P2. Change first line ("You indicated that..." is not in line with what the users believe they have answered previously)
• P2. Change word "package" to "interview" to keep in line with internal terminology. [Different people will know different language. "package" is most correct. How do we handle this? Make "package" a term?]
• [removed] P3 Consider whether hyperlinks at top are appropriate
• [removed that section] P3. Further consider whether output files section is helpful or distracting to userbase
• P3. Clarify instructions:
  • "Complete the following instructions" header
  • Tell them to use their GitHub account ("Log into your GitHub account")
  • [We cannot know that that the heading will stay the same. I tried to make other adjustments, but it might actually be less clear now.] 1. "Read the instructions at this link under "Create Personal Access Token"
  • [Instead, I simplified to remove extraneous descriptions] 2. "Create your Personal Access Token: Tap the checkbox for "workflow" permissions. Ensure that all "Repo" options are also selected."
  • 3. "Finish creating Personal Access Token"
  • 4. "Copy the token and paste below"
• P3. Specify that they are looking for the URL for their interview's repo
• [removed] P5. Consider whether hyperlinks at top are appropriate
• [Actually, the check for failure needs to happen when they see the action page. That check actually needs to happen with that step. It's a lot of text, though.] P5. Instructions should be reorganized and made more specific. Review actions first, then follow other checks
• [removed test writing instructions] P5. Second list may be moved elsewhere (user may close window before doing this optional task)
• [removed test writing instructions] P5. Second list, item 7: "File" should be made more specific

More detailed notes below. They're not really action items, just notes, but may give context to the above and we can pretend they're todos to make sure we address each note:

User 1

• Pointed out AL in title is shortened, but explained below
• Confused about why “organization” was in quotes
• Cued by question about admin to feel like they needed to find an administrator [and they may, so that's fine]
• Unsure whether secrets had already been set up
  • Requires them to check with administrator?
• Clicks every link
  • Check whether each link is necessary or hinders flow
• [Can't do much about this] Unsure initially where to get PAT, but finds page without issue
  • Flips back and forth between interview page with instructions and github documentation
• Unsure what account they should use to get the PAT
• [we've determined that this isn't at all critical and didn't seem to slow progress much] Wasn't sure of PAT expiration date. Chose 7 day expiration
• Unsure which URL to use for their package
  • Follows the patterning of provided URL
  • Could use more clear instructions on which URL they should use
• Appreciates page after inputting PAT--verifies that they did previous step correctly
• [first link removed, second clarified with 'your pull request'] P5-expected pull request link to go to documentation, not repo
• Confusion on red x appearing in action page (due to coding flow on our end, known issue)
• Unsure whether to request review or merge branch (due to our unclear scenario)
  • They do not seem to have clear instructions on their end about which option they should do outside of our test
• P5-Unsure whether “step 1” was a header or an instruction
• Did not delete PAT (read the step but did not follow the instructions)
• Did not know what YAML file looked like, but correctly identified without help
• Believed the “restart” button meant to do another test setup
• [I think I must have deleted this line as I can't find it anymore] Confused by “at github secrets” at end of test.

User 2

• Unsure whether they were an admin, confused by line “depends on your situation”
• P2-did not like first sentence (says because of repeated word “repo” but user 1 identified same sentence was weird due to “you identified that...”)
• Didn’t click alt-text “about github secrets”
  • Would have emailed admin to ask about secrets
• Unsure how to get PAT at first.
  • [What can we do about this?] Appears skeptical that the github documentation link is what they need at this step.
  • [Attempted to emphasize 'your account'] Unsure what account they should use to get the PAT
  • Github did not show their account picture when page was not full-screened (not something we can correct, but something to keep in mind!)
  • [This is fine] Choose 7 day expiration
  • [Not sure what we can do here as we need to use Github docs] Kept continuing on to the “use on command line” in github docs on PAT
    • Appeared to understand that it wasn’t relevant to them after spending a minute or two flipping back between interview, documentation, and create PAT page
• Did not struggle to find the URL for their package
• [Do we need to improve the error message?] Did not properly set up PAT permissions
  • [need to re-test] Was confused by the wording of the sentence (“select workflow. This will also select Repo.” Needed to read several times
  • Initially did not set up workflow permissions
  • Correctly interpreted error page, verbally explained why they got the error, but still struggled to translate that knowledge back to generating PAT
  • [As reflected by absence in summary section above, not something we can address] Used back button on browser to return
• Appeared to have difficulty with terminology (merge/pull/review)
• [Not critical. It's pretty secure.] Did not delete either PAT generated
• [removed test writing instructions] Instructions did not specify to go to docassemble, but knew because the line “playground” or by general understanding of the sentence
• [removed test writing instructions] Unsure which YAML they were testing, but correctly replaced the blank spaces

'User 3', org admin - only got to hear about it after the fact, not actual user testing

• Info for github secrets: Assumed the url could be the url on any server. We might want to break out the url into its own question with a question preceding it asking if they have a window where they are logged into the test user account
• final page: 'Restart' made them think the server would be restarted.

EZ Runthrough from 8/16

• "Before we start" page: Less-technical users may be unclear how to connect the question: "Is this the server where the tests will run" to the URL they are looking at.
• After setting up Github Secrets: page says "3 Github Secrets were created" but the bulletpoint list has 4 items. This may make the user question if things worked properly.
• After setting up Github Secrets: Typo on instruction 1: "Follw this link to the pull request"
• After setting up Github Secrets: "If you want to go for a wild ride." User might be unsure whether they should click on this section. If it's important for a specific person or type of person, write that here. Could rephrase: "If you would like to learn more about x, click here"

[plocket]

This says 1/2, but we discussed both. Is this just the notes from the first user? Can we add the notes about the second user? Sorry I haven't gotten a chance to look through it more thoroughly. Hope I can do that soon.

Notes from an additional user. This user was setting org secrets:

• Assumed the url could be the url of any server. We might want to break out the url into its own question with a question preceding it asking if they have a window where they are logged into the test user account
• 'Restart' made them think the server would be restarted.