Restore multi-language tests set by env vars

[plocket]

[Edit: Nothing really change in ALKiln itself, this is just authors using cucumber features that we didn't know about before.]

As per user request, restore running language tests and write internal tests for such (though I'm not sure how we check that the right tests have been run). These should primarily be triggered manually through the workflow dispatch, though allowing an env var for them might still be useful to carry over from previous behavior. Include:

• Access language page url as per Languages: Add accessing languages by adding to the url params instead of by tapping a button #360
• Add language to Scenario tags
• Internal tests of some kind, maybe using tags in Scenario name section. Check that certain text appears multiple times won't be sufficient as errors can also cause that behavior. [Maybe create our own translation file with different text that we can then look for. Use MADE's methods as a model for that as that is our primary user need at the moment?]
• [If we're creating the language files each time] Ensure InThePlayground deletes the language files during post-test cleanup.

Should close #360.

Edit:

• Add new issue to document this in the documentation repo

[plocket]

Our former method of creating language tests may need to change because of ALKilnInThePlayground. Currently we copy existing .feature files for each language. That's fine in GitHub where the files are temporary, but in the playground those'll be added to the sources and that'll cause problems. First off, it'll confuse the interview author. Secondly, it'll run the tests for those languages in future test runs of that Project.

That said, I'm not exactly sure how to handle this because I'm not sure how we access re-running the Scenario. Ideas are welcome.

[We can, of course, create the files and then clean them up, though generating new files has always felt like a hack to me. If we can figure out a less hacky way, that would be great.]

[plocket]

Of course, we can try to make it impossible, or at least very hard and obscure, to run language tests with the in-Playground interview... Not ideal, but a possible stop-gap.

[plocket]

I got a recommendation to use Examples. cucumberjs has this feature too. This was the example I was given:

Scenario Outline: show feedback on cancel action
Given the user has set the app language to <language_name> 
  And the cancel button shows the corresponding label
   | lang | label   |
   | en   | cancel  |
   | de   | Abbruch |
 When the user clicks on the button 
 Then a message confirms this action
   | lang | label   |
   | en   | Action canceled    |
   | de   | Aktion abgebrochen |

  @lang_en
  Examples: cancellation in english
   | language | language_name |
   | en       | English       |

  @lang_de
  Examples: cancellation in german
   | language | language_name |
   | de       | German        |

Since we want this to be manually triggered, I think we can have an env var that sets a bool - whether to test languages or not. If it's false (the default), we leave the url as it is. If it's true, we can change the url to the language column value. I'm not yet sure how we'd use the tables above in the tests, but it could allow folks to create tests using text on the screen to account for other languages.

[I don't think we can use the method above where they use tags to isolate the language tests as alkiln doesn't have a way to predict what language tags users will use and therefore what to normally exclude. Maybe users can still make use of those tags themselves, though, to be able to isolate different language tests. That would go in the documentation.]

[plocket]

Ok, so MADE doesn't use initial url args from what I can tell. I think you can only get to the language pages by pressing the button of the language you want. Have to think about how to handle this.

[nonprofittechy]

MADE does accept language parameters as url arguments

[plocket]

My mistake. Meant to update here that it was user error - I didn't understand the proper procedure for including url args. It has to be done (as @nonprofittechy kindly informed me) with a url arg of restart or new_session, or come from a /start address.

[plocket]

The first effort I'm making works with the above Examples hook. It's faster for me to implement, but is more burdensome on the developer - the code has to be duplicated for every Scenario Outline. There's no way to abstract it.

There's another method I'm exploring that will avoid those. It involves env vars and loops in bash in the GitHub action. Putting together notes for that. [May just be able to be in run_cucumber.js to make it more accessible to in-Playground.]

[plocket]

So regardless of the implementation, I see two strong possibilities for interfaces for our users.

Examples

In this interface, the dev uses Examples tables on every scenario they want to test with languages. They put tags on each of the tables as below.

Scenario Outline:
  Given I start the interview at "test_interview.yml<url_arg>"
  # The rest of the test
  
  @en
  Examples: English
  | url_arg |
  | &lang=en |
  
  @es
  Examples: Spanish
  | url_arg |
  | &lang=es |
  
  @ar
  Examples: Arabic
  | url_arg |
  | &lang=ar |

By default, ALKiln would run only the test WITHOUT the @alkiln_languages tag. When the author wants to test languages, they can use @alkiln_languages to test both Spanish and Arabic (not English). They can also use @ar to run only tests in Arabic.

This gives the author a lot of very visible control over the tests. Unfortunately, it does force the dev to add these tables to every test. I've done a lot of work trying to find out how to abstract these Examples so they can be used for every test. From what I can tell it's not possible.

This also might allow the author some flexibility as to how they implement accessing the languages. For example, if they have a different way of switching languages, they could use the tables to answer a question about languages on the page, etc.

Language list env var

See #713 (comment) for why we're not implementing this.

A list of comma separated url args that their workflow takes as manual inputs. I think in their action, they'll have to pass that value to us with an env var. Let's call it LANGUAGE_ARGS, or URL_ARGS if we want to make it broader, or something else.

URL_ARGS=lang=es, lang=ar

There are different ways for the developer to get the input from the workflow dispatch (manual input) interface. They can ask for a string or they can offer multiple choices. They just have to be sure that the rest of their workflow passes in the values in the correct format.

With no value for URL_ARGS, ALKiln would run the tests as they are written. If the env var did have a value, ALKiln would loop through the item and run all tests with each string, adding the string to the url of the ...start the interview Step. It would skip tests tagged with @alkiln_skip_for_translations (or something similar).

This method makes the assumption that the way to get interviews with different languages is to use a url argument. That makes it less flexible than Examples.

A down-side of this is that the developer would need to enter the manual input each time, which could be annoying. On the other hand, it gives them fine-grained control over which languages to test. I think that's important because if one language failed and others succeeded, I personally would want to be able to choose to run only the tests for one language.

If they find it too annoying to have to manually enter the information each time, they could create some logic in their workflow to handle that. For example, they could create a boolean input to determine whether to hand in a hard-coded string for the value of URL_ARGS. That way, they could hard code a list of all the languages so they don't have to type them in each time. They could even make that value overridable.

Granted, that's a pain to implement. If it seems to be a popular need someday, we could decide to provide our own abstraction for that.

A down-side for us is that we'd need to provide a way for ALKilnInThePlayground to function in the same way. Maybe just a field for a raw value for that env var as an MVP.

Both

We could implement both, though we'd have to deal with the possibility of developers mixing methods. Not impossible to handle, I think - env vars would override tags. I'd anticipate a point of confusion in this case being the tags, though. One is a positive (@alkiln_languages for marking languages to test using Examples) and one is a negative (@alkiln_skip_for_translations for avoiding non-default language tests with the env var), but they're not really related to each other and misusing the tags could lead to unexpected results. Not sure what to do about that.

I think I had at least one other method or user scenario in mind as well, but I can't recall the details at the moment ¯_(ツ)_/¯

[plocket]

Would especially appreciate your input on the above, @nonprofittechy , when you have a mo.

[plocket]

While the user originally expressed that they preferred to be able to only test languages manually (and avoid automatically testing them all on each push), they now expressed that they may actually prefer to test everything each time, so we're going to let go of that constraint. We'll just test all languages every time and not provide for skipping language tests.

[plocket]

I'm removing behavior 2 ("Language list env var" from a comment above) as a requirement for this issue. It was a way to help the developer avoid typing in all the languages each time they have to manually enter them in. Now that they won't be inputting them manually, there's no need for an env var like that.

[plocket]

Was closed by #713

[plocket]

Also, nothing really change in ALKiln itself, this is just authors using cucumber features that we didn't know about before. We [could] edit the CHANGELOG to reflect this.