Integrate VRT (for those sites that have a framework set up)

[dalin-]

History

Currently this is run manually. See:
https://github.com/fourkitchens/bcj/blob/main/tests/backstop/backstop-setup
Which when run looks like:

And then
2. lando backstop reference
3. lando backstop test

It would be great if this was done in CI to reduce set-up time, and even though these are run in Lando, there still are variations on different laptops. It would also be great to share the final report. Sometimes engineers ask "I think this is a false positive, but I'm not sure", or sometimes clients are interested to know.

Config

There would need to be a config file to store:

• Pantheon site name

We somehow need to get the following message about how to run the VRT:
Edits: Let's keep it simple...

• reference environment could potentially be hardcoded as test, or live, or whatever the CI "canonical" env is
• environment that we're testing CI could be smart enough to assume that it's the current PR environment
• Does the engineer want to either copy DB+files from reference environment to test environment OR leave DB+files as-is (and have lots of false positives due to different content in the two environments) Since we are building the PR multidev from the canonical environment, it already has a copy of the DB/files.

when/how to trigger this to start

Options:

1. Run on every CI run, as a separate task after all the others are complete. Ping back to the ticket with the results.
2. Use a script e.g. run-vrt.sh <site-name> <reference-env> <test-env> --copy-content-reference-to-test
3. A commit message vrt(ticket-id): reference:live test:pr-46 content:copy-reference-to-test

Slack comments from Allan:

The way CircleCI says we should handle this one is make the main workflow that is run on every commit one that parses the commit message and then it calls another job using the Pipeline API. or using Dynamic Configuraiton using a continuation orb.
In the UI this would make every commit pretty much have at least 2 workflows pop up for it.
...
we could even use what you have on backstop-setup as a guide and ask the person running the command to provide
$site = ask_string_question('Enter the machine name of the Pantheon site that you are testing. ');
$environment_test = ask_string_question('What environment are you testing (typically a multidev or dev)? ');
$environment_reference = ask_string_question('What is the reference environment (typically dev or test (of even live if content doesn't change much))? ');
Send that to the remote job and let it do all the other stuff in your script

Discussion

https://fourkitchens.slack.com/archives/C03P6N0ER/p1707509708497879?thread_ts=1707507600.314769&cid=C03P6N0ER

[generalredneck]

We had some more conversation, And this one doesn't have to be Commit MSG based. The goal is to get the VRT setup off the local machine because of the variation between environments over there. So in this case as long as we consistently do VRT the same way, we can create a command wrapper around VRT that the user will fire off locally.