Create wiki page "Hfla Google Apps Script Development/Review Process"

[roslynwythe]

Dependency

• Add latest version of Google Apps Scripts to GitHub #3641
• Updating Wiki: Move Current Wiki to New Repository #3636

Overview

The Google Apps Script development process now includes source control in the Hfla website repository, and the use of the clasp CLI to transfer code between the local workstation and Google Drive, enabling local editing and committing of code. We require a wiki page to guide developers, reviewers, and merge team members how to work with Google Apps Script.

Action Items

• We are currently in the process of moving the old wiki to the new website-wiki repo, so we will not be making any changes or additions to the old wiki at this time. Thus, we will be adding wiki content through a different process now. Read How to Contribute to the Wiki

For Developers, include:

• Installation of required npm modules using package.json and package-lock.json
• Preparation of test environment on dev's personal Google Drive. This is described in step 9 under "How to share files with developers working on an issue" in the WINS Admin Guide
• Login to Google Drive with clasp login
• Cloning Google Apps Script using clasp clone <ID> where <ID> is from the developer's copy of the project.
• Checking that the production Google Apps Script (Code.gs) matches the contents of Code.js in gh-pages.
• Option to edit code locally or in Google Apps Script UI/IDE, and to push code to Google Drive for testing.
• When testing is complete, if any code changes were made in the Google Apps Script UI/IDE, use clasp pull to update the local code file.
• Add/commit Code.js and the manifest file appsscript.json (if any changes were made).
• Push issue branch to dev's fork of gh-pages, then create PR in the usual manner. In the PR, describe the test setup/strategy, including relevant triggers.

For Pull Request Reviewers (testing in their own Google Drive):

• Installation of required npm modules using package.json and package-lock.json
• Preparation of test environment on reviewer's personal Google Drive. This is described in step 9 under "How to share files with developers working on an issue" in the WINS Admin Guide
• Login to Google Drive with clasp login
• Cloning Google Apps Script using clasp clone <ID> where <ID> is from the developer's copy of the project.
• Pull down issue branch, then for testing, run clasp push to copy code to Google Drive.

Merge Instructions:

• Installation of required npm modules using package.json and package-lock.json
• Before bringing down the contributor's issue branch, create a local backup copy of each production code file Code.gs that will be updated. Retain the backups until testing is complete.
• Check that the production code (Code.gs) matches the code (Code.js) in gh-pages.
• Setup local environment, using clasp clone <ID> where is the production project,
• Pull down contributor's issue branch
• Run clasp push to copy code to Google Drive for final testing. Make any change to Triggers that were specified in the PR. If something goes wrong, restore the previous state of the script by copying/pasting from the local backup copy and clicking Save.

After completion, update the following issues with references to the wiki and release any dependencies on this issue, then move issues to New Issue Approval with "ready for dev lead" label:

• Update Google Apps Script to generate wins data feed only on Change #4154
• Update wins Google Apps Script to format wins json feed #4035
• Google Apps Script: Add unique IDs as object keys for each wins entry #2385
• Google Apps Script: Migrate Wins Data - Stop generation of AF file #2147
• Google Apps Script: Migrate Wins Data - Delete AF file  #2148
• Enhance wins sheets to support selection for homepage randomizer #2505

Resources

• WINS Admin Guide
• Add latest version of Google Apps Scripts to GitHub #3641
• CLASP for Google Apps Scripts development evaluation / decision  #3722

[github-actions]

Hi @shavinski, thank you for taking up this issue! Hfla appreciates you :)

Do let fellow developers know about your:-
i. Availability: (When are you available to work on the issue/answer questions other programmers might have about your issue?)
ii. ETA: (When do you expect this issue to be completed?)

You're awesome!

P.S. - You may not take up another issue until this issue gets merged (or closed). Thanks again :)

[shavinski]

Availability: M-F 1pm - 7pm
ETA: October 6th, Friday

[shavinski]

@roslynwythe Hi Roslyn! I just requested access to the WINS admin guide in order to help me complete this issue.

[roslynwythe]

Hi Jakob, these are notes I've created that describe the basics. If you find this useful I can post more notes to detail the parts 2 and 3 that I reference below.

There are two environments that comprise a Google Apps Script Development environment:

• A Google Drive environment associated with your personal (or your personal “development”) Google account. This is the only environment in which you will test Google Apps Script code. In this document it will be referred to as the “Google Drive IDE”. The Google Drive environment is created using copies of documents from the production HfLA Website Admin Google Drive. See Part 2 of this document for instructions on that process.
• A local environment, in the google-apps-script folder of the website repository. In this environment you will use git and clasp along with your choice of IDE or editor.

In the local environment you will be retrieving code from the repo using ‘git’, pushing/pulling code to/from Google Drive using ‘clasp’, and editing using an IDE or text editor. All code files have the filename ‘code.js’ and are located in separate folders under the google-apps-scripts folder, which can be found in the root of the website repository. Currently we have two Google Apps Scripts projects, each associated with a subfolder:

folder structure:
├── google-apps-scripts
│ └── wins-form-responses
│ └── Code.js, appscript.json, package-lock.json, package.json
| |___ gh-requests
| |___ Code.js, appscript.json, package-lock.json, package.json

To get started, you will have to install node.js and npm if they are not already installed. You should also update your local gh-pages with upstream changes, then checkout a new issue branch.
Before using clasp in a subfolder you must execute the commands npm install clasp and npm install @types/google-apps-script

From a google-apps-script subfolder, execute the command clasp login, then authenticate to Google Drive and grant permissions to the clasp app.
Then run clone the script using the command clasp clone <ID> where the script ID can be found on the corresponding Projects Settings page in the Google Drive IDE.

The recommended process is to edit code locally, however testing must be done in the Google Drive IDE. After making local edits, to test your code, run clasp push to push it to Google Drive for testing, where is the script id. If you make any code changes in the GD IDE, use ‘clasp pull’ to bring it down locally for adding/committing to the repository. When testing is complete, code files are added/committed to gh-pages and a PR created in the usual manner. Part 3 of this document will describe the Pull Request and Merge processes for Google Apps Script projects.

[shavinski]

@roslynwythe I am currently trying to get clasp login to work on my local matchine but I am having some issues.

Steps I took as of now:

• I am currently in this subfolder: /website/google-apps-scripts
• I have have pulled from the upstream and then created a new branch for this issue
• I then did the npm install clasp and npm install @types/google-apps-script inside the /website/google-apps-scripts directory
• When running clasp login on my terminal inside /website/google-apps-scripts I get this:
  wiki-google-scripts-4134 website/google-apps-scripts % clasp login
zsh: command not found: clasp

Things I tried to solve this:

• I have tried to do a different download using this npm install @google/clasp, but still got the same error as above
• I have tried going into wiki-google-scripts-4134 google-apps-scripts/wins-form-responses and wiki-google-scripts-4134 google-apps-scripts/gh-requests separately and running the clasp login but I still get the command not found
• I checked my npm version which is 9.5.0
• I checked my node version which is v18.14.2
• I have tried googling but have not found much regarding this issue

[roslynwythe]

Hi @shavinski - you have to be in one of the subfolders of google-apps-scripts to install and run clasp. I suggest testing in google-apps-script/wins-form-responses. I apologize if that was unclear from my instructions. Please improve and reorganize the instructions as you see fit. Please include screenshots.

For the purpose of writing this set of instructions, let us suppose that you are going to work on an issue that will update the script wins-form-responses but that in order to test wins-form-responses, you will require the gh-requests library. I followed the instructions under the heading "How to share files with developers working on an issue" p 10 -12 (through item 8) in the Wins Admin Guide, and so you can start with item 9 on pg 12.

Thanks!

[shavinski]

Hi @roslynwythe! So I was running the npm install clasp command in the subfolder wins-form-responses like you said and was still getting that command not found error. I was able to figure it out finally, I went to the documentation for clasp and ran this command npm install @google/clasp and after I ran npx clasp login which successfully loaded the google drive tab in my browser. I will make sure to add this to the updated instructions.

Let me know if that is ok! Thanks!

[shavinski]

Wins Google Apps Script Development Process (Draft)

@

For Developers

1. Creating an issue branch

a. Open your local command line/terminal and navigate to where you have the Hack for LA project located
b. Create an Issue Branch in the usual manner, from an updated version of gh-pages

2. Installation of npm modules

a. If not already installed, download and install node.js and npm
b. Change directory into the folder google-apps-scripts . Note that it contains two sub-directories, one corresponding to each Google Apps Script project:

• gh-requests
• wins-form-responses

c. The next step is to install clasp and the associated npm packages. This must be done in each subdirectory corresponding to an Apps Script project that you will be modifying. For a local install use npm install and note that you may have to prefix clasp commands with npx, for example: npx clasp login, npx clasp push, etc. For a "global" install run npm install -g.

3. Preparation of test environment on developer's personal Google Drive

a. The developer will need to select (or create) a google account to use for testing. In the testing process it will be necessary to authorize script access to your Google Drive, so you may choose to use (or create) an account separate from your personal google account. Authorization problems can occur if you are logged into multiple Google accounts, so we suggest that you log out of all google accounts then log in using the account you have selected for testing. This needs to happen anytime you want to work with the App Script on the files.
b. The developer will need to ask a team lead to share the files required for the issue, providing a Google email address of the selected test account. The required documents may have been listed in the issue, but guidelines are provided below:

• To understand the WINS project components, the `WINS Form Admin Guide` is helpful
  • Note to team lead: share the `WINS Form Admin Guide` with the developer as VIEWER
• It will almost always be necessary to include `Wins-form (Responses)` sheet which contains the submitted form data, formatted Review data, and the Apps Script `winsFormResponse`
  • Note to team lead: share `Wins-form (Responses)` sheet with the developer as VIEWER.
• If it will be necessary to test using new test form submissions, then the form `Wins-form` is required
  • Note to team lead: make a copy of `Wins-form`, including the issue # in the name, then share that copy with the developer as EDITOR.
• If it will be necessary to modify the Apps Script-GitHub interface or to use that interface for testing (posting issues and/or wins data feed to a GitHub repository), then the Google Doc `gh-requests` will be required. This doc is the container for the Apps Script `ghrequests`.
  • Note to team lead: share `gh-requests` with the developer as VIEWER.

c. Once you receive the necessary shared Google Drive files, make copies of the required documents, placing the copies in your test folder.

d. The following must be done to prepare the test docs for test

• When `Wins-form (Responses)` is copied, the triggers are lost so you may need to recreate some triggers for testing - see https://github.com/Create wiki page "Hfla Google Apps Script Development/Review Process" #4134#issuecomment-2365343902
• Make note the scriptID of the `winsFormResponse` script project from the Project Settings page, for later use with clasp
• If the test environment includes a version of `Wins-form`, edit the `Response` settings so that form data will be stored in the test environment's `Wins-form (Responses)` sheet
• If gh-requests is part of the test environment, and the developer is not on the merge team, it is necessary to follow the steps in Testing WINS in a test repository

4. Login to Google Drive with clasp login

1. Make sure you are working in a sub-directory in which you have installed the packages described above in "Installation of npm modules".
   • google-apps-scripts/gh-requests
   • google-apps-scripts/wins-form-responses
2. Run `clasp login`
3. If successful your local machine should automatically open a new browser tab to sign in to your Google Drive account
4. You will then select (or add) the account that you used in step 3a to save test documents in your test folder, and authorized Google Apps Script CLI for the scopes listed.

5. Cloning Google Apps Script using clasp

1. Cloning is the initial copying of Apps Script and metadata from the Google Drive (cloud) to the local workstation and must be done in the corresponding sub-directory of google-apps-script. Cloning is necessary only for those Apps Scripts that will be modified by your issue, and will not be required if you modified Apps Script just for testing purposes. If your issue calls for modifying and committing multiple Apps Scripts, you will repeat this procedure in each sub-directory.
2. Before the clone you must delete any existing version of the metadata by running rm .clasp.json
   Run clasp clone <Script ID> where ScriptID is the scriptID of the source Google Drive Apps Script project
   • Script ID can be found in the project settings page in the Apps Script project in Google Drive
3. This process is used to confirm that prior to modification, the cloned Apps Script matches the script in the git repository. It is necessary to perform this step for each Google Apps Script project that you will be committing to the git repository.
• Run git diff Code.js. Copy/paste the results of this command in a comment in your issue along with the name of the Apps Script. The warning LF being replaced by CRLF can be ignored, but if any actual file differences are reported, they must be examined to ensure that they are changes you made for testing purposes. If that is not the case, comment-mention a dev lead.

6. Editing code

1. You have two options for editing code
   a. Local code editor
   • Note that Google Apps Script code cannot be tested locally; to test, use clasp push to push code to the Google Drive
     b. Google Apps Script Cloud Editor (ie Google Drive UI)
   • run clasp open or browse to the container from Google Drive UI
   • Note that code must be committed locally; use clasp pull to bring code to the local system.
2. to push script to Google Drive, run clasp push
3. to pull script locally, run clasp pull

7. Testing

a. Perform testing and demonstration as required by your issue.
b. If you run main() in Wins-form (Response), it will generate a Pull Request and new GitHub issue in the target repository.

• If you are using a test respository, see Section 5 of [Testing Wins in a test repository}(https://github.com/Add new section "Testing Wins in a test repository" to the Wins Apps Script Development Wiki Page #5746#issuecomment-1846549069)
• If you are a merge team member testing against the production hackforla/website repository, make appropriate comments in the resulting PR and issue. Apply the label `Ignore: Test` to the issue.

8. Completing an issue

a. Following a successful demonstration, changes may be required before committing the Apps Script code. If you are following instructions in "Testing WINS in a test repository", you need to revert or reverse the changes you made to target the test repository or test Project Board. It may be helpful to refer to the output of the git diff command from step 4 under Cloning Modified Google Apps Script using clasp
b. Commit modified code using git and push the issue branch in the usual manner

• do not add any files other than Code.js to the commit.
• Refer to Part 3: Pull Requests in the contributing guide: https://github.com/hackforla/website/blob/gh-pages/CONTRIBUTING.md
• On the Pull Request, add the label "ready for dev lead"
  c. In your Pull Request, please describe your test setup/strategy

**For Reviewers (Merge Team) **

1. Prepare a Google Drive test environment in a test folder in the HfLA website Admin shared drive (for example, HfLA website Admin/HfLA website admin/testing-issue-4035). If you wish to use a development Google account for this process, share the test folder and the "wins to review" folders as "Contributor" with the dev account, then log out of all accounts and log back in to the dev account.
   a. In general it is ok to target the production repository so Wins-form (Response), gh-requests, and gh-key can be copied into the test folder and used for testing without modification
   b. Note the script ID of the test Apps Script that will be changed by the PR
2. Follow instructions under "Installation of npm modules" but create and pull down the issue branch using the instructions in the PR
3. Follow instructions under "Login to Google Drive with clasp login"
4. Follow the instructions provided in "Cloning Google Apps Script using clasp", but if differences are detected in step 3a, they must be examined to ensure that publishing the changes from this PR does not overwrite previous changes.
5. run `clasp push' to push the changes to Google Drive
6. If multiple Apps Scripts are modified by the PR, repeat steps 3 and 4 for the remaining Apps Script in the corresponding directory
7. test
8. If testing resulted in PR and/or issues created in hackforla/website, close those with a comment and label "Ignore: Test"
9. To update the production code, follow instructions under "Cloning Google Apps Script using clasp" with the production script ID
10. run `clasp push' to push the changes to Google Drive
11. If multiple Apps Scripts are modified by the PR, repeat steps 9 and 10 for the remaining Apps Script in the corresponding directory
12. After successful testing, approve then merge the PR.

[shavinski]

There are some formatting issues regarding line breaks where there should not be.

I felt confident writing about the For Developers section and got less confident as I went from For Pull Request Reviewers to Merge Members.

This task: Google Apps Script: Migrate Wins Data - Delete AF file #2148 I wasn't able to checkmark it on this issue because on the issue linked the format is a bullet point versus a checkbox.

I also had a question about the dependencies, I had to npm install a different dependency, I had to npm install @goolge/clasp to get my clasp commands to work. Should I also submit a PR on this branch with updated dependencies?

Let me know if there are any necessary changes!

[roslynwythe]

@shavinski I have been puzzled by the statement in the Wins Admin Guide, that when setting up a development environment, the developer should create a Personal Access Token, because the guide does not explain how or where that token would be used. Presumably the token would be used so that in the process of testing, the google apps scripts could create issues and/or push wins data to the contributor's fork of the repository instead of to hackforla/website, but in order to accomplish that, it appears that there would have to be code changes in gh-requests. I don't see those code changes described so perhaps that work still has to be done. That would enlarge the scope of this issue and would require the dev to setup a test environment, do testing and more documentation. If you are interested in doing that, I can add to the Action Items of this issue and make this into a Complexity: Large issue. If not, I think we could close this one as completed. What is your preference?

[shavinski]

@roslynwythe Ya I was a little confused about the token portion as well. I can give the Large issue a shot! I will just need to research the best way to set up a test environment and familiarize myself with the Access Token.

[roslynwythe]

That sounds good @shavinski . I'm not certain but my guess is that you should follow the instructions starting at #8 on pg 7 of the WINS Admin Guide, for generating the token, except do it from your GitHub account, then in gh-requests, change every reference to your repository. Please document the steps and add instructions above as you go. For testing to make sure that the test setup process is correct, I would like to see that you can run main() in order to push the wins-data.json to a PR in your repo, and that you could submit a wins form which would create a new "review" issue in your repository. So in your test environment, you will need a copy of the wins form, which must be connected to your copy of the wins-form (Response) spreadsheet, and then also a copy of gh-requests. Try to follow the instructions for setting up the environment so make sure they are complete. On Slack please let me know which files I still need to share with you.

[roslynwythe]

@shavinski I am closing this issue as completed so that you receive credit for your work, and I will create a new issue to create the extra documentation "Testing Wins in a test repository". Thank you.

[roslynwythe]

• New issue Add new section "Testing Wins in a test repository" to the Wins Apps Script Development Wiki Page #5746

[roslynwythe]

Recreating Triggers for testing in Wins-form (Responses)

When Wins-form (Responses) is copied, the triggers are lost, so any triggers required for testing will have to be recreated.

1. Function main, event source time-driven, type Day timer. This trigger is rarely required because main() can be invoked at any time via Debug.
2. Function `insertLatestFormSubmitIntoReviewSheet`, event source `From spreadsheet`, event type `On form submit`. This is required to generate the Review Sheet for a new wins form submission.
3. Function `createIssue`, event source `From spreadsheet`, event type `On form submit`. This is required to generate a GitHub issue to initiate a review of newly submitted form data.