Skip to content

Commit

Permalink
New issue template: Update Truth (#2332)
Browse files Browse the repository at this point in the history 
Co-authored-by: John Halley Gotway <johnhg@ucar.edu>
  • Loading branch information
@georgemccabe @JohnHalleyGotway
georgemccabe and JohnHalleyGotway authored Aug 30, 2023
1 parent 5fbb20a commit 44335f3
Show file tree
Hide file tree
Showing 3 changed files with 168 additions and 76 deletions.
60 changes: 60 additions & 0 deletions .github/ISSUE_TEMPLATE/update_truth.md
View file
Original file line number Diff line number Diff line change
@@ -0,0 +1,60 @@
---
name: Update Truth
about: Review use case differences that are caused by changes in an external repository and update truth dataset if necessary.
title: 'Update Truth: '
labels: 'alert: NEED ACCOUNT KEY, alert: NEED MORE DEFINITION, alert: NEED CYCLE ASSIGNMENT, type: update truth, priority: blocker, component: CI/CD, requestor: METplus Team'
assignees: ''

---

## Describe Expected Changes ##

*Write a short summary of the differences that will likely be found in the GitHub Actions testing workflow that was triggered by the pull request that warranted this issue*

## Define the Metadata ##

### Title ###
- [ ] Add a link to the pull request that warranted this issue to the issue title using format dtcenter/{REPO}#{PR_NUMBER}.

**Example:** Update Truth: For dtcenter/MET#2655

### Assignee ###

*Assign this issue to the author of the pull request that warranted this issue. Optionally assign anyone else who should review the differences in the output.*

- [ ] Select **engineer(s)** or **no engineer** required
- [ ] Select **scientist(s)** or **no scientist** required

### Projects and Milestone ###
- [ ] Select **Repository** and/or **Organization** level **Project(s)** or add **alert: NEED CYCLE ASSIGNMENT** label
- [ ] Select **Milestone** as the next official version or **Future Versions**

## Update Truth Checklist ###
- [ ] Review the GitHub Actions workflow that was triggered by the PR merge
- If no differences were found, note this in a comment.
- If all of the differences are expected, note this in a comment.
Include any details of how the review was performed.
- If unexpected differences are found, the following instructions can
help uncover potential explanations. If none of these apply and the
source of the differences cannot be determined, contact the
METplus wrappers lead engineer (@georgemccabe) for assistance.
- Search for other open issues that have the label `type: update truth`
applied by clicking on the label on this issue. Coordinate with the
author of these issues to ensure all diffs are properly reviewed.
- Check if any additional GitHub Actions testing workflows have been
triggered since the workflow that corresponds to this issue was run.
Review the latest run to ensure that there are no diffs that are
unrelated to this issue.
- If the incorrect differences are caused by the changes from the
issue that warranted this issue, consider reverting the PR and
re-opening the issue.
- Iterate until one of the above conditions apply.
- [ ] Approve the update of the truth data
- Contact the METplus wrappers lead engineer (@georgemccabe) or
backup lead (@jprestop) to let them know that the truth data can
be updated.
- [ ] Update the truth data.
This should be handled by a METplus wrappers engineer.
See the (instructions to update the truth data)[https://metplus.readthedocs.io/en/develop/Contributors_Guide/continuous_integration.html#update-truth-data-update-truth-data-yml]
for more info.
- [ ] Close this issue.
84 changes: 8 additions & 76 deletions docs/Contributors_Guide/add_use_case.rst
View file
Original file line number Diff line number Diff line change
Expand Up @@ -656,36 +656,11 @@ Trigger Input Data Ingest

If working in a forked METplus repository, the newly added input data will not
become available for the tests unless it is triggered from the dtcenter
repository. A METplus developer will need to run the following steps. Please
provide them with the name of the forked repository and the branch that will
be used to create the pull request with the new use case. In this example,
the branch feature_XYZ exists in the *my_fake_user/METplus* repository. First,
clone the *dtcenter/METplus* repository, the run the following::

git remote add my_fake_user https://github.com/my_fake_user/METplus
git checkout develop
git checkout -b feature_XYZ
git pull my_fake_user feature_XYZ
git push origin feature_XYZ
git remote remove my_fake_user

These commands will add a new remote to the forked repository, create a branch
off of the develop branch with the same name as the branch on the fork, pull
in the changes from the forked branch, then push the new branch up to
*dtcenter/METplus* on GitHub. Finally, the remote is removed to avoid clutter.

Once these steps have been completed, go to *dtcenter/METplus* on GitHub
in a web browser and navigate to the
`Actions tab <https://github.com/dtcenter/METplus/actions>`_.
Click on the job named
"Docker Setup - Update Data Volumes" then click on "Update Data Volumes" and
verify that the new data tarfile was found on the DTC web server and the new
Docker data volume was created successfully. See
:ref:`verify-new-input-data-was-found`. If the input data was ingested
properly, then delete the feature branch from *dtcenter/METplus*.
This will avoid
confusion if this branch diverges from the branch on the forked repository that
will be used in the final pull request.
repository. A METplus developer will need to run the
:ref:`cg-ci-update-input-test-data` GitHub Actions workflow to trigger it.
Please provide them with the name of the branch that will
be used to create the pull request with the new use case.


.. _add_use_case_to_test_suite:

Expand Down Expand Up @@ -1143,52 +1118,9 @@ Update the Truth Data
The addition of a new use case results in new output data. When this happens,
the reference branch needs to be updated so that future pull requests will
compare their results to a "truth" data set that contains the new files.
Create a pull request with "develop" as the source branch and "develop-ref" as
the destination branch. This is done so that the pull request number
responsible for the changes in the truth data can be referenced to easily
track where differences occurred.

A GitHub Action workflow is available to handle this step.

* Ensure that the develop data directory has been updated to include all of the
new input data.
Check with the reviewers of recent pull requests that add a new use case to
confirm that the steps under :ref:`update-the-develop-data-directory` have
been completed. If this step has not been completed, then the new use case(s)
will fail and the new output data will not be added to the truth data set.
* Navigate to https://github.com/dtcenter/METplus/actions/workflows/update_truth.yml
or from the METplus GitHub page, click on the Actions tab,
then click on "Update Truth Data" under menu on the left.
* Click on the "Run workflow" button on the right.
* Click on the Branch pull down and select "develop" unless you are updating
the truth data for a bugfix on a main_vX.Y branch.
* Enter the pull request numbers that warranted the update.
Include the '#' symbol before the number to create a link to the PR.
PRs from a repository other than METplus should include
the repository name before '#' symbol.
* Enter a brief summary of the changes.
Developers can navigate to the PRs for more information.

.. figure:: figure/update_truth_data.png

* Click the "Run workflow" button.
* A new workflow run should appear at the top of the list and complete quickly.
* Click on the "Pull Requests" tab.
A new pull request should have been created with the information that
was entered. Click on the new pull request.
* Verify that the information in this pull request is correct.
If the "develop" branch was selected in the "Run workflow" menu,
then the pull request should show **develop-ref <- develop**.
* Add the appropriate project and milestone values on the right hand side.
* Scroll to the bottom of the pull request and click "Squash and merge."
* Click "Confirm squash and merge." It is not necessary to wait for the
automation checks to complete for this step.
* Monitor the Testing automation run for the develop-ref branch and ensure that
all of the use cases run successfully and the final step named
"Create Output Docker Data Volumes" completed successfully.
* If any use cases fail, check that the input data has been updated following
the instructions under :ref:`update-the-develop-data-directory` and rerun
all of the jobs of the -ref workflow.

Follow the instructions for using the :ref:`cg-ci-update-truth-data` GitHub
Actions workflow to perform this step.


Clean Up DTC Web Server
Expand Down
100 changes: 100 additions & 0 deletions docs/Contributors_Guide/continuous_integration.rst
View file
Original file line number Diff line number Diff line change
Expand Up @@ -76,6 +76,106 @@ at the bottom of the workflow summary page when the workflow has completed.

.. figure:: figure/ci-doc-artifacts.png

.. _cg-ci-update-truth-data:

Update Truth Data (update_truth.yml)
------------------------------------

The METplus use case test truth data includes output from use cases that is
used to compare with new use case test results to flag any differences.
Differences can occur due to changes to the METplus wrappers
source code/configuration files or changes to any of its dependent
METplus components such as MET, METplotpy, METcalcpy, and METdataio.
Differences can also occur when a new use case is added, as the new use case
creates output that does not yet exist in the truth dataset.

Once all differences are confirmed to be expected,
the reference branch, e.g. develop-ref, needs to be updated. This triggers a
:ref:`cg-ci-testing-workflow` that runs all of the use cases, creates
Docker images with the new truth data, and pushes them to DockerHub.
This is done so that future pull requests will
compare their results to the updated truth dataset.

This process involves creating a pull request with "develop" as the source
branch and "develop-ref" as the destination branch.
This is done so that the pull request responsible for the changes in the
truth data can be referenced to easily track where differences occurred.

The **Update Truth Data** workflow is available to handle this step.

* Ensure that the develop data directory has been updated to include all of the
new input data.
Check with the reviewers of recent pull requests that add a new use case to
confirm that the steps under :ref:`update-the-develop-data-directory` have
been completed. If this step has not been completed, then the new use case(s)
will fail and the new output data will not be added to the truth data set.
* Navigate to https://github.com/dtcenter/METplus/actions/workflows/update_truth.yml
or from the METplus GitHub page, click on the Actions tab,
then click on "Update Truth Data" under menu on the left.
* Click on the "Run workflow" button on the right.
* Click on the Branch pull down and select "develop" unless you are updating
the truth data for a bugfix on a main_vX.Y branch.
* Enter the pull request numbers that warranted the update.
Include the '#' symbol before the number to create a link to the PR.
PRs from a repository other than METplus should include
the repository name before '#' symbol.
* Enter a brief summary of the changes.
Developers can navigate to the PRs for more information.

.. figure:: figure/update_truth_data.png

* Click the "Run workflow" button.
* A new workflow run should appear at the top of the list and complete quickly.
* Click on the "Pull Requests" tab.
A new pull request should have been created with the information that
was entered. Click on the new pull request.
* Verify that the information in this pull request is correct.
If the "develop" branch was selected in the "Run workflow" menu,
then the pull request should show **develop-ref <- develop**.
* Add the appropriate project and milestone values on the right hand side.
* If a GitHub issue exists to track the review of the differences, click on
the gear icon next to *Development* on the right side menu and add the issue.
* Scroll to the bottom of the pull request and click "Squash and merge."
* Click "Confirm squash and merge." It is not necessary to wait for the
automation checks to complete for this step.
* Monitor the Testing automation run for the develop-ref branch and ensure that
all of the use cases run successfully and the final step named
"Create Output Docker Data Volumes" completed successfully.
* If any use cases fail, check that the input data has been updated following
the instructions under :ref:`update-the-develop-data-directory` and rerun
all of the jobs of the -ref workflow.

.. _cg-ci-update-input-test-data:

Update Input Test Data (update_input_data.yml)
----------------------------------------------

New/updated input data for a METplus use case is read from the
DTC web server as described in the :ref:`use_case_input_data` section of the
**Adding Use Cases** chapter of the METplus Contributor's Guide.
This automatically happens as part of the :ref:`cg-ci-testing-workflow` when
a push event occurs on a dtcenter/METplus branch.
This step can be forced by using the **Update Input Test Data** workflow.

This is workflow is typically used when a new use case is being provided by
an external contributor and their pull request is coming from a forked
repository.
Only dtcenter/METplus workflows have permission to update the input test data.

To force the ingest of this input data, navigate to
https://github.com/dtcenter/METplus/actions/workflows/update_input_data.yml .
Click on the **Run workflow** pull-down, type the name of the branch that
matches the directory that contains the new data on the DTC web server,
and click the **Run workflow** button.

The value in the branch pull-down under the text that says
**Use workflow from** is ignored if there is a value typed for the branch name.
If the branch name exists in the dtcenter/METplus repository, leave the
branch name text box blank and select the branch name from the pull-down menu.

Verify that the workflow ran successfully and properly obtained the new data
by reviewing the log output from the workflow run.


Release Published (release_published.yml) - DEPRECATED
------------------------------------------------------
Expand Down

0 comments on commit 44335f3

Please sign in to comment.