chore: Generate between split repos and main #8442

lqiu96 · 2022-09-23T03:11:11Z

Following Tomo's script advice:

#!/bin/bash

set -e

modules=$(mvn help:evaluate -Dexpression=project.modules | grep '<.*>.*</.*>' | grep 'java' | sed -e 's/<.*>\(.*\)<\/.*>/\1/g')

for module in $modules; do
  echo "Running for ${module}"
  rm -rf "${module}"
  git clone "https://github.com/googleapis/${module}"
  rm -rf "${module}/.git"
  echo "Done running for ${module}"
done

./generation/delete_non_generated_samples.sh

echo "Done running script"

For now:

Ignore pom.xml (assuming they are renovate PRs) -- Can create PR to check only pom.xmls
Ignore CHANGELOG.md (from release-please) -- Might need to copy over from split-repos
Ignore README.md -- Can copy over from split repos
Ignore split repos renovate.json configs
Ignore samples/*
git diff --name-only origin/main | grep -v "pom.xml" | grep -v "CHANGELOG.md" | grep -v "README.md" | grep -v "renovate.json" | grep -v "versions.txt" | grep -v "samples/*" > diff.txt

Probably should ignore owlbot files and .github/*

Shows only the name of the files that have changed: diff.txt

Github has issue displaying the diffs. Might need to only add the files listed in the diff.txt

suztomo · 2022-09-23T14:45:04Z

How about limiting first 1000 diff files in the pull request? (We can't see thousands of files anyway)

Once the script is ready and merged to main, we want a GitHub Actions job that updates a branch periodically. We monitor a pull request from the branch to observe the changes.

suztomo · 2022-09-23T15:25:45Z

This PR reveals Java code diff (maybe manually-maintained tests?). Great.

lqiu96 · 2022-09-23T16:18:56Z

How about limiting first 1000 diff files in the pull request? (We can't see thousands of files anyway)

I didn't want to manually limit/exclude certain files. I've taken a look at a few of the diffs locally and I think quite a few of them can be ignored.

Ignoring these files should help us limit the number of diffs we need to check:

CHANGELOG.md (from release-please) -- Might need to copy over from split-repos
README.md -- Can copy over from split repos
renovate.json configs in each split repos
Ignore samples/* -- As we are no longer maintaining them and can look into them later
Ignore owlbot.yamls

Need to check:

The .java files (either manually or from owlbot)

The pom.xml files shouldn't have custom pom.xml, but I can do a separate and specific diff (either in a new PR or a seperate branch) to check each pom.xml. What do you think about the proposed ignored files above? @suztomo

Once the script is ready and merged to main, we want a GitHub Actions job that updates a branch periodically. We monitor a pull request from the branch to observe the changes.

I can look into a GH action once I flush out the script with the diffs that I want to show.

lqiu96 · 2022-09-23T16:34:32Z

Plan forward is two branches

Java only code (from owlbot changes and samples)
Non-Java code

Script will create two branches and create PRs for each branch that show the diffs

This reverts commit 8a616dc.

suztomo · 2022-09-23T18:09:13Z

We need 1 script generation/generate_diff.sh. The two jobs that run upon updating main branch:

1st GitHub Actions job

steps:
- run: |
  generation/generate_diff.sh
- run: |
  # Commit only Java code.
  git commit ./**/*.java -m 'chore: Java code diff from split repos'
- run: |
  git push -f --set-upstream origin split-repo-diff-non-java

2nd GitHub Actions job

steps:
- run: |
  generation/generate_diff.sh
- run: |
  # Commit non-Java code. How??
  git commit  ???????? -m 'chore: non-java diff from split repos'
- run: |
  git push -f --set-upstream origin split-repo-diff-non-java

suztomo · 2022-09-23T20:06:14Z

generation/generate_diff.sh

+git stash
+git checkout "${current_branch}"
+git stash pop
+
+git checkout -b "${diff_non_java_branch}"
+git add .


I like the use of git stash to separete Java change and non-Java changes.

Let me know if you have a better option. I couldn't think besides this that wouldn't require either two jobs or running the script twice.

suztomo · 2022-09-23T20:18:01Z

.github/workflows/generate_diffs.yaml

+    - name: Run generate-diff script
+      run: ./generation/generate_diff.sh
+      env:
+        USERNAME: ${{ github.actor }}


Did it work? If not YOSHI_CODE_BOT_TOKEN is available.

https://github.com/googleapis/google-cloud-java/blob/main/.github/workflows/googleapis_hermetic_sync.yaml#L41

Yep, #8448 and #8449 both have github-actions force-pushed...

suztomo · 2022-09-23T20:22:36Z

.github/workflows/generate_diffs.yaml

+  push:
+    branches:
+    - main
+  pull_request:


I assume you're testing. We don't need this job at pull request checks.

Yep! Just added in fa84ab5 so the CI only runs on non-PR push events (should be merge only)

It appears in the checks. Can you fix this?

Hmm, I'm not sure if I can prevent it from showing up on the checks. It seems to show up on a push event.

I can skip it from running all together.

Would you explain why you need this line 19 " pull_request:" ?

I can skip it from running all together.

Yes, please do so. Remove line 19.

Ah good catch! I've removed the check.

the job is appearing pull request checks

…ain-diff

lqiu96 · 2022-09-26T15:30:11Z

I'll merge this now to have it regenerate the diffs on merge. I'll create a separate PR to split the ITs and other Java code.

* chore: Generate diff from current split repos * chore: Call delete samples script afterwards * chore: Display diff on GH * chore: Push diffs to seperate branches * Revert "chore: Display diff on GH" This reverts commit 8a616dc. * chore: Create seperate branches * chore: Add push to multiple branches * chore: Stash the non staged changes * chore: Delete non-generated samples * chore: Add retry for git clone * chore: Remove untracked filed * chore: Run from main-diff branch * chore: Fix Typo * chore: cleanup branches * chore: Test if it can ci works * chore: Delete branch only if doesn't exist * chore: Set git username/email * chore: Use fetch-depth 0 to allow unshallow update * chore: Only run on merge * chore: Run action only on merge * chore: Remove run on pull_request

lqiu96 added 3 commits September 23, 2022 02:37

chore: Generate diff from current split repos

6bbe456

chore: Call delete samples script afterwards

2dc9964

chore: Display diff on GH

8a616dc

lqiu96 added 3 commits September 23, 2022 17:56

chore: Push diffs to seperate branches

8d74950

Revert "chore: Display diff on GH"

d2b37ac

This reverts commit 8a616dc.

chore: Create seperate branches

2977762

lqiu96 added 11 commits September 23, 2022 18:16

chore: Add push to multiple branches

1e47be8

chore: Stash the non staged changes

eb082d2

chore: Delete non-generated samples

aea3208

chore: Add retry for git clone

7758d31

chore: Remove untracked filed

3e559b5

chore: Run from main-diff branch

f2947db

chore: Fix Typo

76e00b8

chore: cleanup branches

0264eba

chore: Test if it can ci works

5657d28

chore: Delete branch only if doesn't exist

08ddf73

chore: Set git username/email

abb4b92

suztomo previously approved these changes Sep 23, 2022

View reviewed changes

suztomo reviewed Sep 23, 2022

View reviewed changes

chore: Use fetch-depth 0 to allow unshallow update

e2b6761

suztomo reviewed Sep 23, 2022

View reviewed changes

chore: Only run on merge

fa84ab5

suztomo reviewed Sep 23, 2022

View reviewed changes

lqiu96 marked this pull request as ready for review September 23, 2022 20:33

lqiu96 added the owlbot:run Add this label to trigger the Owlbot post processor. label Sep 23, 2022

gcf-owl-bot bot removed the owlbot:run Add this label to trigger the Owlbot post processor. label Sep 23, 2022

lqiu96 added 3 commits September 23, 2022 21:00

chore: Run action only on merge

7fa55ff

Merge branch 'main' of github.com:googleapis/google-cloud-java into m…

d2ee138

…ain-diff

chore: Remove run on pull_request

a14bffc

suztomo approved these changes Sep 23, 2022

View reviewed changes

lqiu96 merged commit d01ae48 into main Sep 26, 2022

lqiu96 deleted the main-diff branch September 26, 2022 15:40

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

chore: Generate between split repos and main #8442

chore: Generate between split repos and main #8442

lqiu96 commented Sep 23, 2022

suztomo commented Sep 23, 2022

suztomo commented Sep 23, 2022

lqiu96 commented Sep 23, 2022 •

edited

Loading

lqiu96 commented Sep 23, 2022

suztomo commented Sep 23, 2022 •

edited

Loading

suztomo Sep 23, 2022

lqiu96 Sep 23, 2022

suztomo Sep 23, 2022

lqiu96 Sep 23, 2022

suztomo Sep 23, 2022

lqiu96 Sep 23, 2022 •

edited

Loading

suztomo Sep 23, 2022

lqiu96 Sep 23, 2022 •

edited

Loading

suztomo Sep 23, 2022

suztomo Sep 23, 2022

lqiu96 Sep 23, 2022

lqiu96 commented Sep 26, 2022

chore: Generate between split repos and main #8442

chore: Generate between split repos and main #8442

Conversation

lqiu96 commented Sep 23, 2022

suztomo commented Sep 23, 2022

suztomo commented Sep 23, 2022

lqiu96 commented Sep 23, 2022 • edited Loading

lqiu96 commented Sep 23, 2022

suztomo commented Sep 23, 2022 • edited Loading

1st GitHub Actions job

2nd GitHub Actions job

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

lqiu96 Sep 23, 2022 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

lqiu96 Sep 23, 2022 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

lqiu96 commented Sep 26, 2022

lqiu96 commented Sep 23, 2022 •

edited

Loading

suztomo commented Sep 23, 2022 •

edited

Loading

lqiu96 Sep 23, 2022 •

edited

Loading

lqiu96 Sep 23, 2022 •

edited

Loading