Improve git change detection logic #174

SamTheisens · 2023-07-31T03:02:43Z

Attempts to make the change detection logic on a PR more robust.

Improve build status output, making it more instructive
Remove ad-hoc pull in "determine changes on branch", because it smells like voodoo cult. If the state of the branch is not the way it should be we should exit with a warning, not try to fix it and muddle on.
Allow for Jenkins to not be configured for scenarios in which another build tool is used
Allow for minimal clone scenarios in which the default branch is not present (Is it possible to checkout the whole history since the branch creation? actions/checkout#266 (comment))
Create repo command to optionally bring the branch into the required state for a PR

Also fixes a confusing problem in Jenkinsfile. The mpyl_config needs to be pulled from the Vandebron/mpyl_config repo. This created a branch called main in the local .git folder that pointed to a revision that doesn't exist in the Vandebron/mpyl repo. As a result we compare to a base that doesn't exist.
Now the file is curled in from via the github api instead of by checking out the repo.

@Jorg88, it looks like I can't edit the github actions in this repo anymore.
Suggesting to put this

New release `${{ steps.findPr.outputs.pr }}.${{ github.run_number }}` deployed at [Test Pypi](https://test.pypi.org/project/mpyl/). 
Install with `pip install -i https://test.pypi.org/simple/ --extra-index-url https://pypi.org/simple mpyl==${{ steps.findPr.outputs.pr }}.${{ github.run_number }}`

at line 156 of .github/workflows/build-package.yml

Test scenario

New branch gco -b feature/test-174
mpyl build status:

Branch not specified at build.versioning.branch in run_properties.yml. Branch determined via git: feature/test-174
Branch: feature/test-174 at 96b6be3d121ff1651506da3d6330f7df13c7a959
Make change to projects/service/src/sum.js

Branch: feature/test-174 at 96b6be3d121ff1651506da3d6330f7df13c7a959
Execution plan:
🏗️ nodeservice
📋 nodeservice
🚀 nodeservice
Commit change

Branch: feature/test-174 at e9ff18931070de4803da2190274d5fccb0362824
Execution plan:
🏗️ nodeservice
📋 nodeservice
🚀 nodeservice
Back to main branch
1. Make change to sbtservice on main and commit.
2. gco feature/test-174
3. git merge main
  Branch: feature/test-174 at 9215d9c863c1fbf842de46bda189c92fa2226716
  Execution plan:
  🏗️ nodeservice
  📋 nodeservice
  🚀 nodeservice

⚠️ Could not find ticket corresponding to `feature/TECH-XXX-fix-change-detection-logic. Does your branch name follow the correct pattern?

github-actions · 2023-07-31T03:04:48Z

Package	Line Rate	Health
src.mpyl	94%	➖
src.mpyl.cli	66%	❌
src.mpyl.projects	94%	➖
src.mpyl.reporting	100%	✔
src.mpyl.reporting.formatting	96%	➖
src.mpyl.stages	86%	➖
src.mpyl.steps	94%	➖
src.mpyl.steps.build	82%	➖
src.mpyl.steps.deploy	78%	❌
src.mpyl.steps.deploy.k8s	86%	➖
src.mpyl.steps.deploy.k8s.resources	96%	➖
src.mpyl.steps.postdeploy	100%	✔
src.mpyl.steps.test	100%	✔
src.mpyl.utilities	100%	✔
src.mpyl.utilities.cypress	93%	➖
src.mpyl.utilities.github	71%	❌
src.mpyl.utilities.jenkins	69%	❌
src.mpyl.utilities.junit	93%	➖
src.mpyl.utilities.logging	71%	❌
src.mpyl.utilities.pyaml_env	100%	✔
src.mpyl.utilities.repo	77%	❌
src.mpyl.utilities.s3	48%	❌
src.mpyl.utilities.sbt	96%	➖
src.mpyl.utilities.subprocess	78%	❌
Summary	84% (1996 / 2364)	➖

Minimum allowed line rate is 82%

github-actions · 2023-07-31T03:05:17Z

New release 174.2077 deployed at Test Pypi.
Install with

pip install -i https://test.pypi.org/simple/ --extra-index-url https://pypi.org/simple mpyl==174.2077

src/mpyl/cli/build.py

branch:

because it's too implicit and should be done in advance if at all branch:

branch:

e.g. `git clone --shallow-exclude main --single-branch --branch feature/test-invalidation-logic https://github.com/SamTheisens/mpyl-example-gha.git` in which case there will be no default branch, but the first commit is the base from which the branch was created branch:

branch:

and not all refs, including tags branch:

branch:

this should always be origin/main. If a local `main` is merged into the current branch, that should not supersede the original base branch:

branch:

to bring the branch into a state such that PRs can be built branch:

branch: feature/TECH-490-migrate-mpyl-to-python-3.11

with basic implementation of status logic branch:

branch:

Jorg88

Nice solution I would say Sam!

src/mpyl/cli/repository.py

src/mpyl/utilities/repo/__init__.py

branch:

koeves · 2023-08-08T14:23:46Z

I tried running this test version with a real PR in jenkins, it got to

git.exc.GitCommandError: Cmd('git') failed due to: exit code(128)
  cmdline: git describe --tags a6fc2e5d433165644ab29ab1283441dbb9e54b8a
  stderr: 'fatal: No names found, cannot describe anything.'

then based on Jorg's advice added the following to the jenkinsfile:

sh "pipenv run mpyl repo status"
sh "pipenv run mpyl repo init"

which got stuck on Initializing PR-11178...

Maybe it again waits for input in the agent?

EDIT if I try to clone the repo on the agent it asks for the username

Jorg88 · 2023-08-08T14:26:27Z

src/mpyl/cli/commands/repository/__init__.py

@@ -0,0 +1 @@
+"""CVS repository related command group"""


did you add the empty init.py file below (I can't comment on it) on purpose btw?

branch:

because it is not secure branch:

branch:

SamTheisens force-pushed the feature/TECH-XXX-fix-change-detection-logic branch from 8649619 to df5792c Compare July 31, 2023 08:06

Jorg88 self-requested a review July 31, 2023 10:38

SamTheisens force-pushed the feature/TECH-XXX-fix-change-detection-logic branch 2 times, most recently from 25a3712 to 390c5d2 Compare August 1, 2023 05:52

SamTheisens marked this pull request as ready for review August 1, 2023 06:11

SamTheisens changed the title ~~feature/TECH XXX fix change detection logic~~ Improve git change detection logic Aug 1, 2023

SamTheisens requested a review from koeves August 1, 2023 06:20

SamTheisens force-pushed the feature/TECH-XXX-fix-change-detection-logic branch 6 times, most recently from 8d9445a to 7235318 Compare August 2, 2023 12:04

Jorg88 reviewed Aug 4, 2023

View reviewed changes

src/mpyl/cli/build.py Outdated Show resolved Hide resolved

SamTheisens force-pushed the feature/TECH-XXX-fix-change-detection-logic branch 2 times, most recently from 17fe65a to 76bcb53 Compare August 7, 2023 11:08

SamTheisens requested a review from Jorg88 August 7, 2023 11:09

SamTheisens added 11 commits August 7, 2023 19:09

Improve status output

2f29280

branch:

Remove ad-hoc pull

5c6e7f2

because it's too implicit and should be done in advance if at all branch:

Remove status line when just showing plan

ea0d9a5

branch:

Make jenkins config fully optional

aceacd2

branch:

Check changes regardless of whether main was pulled

f3f5052

branch:

Continue until we hit base

88f2aca

branch:

Honor verbosity setting for running builds as well

86e0cc5

branch:

Fix reference to jenkins config

5492c6d

branch:

Pull via request instead of checkout

b3c2351

branch:

Remove unused definitions

915930f

branch:

SamTheisens and others added 10 commits August 7, 2023 19:09

Consider just heads (branches)

bf78eb6

and not all refs, including tags branch:

Add animation about autocompletion

cdd2724

branch:

Add source reference

5d60562

branch:

Simplify logic for determining branch base

42738a4

this should always be origin/main. If a local `main` is merged into the current branch, that should not supersede the original base branch:

Remove dead code

3590092

branch:

Don't determine build status if we can't diff with main

19cdc53

branch:

Add fix-pr parameter to status

e94069a

to bring the branch into a state such that PRs can be built branch:

[TECH-490] Update the gha test version install message

c53c0c3

branch: feature/TECH-490-migrate-mpyl-to-python-3.11

Add repo command

f1db74d

with basic implementation of status logic branch:

Move repo init logic to separate command

d2e5aaf

branch:

SamTheisens force-pushed the feature/TECH-XXX-fix-change-detection-logic branch 3 times, most recently from ceade9a to 4e6ff4e Compare August 8, 2023 04:25

Jorg88 approved these changes Aug 8, 2023

View reviewed changes

src/mpyl/cli/repository.py Outdated Show resolved Hide resolved

src/mpyl/utilities/repo/__init__.py Outdated Show resolved Hide resolved

koeves reviewed Aug 8, 2023

View reviewed changes

src/mpyl/utilities/repo/__init__.py Outdated Show resolved Hide resolved

Add command to help

aef1a39

branch:

SamTheisens force-pushed the feature/TECH-XXX-fix-change-detection-logic branch from 6f90c03 to aef1a39 Compare August 8, 2023 09:08

Jorg88 reviewed Aug 8, 2023

View reviewed changes

SamTheisens added 2 commits August 9, 2023 13:43

Only include credentials in URL when present

46d5c85

branch:

Prevent process from stalling waiting for user input

d71df40

branch:

SamTheisens force-pushed the feature/TECH-XXX-fix-change-detection-logic branch 2 times, most recently from f9dd62b to cfdd777 Compare August 9, 2023 08:31

Remove possibility to include credentials in url

c77a4bc

because it is not secure branch:

SamTheisens force-pushed the feature/TECH-XXX-fix-change-detection-logic branch from cfdd777 to c77a4bc Compare August 9, 2023 08:46

Remove askpass

1b038be

branch:

SamTheisens force-pushed the feature/TECH-XXX-fix-change-detection-logic branch from c9969a4 to 1b038be Compare August 10, 2023 02:59

koeves merged commit ab3a8b4 into main Aug 10, 2023

koeves deleted the feature/TECH-XXX-fix-change-detection-logic branch August 10, 2023 08:06

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Improve git change detection logic #174

Improve git change detection logic #174

SamTheisens commented Jul 31, 2023 •

edited

Loading

github-actions bot commented Jul 31, 2023 •

edited

Loading

github-actions bot commented Jul 31, 2023 •

edited

Loading

Jorg88 left a comment

koeves commented Aug 8, 2023 •

edited

Loading

Jorg88 Aug 8, 2023

Improve git change detection logic #174

Improve git change detection logic #174

Conversation

SamTheisens commented Jul 31, 2023 • edited Loading

Test scenario

⚠️ Could not find ticket corresponding to `feature/TECH-XXX-fix-change-detection-logic. Does your branch name follow the correct pattern?

github-actions bot commented Jul 31, 2023 • edited Loading

github-actions bot commented Jul 31, 2023 • edited Loading

Jorg88 left a comment

Choose a reason for hiding this comment

koeves commented Aug 8, 2023 • edited Loading

Jorg88 Aug 8, 2023

Choose a reason for hiding this comment

SamTheisens commented Jul 31, 2023 •

edited

Loading

github-actions bot commented Jul 31, 2023 •

edited

Loading

github-actions bot commented Jul 31, 2023 •

edited

Loading

koeves commented Aug 8, 2023 •

edited

Loading