Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

kpromo: set git clone depth for pr subcommand to prevent downloading the whole repository #1409

Merged
merged 1 commit into from
Oct 17, 2024

Conversation

chrischdi
Copy link
Member

@chrischdi chrischdi commented Oct 17, 2024

What type of PR is this?

/kind feature

What this PR does / why we need it:

This PR adds git clone options for cloning the kubernetes/k8s.io repository to only clone a depth of 1.

The kubernetes/k8s.io did grow in the past to a big (1,4GB to download) repository:

❯ git clone https://github.com/kubernetes/k8s.io.git
Cloning into 'k8s.io'...
remote: Enumerating objects: 211546, done.
remote: Counting objects: 100% (1397/1397), done.
remote: Compressing objects: 100% (881/881), done.
remote: Total 211546 (delta 428), reused 1294 (delta 359), pack-reused 210149 (from 1)
Receiving objects: 100% (211546/211546), 1.27 GiB | 6.65 MiB/s, done.
Resolving deltas: 100% (122622/122622), done.
Updating files: 100% (15059/15059), done.
❯ du -h -d 0 k8s.io
1.4G    k8s.io

This change should reduce the downloaded size from 1.27 GiB to currently ~29.29 MiB instead.

❯ git clone --depth 1 https://github.com/kubernetes/k8s.io.git
Cloning into 'k8s.io'...
remote: Enumerating objects: 17520, done.
remote: Counting objects: 100% (17520/17520), done.
remote: Compressing objects: 100% (9760/9760), done.
remote: Total 17520 (delta 8344), reused 14982 (delta 7181), pack-reused 0 (from 0)
Receiving objects: 100% (17520/17520), 29.29 MiB | 4.14 MiB/s, done.
Resolving deltas: 100% (8344/8344), done.
Updating files: 100% (15059/15059), done.
❯ du -h -d 0 k8s.io
142M    k8s.io

So this reduces the amount of data which is downloaded when running kpromo pr ... and by that also reduce the time for maintainers to run kpromo.

Which issue(s) this PR fixes:

Special notes for your reviewer:

Does this PR introduce a user-facing change?

kpromo pr: Only clone the head commit to reduce amount of data to download

@k8s-ci-robot k8s-ci-robot added release-note Denotes a PR that will be considered when it comes time to generate release notes. kind/feature Categorizes issue or PR as related to a new feature. labels Oct 17, 2024
@k8s-ci-robot k8s-ci-robot added area/artifacts Issues or PRs related to the hosting of release artifacts for subprojects area/release-eng Issues or PRs related to the Release Engineering subproject cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. sig/release Categorizes an issue or PR as relevant to SIG Release. size/XS Denotes a PR that changes 0-9 lines, ignoring generated files. labels Oct 17, 2024
Copy link
Member

@xmudrii xmudrii left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I agree, lately running kpromo pr takes way too much time.
/lgtm
/approve
/hold
@cpanato @saschagrunert do you want to take a look as well?

@k8s-ci-robot k8s-ci-robot added the do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. label Oct 17, 2024
@k8s-ci-robot k8s-ci-robot added lgtm "Looks good to me", indicates that a PR is ready to be merged. approved Indicates a PR has been approved by an approver from all required OWNERS files. labels Oct 17, 2024
@k8s-ci-robot
Copy link
Contributor

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: chrischdi, saschagrunert, xmudrii

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:
  • OWNERS [saschagrunert,xmudrii]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

Comment on lines +211 to +212
// Set a minimal depth to prevent downloading the whole repository.
Depth: 1,
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The question is if we break something, but yeah let's give it a try.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

That's why I tagged you and @cpanato. 😅 But given that this PR changes only the pr command, I think we can't break that much. And even if we do break it, we would probably realise it very quickly.

@xmudrii
Copy link
Member

xmudrii commented Oct 17, 2024

Let's give this a try!
/hold cancel

@k8s-ci-robot k8s-ci-robot removed the do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. label Oct 17, 2024
@k8s-ci-robot k8s-ci-robot merged commit 2cef208 into kubernetes-sigs:main Oct 17, 2024
8 checks passed
@BenTheElder
Copy link
Member

We should consider --filter=blob:none instead, that's what we use in prow these days.
It doesn't break git describe (unlike depth=1 and similar) but it does significantly reduce clone size.

https://github.blog/open-source/git/get-up-to-speed-with-partial-clone-and-shallow-clone/

@chrischdi
Copy link
Member Author

We should consider --filter=blob:none instead, that's what we use in prow these days. It doesn't break git describe (unlike depth=1 and similar) but it does significantly reduce clone size.

github.blog/open-source/git/get-up-to-speed-with-partial-clone-and-shallow-clone

Oh, this is neat!

I'll gonna try that and file a PR to change to that :-)

@chrischdi
Copy link
Member Author

Unfortunately: --filter=blob:none is not yet supported in the used library: go-git/go-git#713

@BenTheElder
Copy link
Member

BenTheElder commented Dec 18, 2024

Ah, we do clones via exec in prow ... it's kinda nice for debugging but does introduce an external runtime dependency ... (we use a container image with git)

... it does also mean you can use the canonical git implementation ...

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
approved Indicates a PR has been approved by an approver from all required OWNERS files. area/artifacts Issues or PRs related to the hosting of release artifacts for subprojects area/release-eng Issues or PRs related to the Release Engineering subproject cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. kind/feature Categorizes issue or PR as related to a new feature. lgtm "Looks good to me", indicates that a PR is ready to be merged. release-note Denotes a PR that will be considered when it comes time to generate release notes. sig/release Categorizes an issue or PR as relevant to SIG Release. size/XS Denotes a PR that changes 0-9 lines, ignoring generated files.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

5 participants