Cannot remove files from repository above quota

[mikebolt]

Select Topic Area

Question

Body

I am unable to push anything to a public repository I created because it hit the size quota. I want to remove about 10% of the data while I work on compressing it. I could manually delete files in the GitHub Web UI, but there are two problems with that. One problem is that it would take a long time. The other problem is that even after I deleted a few files this way, I was unable to make any kind of push with new data, even a few bytes. I also could not force push a commit with --amend that removed files. So it seems that I have no way to get this repository back under the quota. I might just start over. Any suggestions?

[SMelidoni]

Hi @mikebolt 👋🏽,

It seems like you've reached the repository size limit on GitHub. To resolve this issue, you can reduce the repository size by removing the large files from the history and then force pushing the changes. Here's a step-by-step guide to help you achieve this:

1. Clone the repository: Clone the repository to your local machine, if you haven't done so already.

   git clone https://github.com/your-username/your-repo.git
cd your-repo

2. Find large files: Use git-lfs-migrate (part of Git LFS) or the BFG Repo-Cleaner to find and remove large files from your repository's history.

   With git-lfs-migrate, you can find large files by running:
   git lfs migrate info --top=10

   This command will show you the top 10 largest files in your repository.

3. Remove large files: Use git filter-branch or the BFG Repo-Cleaner to remove the large files from your repository's history.

   Before you proceed with removing large files, it's essential to evaluate the impact of their removal on your project. Here are a few recommendations:

   i. Backup: Before making any changes, create a backup of your repository or the large files in question. This way, you can restore them if needed.

   ii. Evaluate necessity: Assess the importance of the large files for your project. If they are critical to your project's functionality, you might want to consider other options, such as using Git Large File Storage (LFS) to store large files outside of your repository.

   iii. Split the repository: If the large files are part of a specific module or component, consider splitting your repository into multiple repositories. This way, you can keep the large files separate from the main project, reducing the overall repository size.

   iv. Move large files to external storage: If the large files are not critical for your project's functionality or can be fetched during runtime, consider moving them to external storage (e.g., cloud storage or a CDN) and fetch them as needed.

   Continue with caution:

   3.1 Using git filter-branch:

git filter-branch --index-filter "git rm -rf --cached --ignore-unmatch path/to/large_file" HEAD

Replace path/to/large_file with the path of the file you want to remove.

3.2 Using BFG Repo-Cleaner:

bfg --delete-files path/to/large_file

Replace path/to/large_file with the path of the file you want to remove.

5. Clean up and reclaim space:

git reflog expire --expire=now --all && git gc --prune=now --aggressive

6. Force push the changes:

git push origin --force --all
git push origin --force --tags

After completing these steps, your repository should be under the size limit, and you should be able to push new commits. If you still face issues, you might consider creating a new repository and manually adding the necessary files, while making sure to avoid adding the large files that caused the problem initially.

I hope this helps! Let me know if you have any questions or need further assistance.

[mikebolt]

Thanks for the quick reply.

I followed these steps exactly with git filter-branch but I did not use the --aggressive option for git gc. With the aggressive option it took about 1 day when I first tried it.
Everything seems to work until the push command.
This is the error:

remote: fatal: pack exceeds maximum allowed size (2.00 GiB)
error: remote unpack failed: index-pack failed
To github.com:mikebolt/primeland-files.git
 ! [remote rejected]     main -> main (failed)
error: failed to push some refs to 'github.com:mikebolt/primeland-files.git'

I'm going to make a new repo, but I'm curious how to fix this error in case I run into something like this in the future.

[SMelidoni]

Hi @mikebolt,

It seems like your repository still contains large files or packs that exceed the maximum allowed size on GitHub. If the git filter-branch method didn't work as expected, you could try the BFG Repo-Cleaner approach, which is designed specifically for cleaning up repositories with large files.

Here's a step-by-step guide to using BFG Repo-Cleaner:

1. Install the BFG Repo-Cleaner: You can download the BFG Repo-Cleaner from its official website: https://rtyley.github.io/bfg-repo-cleaner/.

2. Clone your repository using the --mirror option:

git clone --mirror https://github.com/your-username/your-repo.git
cd your-repo.git

3. Use BFG Repo-Cleaner to remove large files:

bfg --strip-blobs-bigger-than 100M

This command will remove all blobs larger than 100 MB from the repository's history. You can adjust the size limit as needed.

4. Clean up and reclaim space:

git reflog expire --expire=now --all && git gc --prune=now --aggressive

5. Force push the changes:

git push --force

After completing these steps, your repository should hopefully be under the size limit, and you should be able to push new commits. If you still face issues, creating a new repository (like you mentioned) and manually adding the necessary files while making sure to avoid adding the large files that caused the problem initially is the best option.

In the future, to avoid similar issues, consider using Git LFS (Large File Storage) for storing large files outside of your repository, which will keep your repository size manageable.

I hope this helps! If you have any questions or need further assistance, please let me know.

[mikebolt]

Thanks for the help again. I made the new repo at https://github.com/mikebolt/primeland-files

I actually ran into this issue again and used these steps to remove some files that were over 1GB. It worked without any problems, but the repo was not close to the 100GB limit. So, I think there is something else that needs to be done for repos near the size limit.

[github-actions[bot]]

🕒 Discussion Activity Reminder 🕒

This Discussion has been labeled as dormant by an automated system for having no activity in the last 60 days. Please consider one the following actions:

1️⃣ Close as Out of Date: If the topic is no longer relevant, close the Discussion as out of date at the bottom of the page.

2️⃣ Provide More Information: Share additional details or context — or let the community know if you've found a solution on your own.

3️⃣ Mark a Reply as Answer: If your question has been answered by a reply, mark the most helpful reply as the solution.

Note: This dormant notification will only apply to Discussions with the Question label. To learn more, see our recent announcement.

Thank you for helping bring this Discussion to a resolution! 💬