java.lang.OutOfMemoryError: GC overhead limit exceeded

[wesyq]

Hi,

When converting come really large repos, when the count of objects reaches 400K-500K, I get this error:

Exception in thread "main" java.lang.OutOfMemoryError: GC overhead limit exceede
d
at sun.nio.cs.UTF_8.newDecoder(Unknown Source)
at org.eclipse.jgit.util.RawParseUtils.decode(RawParseUtils.java:1127)
at org.eclipse.jgit.util.RawParseUtils.decodeNoFallback(RawParseUtils.ja
va:1073)
at org.eclipse.jgit.util.RawParseUtils.decode(RawParseUtils.java:1033)
at org.eclipse.jgit.treewalk.TreeWalk.pathOf(TreeWalk.java:1291)
at org.eclipse.jgit.treewalk.AbstractTreeIterator.getEntryPathString(Abs
tractTreeIterator.java:462)
at git.lfs.migrate.GitConverter$4.getEntries(GitConverter.java:198)
at git.lfs.migrate.GitConverter$4.depends(GitConverter.java:221)
at git.lfs.migrate.Main.loadTaskGraph(Main.java:316)
at git.lfs.migrate.Main.processRepository(Main.java:166)
at git.lfs.migrate.Main.main(Main.java:86)

Disabling the GC overhead or using another GC algorithm gives the error about Heap size not enough. I keep increasing the heap size but I still keep getting it.

Thanks,
Waqas

[mmv]

I'm also hitting this problem.
I tried throwing more hardware into the problem but it only gets delayed and I'm unable to convert my repository.

[bozaro]

You can increase memory limit for Java by -Xmx option.
For example:

java -Xmx1024m -jar build/deploy/git-lfs-migrate.jar -s source-repo.git -d target-repo.git -l http://test:test@lfs-server/ "*.psd" "*.zip" "*.bin"

[wesyq]

I acutally did. Even increased it to 10 GB, but it didn't help!

[bozaro]

How big your original repository?

[wesyq]

Just the mirror was above 9 GB insize, I think. Yeah quite a big repository imported from SVN.

[bozaro]

Can you attach full output and command line to this issue?

[wesyq]

Actually I don't have that now. I deleted it as it was a while ago. I will try to recreate the error but cannot say exactly when!

[bozaro]

Looks like I reproduce this issue locally.

[wesyq]

Sorry I closed the issue by mistake :)

[mmv]

In my case I finally completed a conversion with -Xmx20g which actually requires 30GB of RAM to run because in Java8 there's a lot more going into RAM than the heap.

Regarding size of the repository, here's so you can have an idea of the numbers:

# Converted Repo

# Commit count
$ git log --all --pretty=oneline | wc -l
343626

# LFS objects produced
$ find lfs -type f | wc -l
261969

# Total disk size of LFS objects
$ du -sh lfs
130G    lfs

Original repo has a packed size of about 126 GB.

[bozaro]

I created issue #27: There are performance degradation on huge repositories (>250 000 objects).