Reduce memory consumption of graph (pipe sets)

[stefanhaller]

• PR Description

As a followup to #2533, reduce the memory consumption some more by optimizing the storage of the pipe sets used for the commit graph.

Some coarse measurements (taken by opening the respective repo, typing 4 >, and waiting for all commits to be loaded, and then reading the Memory column in Activity Monitor):

	master	this PR
git	2.5 GB	1.0 GB
our work repo	2.3 GB	1.3 GB
linux kernel	94.8 GB	38.0 GB

It's still not really usable for the linux kernel, but for all other repos that I come across in my daily use of lazygit, it works quite well now.

[codacy-production]

Coverage summary from Codacy

See diff coverage on Codacy

Coverage variation	Diff coverage
Report missing for aa688851	✅ 93.42%

Coverage variation details

	Coverable lines	Covered lines	Coverage
Common ancestor commit (aa68885)	Report Missing	Report Missing	Report Missing
Head commit (38ca388)	55874	48469	86.75%

Coverage variation is the difference between the coverage for the head and common ancestor commits of the pull request branch: <coverage of head commit> - <coverage of common ancestor commit>

Diff coverage details

	Coverable lines	Covered lines	Diff coverage
Pull request (#4498)	228	213	93.42%

Diff coverage is the percentage of lines that are covered by tests out of the coverable lines that the pull request added or modified: <covered lines added or modified>/<coverable lines added or modified> * 100%

See your quality gate settings    Change summary preferences

Footnotes

1. Codacy didn't receive coverage data for the commit, or there was an error processing the received data. Check your integration for errors and validate that your coverage setup is correct. ↩

[jesseduffield]

This is some substantial memory saving which is great. I'll need to test this out a bit because I'm worried about nil pointer exceptions.

It's a shame this change requires so many sweeping changes but I can't think of a way to not require sweeping changes of some form. I did some investigating into alternative approaches e.g.:

• using [20]byte instead of string (much better than string, but doesn't address duplicates)
• Having an immutable commit struct which itself lives in a pool. This would represent the actual commit and nothing independent of it like tags. Then each commit in the state can have a pointer to the corresponding immutable commit. You could have the immutable commit have its parents as just pointers to other immutable commits.
• Storing a reference to the pool inside the commit and using .Hash() to encapsulate how to resolve the actual string. Unfortunately means bloating each commit with an 8 byte pointer.

Did you have any thoughts on the above?

[stefanhaller]

It's a shame this change requires so many sweeping changes but I can't think of a way to not require sweeping changes of some form.

I did consider making the Hash property of Commit private (lowercase) and exposing a getter. When doing this in an extra commit, the actual change of turning it into a pointer would require much fewer changes. I didn't do that just because currently all fields are public, but I'm totally open to changing this either just for this field, or for all.

• Having an immutable commit struct which itself lives in a pool.

I don't see how this addresses the problem of the duplicate hash strings; unless you change the Pipes to have pointers to commits instead of pointers to hashes, is that what you meant? But then we need a map of commits by hash, so that we can look up the commit if we only have the hash. I'm not sure I like this direction.

• Storing a reference to the pool inside the commit and using .Hash() to encapsulate how to resolve the actual string. Unfortunately means bloating each commit with an 8 byte pointer.

I'm not sure what you mean by "resolve the actual string", and why you need a reference to the pool for that. The .Hash() getter would just return the pointer; that's basically what I said above.

[jesseduffield]

I did consider making the Hash property of Commit private (lowercase) and exposing a getter. When doing this in an extra commit, the actual change of turning it into a pointer would require much fewer changes. I didn't do that just because currently all fields are public, but I'm totally open to changing this either just for this field, or for all.

I'm not sure what you mean by "resolve the actual string", and why you need a reference to the pool for that. The .Hash() getter would just return the pointer; that's basically what I said above.

I meant that the .Hash() method could just return the string directly so that you didn't need to de-reference it afterwards. But that would require holding a reference to the pool. On reflection I don't actually think it's a good idea.

I don't see how this addresses the problem of the duplicate hash strings; unless you change the Pipes to have pointers to commits instead of pointers to hashes, is that what you meant? But then we need a map of commits by hash, so that we can look up the commit if we only have the hash. I'm not sure I like this direction.

Yep that's what I meant: a map of hash to commit. I like the idea because it would allow us to do some more stuff in memory without having to shell out to git but it's a big piece of work and I'm not clear on the overall value.

[stefanhaller]

I'm not sure what you mean by "resolve the actual string", and why you need a reference to the pool for that. The .Hash() getter would just return the pointer; that's basically what I said above.

I meant that the .Hash() method could just return the string directly so that you didn't need to de-reference it afterwards. But that would require holding a reference to the pool. On reflection I don't actually think it's a good idea.

Still don't get it. We'd have a hash *string field and a Hash() string method that dereferences the field and returns it. Why do we need the pool for that?

[stefanhaller]

When I said above "The .Hash() getter would just return the pointer", I meant the dereferenced pointer of course.

[jesseduffield]

Still don't get it. We'd have a hash *string field and a Hash() string method that dereferences the field and returns it. Why do we need the pool for that?

Ah yes, I mistakenly thought you'd need to access the pool to deference. Well in that case I actually very much like the idea of having a Hash() method.

[codacy-production]

Coverage summary from Codacy

See diff coverage on Codacy

Coverage variation	Diff coverage
Report missing for d87bdaf1	✅ 94.53%

Coverage variation details

	Coverable lines	Covered lines	Coverage
Common ancestor commit (d87bdaf)	Report Missing	Report Missing	Report Missing
Head commit (6ca627d)	55965	48566	86.78%

Coverage variation is the difference between the coverage for the head and common ancestor commits of the pull request branch: <coverage of head commit> - <coverage of common ancestor commit>

Diff coverage details

	Coverable lines	Covered lines	Diff coverage
Pull request (#4498)	274	259	94.53%

Diff coverage is the percentage of lines that are covered by tests out of the coverable lines that the pull request added or modified: <covered lines added or modified>/<coverable lines added or modified> * 100%

See your quality gate settings    Change summary preferences

Footnotes

1. Codacy didn't receive coverage data for the commit, or there was an error processing the received data. Check your integration for errors and validate that your coverage setup is correct. ↩

[stefanhaller]

@jesseduffield I reworked the branch completely, please have another look. I'm quite happy with this now.

[jesseduffield]

Massive improvement using the Hash() method. Nice one. Just one question

[jesseduffield]

It's a bit confusing to read that we're comparing a hash pointer with a hash, when in fact selectedCommitHash is not a hash: it's a hash pointer. Although that's clear enough when you're looking at the argument's type in the function signature, it's not clear when you read it in isolation e.g. in this line here. I think we should have a rule that if a variable contains a hash pointer, it should be end in HashPtr instead of just Hash.

Playing devils advocate against myself: often we pass around pointers to structs without needing to specify that it's a pointer in the variable name, so why the double standard when it comes to hashes? I think the answer is that scalar values (e.g. strings, integers, etc) are just expected not to be pointers and if they actually are pointers the default assumption is that they're only pointers for the sake of being possibly null.

Having said that, does it actually make a measurable difference to compare the pointers vs comparing the values themselves? Cos it would make the code simpler if we could fully encapsulate the pointer stuff inside the Hash() method and not have to sometimes pass hash pointers through functions.

[stefanhaller]

Very interesting discussion. At my workplace we have the convention to name all pointer variables with a "p" prefix, i.e. pSelectedCommitHash, this would solve it. Some of my coworkers object to that though, and find it redundant. I kind of like it myself, but I'm not proposing we adopt the convention in lazygit. 😄

But I'm happy to rename selectedCommitHash to selectedCommitHashPtr (see b438e07).

Having said that, does it actually make a measurable difference to compare the pointers vs comparing the values themselves? Cos it would make the code simpler if we could fully encapsulate the pointer stuff inside the Hash() method and not have to sometimes pass hash pointers through functions.

I would agree if equalHashes were a public function, but it's a private implementation detail, so I don't think it matters as much. And it is sometimes called with pipe.toHash as an argument, which is a pointer, so we would have to dereference it then; I don't feel this would be an improvement.

[jesseduffield]

But I'm happy to rename selectedCommitHash to selectedCommitHashPtr (see b438e07).

Looks good to me. I much prefer having it as a 'Ptr' suffix to a 'p' prefix.

I would agree if equalHashes were a public function, but it's a private implementation detail, so I don't think it matters as much. And it is sometimes called with pipe.toHash as an argument, which is a pointer, so we would have to dereference it then; I don't feel this would be an improvement.

Although that's true, RenderAux and RenderCommitGraph are public and they require a pointer for the sake of passing to equalHashes meaning we do get some blast radius from this performance (as we see in your commit). It's not so bad, but I hope there is a good performance gain we're getting in return.

[jesseduffield]

LGTM