Hashcons NormalizedFilePath values for efficient heap usage

[pepeiborra]

NormalizedFilePath is one of the biggest costs in terms of heap space. The solution adopted here relies on the MemoTrie package to create a global trie of normalised file paths. The trie is branched by dir name and not by character, which hopefully leads to a more compact representation.

The profiles below have been collected using an Edit experiment on a synthetic repo containing 300 empty modules with maximal imports, (every module imports all the modules below it).

BEFORE

AFTER

[wz1000]

The trie is branched by dir name and not by character, which hopefully leads to a more compact representation.

The HasTrie instance for lists looks like HasTrie x => HasTrie [x]. Won't this end up branching on characters anyway?

Perhaps you can store the filepath chunks as a ByteString and define a HasTrie instance for ByteString in terms of a hashmap.

[pepeiborra]

You are right, it will end up branching on the characters, which is a bit silly.

I implemented your suggestion here: pepeiborra@49120e8

However, the heap profile doesn't change. Either I've done something wrong, or it's likely that the heap cost is dominated by the interned FilePath values, and the trie is just a blip. We still may want the HashMap trie for faster performance though

[pepeiborra]

@wz1000 I don't think there's a material perf difference given most dir names are ~10 characters, and the cost of converting a String to Text is probably the same as 10 function calls in the Trie

[pepeiborra]

Ended up rolling my own internIO for simplicity. It would be better to intern every path component separately, I might give that a go tomorrow. But I'll need a more realistic benchmark, because now toNormalizedFilePath doesn't even show in the top 15:

[pepeiborra]

I tested several variations but didn't find any to be an overall improvement:

1. Represent the file path as Text
2. Represent the file path as path segments ([Text]) and intern the segments
3. Represent the uri as Maybe NormalizedUri

3 was a very minor improvement but not worth the extra complexity.

For niceness, I have move the intern to a smart constructor and made the real constructor abstract. I could add a pattern synonym instead if that's preferred?

[pepeiborra]

This is ready to merge and upload to Hackage please