Perf improvements #58
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This allows easier REPL testing and profiling.
Pretty small performance improvement, just 200-400ms on my 1.4mb input from ~10.4s to just under ~10s when running: (time (inner-main ["-t" "out/hmrc-rts-small-area.csv" "-u" "out/hmrc-rts-small-area.csv-metadata.json" "-m" "minimal" "-o" "cube.nt"])) However makes sense to avoid this work. We could certainly avoid more work on things like this, but it would be a bigger change.
On small to moderate inputs there is little difference in performance for this fix. e.g. the 4mb cn8.csv currently takes ~15.5 seconds to process with or without this change.
Another small perf advantage on hmrc-rts-small-area.csv ~1% faster
|
Thanks for checking @lkitching 🙇 |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
I've seen these changes improve real world pipeline performance somewhere between 30-50%.
By far the most significant performance increase is the fix for #56.
Avoiding the work of preping length validations when there aren't any makes is a very small improvement (maybe 200-400ms quicker on a 15s) input, but I think it's at least sensible to try and avoid the cost/complexity of a feature if we're not using it (though the approach here could be improved further, by compiling a function to do the specific validations earlier).
The fix for #59 is a very small performance improvement on inputs I've tested (barely noticeable) but should the interning scares me and may cause GC problems in large pipelines, so better to avoid this altogether.