[PGA] fix stars handling, reduce memory #80
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
vmarkovtsev
reviewed
Sep 19, 2018
| commaPos2 = strings.LastIndex(line[:commaPos2], ",") | ||
| commaPos1 := strings.LastIndex(line[:commaPos2], ",") | ||
| deletedFlag := line[commaPos1+1 : commaPos2] | ||
| if deletedFlag != "0" { |
There was a problem hiding this comment.
How about the number of stars? If a repo is deleted, does it lose all the stars and the new one starts with 0?
Maybe we should take the maximum from the old and the new stars here.
There was a problem hiding this comment.
Yes, deleted repositories do not keep their stars if re-created. I think getting actual stars is the least surprising one, since it accurately reflects current status. This is particularly important for cases where the re-created repository is not the main one anymore, but a fork.
vmarkovtsev
approved these changes
Sep 19, 2018
- On repositories, skip every deleted repository. This means we have no more duplicated project entries anymore. - ID deduplication logic removed from writeWatchers. This also means reduced memory usage. - Use uint32 instead of int to hold project IDs and stars in memory. - Use gzip for stars file. Signed-off-by: Santiago M. Mola <santi@mola.io>
smola
changed the title
campoy
reviewed
Oct 8, 2018
| @@ -70,71 +70,78 @@ func reduceWatchers(stream io.Reader) map[int]int { | |||
| if err != nil { | |||
There was a problem hiding this comment.
instead of Atoi you should use strconv.ParseUint(line[:commaPos], 10, 32) to detect any overflow
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
pga-create: fix logic for deleted repositories, reduce mem
On repositories, skip every deleted repository. This
means we have no more duplicated project entries anymore.
ID deduplication logic removed from writeWatchers. This
also means reduced memory usage.
Use uint32 instead of int to hold project IDs and stars in memory.
Use gzip for stars file.