Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Use generators instead of lists and remove extra calls #34

Merged
merged 2 commits into from
Jan 16, 2024
Merged

Use generators instead of lists and remove extra calls #34

merged 2 commits into from
Jan 16, 2024

Conversation

krishnasism
Copy link
Contributor

  • In memory map, moved everything to use generators instead of lists
  • Removed extra calls that are not required, such as int(ceil... etc

@krishnasism
Copy link
Contributor Author

@callegarimattia this reduces the total memory of the function you were talking about from 900 MB to 700 MB although the total size (1.3GB) remains the same.

But with the Bible sample that you provided, I was able to shave off 2-3 seconds on average in this PR.

@krishnasism krishnasism marked this pull request as draft January 15, 2024 23:31
@krishnasism krishnasism marked this pull request as ready for review January 15, 2024 23:42
@krishnasism
Copy link
Contributor Author

krishnasism commented Jan 16, 2024 •
edited
Loading

@callegarimattia I identified one place we could optimise. In Span Map instead of storing all "HotCharacters" in the Span, we can only store the str values. This way we will save some memory.

Can you create an issue after merging this PR and we could work on it?

The function in question is: extract_spans - which you already flagged to be deprecated

callegarimattia
callegarimattia approved these changes Jan 16, 2024
View reviewed changes
Copy link
Contributor

@callegarimattia callegarimattia left a comment •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nice job cleaning up
good progress
Linking: #15

@callegarimattia callegarimattia merged commit e720997 into weareprestatech:main Jan 16, 2024
2 checks passed
This was referenced Jan 16, 2024
Open
Open
@callegarimattia callegarimattia added the enhancement New feature or request label Jan 16, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@callegarimattia callegarimattia callegarimattia approved these changes

Assignees
No one assigned
Labels
enhancement New feature or request
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
2 participants
@krishnasism @callegarimattia