Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Allow synteny data adapters to open files larger than 512Mb #3711

Merged
merged 2 commits into from
May 19, 2023

Conversation

cmdcolin
Copy link
Collaborator

This changes from string based parsing (which has a max string length of ~512Mb on chrome) to buffer based parsing (which can be up to maybe 2Gb or larger perhaps https://stackoverflow.com/a/8974841/2129219)

It may be that we need to improve performance in ways beyond just this PR to truly handle the larger files, but this would be a step towards enabling it

Fixes #3664

@github-actions github-actions bot added the needs label triage Needs a label to show in changelog (breaking, enhancement, bug, documentation, or internal) label May 18, 2023
@cmdcolin cmdcolin changed the title Allow opening files larger than 512Mb Allow synteny data adapters to open files larger than 512Mb May 18, 2023
@cmdcolin cmdcolin added enhancement New feature or request and removed needs label triage Needs a label to show in changelog (breaking, enhancement, bug, documentation, or internal) labels May 18, 2023
@codecov
Copy link

codecov bot commented May 18, 2023

Codecov Report

Merging #3711 (d3db247) into main (5448370) will increase coverage by 0.00%.
The diff coverage is 21.23%.

@@           Coverage Diff           @@
##             main    #3711   +/-   ##
=======================================
  Coverage   64.24%   64.24%           
=======================================
  Files         934      934           
  Lines       29981    29996   +15     
  Branches     7258     7265    +7     
=======================================
+ Hits        19261    19272   +11     
- Misses      10555    10559    +4     
  Partials      165      165           
Impacted Files Coverage Δ
...parative-adapters/src/ChainAdapter/ChainAdapter.ts 0.00% <0.00%> (ø)
...gins/comparative-adapters/src/ChainAdapter/util.ts 0.00% <0.00%> (ø)
...parative-adapters/src/DeltaAdapter/DeltaAdapter.ts 0.00% <0.00%> (ø)
...gins/comparative-adapters/src/DeltaAdapter/util.ts 0.00% <0.00%> (ø)
...tive-adapters/src/MashMapAdapter/MashMapAdapter.ts 0.00% <0.00%> (ø)
plugins/comparative-adapters/src/util.ts 89.65% <92.85%> (+2.98%) ⬆️
.../comparative-adapters/src/PAFAdapter/PAFAdapter.ts 90.24% <100.00%> (+0.83%) ⬆️
...lugins/comparative-adapters/src/PAFAdapter/util.ts 40.00% <100.00%> (-2.86%) ⬇️

... and 3 files with indirect coverage changes

📣 We’re building smart automated test selection to slash your CI/CD build times. Learn more

@cmdcolin cmdcolin merged commit 3ef1ed1 into main May 19, 2023
@cmdcolin cmdcolin deleted the large_file_synteny branch May 19, 2023 03:38
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Synteny views: Error: Data exceeds maximum string length (512Mb)
1 participant