Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

TASK-6722 - Variant Walker to enable user defined variant analysis #2522

Merged
merged 68 commits into from
Dec 20, 2024
Merged

TASK-6722 - Variant Walker to enable user defined variant analysis #2522

merged 68 commits into from
Dec 20, 2024

Conversation

j-coll
Copy link
Member

@j-coll j-coll commented Oct 21, 2024

No description provided.

@halender
Copy link
Contributor

Task linked: TASK-6722 Variant Walker to enable user defined variant analysis

j-coll added 23 commits October 24, 2024 18:58
@j-coll
storage: Ensure walker output is sorted. #TASK-6722
c5375ea
@j-coll
storage: Extract walker STDERR file from MR execution. #TASK-6722
663c03a
@j-coll
storage: Do not write multiple headers. #TASK-6722
154befa
@j-coll
storage: Fix NoSuchMethodError creating StopWatch. #TASK-6722
85aac6d
@j-coll
storage: Ensure stderr file is moved from scratch dir. #TASK-6722
697b08b
@j-coll
storage: Fix stderr sorting. #TASK-6722
356567e
@j-coll
storage: Write \n after the json header #TASK-6722
6253da3
@j-coll
storage: Do not interrupt header with empty records. #TASK-6722
5789628
@j-coll
storage: Add a custom Partitioner to ensure sorted data with multiple…
4ff0655 
… reducers #TASK-6722
@j-coll
storage: Fix partitioner. #TASK-6722
8268266
@j-coll
storage: Restart process when changing chromosome to ensure correct s…
4147d01 
…orting. #TASK-6722
@j-coll
storage: Fix GenomeHellper generateBootPreSplits. #TASK-6722
7fd439a
@j-coll
storage: Do not interrupt header with empty lines while concat. #TASK…
e6128b0 
…-6722
@j-coll
storage: Replace ImmutableBytesWritable with VariantLocusKey as map o…
100fecf 
…utput key. #TASK-6722
@j-coll
storage: Use VariantLocusKey and VariantLocusPartitioner in VariantEx…
0df69dc 
…portDirver. #TASK-6722
@j-coll
storage: Fix VariantLocusKey serialization. #TASK-6722
f6fd3d4
@j-coll
storage: Fix "Request body si too large" #TASK-6722
fa3c9f2
@j-coll
analysis: Do not try to close twice the same ERM. #TASK-6722
b528c03
@j-coll
storage: Do not use flush on outputstream. HADOOP-16548 #TASK-6722
96e5679
@j-coll
storage: Add VariantExporterDirectMultipleOutputsMapper to ensure sor…
bcd8185 
…ted export without reduce step. #TASK-6722
@j-coll
storage: Do not use reduce step on variant-walker. #TASK-6722
c4c3d3b
@j-coll
storage: Fix VariantRecordWriter bytes_written counter. #TASK-6722
0100097
@j-coll
storage: Reduce number of intermediate mapper files. #TASK-6722
b52ca27
j-coll added 19 commits November 28, 2024 17:11
@j-coll
storage: Fix NPE running SampleVariantStats #TASK-6722
48e1592
@j-coll
storage: Fix CustomPhoenixInputFormat generateSplit for first and las…
1d86756 
…t splits. #TASK-6722
@j-coll
analysis: Fix NPE at relatedness tool. #TASK-6722
5141031
@j-coll
Merge branch 'release-3.x.x' into TASK-6722
c48ce0a
@j-coll
cicd: Upload tests logs as artifacts. Reduce action log size. #TASK-6722
f2bc782
@j-coll
storage: Fix NPE at CohortVariantStatsDriver. #TASK-6722
dd684aa
@j-coll
cicd: Fix NPE. #TASK-6722
9795c6a
@j-coll
storage: Fix AIOOBE SampleVariantStatsDriver #TASK-6722
923651c
@j-coll
storage: Do not produce a .crc checksum file copying from hdfs. #TASK…
90010ac 
…-6722
@j-coll
storage: Improve docker process failure. Do not close the stdin twice…
9f326d9 
…. #TASK-6722
@j-coll
storage: Fix AIOOBE SampleVariantStatsDriver #TASK-6722
627e56a
@j-coll
storage: Do not produce a .crc checksum file copying from hdfs. #TASK…
98ce6f8 
…-6722
@j-coll
analysis: Do not use the scratchDir as intermediate folder for export…
14c07d9 
… and walk. #TASK-6722
@j-coll
storage: Improve collections usage in SampleVariantStatsDriver. #TASK…
050c1ee 
…-6722
@j-coll
analysis: Fix VariantAnalysisTest. #TASK-6722
a0c2a5f
@j-coll
app: Regenerate cli. #TASK-6722
3853c63
@j-coll
storage: Fix junit tests. #TASK-6722
eb61609
@j-coll
cicd: Increase "Publish Test Report on GitHub" memory #TASK-6722
54acc28
@j-coll
core: Fix NumberFormatException from IOUtils. #TASK-6722
4e96492
@j-coll j-coll requested a review from pfurio December 10, 2024 12:42
pfurio
pfurio previously approved these changes Dec 19, 2024
View reviewed changes
juanfeSanahuja
juanfeSanahuja reviewed Dec 20, 2024
View reviewed changes
Copy link
Member

@juanfeSanahuja juanfeSanahuja left a comment •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

In the compilation process with maven you should add --no-transfer-progress to avoid writing the progress logs of the download of the libraries every time

.github/workflows/test-analysis.yml
Copy link
Member

@juanfeSanahuja juanfeSanahuja Dec 20, 2024 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You should add --no-transfer-progress to avoid writing the progress logs of the download of the libraries every time

@j-coll j-coll merged commit bdccb74 into release-3.x.x Dec 20, 2024
48 of 58 checks passed
@j-coll j-coll deleted the TASK-6722 branch December 20, 2024 09:25
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@juanfeSanahuja juanfeSanahuja juanfeSanahuja left review comments

@pfurio pfurio pfurio approved these changes

Assignees
No one assigned
Labels
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
4 participants
@j-coll @halender @pfurio @juanfeSanahuja