Add support for writing Anopheles SNP data to the plink binary file format #515

tristanpwdennis · 2024-03-26T11:01:53Z

This PR adds a new function biallelic_snps_to_plink() to the Ag3 and Af1 APIs which allows for selection of a set of biallelic SNPs for export to the plink binary file format. This then allows for data to be imported into tools like ADMIXTURE which can read the plink format.

Resolves #248.

review-notebook-app · 2024-03-26T11:01:58Z

Check out this pull request on

See visual diffs & provide feedback on Jupyter Notebooks.

Powered by ReviewNB

sanjaynagi · 2024-03-26T13:19:42Z

Hey Tristan. Nice work!

Ill save comments for now but FYI - when you add notebooks to malariagen_data, make sure you have cleared all outputs, otherwise they can become quite hefty in size and then the repo balloons in size over time (all of it is stored in git history).

…s, avoids using dask

…variant_allele mapping

tristanpwdennis · 2024-04-11T10:58:50Z

I've found the source of the AssertionError (also see issue #516) - something to do with how dask.array.map_blocks computes variant_allele at line 1629 of snp_data.py.

I haven't managed to get to the bottom of it yet but in this PR there's a temporary fix that just applies apply_allele_mapping to an in-memory np array of variant_allele, and I've now added biallelic_snp_calls to to_plink.py instead of calling snp_calls and thinning them manually.

Removed numba decorator for the apply_allele_mapping function, for now

tristanpwdennis · 2024-08-08T08:32:57Z

I've (I hope!) made a fix to the above error (issue #516), and described it in more detail there

jonbrenas · 2024-08-08T14:11:20Z

I think it should work. Feel free to mark it as "Ready for review" when you think that it is appropriate.

jonbrenas

Hi @tristanpwdennis. Before we merge this PR, could I ask you to do some clean-up?
test-1.ipynb needs to be removed and you made some changes to .gitignore and test_snp_data.py. There are also quite a few print commands that need to be removed or switched to debug mode.
Could you also add a test that checks (at least) that the file is created?

jonbrenas · 2024-09-11T08:24:12Z

malariagen_data/anoph/to_plink.py

+        # Filter SNPs for segregating sites only
+        with self._spinner("Subsetting to segregating sites"):
+            gt = ds_snps["call_genotype"].data.compute()
+            print("count alleles")


Can you remove this print statement?

jonbrenas · 2024-09-11T08:24:27Z

malariagen_data/anoph/to_plink.py

+            print("count alleles")
+            with ProgressBar():
+                ac = allel.GenotypeArray(gt).count_alleles(max_allele=3)
+                print("ascertain segregating  sites")


Same thing here.

jonbrenas · 2024-09-11T08:24:43Z

malariagen_data/anoph/to_plink.py

+                    & (ac[:, 0] <= max_ref_ac)
+                    & (an_missing <= max_missing_an)
+                )
+                print(f"ascertained {np.count_nonzero(loc_sites):,} sites")


Same thing here.

jonbrenas · 2024-09-11T08:24:50Z

malariagen_data/anoph/to_plink.py

+                print(f"ascertained {np.count_nonzero(loc_sites):,} sites")
+
+        # Set up dataset with required vars for plink conversion
+        print("Set up dataset")


Same thing here.

jonbrenas · 2024-09-11T08:26:07Z

tests/anoph/test_snp_data.py

@@ -88,6 +88,7 @@ def test_open_snp_sites(fixture, api: AnophelesSnpData):
        assert "variants" in contig_grp
        variants = contig_grp["variants"]
        assert "POS" in variants
+        assert False


Not sure what this is here for.

jonbrenas · 2024-09-11T08:29:10Z

Thank you very much @tristanpwdennis. This is great. I added a few comments where there are still some print statement that need to be removed. There is also still a .png file that should be removed. Please feel free to ask if you have a question or if some of the requested changes do not make sense to you.

tristanpwdennis · 2024-09-11T08:30:40Z

Hi Jon,
Thanks! Will tidy further and add the test. Cheers :)

tristanpwdennis · 2024-09-19T04:34:47Z

Hi @jonbrenas, I had a tidy and removed some redundant code from to_plink.py. I also added a test (test_plink_converter.py) to make sure the files are created. Let me know how everything looks & if this is sufficient, or if I can add any more tests. Hope this works ok!
Thanks
-t

alimanfoo

Hi @tristanpwdennis, all checks are passing in the shadow PR so we're looking great here and almost ready to merge!

Just a couple more things here to add documentation for the new function...

malariagen_data/anoph/to_plink.py

tests/anoph/test_plink_converter.py

alimanfoo · 2024-11-21T12:19:47Z

Btw have taken the liberty to edit the PR title and description just so this PR will look good in the release notes :)

Co-authored-by: Alistair Miles <alimanfoo@googlemail.com>

…anpwdennis/malariagen-data-python into plink-converter-2024-03-26

…fs and docstring

tristanpwdennis · 2024-11-25T00:40:22Z

Addressed comments and feeling the urge to merge

alimanfoo

Thanks @tristanpwdennis, noticed a couple more small things. Promise then I'll stop nit picking! If we could make these changes, then get all the checks to pass over in the shadow PR, happy to merge.

malariagen_data/anoph/plink_params.py

malariagen_data/anoph/to_plink.py

Co-authored-by: Alistair Miles <alimanfoo@googlemail.com>

tristanpwdennis · 2024-11-26T22:49:49Z

Accepted corrections :)

jonbrenas · 2024-11-27T08:28:50Z

Thanks @tristanpwdennis. You probably saw that the tests failed. It looks like the tmp_path used during the tests ends up being a pathlib.PosixPath instead of the required str.

tristanpwdennis · 2024-11-27T08:53:11Z

EDIT: I see this now on github, let me try to fix

alimanfoo

All looks good to me, happy to be merged once checks have passed.

alimanfoo · 2024-12-02T17:10:27Z

Wooohoo!

tristanpwdennis · 2024-12-02T22:29:49Z

Hooray, thanks guys! I wonder if I get a GH badge for being the longest open PR on a given repo...

add plink converyt function, faff around with some tests

ea54da7

tristanpwdennis marked this pull request as draft March 26, 2024 11:02

Tristan Dennis added 2 commits April 11, 2024 09:44

temp fix for avoiding error in mapping all-allele -> biallelic allele…

6afd608

…s, avoids using dask

updated plink converter to use biallelic_snp_calls with tmp. amended …

3268640

…variant_allele mapping

tristanpwdennis and others added 3 commits August 8, 2024 08:22

Merge branch 'master' into plink-converter-2024-03-26

ab6f28c

Added chunking of allele_mapping array so that da.map blocks works

5e502bf

Removed numba decorator for the apply_allele_mapping function, for now

numba works fine

31f6469

clear up notebook

4ef5e9c

tristanpwdennis marked this pull request as ready for review August 9, 2024 07:12

alimanfoo mentioned this pull request Aug 9, 2024

'AssertionError' when trying to return 'variant_allele' from biallelic_snp_calls() #516

Closed

alimanfoo mentioned this pull request Aug 19, 2024

Add support for writing Anopheles SNP data to the plink binary file format (shadow PR) #584

Merged

Merge branch 'master' into plink-converter-2024-03-26

98767fc

jonbrenas requested changes Aug 20, 2024

View reviewed changes

remove dud notebook, tidy up to_plink.py

af47433

jonbrenas reviewed Sep 11, 2024

View reviewed changes

Tristan Dennis added 3 commits September 19, 2024 10:47

add plink converter test

96eb157

update plink converter test to include test for created files

98700e3

tidy, add tests, refactor snp_data call

6db9db8

update notebook to fix output dir and relative output path

7d10281

alimanfoo reviewed Nov 21, 2024

View reviewed changes

alimanfoo changed the title ~~Add plink converter function~~ Add support for writing Anopheles SNP data to the plink binary file format Nov 21, 2024

tristanpwdennis and others added 9 commits November 25, 2024 09:49

Update malariagen_data/anoph/to_plink.py

38b9510

Co-authored-by: Alistair Miles <alimanfoo@googlemail.com>

add bool type definition

b48abf4

Merge branch 'plink-converter-2024-03-26' of https://github.com/trist…

cc5ed2d

…anpwdennis/malariagen-data-python into plink-converter-2024-03-26

add docstring

6730123

fix function definitions

92e8340

add overwrite to base params and finish docstring

fb08e1c

fix docstring and base params

dab86ec

update output dir type def

033fc8a

make plink_params.py containing overwrite and output dir, fix type de…

c3d7118

…fs and docstring

alimanfoo reviewed Nov 26, 2024

View reviewed changes

tristanpwdennis and others added 5 commits November 27, 2024 08:47

Update malariagen_data/anoph/plink_params.py

d235b76

Co-authored-by: Alistair Miles <alimanfoo@googlemail.com>

Update malariagen_data/anoph/to_plink.py

c62307c

Co-authored-by: Alistair Miles <alimanfoo@googlemail.com>

Update malariagen_data/anoph/to_plink.py

4605612

Co-authored-by: Alistair Miles <alimanfoo@googlemail.com>

Update malariagen_data/anoph/to_plink.py

23d8c3a

Co-authored-by: Alistair Miles <alimanfoo@googlemail.com>

Update malariagen_data/anoph/to_plink.py

c562106

Co-authored-by: Alistair Miles <alimanfoo@googlemail.com>

coerce tmp_path from posix to str

de0b36c

alimanfoo approved these changes Nov 27, 2024

View reviewed changes

Merge branch 'master' into plink-converter-2024-03-26

31abfea

jonbrenas merged commit b7af935 into malariagen:master Dec 2, 2024

alimanfoo added the BMGF-068808 Work supported by BMGF grant INV-068808 (MalariaGEN 2024-2027). label Dec 4, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add support for writing Anopheles SNP data to the plink binary file format #515

Add support for writing Anopheles SNP data to the plink binary file format #515

tristanpwdennis commented Mar 26, 2024 •

edited by alimanfoo

Loading

review-notebook-app bot commented Mar 26, 2024

sanjaynagi commented Mar 26, 2024

tristanpwdennis commented Apr 11, 2024

tristanpwdennis commented Aug 8, 2024

jonbrenas commented Aug 8, 2024

jonbrenas left a comment

jonbrenas Sep 11, 2024

jonbrenas Sep 11, 2024

jonbrenas Sep 11, 2024

jonbrenas Sep 11, 2024

jonbrenas Sep 11, 2024

jonbrenas commented Sep 11, 2024

tristanpwdennis commented Sep 11, 2024

tristanpwdennis commented Sep 19, 2024 •

edited

Loading

alimanfoo left a comment

alimanfoo commented Nov 21, 2024

tristanpwdennis commented Nov 25, 2024

alimanfoo left a comment

tristanpwdennis commented Nov 26, 2024

jonbrenas commented Nov 27, 2024 •

edited

Loading

tristanpwdennis commented Nov 27, 2024 •

edited

Loading

alimanfoo left a comment

alimanfoo commented Dec 2, 2024

tristanpwdennis commented Dec 2, 2024

Add support for writing Anopheles SNP data to the plink binary file format #515

Add support for writing Anopheles SNP data to the plink binary file format #515

Conversation

tristanpwdennis commented Mar 26, 2024 • edited by alimanfoo Loading

review-notebook-app bot commented Mar 26, 2024

sanjaynagi commented Mar 26, 2024

tristanpwdennis commented Apr 11, 2024

tristanpwdennis commented Aug 8, 2024

jonbrenas commented Aug 8, 2024

jonbrenas left a comment

Choose a reason for hiding this comment

jonbrenas Sep 11, 2024

Choose a reason for hiding this comment

jonbrenas Sep 11, 2024

Choose a reason for hiding this comment

jonbrenas Sep 11, 2024

Choose a reason for hiding this comment

jonbrenas Sep 11, 2024

Choose a reason for hiding this comment

jonbrenas Sep 11, 2024

Choose a reason for hiding this comment

jonbrenas commented Sep 11, 2024

tristanpwdennis commented Sep 11, 2024

tristanpwdennis commented Sep 19, 2024 • edited Loading

alimanfoo left a comment

Choose a reason for hiding this comment

alimanfoo commented Nov 21, 2024

tristanpwdennis commented Nov 25, 2024

alimanfoo left a comment

Choose a reason for hiding this comment

tristanpwdennis commented Nov 26, 2024

jonbrenas commented Nov 27, 2024 • edited Loading

tristanpwdennis commented Nov 27, 2024 • edited Loading

alimanfoo left a comment

Choose a reason for hiding this comment

alimanfoo commented Dec 2, 2024

tristanpwdennis commented Dec 2, 2024

tristanpwdennis commented Mar 26, 2024 •

edited by alimanfoo

Loading

tristanpwdennis commented Sep 19, 2024 •

edited

Loading

jonbrenas commented Nov 27, 2024 •

edited

Loading

tristanpwdennis commented Nov 27, 2024 •

edited

Loading