Skip to content

Commit

Permalink
Merge pull request #1 from wtsi-npg/devel
Browse files Browse the repository at this point in the history
Update
  • Loading branch information
jidur authored Jan 27, 2025
2 parents b226de0 + f308f23 commit b002cdb
Show file tree
Hide file tree
Showing 102 changed files with 7,206 additions and 1,794 deletions.
4 changes: 4 additions & 0 deletions .dockerignore
Original file line number Diff line number Diff line change
@@ -0,0 +1,4 @@
.DS_Store
*~
*.iml
.idea/
10 changes: 10 additions & 0 deletions .github/dependabot.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,10 @@
# Referenced from:
# https://docs.github.com/en/code-security/dependabot/working-with-dependabot/keeping-your-actions-up-to-date-with-dependabot

version: 2
updates:

- package-ecosystem: "github-actions"
directory: "/"
schedule:
interval: "weekly"
65 changes: 65 additions & 0 deletions .github/workflows/publish-docker-image.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,65 @@
name: "Publish Docker Image"

on:
push:
tags:
- "**"

jobs:
build:
runs-on: ubuntu-latest
defaults:
run:
shell: bash -l -e -o pipefail {0}

env:
IMAGE_NAME: p4
REPOSITORY_OWNER: ${{ github.repository_owner }}

steps:
- name: Checkout code
uses: actions/checkout@v4
with:
fetch-depth: 0

- name: "Fetch Tags"
# Workaround for https://github.com/actions/checkout/issues/290
run: git fetch --tags --force

- name: "Get release variables"
# RELEASE_VERSION assignment must match the command used in autoconf's
# configure.ac file. On update, care should be taken to maintain both
# sides congruent.
run: |
echo 'RELEASE_VERSION='$(git describe --always --tags --dirty) >> $GITHUB_ENV
echo 'GIT_URL='$(git remote get-url origin) >> $GITHUB_ENV
echo 'GIT_COMMIT='$(git log --pretty=format:'%H' -n 1) >> $GITHUB_ENV
- name: "Login to Docker registry"
uses: docker/login-action@v3
with:
registry: ghcr.io
username: ${{ github.repository_owner }}
password: ${{ secrets.GITHUB_TOKEN }}

- name: "Build Docker image"
run: |
docker build --rm \
--file Dockerfile \
--platform linux/amd64 \
--progress plain \
--load \
--label "org.opencontainers.image.title=${IMAGE_NAME}" \
--label "org.opencontainers.image.source=${GIT_URL}" \
--label "org.opencontainers.image.revision=${GIT_COMMIT}" \
--label "org.opencontainers.image.version=${RELEASE_VERSION}" \
--label "org.opencontainers.image.created=$(date --utc --iso-8601=seconds)" \
--label "org.opencontainers.image.vendor=npg.sanger.ac.uk" \
--tag "ghcr.io/${REPOSITORY_OWNER}/${IMAGE_NAME}:${RELEASE_VERSION}" \
--tag "ghcr.io/${REPOSITORY_OWNER}/${IMAGE_NAME}:latest" \
.
- name: "Push Docker image to registry"
run: |
docker push "ghcr.io/$REPOSITORY_OWNER/$IMAGE_NAME:$RELEASE_VERSION"
docker push "ghcr.io/$REPOSITORY_OWNER/$IMAGE_NAME:latest"
65 changes: 65 additions & 0 deletions .github/workflows/testing_and_building_repo.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,65 @@
name: testing_and_building_repo
on: [push, pull_request]
jobs:
build:
runs-on: ubuntu-latest
name: Distribution Perl on ubuntu-latest
steps:
- uses: actions/checkout@v4

# Caching cpanm external modules
- name: Cache cpanm external modules
id: cpanmCache
uses: actions/cache@v4
with:
path: ~/perl5ext
key: ${{ runner.os }}-build-cpanm-external

- name: install cpanm
run: |
wget -qO - https://cpanmin.us | /usr/bin/perl - --sudo App::cpanminus
# run the perl install script
- name: install NPG Perl dependencies, and their CPAN dependencies
run: |
cpanm --local-lib=~/perl5ext local::lib && eval $(perl -I ~/perl5ext/lib/perl5/ -Mlocal::lib)
${GITHUB_WORKSPACE}/scripts/install_npg_perl_dependencies.sh $WTSI_NPG_GITHUB_URL $WTSI_NPG_BUILD_BRANCH
env:
WTSI_NPG_GITHUB_URL: https://github.com/wtsi-npg
WTSI_NPG_BUILD_BRANCH: ${GITHUB_HEAD_REF}

- name: install cpanm dependencies
run: |
eval $(perl -I ~/perl5ext/lib/perl5/ -Mlocal::lib=~/perl5npg)
eval $(perl -I ~/perl5ext/lib/perl5/ -Mlocal::lib=~/perl5ext)
cpanm --installdeps --notest .
- name: run Build.PL and ./Build
run: |
eval $(perl -I ~/perl5ext/lib/perl5/ -Mlocal::lib=~/perl5ext)
eval $(perl -I ~/perl5ext/lib/perl5/ -Mlocal::lib=~/perl5npg)
export TEST_AUTHOR=1
perl Build.PL
./Build test --verbose
./Build install
# running ./Build dist and exporting files
- name: run ./Build dist
run: |
eval $(perl -I ~/perl5ext/lib/perl5/ -Mlocal::lib=~/perl5ext)
export TEST_AUTHOR=1
./Build dist
export DIST_FILE=$(ls p4-*.tar.gz)
export MD5_FILE=$DIST_FILE.md5
md5sum $DIST_FILE > $MD5_FILE
export SHA256_FILE=$DIST_FILE.sha256
shasum -a 256 $DIST_FILE > $SHA256_FILE
# Archive logs if failure
- name: Archive CPAN logs
if: ${{ failure() }}
uses: actions/upload-artifact@v4
with:
name: cpan_log
path: /home/runner/.cpanm/work/*/build.log
retention-days: 5
30 changes: 0 additions & 30 deletions .travis.yml

This file was deleted.

4 changes: 4 additions & 0 deletions Build.PL
Original file line number Diff line number Diff line change
Expand Up @@ -58,6 +58,10 @@ my $builder = $class->new(
'ExtUtils::CBuilder' => 0
},

'test_requires' => {
'Test::Compile' => 0,
},

'build_requires' => {
'Carp' => 0,
'Cwd' => 0,
Expand Down
172 changes: 172 additions & 0 deletions Changes
Original file line number Diff line number Diff line change
@@ -1,6 +1,178 @@
CHANGES LOG
-----------

- Added .github/dependabot.yml file to auto-update GitHub actions

0.40.0
- change permissions of STAR working directory to allow deletion by standard pipeline tools
- add Dockerfile to create container with appropriate tools for basic analyses
- Updated image labels

0.39.0
- use "samtools index" instead of "cram_index to create cram indexes
- for bam and cram index creation, use "samtools index -o <index_file>" to explicitly indicate the index file

0.38.0
- add additional autosome specific coverage threshold to final_output_prep.json

0.37.2
- change bambi i2b command in stage1 analysis to use new boolean version of --nocall-quality flag

0.37.1
- amend markdup command chain for duplexseq (nanoseq) analyses
- updated to use samtools merge instead of bammerge for library merging

0.37.0
- add i2b_nocall_qual_switch to stage1 analysis to enable "--nocall-quality=2" to be set for bambi i2b

0.36.0
- add hs_bwa_executable parameter to allow different bwa executables for target and nchs alignments (e.g. bwa-mem2 and bwa0_6)
- add optminpixeldif flag (optical distance for duplicates) to bamstreamingmarkduplicates command (biobambam markdup method)

0.35.0
- replace scramble with samtools in analysis pipeline templates

0.34.0
- add calmd to biobambam,samtools and duplexseq markdup methods

0.33.1
- update CI and tests

0.33.0
- extend list of preserved auxtags in realignment bamreset (for Duplex-Seq)

0.32.4
- add supplementary files used in dehumanising CLIMB uploads for ENA (Heron)
- add maxreadlen flag to bamstreamingduplicates in markdup_biobambam with default of 500

0.32.3
- force markdup_method:none (and primer_clip_method:no_clip) for untagged phiX and non-consented human split

0.32.2
- change prefix for SamHaplotag log files to include a trailing '.' (simplifies npg_irods archival)

0.32.1
- add missing edge to top-level human split template

0.32.0
- add haplotag processing to stage2 analysis
- move C2A analysis from bam to cram output stream (to void possible malformed MD tag values)
- remove intermediate files from duplexseq processing
- flatten edges arrays (burst sub-arrays) as part of vtfp processing

0.31.0
- add C2A analysis to stage2 of standard pipelines

0.30.0
- add optional flags to bwa mem command for Hi-C library stage 2 analyses
- bam files (and md5, index files) produced using samtools and md5sum instead of bamrecompress

0.29.0
- renamed markdup_botseq.json markdup_duplexseq.json
- add markdup_botseq.json

0.28.2
- move samtools calmd to correct position downstream of coord sort, and make calmd -Q flag (quiet) the default (alignment with markdup_method:none)

0.28.1
- amend samtools command to "ampliconclip" (previously "amplicon-clip")

0.28.0
- a new template for cases when marking duplicates is not required, with
an option to clip primers, which relies on a new feature in samtools

0.27.0
- check for divide by 0 when calc samtools subsample value
- drop file check, explictly assume fastq.gz files are gzipped when calc #reads for salmon
- make the code flow more robust in cases of insufficient reads for salmon
- unconditionally remove auxtags before adapter clipping when realignment_switch is 1

0.26.0
- add parameters file for top-up merge
- functional equivalence: enable selection of markdup method - biobambam (default), samtools or picard

0.25.0
- add target autosome stats file generation to final_output_prep
- library merge:
cram_write_option (default: use_tears, option use_local)
updated templates to v2.0 format
- add options to standard stage2 templates to handle realignment

release 0.24.2
- stage 1 and seq alignment templates: remove fastqcheck file generation
- stage 1 template: remove generation of a fastq file for index read

release 0.24.1
- correct count of minimum reads to run Salmon

release 0.24.0
- allow format selection for stage1 outputs/stage2 inputs (default: cram)
- stage1 adapter detection now done in parallel with phix alignment
- add samtools stats qc check to stage1 analysis
- stage2 analysis templates amended to allow merging of multiple inputs; application of spatial_filter moved from stage1
- RNA alignments handle single-end runs
- new hisat2 alignment method
- use compressed fastq input for star alignment
- salmon quantification
check that there are enough reads in the fastq file inputs
check if quant file is present before copying it
connect zip node to quant_genes node to ensure zip waits for salmon
- post_alignment_realignment.json: new port specification format

release 0.23.0
- target stats (samtools) addition to final output prep

release 0.22.1
- add default ("ifnull":"salmon") value for quant_method parameter to fix star/tophat2 tag#0 problem

release 0.22.0
- new template for minimap2 alignment, with optional post-alignment filtering of secondary and supplementary
alignments, and minimap2_F_value parameter to allow non-default insert size to be specified (-F flag)
- subsampling of 10K reads (fastq produced by samtools view -s SEED.FRAC) and production of fastqcheck files
added to stage1 and stage2 analyses (nchs, single-/paired-end)
- add lane-level samtools stats to stage1
- spatial filter application moved to stage2 from stage1
- amend spatial_filter input/output file names for easier identification
- empty edge entries recognised as noop - filter in viv.pl, ignore in visualisation, do not apply VTFILE
prefixes when there are no to and from attributes
- added -F option to minimap2

release 0.21.0
- salmon_alignment: copy quant.genes.sf to archive directory
- star_alignment: assign appropriate suffixes to bamtofastq output files
- merge: flag change for use with tears 1.2.4

release 0.20.1
- star_alignment
correct version number (to 2.0)
do queryname sort after alignment
- upgrade realignment template to version 2
- viv.pl: loosen overly strict checks when using output file nodes with subtype dummy

release 0.20.0
- phase 1 of new format templates (port spec)
- replacement of AlignmentFilter and spatial filter with bambi commands
- now handles XA/Y-split with no target alignment

release 0.19.3
- fixes for nonconsented human split with no target alignment
change intermediate OUTFILE node to RAFILE
remove unneeded bammarkduplicates and parameters only used by sort and duplicate marking

release 0.19.2
- fix for single-end alignments (bwa aln): specify use_STD[IN|OUT] attributes for EXEC nodes in bwa_aln_se_alignment

release 0.19.1
- fix split_by_chromosome template (ysplit)

release 0.19.0
- add new templates for STAR alignment and Salmon
- Small change in final_output_prep to allow bamsort_cmd to have configurable executable and scramble to have optional embed reference param.
- add select directive
- splice/prune operations can be specified in param_vals file
- new version2 format of port specification in templates
- correct failure to report errors lower than one VTFILE level

release 0.18.6
- Y-split fixes
When processing VTFILE nodes, make sure that parameters are reevaluated locally instead of naively inheriting values
Expand Down
Loading

0 comments on commit b002cdb

Please sign in to comment.