Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

cloudspades: add linux-aarch64 build #51446

Merged
merged 3 commits into from
Oct 17, 2024

Conversation

martin-g
Copy link
Contributor

Describe your pull request here


Please read the guidelines for Bioconda recipes before opening a pull request (PR).

General instructions

  • If this PR adds or updates a recipe, use "Add" or "Update" appropriately as the first word in its title.
  • New recipes not directly relevant to the biological sciences need to be submitted to the conda-forge channel instead of Bioconda.
  • PRs require reviews prior to being merged. Once your PR is passing tests and ready to be merged, please issue the @BiocondaBot please add label command.
  • Please post questions on Gitter or ping @bioconda/core in a comment.

Instructions for avoiding API, ABI, and CLI breakage issues

Conda is able to record and lock (a.k.a. pin) dependency versions used at build time of other recipes.
This way, one can avoid that expectations of a downstream recipe with regards to API, ABI, or CLI are violated by later changes in the recipe.
If not already present in the meta.yaml, make sure to specify run_exports (see here for the rationale and comprehensive explanation).
Add a run_exports section like this:

build:
  run_exports:
    - ...

with ... being one of:

Case run_exports statement
semantic versioning {{ pin_subpackage("myrecipe", max_pin="x") }}
semantic versioning (0.x.x) {{ pin_subpackage("myrecipe", max_pin="x.x") }}
known breakage in minor versions {{ pin_subpackage("myrecipe", max_pin="x.x") }} (in such a case, please add a note that shortly mentions your evidence for that)
known breakage in patch versions {{ pin_subpackage("myrecipe", max_pin="x.x.x") }} (in such a case, please add a note that shortly mentions your evidence for that)
calendar versioning {{ pin_subpackage("myrecipe", max_pin=None) }}

while replacing "myrecipe" with either name if a name|lower variable is defined in your recipe or with the lowercase name of the package in quotes.

Bot commands for PR management

Please use the following BiocondaBot commands:

Everyone has access to the following BiocondaBot commands, which can be given in a comment:

@BiocondaBot please update Merge the master branch into a PR.
@BiocondaBot please add label Add the please review & merge label.
@BiocondaBot please fetch artifacts Post links to CI-built packages/containers.
You can use this to test packages locally.

Note that the @BiocondaBot please merge command is now depreciated. Please just squash and merge instead.

Also, the bot watches for comments from non-members that include @bioconda/<team> and will automatically re-post them to notify the addressed <team>.

Signed-off-by: Martin Tzvetanov Grigorov <mgrigorov@apache.org>
@martin-g martin-g added the aarch64 Related to adding linux-aarch64 support label Oct 17, 2024
Copy link
Contributor

coderabbitai bot commented Oct 17, 2024

📝 Walkthrough
📝 Walkthrough

Walkthrough

The pull request modifies the meta.yaml file for the cloudspades package. The version number remains at 3.16.0, while the build number is incremented from 0 to 1. The source section is unchanged except for corrected indentation. A new entry is added to the extra section, specifying additional-platforms with the value linux-aarch64. Additionally, the indentation of the doc_url line is corrected. There are no modifications to the requirements or test commands in this update. The build.sh script is also updated to include an additional flag in the CXXFLAGS environment variable, appending -fsigned-char to the existing flags, which alters the compilation settings for C++ files. The CircleCI configuration file .circleci/config.yml is updated to increase the resource class for the linux-aarch64 executor from arm.medium to arm.large, and various job definitions are refined to accommodate builds on the specified platforms.

Possibly related PRs

Suggested labels

please review & merge

Suggested reviewers

  • mencian

Thank you for using CodeRabbit. We offer it for free to the OSS community and would appreciate your support in helping us grow. If you find it useful, would you consider giving us a shout-out on your favorite social media?

❤️ Share
🪧 Tips

Chat

There are 3 ways to chat with CodeRabbit:

  • Review comments: Directly reply to a review comment made by CodeRabbit. Example:
    • I pushed a fix in commit <commit_id>, please review it.
    • Generate unit testing code for this file.
    • Open a follow-up GitHub issue for this discussion.
  • Files and specific lines of code (under the "Files changed" tab): Tag @coderabbitai in a new review comment at the desired location with your query. Examples:
    • @coderabbitai generate unit testing code for this file.
    • @coderabbitai modularize this function.
  • PR comments: Tag @coderabbitai in a new PR comment to ask questions about the PR branch. For the best results, please provide a very specific query, as very limited context is provided in this mode. Examples:
    • @coderabbitai gather interesting stats about this repository and render them as a table. Additionally, render a pie chart showing the language distribution in the codebase.
    • @coderabbitai read src/utils.ts and generate unit testing code.
    • @coderabbitai read the files in the src/scheduler package and generate a class diagram using mermaid and a README in the markdown format.
    • @coderabbitai help me debug CodeRabbit configuration file.

Note: Be mindful of the bot's finite context window. It's strongly recommended to break down tasks such as reading entire modules into smaller chunks. For a focused discussion, use review comments to chat about specific files and their changes, instead of using the PR comments.

CodeRabbit Commands (Invoked using PR comments)

  • @coderabbitai pause to pause the reviews on a PR.
  • @coderabbitai resume to resume the paused reviews.
  • @coderabbitai review to trigger an incremental review. This is useful when automatic reviews are disabled for the repository.
  • @coderabbitai full review to do a full review from scratch and review all the files again.
  • @coderabbitai summary to regenerate the summary of the PR.
  • @coderabbitai resolve resolve all the CodeRabbit review comments.
  • @coderabbitai configuration to show the current CodeRabbit configuration for the repository.
  • @coderabbitai help to get help.

Other keywords and placeholders

  • Add @coderabbitai ignore anywhere in the PR description to prevent this PR from being reviewed.
  • Add @coderabbitai summary to generate the high-level summary at a specific location in the PR description.
  • Add @coderabbitai anywhere in the PR title to generate the title automatically.

CodeRabbit Configuration File (.coderabbit.yaml)

  • You can programmatically configure CodeRabbit by adding a .coderabbit.yaml file to the root of your repository.
  • Please see the configuration documentation for more information.
  • If your editor has YAML language server enabled, you can add the path at the top of this file to enable auto-completion and validation: # yaml-language-server: $schema=https://coderabbit.ai/integrations/schema.v2.json

Documentation and Community

  • Visit our Documentation for detailed information on how to use CodeRabbit.
  • Join our Discord Community to get help, request features, and share feedback.
  • Follow us on X/Twitter for updates and announcements.

Signed-off-by: Martin Tzvetanov Grigorov <mgrigorov@apache.org>
Signed-off-by: Martin Tzvetanov Grigorov <mgrigorov@apache.org>
Copy link
Contributor

@coderabbitai coderabbitai bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 0

🧹 Outside diff range and nitpick comments (1)
.circleci/config.yml (1)

Line range hint 1-385: Overall configuration review: ARM support looks comprehensive

The CircleCI configuration for ARM support (both osx-arm64 and linux-aarch64) appears comprehensive and well-structured. It covers various scenarios including PR builds, master branch builds, bulk builds, and nightly builds. The separation of concerns between different job types is commendable.

Consider the following suggestions for further improvements:

  1. Caching: Implement caching strategies for dependencies to speed up builds.
  2. Parallelization: Explore opportunities to parallelize steps within jobs where possible.
  3. Conditional Execution: Implement more granular conditional execution to skip unnecessary steps based on the changes in a PR.
  4. Error Handling: Enhance error handling and reporting, especially for the bulk and nightly builds.
  5. Documentation: Add inline comments explaining the purpose of critical or complex steps.

To verify the effectiveness of the current configuration, you can run the following script to analyze job success rates:

#!/bin/bash
# Description: Analyze job success rates for ARM builds

# Note: Ensure you have the CircleCI CLI installed and configured

# Function to get success rate for a job
get_success_rate() {
  local job_name=$1
  local total=$(circleci job list bioconda/bioconda-recipes --filter "$job_name" --limit 100 | jq '.total_items')
  local success=$(circleci job list bioconda/bioconda-recipes --filter "$job_name" --limit 100 | jq '.items[] | select(.status == "success") | .id' | wc -l)
  echo "scale=2; $success / $total * 100" | bc
}

echo "Success rates for ARM builds:"
echo "build_and_test (osx-arm64): $(get_success_rate 'build_and_test-osx-arm64')%"
echo "build_and_test (linux-aarch64): $(get_success_rate 'build_and_test-linux-aarch64')%"
echo "build_and_upload (osx-arm64): $(get_success_rate 'build_and_upload-osx-arm64')%"
echo "build_and_upload (linux-aarch64): $(get_success_rate 'build_and_upload-linux-aarch64')%"
echo "bulk_build (linux-aarch64): $(get_success_rate 'bulk_build-linux-aarch64')%"
echo "nightly_build (linux-aarch64): $(get_success_rate 'nightly_build-linux-aarch64')%"

This script will help identify any jobs that may need attention due to low success rates.

📜 Review details

Configuration used: CodeRabbit UI
Review profile: CHILL

📥 Commits

Files that changed from the base of the PR and between cafa0cf and 6820164.

📒 Files selected for processing (1)
  • .circleci/config.yml (1 hunks)
🧰 Additional context used
🔇 Additional comments (1)
.circleci/config.yml (1)

12-12: Approved: Increased resource class for linux-aarch64 executor

The change from arm.medium to arm.large for the linux-aarch64 executor is appropriate for supporting the new ARM64 build. This should provide more resources for potentially resource-intensive builds.

To ensure this change is beneficial:

  1. Monitor build times and success rates for linux-aarch64 builds after this change.
  2. Compare resource utilization between arm.medium and arm.large to justify the increased cost.

Run the following script to check recent build times for linux-aarch64:

This will help in assessing the impact of the resource class change on build performance.

@martin-g
Copy link
Contributor Author

@BiocondaBot please fetch artifacts

@BiocondaBot
Copy link
Collaborator

Package(s) built are ready for inspection:

Arch Package Zip File / Repodata CI Instructions
linux-64 cloudspades-3.16.0-h6dccd9a_1.tar.bz2 LinuxArtifacts.zip Azure
showYou may also use conda to install after downloading and extracting the zip file. From the LinuxArtifacts directory: conda install -c ./packages <package name>
linux-aarch64 cloudspades-3.16.0-hf9a8d21_1.tar.bz2 repodata.json CircleCI
showYou may also use conda to install:conda install -c https://output.circle-artifacts.com/output/job/f1592535-18b2-475c-8caf-499a7d4b2b40/artifacts/0/tmp/artifacts/packages <package name>

Docker image(s) built:

Package Tag CI Install with docker
cloudspades 3.16.0--h6dccd9a_1 Azure
showImages for Azure are in the LinuxArtifacts zip file above.gzip -dc LinuxArtifacts/images/cloudspades:3.16.0--h6dccd9a_1.tar.gz | docker load

@martin-g
Copy link
Contributor Author

mgrigorov in 🌐 euler-arm-22 in /tmp/cloudspades via 🅒 test123 took 40s 
❯ tar xvf cloudspades-3.16.0-hf9a8d21_1.tar.bz2                                                                                                                                                            (test123) 
info/run_exports.json
info/hash_input.json
info/test/run_test.sh
info/index.json
info/recipe/build.sh
info/recipe/meta.yaml.template
info/recipe/0001-change-verbosity-remove-expr.patch
info/licenses/assembler/LICENSE
info/recipe/meta.yaml
info/files
info/recipe/conda_build_config.yaml
info/about.json
info/paths.json
info/git
share/spades/sewage/usher_barcodes.csv
share/spades/rnaspades_manual.html
share/spades/manual.html
bin/rnaviralspades.py
bin/plasmidspades.py
bin/metaspades.py
bin/metaplasmidspades.py
bin/coronaspades.py
bin/rnaspades.py
bin/metaviralspades.py
share/spades/joblib3/_compat.py
share/spades/joblib3/testing.py
share/spades/spades_pipeline/executors/executors.py
share/spades/spades_pipeline/scripts/copy_files.py
share/spades/spades_pipeline/stages/stage.py
share/spades/spades_pipeline/executors/executor_save_yaml.py
share/spades/spades_pipeline/scripts/plasmid_glue.py
share/spades/pyyaml3/loader.py
share/spades/joblib3/_multiprocessing_helpers.py
share/spades/spades_pipeline/stages/terminating_stage.py
share/spades/pyyaml3/nodes.py
share/spades/spades_pipeline/stages/pipeline.py
share/spades/spades_pipeline/stages/breaking_scaffolds_stage.py
bin/spades_init.py
share/spades/spades_pipeline/stages/before_start_stage.py
share/spades/spades_pipeline/stages/check_test_stage.py
share/spades/spades_pipeline/stages/scaffold_correction_stage.py
share/spades/spades_pipeline/scripts/breaking_scaffolds_script.py
share/spades/spades_pipeline/commands_parser.py
share/spades/spades_pipeline/common/parallel_launcher.py
share/spades/pyyaml3/events.py
share/spades/spades_pipeline/scripts/preprocess_contigs.py
share/spades/pyyaml3/error.py
share/spades/pyyaml3/tokens.py
share/spades/pyyaml3/dumper.py
share/spades/spades_pipeline/common/alignment.py
share/spades/spades_pipeline/executors/executor_local.py
share/spades/joblib3/my_exceptions.py
share/spades/joblib3/disk.py
share/spades/pyyaml3/cyaml.py
share/spades/spades_pipeline/scripts/correction_iteration_script.py
share/spades/spades_pipeline/scripts/check_test_script.py
share/spades/joblib3/_memory_helpers.py
share/spades/spades_pipeline/options_storage.py
share/spades/pyyaml3/serializer.py
share/spades/spades_pipeline/scripts/preprocess_interlaced_reads.py
share/spades/spades_pipeline/process_cfg.py
share/spades/joblib3/__init__.py
share/spades/spades_pipeline/scripts/compress_all.py
share/spades/pyyaml3/composer.py
share/spades/joblib3/logger.py
share/spades/spades_pipeline/common/SeqIO.py
share/spades/spades_pipeline/stages/correction_stage.py
share/spades/pyyaml3/reader.py
share/spades/spades_pipeline/stages/spades_iteration_stage.py
share/spades/pyyaml3/resolver.py
share/spades/spades_pipeline/common/sam_parser.py
share/spades/joblib3/hashing.py
share/spades/spades_pipeline/stages/error_correction_stage.py
share/spades/pyyaml3/__init__.py
share/spades/joblib3/func_inspect.py
share/spades/spades_pipeline/stages/preprocess_reads_stage.py
share/spades/pyyaml3/representer.py
share/spades/joblib3/format_stack.py
share/spades/joblib3/numpy_pickle.py
share/spades/spades_pipeline/stages/spades_stage.py
share/spades/joblib3/pool.py
share/spades/pyyaml3/parser.py
share/spades/pyyaml3/constructor.py
share/spades/spades_pipeline/support.py
bin/spades.py
share/spades/joblib3/parallel.py
share/spades/joblib3/memory.py
share/spades/pyyaml3/emitter.py
share/spades/pyyaml3/scanner.py
share/spades/spades_pipeline/options_parser.py
share/spades/spades_pipeline/common/__init__.py
share/spades/spades_pipeline/stages/__init__.py
share/spades/test_dataset/reference_1K.fa.gz
share/spades/biosynthetic_spades_hmms/KR.hmm.gz
share/spades/biosynthetic_spades_hmms/TE.hmm.gz
share/spades/biosynthetic_spades_hmms/CStart.hmm.gz
share/spades/biosynthetic_spades_hmms/AT.hmm.gz
share/spades/biosynthetic_spades_hmms/KS.hmm.gz
share/spades/biosynthetic_spades_hmms/AMP.hmm.gz
share/spades/test_dataset_plasmid/pl1.fq.gz
share/spades/test_dataset_plasmid/pl2.fq.gz
share/spades/test_dataset/ecoli_1K_1.fq.gz
share/spades/test_dataset/ecoli_1K_2.fq.gz
share/spades/coronaspades_hmms/Phan-CoV.hmm.gz
share/spades/coronaspades_hmms/Pfam-A.SARS-CoV-2.hmm.gz
share/spades/test_dataset/operons_1K.gff
share/spades/test_dataset/genes_1K.gff
share/spades/VERSION
share/spades/LICENSE
share/spades/README.md
bin/spades-bwa
bin/spades-convert-bin-to-fasta
bin/spades-corrector-core
bin/spades-hammer
bin/spades-kmercount
bin/spades-kmer-estimating
bin/spades-ionhammer
bin/spades-read-filter
bin/spades-gbuilder
bin/spades-gsimplifier
bin/spades-gmapper
filebin/spades-core
^[[Cshare/spades/configs/debruijn/toy.info
share/spades/configs/debruijn/isolate_mode.info
share/spades/configs/debruijn/metaplasmid_mode.info
share/spades/configs/debruijn/hmm_mode.info
share/spades/configs/debruijn/large_genome_mode.info
share/spades/configs/debruijn/tsa.info
share/spades/configs/corrector/corrector.info
share/spades/configs/debruijn/plasmid_mode.info
share/spades/configs/debruijn/careful_mda_mode.info
share/spades/configs/debruijn/careful_mode.info
share/spades/configs/debruijn/rnaviral_mode.info
share/spades/configs/debruijn/construction.info
share/spades/configs/debruijn/detail_info_printer.info
share/spades/configs/debruijn/metaviral_mode.info
share/spades/configs/debruijn/sewage_mode.info
share/spades/configs/hammer/config.info
share/spades/configs/debruijn/distance_estimation.info
share/spades/configs/debruijn/mda_mode.info
share/spades/configs/debruijn/meta_mode.info
share/spades/configs/debruijn/config.info
share/spades/configs/debruijn/rna_mode.info
share/spades/configs/debruijn/simplification.info
share/spades/configs/debruijn/pe_params.info
share/spades/configs/debruijn/tslr_resolver.info
share/spades/test_dataset/operons_1K.txt
share/spades/test_dataset/genes_1K.txt
share/spades/GPLv2.txt
share/spades/configs/ionhammer/ionhammer.cfg

mgrigorov in 🌐 euler-arm-22 in /tmp/cloudspades via 🅒 test123 took 7s 
❯ file bin/*                                                                                                                                                                                               (test123) 
bin/coronaspades.py:             symbolic link to spades.py
bin/metaplasmidspades.py:        symbolic link to spades.py
bin/metaspades.py:               symbolic link to spades.py
bin/metaviralspades.py:          symbolic link to spades.py
bin/plasmidspades.py:            symbolic link to spades.py
bin/rnaspades.py:                symbolic link to spades.py
bin/rnaviralspades.py:           symbolic link to spades.py
bin/spades-bwa:                  ELF 64-bit LSB pie executable, ARM aarch64, version 1 (SYSV), dynamically linked, interpreter /lib/ld-linux-aarch64.so.1, for GNU/Linux 3.7.0, with debug_info, not stripped
bin/spades-convert-bin-to-fasta: ELF 64-bit LSB pie executable, ARM aarch64, version 1 (GNU/Linux), dynamically linked, interpreter /lib/ld-linux-aarch64.so.1, for GNU/Linux 3.7.0, with debug_info, not stripped
bin/spades-core:                 ELF 64-bit LSB pie executable, ARM aarch64, version 1 (GNU/Linux), dynamically linked, interpreter /lib/ld-linux-aarch64.so.1, for GNU/Linux 3.7.0, with debug_info, not stripped
bin/spades-corrector-core:       ELF 64-bit LSB pie executable, ARM aarch64, version 1 (GNU/Linux), dynamically linked, interpreter /lib/ld-linux-aarch64.so.1, for GNU/Linux 3.7.0, with debug_info, not stripped
bin/spades-gbuilder:             ELF 64-bit LSB pie executable, ARM aarch64, version 1 (GNU/Linux), dynamically linked, interpreter /lib/ld-linux-aarch64.so.1, for GNU/Linux 3.7.0, with debug_info, not stripped
bin/spades-gmapper:              ELF 64-bit LSB pie executable, ARM aarch64, version 1 (GNU/Linux), dynamically linked, interpreter /lib/ld-linux-aarch64.so.1, for GNU/Linux 3.7.0, with debug_info, not stripped
bin/spades-gsimplifier:          ELF 64-bit LSB pie executable, ARM aarch64, version 1 (GNU/Linux), dynamically linked, interpreter /lib/ld-linux-aarch64.so.1, for GNU/Linux 3.7.0, with debug_info, not stripped
bin/spades-hammer:               ELF 64-bit LSB pie executable, ARM aarch64, version 1 (GNU/Linux), dynamically linked, interpreter /lib/ld-linux-aarch64.so.1, for GNU/Linux 3.7.0, with debug_info, not stripped
bin/spades-ionhammer:            ELF 64-bit LSB pie executable, ARM aarch64, version 1 (GNU/Linux), dynamically linked, interpreter /lib/ld-linux-aarch64.so.1, for GNU/Linux 3.7.0, with debug_info, not stripped
bin/spades-kmer-estimating:      ELF 64-bit LSB pie executable, ARM aarch64, version 1 (GNU/Linux), dynamically linked, interpreter /lib/ld-linux-aarch64.so.1, for GNU/Linux 3.7.0, with debug_info, not stripped
bin/spades-kmercount:            ELF 64-bit LSB pie executable, ARM aarch64, version 1 (GNU/Linux), dynamically linked, interpreter /lib/ld-linux-aarch64.so.1, for GNU/Linux 3.7.0, with debug_info, not stripped
bin/spades-read-filter:          ELF 64-bit LSB pie executable, ARM aarch64, version 1 (GNU/Linux), dynamically linked, interpreter /lib/ld-linux-aarch64.so.1, for GNU/Linux 3.7.0, with debug_info, not stripped
bin/spades.py:                   Python script, ASCII text executable
bin/spades_init.py:              Python script, ASCII text executable

mgrigorov in 🌐 euler-arm-22 in /tmp/cloudspades via 🅒 test123 
❯ ./bin/spades-bwa                                                                                                                                                                                         (test123) 

Program: bwa (alignment via Burrows-Wheeler transformation)
Version: 0.7.16a-r1185-dirty
Contact: Heng Li <lh3@sanger.ac.uk>

Usage:   bwa <command> [options]

Command: index         index sequences in the FASTA format
         mem           BWA-MEM algorithm
         fastmap       identify super-maximal exact matches
         pemerge       merge overlapping paired ends (EXPERIMENTAL)
         aln           gapped/ungapped alignment
         samse         generate alignment (single ended)
         sampe         generate alignment (paired ended)
         bwasw         BWA-SW for long queries

         shm           manage indices in shared memory
         fa2pac        convert FASTA to PAC format
         pac2bwt       generate BWT from PAC
         bwtupdate     update .bwt to the new format
         bwt2sa        generate SA from BWT and Occ

Note: To use BWA, you need to first index the genome with `bwa index'.
      There are three alignment algorithms in BWA: `mem', `bwasw', and
      `aln/samse/sampe'. If you are not sure which to use, try `bwa mem'
      first. Please `man ./bwa.1' for the manual.

LGTM!

@martin-g
Copy link
Contributor Author

@BiocondaBot please add label

@BiocondaBot BiocondaBot added the please review & merge set to ask for merge label Oct 17, 2024
@rpetit3 rpetit3 merged commit 67dbb42 into bioconda:master Oct 17, 2024
7 checks passed
@coderabbitai coderabbitai bot mentioned this pull request Oct 17, 2024
@martin-g martin-g deleted the aarch64-cloudspades branch October 18, 2024 08:22
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
aarch64 Related to adding linux-aarch64 support please review & merge set to ask for merge
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants