Add nested recipe check to get_recipes (#308)

* Pin to conda-build 3. * Depend on latest conda-forge-pinning package. This will ensure that conda-build 3 is always using the pinnings defined in https://github.com/conda-forge/conda-forge-pinning-feedstock/blob/master/recipe/conda_build_config.yaml By contributing to this list, we can maintain compatibility with conda-forge. * Try downgrading to last minor release of conda. * Try using conda 4.3. * Try 4.5.0 * Handle UnableToParse * Remove galaxy-lib version restriction. * adapt to conda 4.5 * use get_output_file_paths instead of get_output_file_path * Add lint for deprecated numpy spec. * Document new lint. * register lint. * fix tests * fix lint * adapt test cases and remove obsolete ones. * Remove env_matrix. * fix typo * fixes and further cleanup * fixes * Fixes. * fix typo * fixes * fixes * fixes. * fix DAG * Linting now uses fully rendered recipes, and checks all combinations of the variant matrix and both linux and osx. * fix conda execution in docker * fix argument * fixes. * Adapt numpy x.x lint to new multi-metadata strategy. * properly render metadata when finding built package paths. * fixes. * fixes * fix test for rendering sandboxing * disable skip-existing * Use same build args for the copying as for building. * add test for conda-forge pinning * add ability to have separate bioconda pins * pep8 * add some more docs on testing * add bioconda conda_build_config.yaml * add first metadata fix * update tests for load_first_metadata * first pass of cb3 migration docs * port bioconductor_skeleton to use cb3 compilers; improve detection * update bioconductor_skeleton test * more cb3 docs * add anaconda to channels * add anaconda channel to circleci setup * short-circuit filtering if skipped * fix text to only check names * add test for when CI=true * add extra checks that built pkgs exist * add test for cb3 outputs * clean up cb3 output test; typos * utils.Progress: remove unnecessary sleep * pkg_test: use --involucro-path to prevent download in galaxy-lib * remove anaconda channel * simplify htslib in test * add compiler test * fix htslib max pin * update docs to reflect run_exports * Update cb3.rst * update docs * add lint functions for compilers and fn * add linting docs * bring back the noarch lint docs * render with finalize=False where possible * rename load_conda_config to load_conda_build_config * use conda.exports (public API) * pin conda-forge-pinning, update conda/conda-build pins * lint: handle if source is a list * adjust tests * show test durations * lint: use decorator instead explicit loops * lint: fix decorator, pass through __name__ for registry * lint: fix decorator, pass through __name__ for registry (2) * run long-running tests separately * docker_utils: use base image acc. to BIOCONDA_UTILS_TAG * Remove a priori filtering. Instead filter when each recipe is considered. * Naming. * Skip filter tests for now. * Disable DAG generation. * Activate DAG calculation again (we cannot avoid it because we need the topological sorting). * Fast loading of metadata for DAG. * Only use host dependencies when building DAG. * get_package_paths returns tuple * utils.get_package_paths: only return paths * requirements: update conda, conda-build, conda-forge-pinning * use bypass_env_check=True when we use non-finalized renderings * requirements: use conda-build=3.10.3 for now * test_utils: disable the remaining filter_recipes test * Abort building if build strings in repo are divergent * fix previous commit: add Python's redundant colon * fix generate_docs.py * fix another typo * support linting just before building * sort recipes for linting * better noarch detection * default df * be clear which recipe the problem is on * add back in debug info on linting * more generic check for compilers * add conda_build_config files consistently * fix previous commit * fix previous commit again * fix mulled upload: iterate over all generated images. * build: fix collecting mulled_images * requirements: require python >=3.6 * build: skip builds early based on num of existing builds in channel * build: topo-sort connected components before merging * docs: handle extra.notes being a list * build: handle UnsatisfiableError * fix import * [WIP] update to conda-build 3.10.9 (#312) * build: rename --prelint to --lint, add --lint-only/--lint-exclude args * requirements: update to conda-build 3.10.9 * build: remove default value for --lint-exclude * requirements: require python >=3.6 * meta.get_section might return None (conda-build 3.10.9 compat) * silence most flake8 nagging * bioconductor: fix missing newline at EOF for post-link.sh * fix typo * use meta.get_value preferably * make flake8 shut up already * meta.get_value only works for path with depth 2 * lint: fix should_not_be_noarch for conda_build >3.10.3 * lint._has_compilers: also check for old compiler packages * lint._get_deps: fix sections being unset on default * docs: use only one job for debugging * Add nested recipe check to get_recipes Currently get_recipes only support recipes nested up to two deep from the recipe_folder provided. Our application of get_recipes, and potential future uses from others, has recipes nested deeper. This purposed implementation will provide the ability to locate all recipes, no matter how deep, within the user provided recipe_folder. * Add test for new get_recipes deep nesting check get_recipes was changed to allow for different levels of recipe nesting. test_depthcheck_get_recipe() will test get_recipes ability to find different levels of nested recipes. * Update test_utils.py * update test_depthcheck_get_recipe * update test_depthcheck_get_recipe * Update test_utils.py * Update test_utils.py * Update test_utils.py * update test_depthcheck_get_recipe * Re-test with circle ci * Update utils.py * Update test_utils.py * Update test_utils.py * Update test_utils.py * Update utils.py
bioconda · Jul 20, 2018 · 9aa0ef2 · 9aa0ef2
1 parent d67288b
commit 9aa0ef2
Show file tree

Hide file tree

Showing 3 changed files with 100 additions and 4 deletions.
diff --git a/bioconda_utils/build.py b/bioconda_utils/build.py
@@ -1,4 +1,5 @@
 import subprocess as sp
+from itertools import chain
 from collections import defaultdict, namedtuple
 import os
 import logging
@@ -371,6 +372,7 @@ def build_recipes(
     # merge subdags of the selected chunk
     subdag = dag.subgraph(chunks[subdag_i])
 
+
     recipes = [recipe
                for package in subdag
                for recipe in name2recipes[package]]

diff --git a/bioconda_utils/utils.py b/bioconda_utils/utils.py
@@ -494,6 +494,7 @@ def get_deps(meta, sec):
             return []
         return [dep.split()[0] for dep in deps if dep]
 
+
     def get_inner_deps(dependencies):
         dependencies = list(dependencies)
         for dep in dependencies:
@@ -537,10 +538,10 @@ def get_recipes(recipe_folder, package="*"):
         logger.debug(
             "get_recipes(%s, package='%s'): %s", recipe_folder, package, p)
         path = os.path.join(recipe_folder, p)
-        yield from map(os.path.dirname,
-                       glob.glob(os.path.join(path, "meta.yaml")))
-        yield from map(os.path.dirname,
-                       glob.glob(os.path.join(path, "*", "meta.yaml")))
+        for new_dir in glob.glob(path):
+            for dir_path,dir_names,file_names in os.walk(new_dir):
+                if "meta.yaml" in file_names:
+                    yield dir_path 
 
 
 def get_latest_recipes(recipe_folder, config, package="*"):

diff --git a/test/test_utils.py b/test/test_utils.py
@@ -976,3 +976,96 @@ def test_compiler():
         for i in utils.built_package_paths(v):
             assert os.path.exists(i)
             ensure_missing(i)
+
+def test_nested_recipes():
+    """  
+    Test get_recipes ability to identify different nesting depths of recipes 
+    """
+    r = Recipes(
+
+        """
+        shallow:
+            meta.yaml: |
+                package:
+                    name: shallow
+                    version: "0.1"
+            build.sh: |
+                #!/bin/bash
+                echo "Shallow Created"
+                pwd
+        normal/normal:
+            meta.yaml: |
+                package:
+                    name: normal
+                    version: "0.1"
+                build:
+                    skip: true
+                requirements:
+                    build:
+                        - python 3.6
+            build.sh: |
+                #!/bin/bash
+                echo "Testing build.sh through python"
+                python -h
+        deep/deep/deep:
+            meta.yaml: |
+                package:
+                    name: deep
+                    version: "0.1"
+                requirements:
+                    build:
+                        - python
+                    run:
+                        - python
+            build.sh: |
+                #!/bin/bash
+                ## Empty script
+        deeper/deeper/deeper/deeper:
+            meta.yaml: |
+                package:
+                    name: deeper
+                    version: "0.1"
+        F/I/V/E/deep:
+            meta.yaml: |
+                package:
+                    name: fivedeep
+                    version: "0.1"
+                requirements:
+                    build:
+                        - python 3.6
+                    run:
+                        - python 3.6
+        S/I/X/De/e/ep:
+            meta.yaml: |
+                package:
+                    name: ep
+                    version: "0.1"
+        S/E/V/E/N/D/eep:
+            meta.yaml: |
+                package:
+                    name: eep
+                    version: "0.1"
+        T/W/E/N/T/Y/N/E/S/T/D/I/R/E/C/T/O/R/Y/DEEP:
+            meta.yaml: |
+                package:
+                    name: twentydeep
+                    version: "0.1"
+        """, from_string=True)
+    r.write_recipes()
+
+    build_results = build.build_recipes(
+        r.basedir,
+        config={},
+        packages="*",
+        testonly=False,
+        force=False,
+        mulled_test=False,
+    )
+    assert build_results
+
+    assert len(list(utils.get_recipes(r.basedir))) == 8 
+
+    for k, v in r.recipe_dirs.items():
+        for i in utils.built_package_paths(v):
+            assert os.path.exists(i)
+            ensure_missing(i)