Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Migrate to conda-build 3. #293

Merged
merged 127 commits into from
Jul 19, 2018
+2,760 −1,926
Merged

Migrate to conda-build 3. #293

Changes from 1 commit
Commits
Show all changes
127 commits
Select commit Hold shift + click to select a range
a6e402a
Pin to conda-build 3.
johanneskoester Apr 23, 2018
12f128e
Depend on latest conda-forge-pinning package.
johanneskoester Apr 23, 2018
66c1a4c
Try downgrading to last minor release of conda.
johanneskoester Apr 23, 2018
9d6636c
Try using conda 4.3.
johanneskoester Apr 23, 2018
6ee9510
Try 4.5.0
johanneskoester Apr 23, 2018
63f809f
Handle UnableToParse
johanneskoester Apr 23, 2018
3664db1
Remove galaxy-lib version restriction.
johanneskoester Apr 24, 2018
20d58eb
adapt to conda 4.5
johanneskoester Apr 24, 2018
14aafd2
use get_output_file_paths instead of get_output_file_path
johanneskoester Apr 24, 2018
76035ea
Merge branch 'cb3-migration' of github.com:bioconda/bioconda-utils in…
johanneskoester Apr 24, 2018
c847294
Merge branch 'master' into cb3-migration
johanneskoester Apr 24, 2018
f6b5669
Add lint for deprecated numpy spec.
johanneskoester Apr 24, 2018
0d7d1cc
Document new lint.
johanneskoester Apr 24, 2018
55e1e14
register lint.
johanneskoester Apr 24, 2018
82abcf7
fix tests
johanneskoester Apr 24, 2018
3384875
fix lint
johanneskoester Apr 24, 2018
38aaf26
adapt test cases and remove obsolete ones.
johanneskoester Apr 24, 2018
07eb371
Remove env_matrix.
johanneskoester Apr 25, 2018
de2e4a2
fix typo
johanneskoester Apr 25, 2018
6014495
fixes and further cleanup
johanneskoester Apr 25, 2018
9a4a3d9
fixes
johanneskoester Apr 25, 2018
312f982
Fixes.
johanneskoester Apr 25, 2018
06fa571
fix typo
johanneskoester Apr 26, 2018
742e61d
fixes
johanneskoester Apr 26, 2018
2aaf71e
fixes
johanneskoester Apr 26, 2018
d888281
fixes.
johanneskoester Apr 26, 2018
7d97668
fix DAG
johanneskoester Apr 26, 2018
be141e0
Linting now uses fully rendered recipes, and checks all combinations …
johanneskoester Apr 26, 2018
660dc84
fix conda execution in docker
johanneskoester Apr 26, 2018
b3e2eda
fix argument
johanneskoester Apr 26, 2018
7934506
fixes.
johanneskoester Apr 26, 2018
a2aa8e1
Adapt numpy x.x lint to new multi-metadata strategy.
johanneskoester Apr 30, 2018
26132ed
properly render metadata when finding built package paths.
johanneskoester Apr 30, 2018
7aa4a7c
fixes.
johanneskoester Apr 30, 2018
69b0123
fixes
johanneskoester Apr 30, 2018
2102ad0
fix test for rendering sandboxing
daler May 1, 2018
ff204fe
disable skip-existing
daler May 1, 2018
3e1533a
Use same build args for the copying as for building.
johanneskoester May 2, 2018
3cd0ff6
add test for conda-forge pinning
daler May 4, 2018
c6ed119
add ability to have separate bioconda pins
daler May 5, 2018
31bd484
pep8
daler May 5, 2018
8139e77
add some more docs on testing
daler May 5, 2018
788f6d5
add bioconda conda_build_config.yaml
daler May 5, 2018
cfa4060
add first metadata fix
daler May 5, 2018
04543f3
update tests for load_first_metadata
daler May 5, 2018
526e48d
first pass of cb3 migration docs
daler May 5, 2018
ed371bd
port bioconductor_skeleton to use cb3 compilers; improve detection
daler May 5, 2018
9473f74
update bioconductor_skeleton test
daler May 5, 2018
02e89ae
more cb3 docs
daler May 5, 2018
3f57264
add anaconda to channels
daler May 5, 2018
4494c42
add anaconda channel to circleci setup
daler May 5, 2018
2b97c0b
short-circuit filtering if skipped
daler May 5, 2018
0969bce
fix text to only check names
daler May 5, 2018
a3441ea
add test for when CI=true
daler May 5, 2018
a704232
add extra checks that built pkgs exist
daler May 6, 2018
2da3a20
add test for cb3 outputs
daler May 6, 2018
08562bf
clean up cb3 output test; typos
daler May 6, 2018
9a3a9d8
utils.Progress: remove unnecessary sleep
mbargull May 6, 2018
6a86254
pkg_test: use --involucro-path to prevent download in galaxy-lib
mbargull May 6, 2018
71f52fa
remove anaconda channel
daler May 7, 2018
7999eec
simplify htslib in test
daler May 7, 2018
71b6b0c
add compiler test
daler May 7, 2018
ce67dff
fix htslib max pin
daler May 7, 2018
d6ea7a1
update docs to reflect run_exports
daler May 7, 2018
b2c82d9
Merge branch 'master' into cb3-migration
daler May 7, 2018
d374363
Update cb3.rst
kyleabeauchamp May 8, 2018
232e160
Merge pull request #297 from kyleabeauchamp/patch-1
daler May 9, 2018
ea608fb
Merge branch 'master' into cb3-migration
johanneskoester May 13, 2018
c9ec48b
update docs
daler May 13, 2018
fd7bfee
add lint functions for compilers and fn
daler May 13, 2018
3ec7681
add linting docs
daler May 13, 2018
80d25c7
bring back the noarch lint docs
daler May 13, 2018
7079bb3
render with finalize=False where possible
mbargull May 16, 2018
0ae67ab
rename load_conda_config to load_conda_build_config
mbargull May 16, 2018
884e757
use conda.exports (public API)
mbargull May 16, 2018
91639b8
pin conda-forge-pinning, update conda/conda-build pins
mbargull May 16, 2018
77b4f84
lint: handle if source is a list
mbargull May 16, 2018
7002683
adjust tests
mbargull May 16, 2018
a089bd0
show test durations
mbargull May 16, 2018
b6121dc
lint: use decorator instead explicit loops
mbargull May 16, 2018
8f026ad
lint: fix decorator, pass through __name__ for registry
mbargull May 16, 2018
933a954
lint: fix decorator, pass through __name__ for registry (2)
mbargull May 16, 2018
03960fe
run long-running tests separately
mbargull May 16, 2018
4686605
Merge branch 'master' into cb3-migration
johanneskoester May 22, 2018
b122120
docker_utils: use base image acc. to BIOCONDA_UTILS_TAG
mbargull May 25, 2018
b738ca5
Remove a priori filtering. Instead filter when each recipe is conside…
johanneskoester May 25, 2018
5ea1d5e
Naming.
johanneskoester May 25, 2018
325f0f2
Skip filter tests for now.
johanneskoester May 25, 2018
d3530ad
Disable DAG generation.
johanneskoester May 25, 2018
26ec23c
Activate DAG calculation again (we cannot avoid it because we need th…
johanneskoester May 25, 2018
668ec38
Fast loading of metadata for DAG.
johanneskoester May 25, 2018
a6dabe3
Only use host dependencies when building DAG.
johanneskoester May 26, 2018
2fb47ea
get_package_paths returns tuple
daler May 26, 2018
6072d39
utils.get_package_paths: only return paths
mbargull May 26, 2018
62f1d52
requirements: update conda, conda-build, conda-forge-pinning
mbargull May 26, 2018
9821159
use bypass_env_check=True when we use non-finalized renderings
mbargull May 26, 2018
62efbd5
requirements: use conda-build=3.10.3 for now
mbargull May 26, 2018
66497f2
test_utils: disable the remaining filter_recipes test
mbargull May 26, 2018
c074b13
Abort building if build strings in repo are divergent
mbargull May 29, 2018
4085e6a
fix previous commit: add Python's redundant colon
mbargull May 29, 2018
b002ec0
fix generate_docs.py
mbargull May 29, 2018
b4917aa
fix another typo
mbargull May 29, 2018
69a1f35
support linting just before building
daler Jun 2, 2018
ec47a17
sort recipes for linting
daler Jun 2, 2018
77d673b
better noarch detection
daler Jun 2, 2018
7a0c70b
Merge branch 'cb3-migration' of github.com:bioconda/bioconda-utils in…
daler Jun 2, 2018
0302291
default df
daler Jun 2, 2018
7d35d0b
be clear which recipe the problem is on
daler Jun 2, 2018
b2fb765
add back in debug info on linting
daler Jun 2, 2018
b6763c5
more generic check for compilers
daler Jun 2, 2018
41f39c0
add conda_build_config files consistently
mbargull Jun 3, 2018
e0603ec
fix previous commit
mbargull Jun 3, 2018
190ff4f
fix previous commit again
mbargull Jun 3, 2018
9fdbc8f
fix mulled upload: iterate over all generated images.
johanneskoester Jun 14, 2018
6c45c87
build: fix collecting mulled_images
mbargull Jun 19, 2018
3ab8023
requirements: require python >=3.6
mbargull Jun 21, 2018
d4701c6
build: skip builds early based on num of existing builds in channel
mbargull Jun 23, 2018
99782c2
build: topo-sort connected components before merging
mbargull Jun 24, 2018
e033e47
docs: handle extra.notes being a list
mbargull Jun 24, 2018
44565d0
build: handle UnsatisfiableError
mbargull Jun 24, 2018
86fc487
fix import
mbargull Jun 24, 2018
4f6b405
[WIP] update to conda-build 3.10.9 (#312)
mbargull Jun 26, 2018
4e35af8
Add paper to the homepage.
johanneskoester Jul 2, 2018
b0df159
Handle dependency cycles (#313)
mbargull Jul 4, 2018
5800ebe
requirements: update to conda-forge-pinning 2018.07.18
mbargull Jul 18, 2018
9349490
Merge pull request #315 from bioconda/cf-pinning-2018-07-18
bgruening Jul 18, 2018
9e40c85
Merge branch 'master' into cb3-migration
johanneskoester Jul 19, 2018
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
10 changes: 10 additions & 0 deletions bioconda_utils/build.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -4,6 +4,7 @@
import os
import logging

from conda.exports import UnsatisfiableError
import networkx as nx
import pandas

Expand Down Expand Up @@ -384,6 +385,15 @@ def build_recipes(
for n in nx.algorithms.descendants(subdag, name):
skip_dependent[n].append(recipe)
continue
except UnsatisfiableError as e:
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Without this the whole build in https://circleci.com/gh/bioconda/bioconda-recipes/14705 failed with

Traceback (most recent call last):
  File "/home/circleci/project/miniconda/bin/bioconda-utils", line 11, in <module>
    load_entry_point('bioconda-utils==0.12.3+118.g99782c2e', 'console_scripts', 'bioconda-utils')()
  File "/home/circleci/project/miniconda/lib/python3.6/site-packages/bioconda_utils/cli.py", line 611, in main
    argh.dispatch_commands([build, dag, dependent, lint, duplicates, bioconductor_skeleton, pypi_check, clean_cran_skeleton])
  File "/home/circleci/project/miniconda/lib/python3.6/site-packages/argh/dispatching.py", line 328, in dispatch_commands
    dispatch(parser, *args, **kwargs)
  File "/home/circleci/project/miniconda/lib/python3.6/site-packages/argh/dispatching.py", line 174, in dispatch
    for line in lines:
  File "/home/circleci/project/miniconda/lib/python3.6/site-packages/argh/dispatching.py", line 277, in _execute_command
    for line in result:
  File "/home/circleci/project/miniconda/lib/python3.6/site-packages/argh/dispatching.py", line 260, in _call
    result = function(*positional, **keywords)
  File "/home/circleci/project/miniconda/lib/python3.6/site-packages/bioconda_utils/cli.py", line 391, in build
    prelint=prelint,
  File "/home/circleci/project/miniconda/lib/python3.6/site-packages/bioconda_utils/build.py", line 376, in build_recipes
    force=force)
  File "/home/circleci/project/miniconda/lib/python3.6/site-packages/bioconda_utils/utils.py", line 880, in get_package_paths
    platform, metas = _load_platform_metas(recipe, finalize=True)
  File "/home/circleci/project/miniconda/lib/python3.6/site-packages/bioconda_utils/utils.py", line 806, in _load_platform_metas
    return platform, load_all_meta(recipe, config=config, finalize=finalize)
  File "/home/circleci/project/miniconda/lib/python3.6/site-packages/bioconda_utils/utils.py", line 172, in load_all_meta
    bypass_env_check=bypass_env_check,
  File "/home/circleci/project/miniconda/lib/python3.6/site-packages/conda_build/api.py", line 44, in render
    permit_unsatisfiable_variants=permit_unsatisfiable_variants)
  File "/home/circleci/project/miniconda/lib/python3.6/site-packages/conda_build/render.py", line 739, in render_recipe
    allow_no_other_outputs=True, bypass_env_check=bypass_env_check)
  File "/home/circleci/project/miniconda/lib/python3.6/site-packages/conda_build/render.py", line 645, in distribute_variants
    bypass_env_check=bypass_env_check)
  File "/home/circleci/project/miniconda/lib/python3.6/site-packages/conda_build/metadata.py", line 965, in parse_until_resolved
    bypass_env_check=bypass_env_check)
  File "/home/circleci/project/miniconda/lib/python3.6/site-packages/conda_build/metadata.py", line 888, in parse_again
    bypass_env_check=bypass_env_check),
  File "/home/circleci/project/miniconda/lib/python3.6/site-packages/conda_build/metadata.py", line 1450, in _get_contents
    rendered = template.render(environment=env)
  File "/home/circleci/project/miniconda/lib/python3.6/site-packages/jinja2/asyncsupport.py", line 76, in render
    return original_render(self, *args, **kwargs)
  File "/home/circleci/project/miniconda/lib/python3.6/site-packages/jinja2/environment.py", line 1008, in render
    return self.environment.handle_exception(exc_info, True)
  File "/home/circleci/project/miniconda/lib/python3.6/site-packages/jinja2/environment.py", line 780, in handle_exception
    reraise(exc_type, exc_value, tb)
  File "/home/circleci/project/miniconda/lib/python3.6/site-packages/jinja2/_compat.py", line 37, in reraise
    raise value.with_traceback(tb)
  File "/home/circleci/project/recipes/anvio/3.0.0/meta.yaml", line 39, in top-level template code
    - cython
  File "/home/circleci/project/miniconda/lib/python3.6/site-packages/conda_build/jinja_context.py", line 227, in pin_compatible
    pins, _, _ = get_env_dependencies(m, 'host', m.config.variant)
  File "/home/circleci/project/miniconda/lib/python3.6/site-packages/conda_build/render.py", line 124, in get_env_dependencies
    channel_urls=tuple(m.config.channel_urls))
  File "/home/circleci/project/miniconda/lib/python3.6/site-packages/conda_build/environ.py", line 744, in get_install_actions
    channel_urls=tuple(channel_urls))
  File "/home/circleci/project/miniconda/lib/python3.6/site-packages/conda_build/environ.py", line 744, in get_install_actions
    channel_urls=tuple(channel_urls))
  File "/home/circleci/project/miniconda/lib/python3.6/site-packages/conda_build/environ.py", line 744, in get_install_actions
    channel_urls=tuple(channel_urls))
  File "/home/circleci/project/miniconda/lib/python3.6/site-packages/conda_build/environ.py", line 708, in get_install_actions
    actions = install_actions(prefix, index, specs, force=True)
  File "/home/circleci/project/miniconda/lib/python3.6/site-packages/conda/common/io.py", line 46, in decorated
    return f(*args, **kwds)
  File "/home/circleci/project/miniconda/lib/python3.6/site-packages/conda/plan.py", line 541, in install_actions
    txn = solver.solve_for_transaction(prune=prune, ignore_pinned=not pinned)
  File "/home/circleci/project/miniconda/lib/python3.6/site-packages/conda/core/solve.py", line 505, in solve_for_transaction
    force_remove, force_reinstall)
  File "/home/circleci/project/miniconda/lib/python3.6/site-packages/conda/core/solve.py", line 438, in solve_for_diff
    final_precs = self.solve_final_state(deps_modifier, prune, ignore_pinned, force_remove)
  File "/home/circleci/project/miniconda/lib/python3.6/site-packages/conda/core/solve.py", line 331, in solve_final_state
    solution = r.solve(tuple(final_environment_specs))  # return value is List[dist]
  File "/home/circleci/project/miniconda/lib/python3.6/site-packages/conda/common/io.py", line 46, in decorated
    return f(*args, **kwds)
  File "/home/circleci/project/miniconda/lib/python3.6/site-packages/conda/resolve.py", line 857, in solve
    self.find_conflicts(specs)
  File "/home/circleci/project/miniconda/lib/python3.6/site-packages/conda/resolve.py", line 244, in find_conflicts
    raise UnsatisfiableError(bad_deps)
conda.exceptions.UnsatisfiableError: The following specifications were found to be in conflict:
  - numpy=1.9
  - pandas==0.20.1 -> numpy=1.12
Use "conda info <package>" to see the dependencies for each package.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Mh, this never crashed utils before yesterday ... wired.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yep, I'm also a bit puzzled we didn't hit this before...
Looks like it was only fixed on conda-build:master lately in: conda/conda-build#2974

logger.error(
'BUILD ERROR: '
'could not determine dependencies for recipe %s: %s',
recipe, e)
failed.append(recipe)
for n in nx.algorithms.descendants(subdag, name):
skip_dependent[n].append(recipe)
continue
if not pkg_paths:
logger.info("Nothing to be done for recipe %s", recipe)
continue
Expand Down