Migrate to conda-build 3. #293

johanneskoester · 2018-04-23T09:23:53Z

This is a start to finally do the migration to CB3. Feel free to contribute, as my time is quite limited at the moment (sorry).

xref: #284

adapt bioconda-utils
update docs as suggested in Conda-Build 3 support #284
try to implement all lints from Conda-Build 3 support #284

This will ensure that conda-build 3 is always using the pinnings defined in https://github.com/conda-forge/conda-forge-pinning-feedstock/blob/master/recipe/conda_build_config.yaml By contributing to this list, we can maintain compatibility with conda-forge.

johanneskoester · 2018-04-23T09:31:28Z

@mbargull in case you already have some local changes for CB3 or just your conceptual work, please move that into this PR.

johanneskoester · 2018-04-23T09:35:45Z

Seems we are running into this bug: conda/conda#6691

johanneskoester · 2018-04-23T13:31:24Z

@bgruening the restriction of the galaxy-lib version (<18.6.0a0) triggers a bug in the latest conda package (see above). Is the restriction crucial?

bgruening · 2018-04-23T13:37:45Z

No and if so, I will fix galaxy-lib.

…to cb3-migration

johanneskoester · 2018-04-25T12:37:20Z

Update: this works fine. I am now migrating to the new way of handling pinnings.

kyleabeauchamp · 2018-04-25T15:03:35Z

Thanks for tackling this one!

kyleabeauchamp · 2018-06-15T20:26:00Z

Is this ready to merge into master, or are we pointing to this branch now in production?

johanneskoester · 2018-06-24T11:26:32Z

Good question. @mbargull @bgruening @daler how about merging to master? Any Idea about this error?

waiting for workers...

Sphinx parallel build error:
TypeError: expected string or bytes-like object

Maybe we should just update to the latest sphinx version?

mbargull · 2018-06-24T11:45:25Z

Maybe we should just update to the latest sphinx version?

I think it's tripping on some recipe that has some malformation or at least unexpected content.
Last time I checked, we were already using the latest sphinx. (I also checked that the failing build was not related to the Sphinx update, too. Hence I assume it's something in the recipes.)
I'll change build-docs.sh on another branch to use only one job -- maybe that'll show some more information. Following that, we can merge into master, sure.

mbargull · 2018-06-24T12:33:22Z

There is at least one recipe (atactk) that uses a list for extra/notes. Although this should be a string, I added a change to support this anyway since I've seen extra/notes being a list in multiple PRs in the past (which I guess makes sense since "notes" is a plural, hence contributors make it a list)

mbargull · 2018-06-24T12:34:14Z

Would be nice to get the error output from https://circleci.com/gh/bioconda/bioconda-utils/1037 when using multiple jobs, though...

mbargull · 2018-06-24T15:22:45Z

bioconda_utils/build.py

@@ -384,6 +385,15 @@ def build_recipes(
            for n in nx.algorithms.descendants(subdag, name):
                skip_dependent[n].append(recipe)
            continue
+        except UnsatisfiableError as e:


Without this the whole build in https://circleci.com/gh/bioconda/bioconda-recipes/14705 failed with

Traceback (most recent call last): File "/home/circleci/project/miniconda/bin/bioconda-utils", line 11, in <module> load_entry_point('bioconda-utils==0.12.3+118.g99782c2e', 'console_scripts', 'bioconda-utils')() File "/home/circleci/project/miniconda/lib/python3.6/site-packages/bioconda_utils/cli.py", line 611, in main argh.dispatch_commands([build, dag, dependent, lint, duplicates, bioconductor_skeleton, pypi_check, clean_cran_skeleton]) File "/home/circleci/project/miniconda/lib/python3.6/site-packages/argh/dispatching.py", line 328, in dispatch_commands dispatch(parser, *args, **kwargs) File "/home/circleci/project/miniconda/lib/python3.6/site-packages/argh/dispatching.py", line 174, in dispatch for line in lines: File "/home/circleci/project/miniconda/lib/python3.6/site-packages/argh/dispatching.py", line 277, in _execute_command for line in result: File "/home/circleci/project/miniconda/lib/python3.6/site-packages/argh/dispatching.py", line 260, in _call result = function(*positional, **keywords) File "/home/circleci/project/miniconda/lib/python3.6/site-packages/bioconda_utils/cli.py", line 391, in build prelint=prelint, File "/home/circleci/project/miniconda/lib/python3.6/site-packages/bioconda_utils/build.py", line 376, in build_recipes force=force) File "/home/circleci/project/miniconda/lib/python3.6/site-packages/bioconda_utils/utils.py", line 880, in get_package_paths platform, metas = _load_platform_metas(recipe, finalize=True) File "/home/circleci/project/miniconda/lib/python3.6/site-packages/bioconda_utils/utils.py", line 806, in _load_platform_metas return platform, load_all_meta(recipe, config=config, finalize=finalize) File "/home/circleci/project/miniconda/lib/python3.6/site-packages/bioconda_utils/utils.py", line 172, in load_all_meta bypass_env_check=bypass_env_check, File "/home/circleci/project/miniconda/lib/python3.6/site-packages/conda_build/api.py", line 44, in render permit_unsatisfiable_variants=permit_unsatisfiable_variants) File "/home/circleci/project/miniconda/lib/python3.6/site-packages/conda_build/render.py", line 739, in render_recipe allow_no_other_outputs=True, bypass_env_check=bypass_env_check) File "/home/circleci/project/miniconda/lib/python3.6/site-packages/conda_build/render.py", line 645, in distribute_variants bypass_env_check=bypass_env_check) File "/home/circleci/project/miniconda/lib/python3.6/site-packages/conda_build/metadata.py", line 965, in parse_until_resolved bypass_env_check=bypass_env_check) File "/home/circleci/project/miniconda/lib/python3.6/site-packages/conda_build/metadata.py", line 888, in parse_again bypass_env_check=bypass_env_check), File "/home/circleci/project/miniconda/lib/python3.6/site-packages/conda_build/metadata.py", line 1450, in _get_contents rendered = template.render(environment=env) File "/home/circleci/project/miniconda/lib/python3.6/site-packages/jinja2/asyncsupport.py", line 76, in render return original_render(self, *args, **kwargs) File "/home/circleci/project/miniconda/lib/python3.6/site-packages/jinja2/environment.py", line 1008, in render return self.environment.handle_exception(exc_info, True) File "/home/circleci/project/miniconda/lib/python3.6/site-packages/jinja2/environment.py", line 780, in handle_exception reraise(exc_type, exc_value, tb) File "/home/circleci/project/miniconda/lib/python3.6/site-packages/jinja2/_compat.py", line 37, in reraise raise value.with_traceback(tb) File "/home/circleci/project/recipes/anvio/3.0.0/meta.yaml", line 39, in top-level template code - cython File "/home/circleci/project/miniconda/lib/python3.6/site-packages/conda_build/jinja_context.py", line 227, in pin_compatible pins, _, _ = get_env_dependencies(m, 'host', m.config.variant) File "/home/circleci/project/miniconda/lib/python3.6/site-packages/conda_build/render.py", line 124, in get_env_dependencies channel_urls=tuple(m.config.channel_urls)) File "/home/circleci/project/miniconda/lib/python3.6/site-packages/conda_build/environ.py", line 744, in get_install_actions channel_urls=tuple(channel_urls)) File "/home/circleci/project/miniconda/lib/python3.6/site-packages/conda_build/environ.py", line 744, in get_install_actions channel_urls=tuple(channel_urls)) File "/home/circleci/project/miniconda/lib/python3.6/site-packages/conda_build/environ.py", line 744, in get_install_actions channel_urls=tuple(channel_urls)) File "/home/circleci/project/miniconda/lib/python3.6/site-packages/conda_build/environ.py", line 708, in get_install_actions actions = install_actions(prefix, index, specs, force=True) File "/home/circleci/project/miniconda/lib/python3.6/site-packages/conda/common/io.py", line 46, in decorated return f(*args, **kwds) File "/home/circleci/project/miniconda/lib/python3.6/site-packages/conda/plan.py", line 541, in install_actions txn = solver.solve_for_transaction(prune=prune, ignore_pinned=not pinned) File "/home/circleci/project/miniconda/lib/python3.6/site-packages/conda/core/solve.py", line 505, in solve_for_transaction force_remove, force_reinstall) File "/home/circleci/project/miniconda/lib/python3.6/site-packages/conda/core/solve.py", line 438, in solve_for_diff final_precs = self.solve_final_state(deps_modifier, prune, ignore_pinned, force_remove) File "/home/circleci/project/miniconda/lib/python3.6/site-packages/conda/core/solve.py", line 331, in solve_final_state solution = r.solve(tuple(final_environment_specs)) # return value is List[dist] File "/home/circleci/project/miniconda/lib/python3.6/site-packages/conda/common/io.py", line 46, in decorated return f(*args, **kwds) File "/home/circleci/project/miniconda/lib/python3.6/site-packages/conda/resolve.py", line 857, in solve self.find_conflicts(specs) File "/home/circleci/project/miniconda/lib/python3.6/site-packages/conda/resolve.py", line 244, in find_conflicts raise UnsatisfiableError(bad_deps) conda.exceptions.UnsatisfiableError: The following specifications were found to be in conflict: - numpy=1.9 - pandas==0.20.1 -> numpy=1.12 Use "conda info <package>" to see the dependencies for each package.

Mh, this never crashed utils before yesterday ... wired.

Yep, I'm also a bit puzzled we didn't hit this before...
Looks like it was only fixed on conda-build:master lately in: conda/conda-build#2974

* build: rename --prelint to --lint, add --lint-only/--lint-exclude args * requirements: update to conda-build 3.10.9 * build: remove default value for --lint-exclude * requirements: require python >=3.6 * meta.get_section might return None (conda-build 3.10.9 compat) * silence most flake8 nagging * bioconductor: fix missing newline at EOF for post-link.sh * fix typo * use meta.get_value preferably * make flake8 shut up already * meta.get_value only works for path with depth 2 * lint: fix should_not_be_noarch for conda_build >3.10.3 * lint._has_compilers: also check for old compiler packages * lint._get_deps: fix sections being unset on default * docs: use only one job for debugging

mbargull · 2018-07-04T16:37:04Z

.circleci/build-docs.sh

@@ -58,7 +58,9 @@ rm -r *

 # build docs and copy over to tmpdir
 cd ${DOCSOURCE}
-make clean html SPHINXOPTS="-j2" 2>&1 | grep -v "WARNING: nonlocal image URL found:"
+# TODO: reenable "-j2" when docs build fine


Looking at the build times, it doesn't make much sense to use multiple jobs here. It's less than a minute faster with -j2 but also "obfuscates" the progress/error output...

* fix minor bug in `build --lint-exclude` * utils.check_recipe_skippable: avoid additional FILTER msg if nothing to do * utils.get_dag: 'dep' can be empty string for unevaluated jinja expressions * utils.get_dag: respect dependencies from build and run environments * build.build_recipes: gracefully handle(=skip) packages in dependency cycles

requirements: update to conda-forge-pinning 2018.07.18

bgruening · 2018-07-18T16:03:58Z

Bulk is green and we have rebuild our pkgs, a substantial amount is still blacklisted and we will work on this in the coming weeks.
But I think it is important to merge this to get the changes onto our homepage.

johanneskoester · 2018-07-19T11:24:24Z

Ok, I will merge once the tests pass.

johanneskoester · 2018-07-19T12:38:02Z

@bioconda/core I will shortly update bioconda-recipes:master to point to the new release 0.13 of bioconda-utils. I am just waiting for the docker build to finish.

johanneskoester · 2018-07-19T12:48:12Z

@bioconda/core done. Migration is complete (at least from the bioconda-utils side)!

johanneskoester added 2 commits April 23, 2018 11:18

Pin to conda-build 3.

a6e402a

johanneskoester added 4 commits April 23, 2018 11:36

Try downgrading to last minor release of conda.

66c1a4c

Try using conda 4.3.

9d6636c

Try 4.5.0

6ee9510

Handle UnableToParse

63f809f

johanneskoester added 11 commits April 24, 2018 09:09

Remove galaxy-lib version restriction.

3664db1

adapt to conda 4.5

20d58eb

use get_output_file_paths instead of get_output_file_path

14aafd2

Merge branch 'cb3-migration' of github.com:bioconda/bioconda-utils in…

76035ea

…to cb3-migration

Merge branch 'master' into cb3-migration

c847294

Add lint for deprecated numpy spec.

f6b5669

Document new lint.

0d7d1cc

register lint.

55e1e14

fix tests

82abcf7

fix lint

3384875

adapt test cases and remove obsolete ones.

38aaf26

johanneskoester changed the title ~~Migrate to conda-build 3.~~ [WIP] Migrate to conda-build 3. Apr 24, 2018

johanneskoester mentioned this pull request Apr 25, 2018

pin libnetcdf to version 4.4.* bioconda/bioconda-recipes#8706

Merged

5 tasks

johanneskoester added 2 commits April 25, 2018 16:56

Remove env_matrix.

07eb371

fix typo

de2e4a2

johanneskoester added 3 commits April 25, 2018 20:31

fixes and further cleanup

6014495

fixes

9a4a3d9

Fixes.

312f982

dlaehnemann mentioned this pull request Jun 11, 2018

unblacklist rust recipes bioconda/bioconda-recipes#9266

Closed

fix mulled upload: iterate over all generated images.

9fdbc8f

mbargull added 4 commits June 19, 2018 18:55

build: fix collecting mulled_images

6c45c87

requirements: require python >=3.6

3ab8023

build: skip builds early based on num of existing builds in channel

d4701c6

build: topo-sort connected components before merging

99782c2

docs: handle extra.notes being a list

e033e47

build: handle UnsatisfiableError

44565d0

mbargull reviewed Jun 24, 2018

View reviewed changes

mbargull and others added 3 commits June 24, 2018 17:40

fix import

86fc487

Add paper to the homepage.

4e35af8

mbargull reviewed Jul 4, 2018

View reviewed changes

mbargull and others added 3 commits July 4, 2018 23:34

requirements: update to conda-forge-pinning 2018.07.18

5800ebe

Merge pull request #315 from bioconda/cf-pinning-2018-07-18

9349490

requirements: update to conda-forge-pinning 2018.07.18

bgruening removed the WIP label Jul 18, 2018

Merge branch 'master' into cb3-migration

9e40c85

johanneskoester merged commit d67288b into master Jul 19, 2018

bgruening deleted the cb3-migration branch September 29, 2018 08:57

epruesse mentioned this pull request Jan 8, 2019

Migrate to get_output_file_paths with conda-build api #195

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Migrate to conda-build 3. #293

Migrate to conda-build 3. #293

johanneskoester commented Apr 23, 2018 •

edited

Loading

johanneskoester commented Apr 23, 2018

johanneskoester commented Apr 23, 2018

johanneskoester commented Apr 23, 2018

bgruening commented Apr 23, 2018

johanneskoester commented Apr 25, 2018

kyleabeauchamp commented Apr 25, 2018

kyleabeauchamp commented Jun 15, 2018

johanneskoester commented Jun 24, 2018

mbargull commented Jun 24, 2018

mbargull commented Jun 24, 2018

mbargull commented Jun 24, 2018

mbargull Jun 24, 2018

bgruening Jun 24, 2018

mbargull Jun 24, 2018

mbargull Jul 4, 2018 •

edited

Loading

bgruening commented Jul 18, 2018

johanneskoester commented Jul 19, 2018

johanneskoester commented Jul 19, 2018

johanneskoester commented Jul 19, 2018

Migrate to conda-build 3. #293

Migrate to conda-build 3. #293

Conversation

johanneskoester commented Apr 23, 2018 • edited Loading

johanneskoester commented Apr 23, 2018

johanneskoester commented Apr 23, 2018

johanneskoester commented Apr 23, 2018

bgruening commented Apr 23, 2018

johanneskoester commented Apr 25, 2018

kyleabeauchamp commented Apr 25, 2018

kyleabeauchamp commented Jun 15, 2018

johanneskoester commented Jun 24, 2018

mbargull commented Jun 24, 2018

mbargull commented Jun 24, 2018

mbargull commented Jun 24, 2018

mbargull Jun 24, 2018

Choose a reason for hiding this comment

bgruening Jun 24, 2018

Choose a reason for hiding this comment

mbargull Jun 24, 2018

Choose a reason for hiding this comment

mbargull Jul 4, 2018 • edited Loading

Choose a reason for hiding this comment

bgruening commented Jul 18, 2018

johanneskoester commented Jul 19, 2018

johanneskoester commented Jul 19, 2018

johanneskoester commented Jul 19, 2018

johanneskoester commented Apr 23, 2018 •

edited

Loading

mbargull Jul 4, 2018 •

edited

Loading