New version of conda-build throws error if channel URL path is conda

[Zaharid]

This seems like a specific conda one. If I look at

https://travis-ci.com/github/NNPDF/nnpdf/jobs/380787948

I see a weird error at the very end.

It can be that the version of conda build is different. Or something specific to our package repositories (given the error).

compiling .pyc files...

number of files: 1703

Traceback (most recent call last):

  File "/Users/travis/miniconda3/bin/conda-build", line 11, in <module>

    sys.exit(main())

  File "/Users/travis/miniconda3/lib/python3.8/site-packages/conda_build/cli/main_build.py", line 474, in main

    execute(sys.argv[1:])

  File "/Users/travis/miniconda3/lib/python3.8/site-packages/conda_build/cli/main_build.py", line 463, in execute

    outputs = api.build(args.recipe, post=args.post, test_run_post=args.test_run_post,

  File "/Users/travis/miniconda3/lib/python3.8/site-packages/conda_build/api.py", line 209, in build

    return build_tree(sorted(absolute_recipes), config, stats, build_only=build_only, post=post,

  File "/Users/travis/miniconda3/lib/python3.8/site-packages/conda_build/build.py", line 3031, in build_tree

    packages_from_this = build(metadata, stats,

  File "/Users/travis/miniconda3/lib/python3.8/site-packages/conda_build/build.py", line 2314, in build

    newly_built_packages = bundlers[pkg_type](output_d, m, env, stats)

  File "/Users/travis/miniconda3/lib/python3.8/site-packages/conda_build/build.py", line 1595, in bundle_conda

    files = post_process_files(metadata, initial_files)

  File "/Users/travis/miniconda3/lib/python3.8/site-packages/conda_build/build.py", line 1444, in post_process_files

    post_build(m, new_files, build_python=python)

  File "/Users/travis/miniconda3/lib/python3.8/site-packages/conda_build/post.py", line 1314, in post_build

    check_overlinking(m, files, host_prefix)

  File "/Users/travis/miniconda3/lib/python3.8/site-packages/conda_build/post.py", line 1220, in check_overlinking

    return check_overlinking_impl(m.get_value('package/name'),

  File "/Users/travis/miniconda3/lib/python3.8/site-packages/conda_build/post.py", line 1055, in check_overlinking_impl

    package_nature = {package: library_nature(package, run_prefix, subdir, bldpkgs_dirs, output_folder, channel_urls)

  File "/Users/travis/miniconda3/lib/python3.8/site-packages/conda_build/post.py", line 1055, in <dictcomp>

    package_nature = {package: library_nature(package, run_prefix, subdir, bldpkgs_dirs, output_folder, channel_urls)

  File "/Users/travis/miniconda3/lib/python3.8/site-packages/conda_build/post.py", line 601, in library_nature

    dsos, run_exports, _ = determine_package_nature(pkg, prefix, subdir, bldpkgs_dirs, output_folder, channel_urls)

  File "/Users/travis/miniconda3/lib/python3.8/site-packages/conda_build/post.py", line 583, in determine_package_nature

    assert isinstance(channeldata, dict) or isinstance(pkg, FakeDist)

[scarlehoff]

Is it possible to have "informative tests"?

So that we have 3 situations:

• Linux fails: failure
• Mac fails: warning
• None fails: all good

[wilsonmr]

So aside from the fact that I use Mac and dislike the builds not being ran for it. I also remember the case where the tests were only failing on one build and it exposed that there was a genuine problem in the code: #779 so I think long term not running mac builds isn't a great idea.

I've been digging around this morning and basically all of the failed builds use conda-build > 3.18 which makes sense because the assertion that is firing was added around 3.19.4/3.20 conda/conda-build#3952

we note that one of the last successful builds after unlocking the conda build version https://travis-ci.com/github/NNPDF/nnpdf/jobs/366999896 uses conda build 3.18.11, I was planning to play around with this and try to confirm if this is true

[scarlehoff]

Just to make my comment clearer, I think we totally should still run the mac builds.
But it would be nice to know when you look at the PR whether the problem was mac or linux (the quick-obvious option is to have two different travis jobs but I don't know if we can do that without paying).

[wilsonmr]

Ah ok, I misunderstood.. That does sound helpful!

[wilsonmr]

Ok I either have no idea what's happening or may be close to sorting this (actually both is probably a fairer assessment)

It would be really helpful if we could see the version of conda build being used in the linux build for comparison between mac and linux in future. If I manage to make a fix for this I will also add that for future debugging.

[wilsonmr]

ok this is super rogue but basically I think we would start getting builds breaking on linux once we moved to 3.8

basically naming a channel conda* appears to be a hook for some internal thing: https://github.com/conda/conda/blob/bf8ad50a694fec32ac09dc6ad83e77f2bdb3024e/conda/models/channel.py#L442

and so the channel name was not being filled correctly - I tested this by temporarily copying https://packages.nnpdf.science/conda/ to https://packages.nnpdf.science/nnpdf-packages/ (I'd previously tried to just change the name using the command line option with conda index but it didnt work) and updating my condarc to point at it and it fixes it for me..

I can't believe how long this has taken me to work out 🤦

[wilsonmr]

the TL;DR is can we change the name of our package channel URLs please 😆

[wilsonmr]

would probably also be nice if we were using the same version of python/conda build on both linux and mac builds but not sure how easy this is

[wilsonmr]

as suspected the linux builds are using an old version of codna build https://travis-ci.com/github/NNPDF/nnpdf/jobs/383591803 and an older version of python

in fact I think that the section in the dockerfile is doing nothing:

RUN wget "$CONDA_URL"      && \
    chmod +x "$CONDA_FILE" && \
    ./"$CONDA_FILE" -b     && \
    rm "$CONDA_FILE"

because I assume this installs into the home directory but from then onwards we use the conda installed in root, unless I am mistaken with how docker works

EDIT: in that build log some info on conda is printed out

[wilsonmr]

@Zaharid it seems to me there are a few things to sort out. Firstly do you agree that we should change the channel directory names? I think this would involve:

• updating directories on server (not sure if there are extra steps with the private channel?)
• updating indexing scripts on server
• update ci scripts to set channels to new locations
• update ci scripts to upload to new location
• update binary bootstrap and documentation
• email everyone and let them know

The second issue is whether we should be using the new version of conda/miniconda on travis? At the moment the Mac build is always performed with latest miniconda and the linux build is always performed with (I guess some preinstalled on the image) python 3.7 conda.

[scarlehoff]

Out of curiosity, is it possible in conda to have legacy channel so that when you use it it still works but you get a message saying "change to this other channel" @Zaharid ?

[wilsonmr]

I don't believe that is supported. The closest you can get it probably to keep the old channels in their current state the downside is there would be no warning and so people would be using out of date packages probably without knowing.

Is there something you're worried about with changing the channel names?

[scarlehoff]

No, just wondering as I thought it would be somewhat cool.

[wilsonmr]

honestly the documentation for setting up your own channel is basically non-existing limited AFAICT to this: https://docs.conda.io/projects/conda/en/latest/user-guide/tasks/create-custom-channels.html

It seems relatively unsupported/undocumented hence why it took so long to find this issue. I guess an alternative would be to just to put the channels on an anaconda account but I guess there was a reason why this wasn't done, maybe it costs money?

[Zaharid]

the TL;DR is can we change the name of our package channel URLs please laughing

Frankly the behaviour of conda is more than a bit rogue here. I would suggest pinning conda build to the latest version known to work and trying to open an issue with them. I am not sure they would be very receptive (seems non trivial to fix) but maybe at least offer some workaround.

Changing the channel names is rather annoying in that you have to ask everyone to do it by hand on every computer, plus a ton of documentation plus the logic in the server, so I'd like to avoid that if possible.

[wilsonmr]

I think opening an issue is a good idea but it wasn't clear to me where to open it:

• conda treats a channel with URL pattern: some.url.com/conda as a special case and the comment in the code indicates this is related to anaconda enterprise. At best this is completely undocumented.
• The problem has arisen because conda-build now checks that all packages have a channel with a name or are local channels I think, I'm not sure why this check was introduced

So is this an issue for conda-build or conda? I agree that I don't think they will be that receptive to changing it.

We can pin conda-build to a set version of course. Would it make sense to be building on both platforms with same version conda-build?

[Zaharid]

I think conda build which has the funky special case. I'd say it's good to pin to the same older version for now. I guess if we find no better solution we will have to resort to changing the channels, but let's leave that for later.

[Zaharid]

@wilsonmr did you open an issue with conda build?

[wilsonmr]

Not yet! I will do

[wilsonmr]

ok opened: conda/conda-build#4057

[Zaharid]

I think that given the lack of response on the conda issue and that the "conda-private" channel name is not going to make a whole lot of sense, we may want to byte the bullet and change the names...

[Zaharid]

I have simplinked the channels so that

channels:
  - https://packages.nnpdf.science/private
  - https://packages.nnpdf.science/public
  - defaults
  - conda-forge

works.