macOS builds failing universally

[matthewfeickert]

Bug:

@conda-forge/staged-recipes This is already known to people given #28214 (comment), but to have a tracking issue for people, all macOS builds are failing with errors along the lines of this snippet taken from the full build_log_failure.txt (this is taken from PR #28224).

Errors:

Building oneloop with /Users/runner/bld,conda-forge
2024-11-16T23:06:14 Subdir: noarch Gathering repodata
2024-11-16T23:06:14 noarch Writing pre-patch repodata
2024-11-16T23:06:14 noarch Applying patch instructions
2024-11-16T23:06:14 noarch Writing patched repodata
2024-11-16T23:06:14 noarch Building current_repodata subset
2024-11-16T23:06:14 noarch Writing current_repodata subset
2024-11-16T23:06:14 noarch Writing index HTML
Traceback (most recent call last):
  File "/Users/runner/miniforge3/lib/python3.12/site-packages/conda/core/index.py", line 310, in data
    return self._data
           ^^^^^^^^^^
AttributeError: 'Index' object has no attribute '_data'. Did you mean: 'data'?

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/Users/runner/miniforge3/lib/python3.12/site-packages/conda/gateways/repodata/jlap/fetch.py", line 309, in request_url_jlap_state
    response = download_and_hash(
               ^^^^^^^^^^^^^^^^^^
  File "/Users/runner/miniforge3/lib/python3.12/site-packages/conda/gateways/repodata/jlap/fetch.py", line 245, in download_and_hash
    response.raise_for_status()
  File "/Users/runner/miniforge3/lib/python3.12/site-packages/requests/models.py", line 1024, in raise_for_status
    raise HTTPError(http_error_msg, response=self)
requests.exceptions.HTTPError: 404 Client Error: None for url: file:///Users/runner/bld/noarch/repodata.json.zst

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/Users/runner/miniforge3/lib/python3.12/site-packages/conda/gateways/repodata/__init__.py", line 203, in conda_http_errors
    yield
  File "/Users/runner/miniforge3/lib/python3.12/site-packages/conda/gateways/repodata/jlap/interface.py", line 87, in repodata_parsed
    repodata_json_or_none = fetch.request_url_jlap_state(
                            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/runner/miniforge3/lib/python3.12/site-packages/conda/gateways/repodata/jlap/fetch.py", line 332, in request_url_jlap_state
    response = download_and_hash(
               ^^^^^^^^^^^^^^^^^^
  File "/Users/runner/miniforge3/lib/python3.12/site-packages/conda/gateways/repodata/jlap/fetch.py", line 245, in download_and_hash
    response.raise_for_status()
  File "/Users/runner/miniforge3/lib/python3.12/site-packages/requests/models.py", line 1024, in raise_for_status
    raise HTTPError(http_error_msg, response=self)
requests.exceptions.HTTPError: 404 Client Error: None for url: file:///Users/runner/bld/noarch/repodata.json

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/Users/runner/work/1/s/.ci_support/build_all.py", line 345, in <module>
    build_all(os.path.join(root_dir, "recipes"), args.arch)
  File "/Users/runner/work/1/s/.ci_support/build_all.py", line 184, in build_all
    build_folders(recipes_dir, folders, arch, channel_urls)
  File "/Users/runner/work/1/s/.ci_support/build_all.py", line 220, in build_folders
    conda_resolve = conda.resolve.Resolve(index)
                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/runner/miniforge3/lib/python3.12/site-packages/conda/resolve.py", line 130, in __init__
    groups = groupby(lambda x: x.name, index.values())
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/runner/miniforge3/lib/python3.12/site-packages/conda/common/iterators.py", line 21, in groupby_to_dict
    for key, group in itertools.groupby(sequence, keyfunc):
                      ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "<frozen _collections_abc>", line 912, in __iter__
  File "/Users/runner/miniforge3/lib/python3.12/collections/__init__.py", line 1144, in __iter__
    return iter(self.data)
                ^^^^^^^^^
  File "/Users/runner/miniforge3/lib/python3.12/site-packages/conda/core/index.py", line 312, in data
    self._realize()
  File "/Users/runner/miniforge3/lib/python3.12/site-packages/conda/core/index.py", line 375, in _realize
    self._data.update((prec, prec) for prec in subdir_data.iter_records())
                                               ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/runner/miniforge3/lib/python3.12/site-packages/conda/core/subdir_data.py", line 288, in iter_records
    self.load()
  File "/Users/runner/miniforge3/lib/python3.12/site-packages/conda/core/subdir_data.py", line 259, in load
    _internal_state = self._load()
                      ^^^^^^^^^^^^
  File "/Users/runner/miniforge3/lib/python3.12/site-packages/conda/core/subdir_data.py", line 305, in _load
    repodata, state = fetcher.fetch_latest_parsed()
                      ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/runner/miniforge3/lib/python3.12/site-packages/conda/gateways/repodata/__init__.py", line 730, in fetch_latest_parsed
    parsed, state = self.fetch_latest()
                    ^^^^^^^^^^^^^^^^^^^
  File "/Users/runner/miniforge3/lib/python3.12/site-packages/conda/gateways/repodata/__init__.py", line 847, in fetch_latest
    raw_repodata = repo.repodata_parsed(cache.state)  # type: ignore
                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/runner/miniforge3/lib/python3.12/site-packages/conda/gateways/repodata/jlap/interface.py", line 86, in repodata_parsed
    with conda_http_errors(self._url, self._repodata_fn):
         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/runner/miniforge3/lib/python3.12/contextlib.py", line 158, in __exit__
    self.gen.throw(value)
  File "/Users/runner/miniforge3/lib/python3.12/site-packages/conda/gateways/repodata/__init__.py", line 268, in conda_http_errors
    raise UnavailableInvalidChannel(
conda.exceptions.UnavailableInvalidChannel: HTTP 404 UNAVAILABLE OR INVALID for channel Users/runner/bld <file:///Users/runner/bld>

The channel is not accessible or is invalid.

You will need to adjust your conda configuration to proceed.
Use `conda config --show channels` to view your configuration's current state,
and use `conda config --show-sources` to view config file locations.

As of conda 4.3, a valid channel must contain a `noarch/repodata.json` and
associated `noarch/repodata.json.bz2` file, even if `noarch/repodata.json` is
empty. Use `conda index /Users/runner/bld`, or create `noarch/repodata.json`
and associated `noarch/repodata.json.bz2`.


##[error]Bash exited with code '1'.
##[section]Finishing: Run OSX build

build_log_failure.txt

[jakirkham]

Thanks Matthew! 🙏

Perhaps we want something like this: #28243

Do you have a candidate PR for testing with that change? If so, could just try merging that into the candidate PR to see if it helps. If it does, we can merge. If not, at least we can start iterating on a solution

[matthewfeickert]

Thanks @jakirkham. I'm AFK currently, but yes, PR #28224 is one I have open. I can give #28243 a look once back.

[matthewfeickert]

@jakirkham I've cherry picked PR #28243 into my PR #28224, but am getting the same failing behavior there as what's included in the above build_log_failure.txt.

[lbianchi-lbl]

Thanks @matthewfeickert for opening this issue and @jakirkham for looking into this issue.

I'm also experiencing this in my PR #28221. I've just rebased my changes on top of 6384969, but I'm still getting the same error: https://dev.azure.com/conda-forge/84710dde-1620-425b-80d0-4cf5baca359d/_apis/build/builds/1082732/logs/24

With the important caveat that I'm not familiar with the repository, I've tried looking around for recent changes to the build infrastructure that could be related with the current macOS failures, and I've noticed that:

• There have been changes to .scripts/run_osx_build.sh (https://github.com/conda-forge/staged-recipes/blame/main/.scripts/run_osx_build.sh) around 4 days ago
• Looking at the Azure pipelines (https://dev.azure.com/conda-forge/feedstock-builds/_build?definitionId=2259&_a=summary), I tried "bisecting by eye" to find the point where this error first appeared, which seems to be between runs 20241114.63 (passing) and 20241114.64 (failing). Note that these are not for the same PR, but Run 20241114.64 for Add recipe for r-glmtrans #28205 is the first instance of this error I could find in the Azure logs
• For Add recipe for r-glmtrans #28205, even though there were other build failures happening at the time, one can still see a difference between:
  • 3af24f8, corresponding to Run 20241114.18, which failed after about 8 minutes; and
  • 52610f1, corresponding to 20241114.64, which failed much earlier
  • A diff of these two build logs (after removing the timestamps) is here: https://www.diffchecker.com/tSmAis99/

[jaimergp]

@zklaus does this ring a bell? I think it's related to your changes in conda. We might need to require a more recent conda-build to ensure it's compatible with latest conda.

Do we have info about the conda and conda-build versions involved in the error? I can't see the logs from mobile, sorry.

[jaimergp]

Ah wait, that diffchecker thing does work (and it's genius!). It does have to do with a recent conda change then. I'll take a look tomorrow, thanks. And sorry!

[matthewfeickert]

Thank you for the fix @jaimergp!