URGENT track_features pushing List instead of String - Breaking Artifactory integrations

[adam-browning]

Checklist

• I added a descriptive title
• I searched open reports and couldn't find a duplicate

What happened?

Following an initial investigation we found that conda-forge added petsc-3.22.3-cuda11_complex_he184e62_101.conda 2 days ago that introduced a breaking change where the "track_features" property is holding a value of type List instead of the expected String.
e.g.

{
    "info": {
        "subdir": "linux-64"
    },
    "packages": {
        "petsc-3.22.3-cuda11_complex_he184e62_101.conda": {
            "build": "cuda11_complex_he184e62_101",
            "build_number": 101,
            "depends": [
                "cuda-version >=11.2,<12",
                "cudatoolkit >=11.2,<12",
                "libgcc-ng >=12",
                "libstdcxx-ng >=12",
                "libgcc-ng >=12",
                "libgfortran5 >=11.4.0",
                "libgfortran-ng",
                "libgcc-ng >=12",
                "__glibc >=2.17,<3.0.a0",
                "fftw >=3.3.10,<4.0a0",
                "fftw * mpi_openmpi_*",
                "hdf5 >=1.14.3,<1.14.4.0a0 mpi_openmpi_*",
                "scalapack >=2.2.0,<2.3.0a0",
                "libptscotch >=7.0.6,<7.0.7.0a0",
                "libamd >=3.3.3,<4.0a0",
                "openmpi >=5.0.6,<6.0a0",
                "libblas >=3.9.0,<4.0a0",
                "superlu >=7.0.0,<7.1.0a0",
                "yaml >=0.2.5,<0.3.0a0",
                "metis >=5.1.0,<5.1.1.0a0",
                "libcholmod >=5.3.0,<6.0a0",
                "parmetis >=4.0.3,<4.1.0a0",
                "libspqr >=4.3.4,<5.0a0",
                "libumfpack >=6.3.5,<7.0a0",
                "superlu_dist >=9.1.0,<10.0a0",
                "libklu >=2.3.5,<3.0a0",
                "mumps-mpi >=5.7.3,<5.7.4.0a0",
                "libscotch >=7.0.6,<7.0.7.0a0",
                "liblapack >=3.9.0,<4.0a0",
                "hypre >=2.32.0,<2.33.0a0"
            ],
            "legacy_bz2_md5": null,
            "license": "BSD-2-Clause",
            "license_family": "BSD",
            "md5": "351cd312e5a222b92583c43a31e294bd",
            "name": "petsc",
            "sha256": "4597af484aee5fc844f66ef5a23bb692f02fabe82cba06006beb8ce7a16579c5",
            "size": 23587752,
            "subdir": "linux-64",
            "timestamp": 1738239766514,
            "track_features": [
                "petsc-p-0",
                "petsc-p-1"
            ],
            "version": "3.22.3"
        }
    }
}

We are unsure how this field value was switched from "String" to "Array" but this is causing many integarations to break including our mutual customers using conda and Artifacotry.

We originally reached out to PETSc to see if anything changed on their end, but their response was that nothing changed.

Additionally, we noticed that there is a difference between the conda client and the server schema:

1. Conda client supports list "track_features": list,
   see: https://github.com/conda/conda-build/blob/main/conda_build/metadata.py#L602
2. Conda server expects String

"track_features": {
      "type": "string",
      "description": "The features 'tracked' by the package. Format is a space-delimited string of items. https://docs.conda.io/projects/conda- 
       build/en/latest/resources/define-metadata.html#track-features",
      ...
    },

see: https://github.com/conda/schemas/blob/main/repodata-record-1.schema.json#L115

We ask to check if any change was made to conda-build or perhaps to the conda server to help fix this issue at the root.
We are looking forward to your assistance and collaboration.

Thanks in advance,
Adam

Conda Info

Conda Config

Conda list

Additional Context

No response

[beckermr]

@wolfv petsc is using the v1 format? Is this a rattler-build bug? If so, we may need to rebuild or fix a significant number of packages?

[hmaarrfk]

xref: conda-forge/petsc-feedstock#227 (comment)

[wolfv]

Yes, this seems to be a bug in rattler/rattler-build. Sorry. I'll look into a fix asap.

[wolfv]

I made a PR to fix it in rattler / rattler-build: conda/rattler#1038

[adam-browning]

Hi @wolfv
We're happy that the issue was found
Is there anything that can be done regarding packages that we're already built using the malformed track_features property?

[wolfv]

Yes, we can delete them from the channel and rebuild. Can you write a script to find the ones based on the repodata.json?

[adam-browning]

My engineering team found these 4, but they came up with them because of direct usage in our pipelines via Artifactory.
There could potentially be more that we're not aware of that other users are being impacted from.
We respectfully ask that someone more experienced in the Conda team please look into this

1. petsc-3.22.3-cuda11_complex_h5ce9cbd_101.conda
2. petsc-3.22.3-cuda11_complex_he184e62_101.conda
3. petsc-3.22.3-cuda12_complex_h82b52a3_101.conda
4. petsc-3.22.3-cuda12_complex_h8442a08_101.conda

[beckermr]

Well mark them broken but yeah. Please submit a pr to conda-forge/admin-requests

[jakirkham]

Well mark them broken but yeah. Please submit a pr to conda-forge/admin-requests

Here are the steps to mark a package broken

Please xref this issue in the PR

[oriyafe]

PR opened:
conda-forge/admin-requests#1363