Starts work on adding recipe upgrade feature

[schuylermartin45]

Description

First-pass implementation supporting the V1 recipe standard (used by rattler-build, as discussed in #486

As of writing this PR (currently in the DRAFT stage), there are a few issues:
1. conda-recipe-manager is not currently in the conda-forge channel. There is an open PR waiting for a review to add it in.
2. Locally, I can't get ruamel to dump souschef's version of the recipe data into a StringIO stream. I had many issues getting conda to use my local copy of conda-recipe-manager and I think it's related to that. I have ruamel.yaml.jinja installed in the local environment, but I still get an error. Here are some logs for more details:

(grayskull) smartin@Schuylers-MacBook-Pro:grayskull> grayskull pypi -u types-toml                       



#### Initializing recipe for types-toml (pypi) ####

Recovering metadata from pypi...
Starting the download of the sdist package types-toml
types-toml 100% Time:  0:00:00   1.8 MiB/s|####################################################################################|
Checking for pyproject.toml
pyproject.toml not found.
Recovering information from setup.py
Executing injected distutils...
Recovering metadata from setup.cfg
No data was recovered from setup.py. Forcing to execute the setup.py as script
Recovering metadata from setup.cfg
Checking >> -- 100% |#                                                                                  |[Elapsed Time: 0:00:00]
Recovering license info from spdx.org ...
INFO:Best match for license Apache-2.0 was Apache-2.0.
Best matches: [('Apache-2.0', 100.0, 0), ('Apache-1.0', 90.0, 1), ('Apache-1.1', 80.0, 2), ('Apache License 1.0', 67.5, 4), ('Zend-2.0', 55.55555555555556, 3)]
INFO:Github url: https://api.github.com/repos/python/typeshed/license?ref=0.10.8.20240310 - recovering license info
Recovering license information from github...
Recovering license info from repository...
Cloning into '/var/folders/mj/y6v4_y555yx3_370k4sk0_bh0000gp/T/gs-clone-repo-amc55x71'...
fatal: Remote branch 0.10.8.20240310 not found in upstream origin
Recovering license info from repository...
Cloning into '/var/folders/mj/y6v4_y555yx3_370k4sk0_bh0000gp/T/gs-clone-repo-a_k02yuu'...
fatal: Remote branch v0.10.8.20240310 not found in upstream origin
License type: Apache-2.0
License file: []
Build requirements:
  <none>
Host requirements:
  - python >=3.8
  - pip
Run requirements:
  - python >=3.8

RED: Package names not available on conda-forge
YELLOW: PEP-725 PURLs that did not map to known package
GREEN: Packages available on conda-forge

Maintainers:
   - schuylermartin45
Traceback (most recent call last):
  File "/Users/smartin/miniconda3/envs/grayskull/bin/grayskull", line 8, in <module>
    sys.exit(main())
             ^^^^^^
  File "/Users/smartin/work/grayskull/grayskull/main.py", line 295, in main
    generate_recipes_from_list(args.pypi_packages, args)
  File "/Users/smartin/work/grayskull/grayskull/main.py", line 340, in generate_recipes_from_list
    generate_recipe(recipe, config, args.output, args.use_v1_format)
  File "/Users/smartin/work/grayskull/grayskull/utils.py", line 230, in generate_recipe
    upgrade_v0_recipe_to_v1(recipe, recipe_path)
  File "/Users/smartin/work/grayskull/grayskull/utils.py", line 247, in upgrade_v0_recipe_to_v1
    yaml.dump(recipe.yaml, recipe_stream)
  File "/Users/smartin/miniconda3/envs/grayskull/lib/python3.11/site-packages/ruamel/yaml/main.py", line 574, in dump
    return self.dump_all([data], stream, transform=transform)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/smartin/miniconda3/envs/grayskull/lib/python3.11/site-packages/ruamel/yaml/main.py", line 583, in dump_all
    self._context_manager.dump(data)
  File "/Users/smartin/miniconda3/envs/grayskull/lib/python3.11/site-packages/ruamel/yaml/main.py", line 913, in dump
    self.init_output(data)
  File "/Users/smartin/miniconda3/envs/grayskull/lib/python3.11/site-packages/ruamel/yaml/main.py", line 906, in init_output
    self._yaml.get_serializer_representer_emitter(self._output, tlca)
  File "/Users/smartin/miniconda3/envs/grayskull/lib/python3.11/site-packages/ruamel/yaml/main.py", line 644, in get_serializer_representer_emitter
    self.emitter.stream = stream
    ^^^^^^^^^^^^^^^^^^^
  File "/Users/smartin/miniconda3/envs/grayskull/lib/python3.11/site-packages/ruamel/yaml/jinja2/__plug_in__.py", line 88, in stream
    self, Rewriter(val, getattr(self.dumper, '_plug_in_' + typ))
                        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
AttributeError: 'YAML' object has no attribute '_plug_in_jinja2'

[schuylermartin45]

Update: I rebuilt my local environment this morning now that conda-recipe-manager is in conda-forge.

Unfortunately I am still getting:

AttributeError: 'YAML' object has no attribute '_plug_in_jinja2'

when I try to run the conversion flag. Does anyone know if I have a missing dependency locally? Do I have to install a plugin not listed in the environment.yaml file for ruamel?

[schuylermartin45]

Update: I rebuilt my local environment this morning now that conda-recipe-manager is in conda-forge.

Unfortunately I am still getting:

AttributeError: 'YAML' object has no attribute '_plug_in_jinja2'

when I try to run the conversion flag. Does anyone know if I have a missing dependency locally? Do I have to install a plugin not listed in the environment.yaml file for ruamel?

Here are the ruamel packages I have installed in this environment:

ruamel.yaml               0.18.6          py312he37b823_0    conda-forge
ruamel.yaml.clib          0.2.8           py312he37b823_0    conda-forge
ruamel.yaml.jinja2        0.2.4                      py_1    conda-forge

[schuylermartin45]

@marcelotrevisani I'd hate to bother you directly, but have you seen these issues before? I get the feeling that I'm missing some dependencies for ruamel and haven't found much when searching with this error message.

Here's how I've configured my development:

conda env create -f environment.yaml --name grayskull --yes
conda run --name grayskull pip install -e .
conda activate grayskull

I've also tried forcing the environment to use 3.11 by editing environment.yaml:

  - python >=3.8,<3.12

[marcelotrevisani]

Hi, sorry, I have been busy in my personal life and job.
I didn't have time to take a look, but did you try to downngrade the version of your ruamel?

[schuylermartin45]

Hi, sorry, I have been busy in my personal life and job. I didn't have time to take a look, but did you try to downngrade the version of your ruamel?

No I have not and did not think to do that. Do you know off the top of your head what the last known good version was? I don't see it listed in the environment file so I'm guessing it transitively is included with sous-chef?

[schuylermartin45]

Hi, sorry, I have been busy in my personal life and job. I didn't have time to take a look, but did you try to downngrade the version of your ruamel?

No I have not and did not think to do that. Do you know off the top of your head what the last known good version was? I don't see it listed in the environment file so I'm guessing it transitively is included with sous-chef?

Yeah I see it in sous-chef as >=0.15.3 so I'll start from there and see what happens. Thanks!

[schuylermartin45]

Latest update: I've struggled to get conda to use an older version of ruamel.yaml so I have started to use venv and pip to experiment. The oldest version of ruamel.yaml I've used so far is 0.16.10 in this setup and I still get the same error message. pip would not allow me to install 0.15.3 so I'm going to try to see what the oldest version between the two is that pip will let me revert to.

I also ran a brief experiment yesterday and tried to have ruamel dump to a file and not a string stream. That causes the same error, but locally souschef's library is performing the same logic on my local environment without issue. I get the impression that somehow, when called from grayskull, ruamel can't find the jinja2 plugin that is readily accessible in souschef.

[schuylermartin45]

pip will not let me install ruamel@0.15.* but will let me try ruamel@0.16.* and 0.16.1 returns the same error. I think I've ruled out a version incompatibility issue at this point.

[dhirschfeld]

IIRC if you conda install a local .bz2/.conda file it will just do it, ignoring all constraints.

[schuylermartin45]

After talking with @beeankha, we found a workaround. I'm not super proud of it, but I've gotten around the JINJA issue by reading the recipe file produced by souschef::recipe.save(), converting that string, and then rewriting the file with the converted text.

Like I said, it's a hack but it works for the time being. Arguably we could modify this further by having the upgrade option return both a meta.yaml and a recipe.yaml file as we transition between formats.

I'll be moving the PR out of the Draft state now that we have a working solution.