Improve maintenance and contribution process

[jezdez]

Checklist

• I added a descriptive title
• I searched open reports and couldn't find a duplicate

Summary

In a recent conda community call and in Anaconda internal discussions, we were reminded that conda-build has had less focus from the conda maintainer team than conda itself.

My guess is that a lack of in-depth experience with conda package building made maintenance more difficult than conda. Plus, many of the use cases and scenarios encoded in conda-build are outside the scope of regular software development and are easier to code review and engage with when being a package builder.

As such, to improve the maintenance of conda-build, we should focus on:

• Simplifying the maintenance and contributing process further
• Adding experienced packagers to the maintenance team
• Tapping into the existing knowledge in the community of using conda-build
• Having at least one person of conda maintainer team skilled up by being mentored in the packaging practitioners

Linked Issues & PRs

Short-term

Beta Give feedback

1. Mentor conda maintainers by packaging practitioners #4707
   locked source::anaconda type::task +3
   
2. Set up build tools team under the conda governance policy #4698
   in-progress locked source::anaconda type::task +4
   
3. Review testing strategy and implement improvements for better developer experience and test quality and coverage #4979
   stale stale::closed +2
4. Improve the PR template so contributors provide enough information to help the review team understand the context of the request #4980
   locked +1
5. Review release policy of conda-build #4754
   locked severity::4 source::anaconda type::task +4
6. Switch from SemVer to CalVer (i.e., adopt CEP-8) #4975
   3 of 7
   epic locked source::anaconda type::task +4
7. Explore and prototype possible applications of a plugin system like conda #4978
   stale stale::closed +2
8. Mark conda-build skeleton as "Pending deprecation" #4640
   source::anaconda stale stale::closed type::feature +4
9. Replace conda_build/index.py with standalone conda-index #4645
   backlog in-progress locked source::anaconda type::feature +5
   
10. Write and vote on CEP to specify the next-gen recipe format #4755
    severity::3 source::anaconda source::community source::partner type::task +5
    
11. Add end user documentation for recipe format v2
12. Add schema for recipe format v2
13. Create a linter for recipe format v2
14. Sync with efforts around using conda-build/boa for WASM for emerging platform needs

Mid-term

Beta Give feedback

1. Implement a plugin infrastructure like conda (CEP 4) and port an existing feature to a plugin #4932
   stale stale::closed +2
2. Get Anaconda build team to try out next-gen recipe format based on CEP (draft), to get experience and provide feedback to software team
3. Support and interview conda-forge community while trialling next-gen recipe format, to learn about their experience
4. Engage with boa maintainers to expand "build tools" team to also include boa and find common ground (e.g. port code improvements to conda-build, including code quality ;)

Long-term

Beta Give feedback

1. Write evolution plan for conda-build users and its recipe format(s), and how it relates to other community initiatives like boa
2. Execute migration plan together with community and commercial users

Resources

• Support boa recipe format #1612
• The boa recipe spec — documentation
• Notes from boa spec discussion
• Commands and tools for building conda packages
• GitHub - mamba-org/boa: The fast conda package builder, based on mamba

[jaimergp]

Thanks for writing this out! And what is this fancy style for issue grouping? 😮

Now, about the proposal itself :D I see a few items around the declarative boa format. This is still WIP, afaik; even if the current draft is functional, last time we talked we mentioned a few ideas to change it a bit so it resembles a GHA-like workflow a bit more (steps with referenceable identifiers, non-semantic keys for conditions, maybe a couple more things?). Is the idea to iterate together, to follow development as it happens, more collaboration? It feels like we might be chasing a moving target if we start blindly.

Also, I don't see as many items talking about the (many) bugs in meta.yaml right now. Is the overall plan to phase out meta.yaml as is, without trying to address the existing problems, or are we missing a couple items talking about that kind of maintenance? All these efforts can be pursued in parallel by several teams, but meta.yaml is in wide usage today and it still bites builders daily (see this thread for a recent example of a long-time filed bug that affected a seasoned maintainer).

I'm mainly talking about the inconsistencies in the outputs feature, lacking documentation and overall how-to. Even official writeups about how to navigate the known limitations would be helpful! The conda-forge linter even has some logic to detect some common pitfalls, but ideally these are fixed instead of avoided. Having a thorough investigation (including triaging but also interviews with builders) to map this out in documentation would be super helpful already, and it doesn't involve touching the (quite daunting) code at all :D

[kenodegard]

And what is this fancy style for issue grouping? 😮

Tasklists

Is the idea to iterate together, to follow development as it happens, more collaboration?

Yes. Nothing has been decided about the future of conda-build recipes, I expect meta.yaml to be supported for the foreseeable future.

I'm mainly talking about the inconsistencies in the outputs feature

There's no reason we cannot address the known issues with meta.yaml. During the ongoing test pruning/review I found several inconsistencies which would be easy to address if we dedicate some time to it.

[wolfv]

I just ticked a few items relating to v2 recipe format.