Support YAML Anchors

[kyeotic]

Note to incoming readers (2024)

This issue has been open for several years, and despite commitment from staff that it would be done there has been no progress. I will update this if anything changes, you don't need to read this entire thread.

UPDATE Aug 2024: The latest comment from staff says they will be looking at this next year

Thanks for engaging on this all, this is on our list to get to in the early part of next year. Right now we are re-working the services in Actions which do our YAML expansion (focusing on how we improve their availability). Once we are live with the new services we will start looking at adding features and this is on our list <3

If you want to help, upvote this comment by reacting to it with 👍. This will help it sort higher in the issues list.

Adding "me too" or "+1" comments does not help. Issues are not sorted by the number of comments, they are sorted by the number of reactions to the top issue comment (this comment). The only thing you accomplish by adding "me too", or "+1" is to send an email to issue subscribers (the other people who have commented, not staff). This will not help the issue get done faster, it does not get seen by GitHub staff.

Original Issue

---

Describe the enhancement
Support YAML anchors in in workflow files

Code Snippet

  - env: &docker_cred
      DOCKER_USER: ${{ secrets.DOCKER_USER }}
      DOCKER_PASS: ${{ secrets.DOCKER_PASS }}
# ...
  - env:
      <<: *docker_cred

OR

anchors:
 docker_hub_credentials: &docker_hub_credentials
   credentials:
     username: my-dockerhub-username
     password: ${{ secrets.DOCKER_HUB_PASSWORD }}

 postgres_service: &postgres_service
   postgres:
     image: mdillon/postgis:9.6
     <<: *docker_hub_credentials

Additional information
This has been asked for before, in other places

• Issue from starter-workflows
• highly-upvoted comment from compose actions issues

Note to implementors

This is generally handled by YAML parsers automatically. Perhaps it has been disabled, or a parser that does not support it is being used. This might be as simple as switching YAML parsers.

[freitasmurillo]

3+ years of people complaining and u don't take any action?
Either, say u won't do or do it already. not a big deal to implement.

[DanySK]

This is truly absurd. I think that, given the existence of so many YAML parsers, it is harder to not support anchors and merge keys than doing it. Just let a normal YAML parser run.

[kyeotic]

@DanySK (and to a lesser degree @freitasmurillo), while I appreciate the support on issues, since upvotes help show priority, I don't think hurling abuse at the maintainers or anyone else is going to get this done any faster. If either of you are professional software developers I'm sure you can relate to working on project's whose feature requests exceed your team's bandwidth. There are 300 open issues on this repository and some of them describe actual bugs. Surely this is not the highest priority ticket.

Please keep in mind there is a human on the other side reading these messages. Consider how you might react to someone who said as much to you.

[DanySK]

Hi @kyeotic,
I have no intention to be rude to people, much less to those whose work benefits the FOSS community so much. I expressed a technical concern that is puzzling me from a software engineering perspective, and that you also highlighted in the first post. I did not think that could hurt others -- usually, there is either some deep issue behind it, or it could just be a matter of applying small correctives.

That said, yes I know that it's impossible to keep up with the pace of feature requests in many cases, and especially so for FOSS. Given that, if someone can help me figure out where the YAML parsing happens, I'm willing to help.

[midgleyc]

The error message appears in YamlObjectReader.cs which references YamlDotNet.Core (and the csproj references version 5.3.0), so I expect that's the parser.

They're a few major versions behind, but it looks like there are bugs even on the latest version, e.g. this one from August. I'm not sure if that can be ignored, because they're not using the serialization from YamlDotNet, only the parser, and they've written their own custom code on top of the library.

So in short, I think it's a custom parser, written on top of another parser. The parser looks very local in how it parses things, so supporting anchors might require a bit of a rearchitect, which would explain why it's been open for so long.

[freitasmurillo]

@kyeotic sorry for being late to get your response. I definitely wasn't abusing anyone, sorry if that is what it sounded.
Not having English as my primary language probably helped a lot. 😞

What I was trying to convey is that 3 years without replying seemed to me a lot, u know?
This feature would be amazing and I know that they have a lot but I think once in a while they could provide updates and let people know what is on their plate or not for the moment.

If they don't have a minimal clue on if they will do or not post it on the thread. 💥

[freitasmurillo]

For the feature request POV I am more than happy to help anyhow.

[aramfe]

As much as we love GitHub Actions, the lack of this really makes us think about to switch back to GitLab. Re-using steps shouldn't be that complicated to work with (composite actions, etc. aren't really comparable to in-file templates).

Right now, GitHub doesn't seem to focus on their enterprise and power users, which need to work with complex workflows. It's understandable, as GitHub is very much more driven by the open-source community, but it nevertheless doesn't change the fact, that a lack of something like this, is very frustrating for enterprise users.

It simply makes it hideous to build complex workflows without a feature like that, even though GitHub has the best fundaments for building complex workflows in comparison to any other CI/CD platform.

When we evaluated both, GitHub and GitLab, we always expected in-file templating support to be released very soon for GitHub and that's why we never really considered the lack of this, as a decision defining point in the outcome of our evaluation. But looking back to it from today, it would have definitely changed the outcome of our evaluation, without any doubt.

GitLab CI / CD supports this since more than like three years already, and I haven't seen anyone, who didn't use it for their pipeline definition files. In recent time, they supported this without any anchors, which made it even easier to work with.

I would really love to see this being supported here as well.

[umarcor]

Right now, GitHub doesn't seem to focus on their enterprise and power users, which need to work with complex workflows. It's understandable, as GitHub is very much more driven by the open-source community, but it nevertheless doesn't change the fact, that a lack of something like this, is very frustrating for enterprise users.

@aramfe, your wording seems to imply that enterprise and power users have more complex requirements than the open-source community. I'd say that most enterprise and power users in GitHub are producing open-source software. Thus, that's a false dicotomy.

[kyeotic]

I'd say that most enterprise and power users in GitHub are producing open-source software

I don't know if you have any data to back this up, but I would be very, very surprised to learn that this was true. I expect most enterprises, even those using GitHub, to be developed closed-source, proprietary software.

[umarcor]

@kyeotic it's an opinion, based on the enterprises I know which use GitHub. Nevertheless, the point holds regardless of quantitative data: licensing of a software project is unrelated to the complexity. Moreover, the difference between enterprise users and others is that enterprise clients pay GitHub, not the licenses they use or the complexity of their projects.

[mehran-sayyah]

Ok

[umarcor]

FTR:

mithro/actions-includes allows to work around the lack of YAML anchor support, by preprocessing Workflow files.

Alternatively, in hdl/containers, a Reusable Workflow is combined with a Python script in order to dynamically generate the matrix by processing a YAML file with anchors.

• The Reusable Workflow: common.yml.
• The YAML file: config.yml (see section jobs)
• Calling the Reusable Workflow in a regular Workflow, and passing the key to decide which job matrix to generate: boolector.yml.
• Wrapping the Reusable Workflow in a dispatchable Workflow: dispatch.yml. See pyTooling/Actions: Callable vs dispatchable workflows.

A similar approach is used in the Params Reusable Workflow of pyTooling/Actions: Params.yml.

[mehran-sayyah]

#1059

[michael132]

would love to have anchor support in github workflow files as well :)

[ethomson]

I agree, we need to do this. In the meantime, we've made a number of improvements in simplifying complex workflows, like adding reusable workflows. There are some tricky bits about this implementation - as we have multiple components that deal with workflows (it's not just the runner in this repo) - but this is on the backlog. I don't have an ETA, but yes, we will do this.

Thanks everybody for the feedback.

[caspiano]

This is a desperately needed feature.
Without it, basic things like abstracting out a single static value requires the use of the env key, which is not supported across all of the GitHub Actions format.

[dreinhardt89]

+1

[kyeotic]

Please stop adding "me too" and "plus one" comments. They do not help, all they do is send an email to all the other people in this thread. If you want to help, upvote the original comment.

[flobernd]

This is a lost case anyways..

[Felixoid]

Please stop adding "me too" and "plus one" comments. They do not help, all they do is send an email to all the other people in this thread. If you want to help, upvote the original comment.

Neither is useful, let's be honest. But at least the pinging brings some hope to this world.

And having such a long case without any action from the dev team can prevent poor souls from jumping into using GH actions.

[kyeotic]

"me too" comments do not bring hope to me, because I know GitHub staff are not seeing them. Actually, they just remind me how unlikely this issue is to be fixed, by bringing my attention back to the issue. My attention here is not useful to anyone.

They are unlikely to deter anyone more than the mere fact that this issue is 3 years old, especially considering they will just get hidden in the "more items" fold.

If you don't think upvoting is useful, then just move on. This issue is a graveyard, let the dead sleep.

[Jackenmen]

GitHub mostly just looks at their community feedback discussions page so I don't think there's even much point in this issue - if you want your voice to have a higher chance of being seen (and heard) then you should upvote/comment on the relevant discussion instead:
https://github.com/orgs/community/discussions/4501

[beeman]

GitHub mostly just looks at their community feedback discussions page so I don't think there's even much point in this issue - if you want your voice to have a higher chance of being seen (and heard) then you should upvote/comment on the relevant discussion instead: https://github.com/orgs/community/discussions/4501

Sounds cute in theory, but GitHub completely ignores the second most upvoted discussion in the history of the platform.

The problem is that nobody at GitHub cares. So we can rant wherever we want, they'll swiftly mute us then rush to get a promotion by shoving a new AI powered something in our face.

[flobernd]

The feedback discussions are not actively monitored as well. I don't blame them - the amount of threads per day is insane.

Regarding the YAML anchors: There were several comments from employees, stating that this feature most likely never gets implemented due to technical constraints introduced by their current architecture (something about the way these workflow files are fetched and interpreted).

[aogier]

Are we redditing over github issues, perchance?

[marcellodesales]

Problem

• Duplication of similar configs

  pull_request:
    branches:
      - develop
      - feature/**
      - bugfix/**
      - main
    paths:
      - '3.11/**'
      - '.github/workflows/python-3.11-docker-image-cicd.yaml'
      
  push:
    branches:
      - develop
      - feature/**
      - bugfix/**
      - main

    tags:
      # PRD environment: once we decide on tagging the pipeline with a semantic version. Placed on master branch
      - 'v*'

    # https://docs.github.com/en/actions/using-workflows/workflow-syntax-for-github-actions#example-including-paths
    paths:
      - '3.11/**'
      - '.github/workflows/python-3.11-docker-image-cicd.yaml'

Achors support

• Why not using Anchors support similarly implemented by the Docker team using x- objects?
  • Say g-

# Proposed support with a prefixed `g-` object like Docker
g-on-branches-and-paths: &on-branches-and-paths
    branches:
      - develop
      - feature/**
      - bugfix/**
      - main
    paths:
      - '3.11/**'
      - '.github/workflows/python-3.11-docker-image-cicd.yaml'
  
# Regular Github workflow object tree
on: 
  pull_request: *on-branches-and-paths
  push: *on-branches-and-paths

[nebuk89]

Thanks for engaging on this all, this is on our list to get to in the early part of next year. Right now we are re-working the services in Actions which do our YAML expansion (focusing on how we improve their availability). Once we are live with the new services we will start looking at adding features and this is on our list <3

[flobernd]

Wow thats good news after such a long wait. Thanks for keeping us up to date @nebuk89 🙂