Feature request: allow workflow configuration in sub-folders

[solarmosaic-kflorence]

Copying this feature request over from https://github.saobby.my.eu.orgmunity/t/feature-request-allow-workflow-configuration-in-sub-folders/16776

We use a monorepo setup and currently everything is organized into sub-folders, so for example all services are located in the “services” folder. Currently workflow configuration is the only code which relates to a service that is not in the applicable “services” folder (e.g. “services/foo-service”). In order to maintain clean code separation and ownership, it would be nice if we could place our workflow configuration for a specific service in it’s appropriate folder. The most straightforward solution would seem to be matching on any .github folder in the repo, not just the root one. This would allow us to place workflows in the “services/foo-service/.github/workflows” folder, for example.

[MunchyYDL]

I was searching for a way to organise many workflows in the UI, like by category or by team for a mono-repo. And I was thinking about just being able to create subfolders in the ./github/workflow/ folder, and that each folder would appear as a collapsable section in the UI.

But this proposal would be even better, at least when it comes to separation of files, but I'm not sure how it would be presented in the UI?

[SiavoshZarrasvand]

@MunchyYDL I actually prefer your suggestion.

[amitfmg]

I like the suggestion about collapsible folders on the UI.

[chaddyc]

@MunchyYDL, keen to see if Github will implement/develop this feature.

[garrmark]

I'd like to add my support for the folders under .githuib/workflow that would be displayed as collapsable folders in the UI

[eliezercazares]

This is enough for my needs. I have setup Automated Tests in the Workflows, using the Runners. And it's so many suites and tests that being able to organize my test suites would be cool. Because I also have workflows that do Slack notifications and are reused. So a common/ folder would be good for those.

[zachsa]

I was trying to do:

.github/
  - workflows/
    - dir_1
      - a.yml
      - b.yml
      - etc
    - dir_2
      - a.yml
      - b.yml
      - etc

With the hope that I would see a list of actions as

dir_1_a
dir_1_b
dir_1_c
dir_2_a
dir_2_b
dir_2_c

or something similar. Since the action names are configured in the yml files it would make sense to namespace actions according to the directory they are in. For my use case, aside from namespacing actions via a directory name, nothing would change (though since I use only a small subset of actions functionality I'm not sure if there are other considerations).

It would be nice if it was possible to put .github folders anywhere.

In any case, thank you for the actions tooling! It's pretty amazing.

[krtschmr]

seems quite tricky to scan all folders for a .github folder. subfolders in the .github would be sufficient i believe

[san7890]

It would also be rather nice if there could be a way to run a workflow nested in a submodule, and then leverage that same submoduled workflow to work on the parent repository. This is definitely a much bigger ask than what the original discussion was most likely tailored for, but it would certainly come in use for a repository I currently maintain.

[alfreddagenais]

This will help a lot my team for a mono-repo! I hope to saw this feature soon

[jsoref]

You can do a lot of this by using a services/foo-service/action.yml and then put uses: ./services/foo-service/ into your .github/workflows/foo-service.yml.

It isn't perfect, but, you can do it.

Or, you could just use CODEOWNERS and have that map ownership of .github/workflows/foo-service-* to @foo-service.

Forcing GitHub to recurse through the entire tree, or even a large slice of a tree whenever it receives any event is almost certainly a non-starter. I appreciate the problem, but don't think that asking them to do this is likely to go anywhere.

What you can do is have your workflow load its configuration from a file at a path in your chosen directory. Many public workflows do this, although typically that directory is .github/actions/{action-short-name}/

[tn3rb]

Forcing GitHub to recurse through the entire tree, or even a large slice of a tree whenever it receives any event is almost certainly a non-starter.

what do you think is going to happen if you have a bunch of files like

.github/workflows/foo-service.yml
.github/workflows/bar-service.yml
.github/workflows/baz-service.yml

that all implement use to point to additional actions in subfolders like:

# in .github/workflows/foo-service.yml
uses: ./services/foo-service/ 

# in .github/workflows/bar-service.yml
uses: ./services/bar-service/ 

# in .github/workflows/baz-service.yml
uses: ./services/baz-service/ 

GitHub is going to load all of those actions in the root of .github/workflows and THEN have to ALSO go into those subfolders anyways.

Sound like even more work than just recursing through a folder structure like:

.github
    - workflows
        - foo
            - service.yml
        - bar
            - service.yml
        - baz
            - service.yml

which is not an unreasonable request in this day and age

[jsoref]

That isn't how the GitHub Action Runtime works.

When it gets an event, it only needs to enumerate objects in a single folder (on at most two branches -- default and the current one).

For each object, it then decides if the workflow is enabled, and then if the workflow applies.

If it's enabled and it applies, then it dispatches it to a mechanism that will send things to runners which will then digest the uses lines. It's done much later in the pipeline, and it isn't done by enumerating all objects in the repository, it looks for very specific paths ./services/baz-service/ {action.yaml | action.yml }

[jeffrey-aguilera]

find .github/ -type file -name *.yml is incredibly fast. There is really no excuse for preventing users from organizing their workflows.

[sparr]

@jeffrey-aguilera Not only do some repos contain yaml files that aren't github workflows, some repos contain valid github workflows as the content of the repo, not as workflows to be run on that repo.

[Anonymous-Coward]

You can do a lot of this by using a services/foo-service/action.yml and then put uses: ./services/foo-service/ into your .github/workflows/foo-service.yml.

My biggest gripe with this is that it doesn't give you any possibility to organize workflows in the UI. I have several dozen workflows, over a hundred, which do exclusively the uses thing with a handful of reusable workflows. It's time-consuming, with the current actions UI, to find the one I'm looking for. Without some structure inside /github/workflows, or an artifficially created one by specifying something like `path: ["sub-dir-1", "sub-dir-2", "sub-dir-3"] for the workflows to be shown in a tree structure, I can't see how reusable actions/workflows can help with this problem.

[sam0x17]

A good workaround is symbolic links in your root directory .github/workflows folder

[raihle]

I'm not sure that works at all, but I know it doesn't work when adding, removing, or renaming workflow files

[socketbox]

This does not work

[sam0x17]

yes it does not work unfortunately. Wish it did

[danielfigueiredo]

I've tried using symbolic links and GHA will raise errors when trying to parse workflow files

[Anonymous-Coward]

A good workaround is symbolic links in your root directory .github/workflows folder

Not really. Even if it would work, the workflow links inside .github/workflows would still be presented as soup. You can't easily work with that soup when you have well over a hundred workflows in the same repo.

[zomgbre]

This feature would be awesome to have for when using both re-usable workflows and the "regular" workflow. Right now, I'm naming things with emojis in front of them to do this. 😂 It gets a little ⚡ if it's a re-usable.

[zomgbre]

All I did was paste in the emoji in vscode in front of the name :P it was a little silly:

name: '⚡ ci'
on:
  workflow_call:
    inputs:
      node-version:
        description: Node version
        type: string
        required: true
      working-directory:
        description: Working directory to use
        type: string
        default: './'

jobs:
  test:
    name: Test
    runs-on: ubuntu-latest
    defaults:
      run:
        working-directory: ${{ inputs.working-directory }}
    steps:
      - name: Checkout repository
        uses: actions/checkout@v3

      - name: Use nodejs ${{ inputs.node-version }}
        uses: actions/setup-node@v3
        with:
          node-version: ${{ inputs.node-version }}
          registry-url: 'https://npm.pkg.github.com'

      - name: Install with package-lock
        run: npm ci
        env:
          NODE_AUTH_TOKEN: ${{ secrets.NODE_AUTH_TOKEN }}

      - name: Compile typescript
        run: npm run build
      - name: Tests
        run: npm run test

I have since abstracted these workflows into a separate repository and now referencing them like so across several similar projects:

name: ci
on:
  pull_request:
    branches:
      - main
      - release/**

permissions:
  contents: read

jobs:
  ci:
    uses: orgname/repo-name/.github/workflows/ci.yml@main
    secrets: inherit
    with:
      node-version: 18

[granthorner]

thanks @zomgbre I guessing there's no way to actually display the emoji directly within the github actions list in a way to help group them?

[jsoref]

You can certainly use emoji in the name: field w/in workflows...
https://github.com/jsoref/sample-nested-workflows/actions

[zomgbre]

You can copy the emojis from here: ❤️
https://gist.github.com/rxaviers/7360908

It's gotta be the actual emoji and not the markdown :heart: Hehe, it seems counter-intuitive a bit because you'd think it would render with the colons.

[granthorner]

Ah that's amazing, that's where I was going wrong....much appreciated for your help @zomgbre

[sam0x17]

Apparently it doesn’t work

…

On Tue, Aug 2, 2022 at 6:32 AM Tim Kretschmer ***@***.***> wrote: How can i achieve this? ln -s ? — Reply to this email directly, view it on GitHub <#18055 (reply in thread)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAOE4LLF6VBUR4F74GVLKSLVXD2KHANCNFSM5YELZWBA> . You are receiving this because you commented.Message ID: ***@***.***>

[ErikBjare]

You mean the symlink suggestion?

[rodrigo-georgian]

Instead of recursing through the whole repo, why not provide a keyword in the root .github/workflows that can be used to point to the "sub-root" of the directory you want to treat as it's own repo (for actions)?

Something like this:

In /.github/workflows/package_1.yaml:

redirects: './package_1'

In /package_1/.github/workflows/package_1.yaml:

# regular github actions config, as if this were the root package

Benefits:

• Almost complete code separability. (Only needs to add that one line / file to the root)
• No need to recurse anything.

[rawnly]

Couldn't this can be archived via the uses keyword as mentioned before?

[Updatephoneme]

Copying this feature request over from https://github.saobby.my.eu.orgmunity/t/feature-request-allow-workflow-configuration-in-sub-folders/16776

We use a monorepo setup and currently everything is organized into sub-folders, so for example all services are located in the “services” folder. Currently workflow configuration is the only code which relates to a service that is not in the applicable “services” folder (e.g. “services/foo-service”). In order to maintain clean code separation and ownership, it would be nice if we could place our workflow configuration for a specific service in it’s appropriate folder. The most straightforward solution would seem to be matching on any .github folder in the repo, not just the root one. This would allow us to place workflows in the “services/foo-service/.github/workflows” folder, for example.

[kr99]

This would be a nice feature for our template projects, to make certain workflows optional and easy to tear out (with related scripts). eg. "Not using the event queue system? tear out the workflow and related integration testing scripts by pulling ONE folder..."

Suggested workaround: strictly enforced naming conventions on the workflow files. e.g. "event_queue.integration_test.stage.yml" mirroring a folder src/event_queue or something (I don't like my example a ton, but it illustrates the point)

[tn3rb]

also running a monorepo and having workflow subfolders would be great as we have a lot of granular separation going on.

this would be ideal for us:

.github
    - workflows
        - foo
            - deploy.yml
            - e2e-test.yml
            - unit-test.yml
        - bar
            - deploy.yml
            - e2e-test.yml
            - unit-test.yml
        - baz
            - deploy.yml
            - e2e-test.yml
            - unit-test.yml

[rawnly]

I've been following this thread for a while, if you're interest I made a simple cli in rust that can help with workflows context separation. Is still a WIP, nothing fancy. Hope this helps someone ✌️

https://github.com/rawnly/hawk

[rodrigo-georgian]

Pragmatic solution, thanks @rawnly

[rawnly]

If anyone willing to use it, I just updated and released hawk, now is "generally available" ✌️

[dsdavis4]

Has there been any progress made on this? Being able to define a .github directory within subfolders would be ideal, however what @tn3rb suggested would also be great.

[adikari]

We are using monorepo setup and our .github/worflows is getting out of control. Before Actions, we were using Buildkite and it allowed to add workflows in specific folders for services as suggested by the OP. The root workflow is getting too big and unmanageable and we are considering moving back to Buildkite just for this reason :(

[DavidPriceNBS]

+1

[damiromero-uala]

+1

[fishpen0]

If you don't actually vote on the issue and only reply this way, it does nothing to how GH prioritizes the issue

[lkfortuna]

Hey there from Actions 👋 To provide an update here, this is not something we have planned for this year. Just to set expectations, we will not be revisiting this until next calendar year.

[marcin-sg]

Not a great news, but thank you very much for providing an update!

[jonatasteixeira]

Sad sad. But we will survive! I guess.

[geekmush]

+1

[fishpen0]

If you don't actually vote on the issue and only reply this way, it does nothing to how GH prioritizes the issue

[tt-rkim]

+1

[fishpen0]

If you don't actually vote on the issue and only reply this way, it does nothing to how GH prioritizes the issue

[ashishgupta-in]

Please prioritise this in upcoming releases, much needed feature.

[studiotomi]

I would really like the ability to use subdirectories in the .github/workflows/ directory. that would solve 95% of our issues.

[github-actions[bot]]

💬 Your Product Feedback Has Been Submitted 🎉

Thank you for taking the time to share your insights with us! Your feedback is invaluable as we build a better GitHub experience for all our users.

Here's what you can expect moving forward ⏩

• Your input will be carefully reviewed and cataloged by members of our product teams. 
  • Due to the high volume of submissions, we may not always be able to provide individual responses.
  • Rest assured, your feedback will help chart our course for product improvements.
• Other users may engage with your post, sharing their own perspectives or experiences. 
• GitHub staff may reach out for further clarification or insight. 
  • We may 'Answer' your discussion if there is a current solution, workaround, or roadmap/changelog post related to the feedback.

Where to look to see what's shipping 👀

• Read the Changelog for real-time updates on the latest GitHub features, enhancements, and calls for feedback.
• Explore our Product Roadmap, which details upcoming major releases and initiatives.

What you can do in the meantime 💻

• Upvote and comment on other user feedback Discussions that resonate with you.
• Add more information at any point! Useful details include: use cases, relevant labels, desired outcomes, and any accompanying screenshots.

As a member of the GitHub community, your participation is essential. While we can't promise that every suggestion will be implemented, we want to emphasize that your feedback is instrumental in guiding our decisions and priorities.

Thank you once again for your contribution to making GitHub even better! We're grateful for your ongoing support and collaboration in shaping the future of our platform. ⭐

[nebuk89]

I think I asked this on another of these, but anyone who has feedback on how you would like branches to work with a folder view would be <3 (especially if people are moving workflows between folders etc).

[WesleyWaffles]

I would like it the folder view reflected whatever the current branch's structure is 🤷‍♂️. But give me a way in the Actions UI to surface all of the workflows defined in any branch and run them, similar to how Azure DevOps pipelines work. Honestly if you just gave folks a way in the Actions UI to point to a yaml file and say, "Hey, that's a workflow for this repo" regardless of where it's located that would be good.

[nebuk89]

Thank you for the feedback (and sorry I have been quiet, I was out on personal leave for a couple of months!)

I don't have any news on 'what/when' for this issue yet sadly :( It is still something we are looking at and I am still going to see if we can sneak out search (as we have a PR with it...) in the near future 😄

along with your feedback @WesleyWaffles the general feedback I have heard is just respect branches better in the Actions UI :) I think we have some tidying we can do to make this far better and make folders a simpler UX change (we just need to get the work done.)

Thank y'all for continuing to engage - please keep sharing and upvoting at the top as this gives me the core numbers to keep pushing for changes going forward <3

[sparr]

AFAIK to get workflows to show up in the web view in the first place, they have to be present in the default branch. I'd be ok with a version of this feature that only honored folder structure in that branch.

[serpro69]

They most certainly do not need to be present in a default branch.
Try adding a new workflow to a non-default branch and create a pull-request from it - your workflow will be available.

[nebuk89]

Yep exactly @serpro69 :) thus my question on how we want the UX to behave here. I would love to couple the support for folders with a UX change here and was just curious on others thoughts.

[piotrekkr]

Workflow need to run at least once so it will be "registered" by GitHub and will show up on list in UI. Some workflows will never show up unless merged. For example workflows triggered on schedule event or on workflow_run will not work unless merged to default branch.

[serpro69]

I have a different reason for wanting support for sub-directories (I couldn't care less about lookup of all .github directories in a repo though), e.g. .github/workflows/{foo,bar,baz}/build.yml - and that's versioning of workflows.
We have a single repo with reusable workflows, and you can imagine that there are quire a few of those there. Now, how could I possibly version them? Git tags is not an option, because they're not flexible enough (a tag versions the whole repo, not each individual reusable workflow).
Now, what I could do is version by sub-paths, e.g. .github/workflows/v{1,2,3}. With this approach each workflow file could effectively be versioned individually and could be promoted to a next version w/o promoting workflows that haven't changed.
Imagine my surprise when I discovered that sub-directories in workflow paths are not supported... 🤯
It's almost 2025 for crying out loud. To think that something so simple wouldn't be supported is really unbelievable. Once again I question the usefulness of github actions for anything beyond single/standalone open-source projects.

[romani]

One more use case for folders is usage of README.md to document usage of workflow.
checkstyle/checkstyle#15380

Having documentation for all workflow in single readme is not ideal.

[WesleyWaffles]

Bump. This many years on and GHA is still forcing me into a very specific dir structure for my build definitions 😞

[dkfellows]

At the very least, can we have workflow_call workflows be allowed elsewhere, provided we use the full path to them? There'd be no searching involved. This definitely doesn't work right now.

(I'd be fine with the location being required to be below .github; a .github/reusable_workflows sibling of .github/workflows would still clean up things a lot in some of my repositories.)

[skucherenko]

I don't get it to be honest, it's a feature that makes GH Actions close to unusable and the reason why we avoid using it. It's enough to have 3 envs (dev, prod, staging) and the tasks for each of them to have an unmanageable mess. I don't get why people are adding new features to the product ignoring the glaring critical UX issue for years.

[keriati]

As we have a bigger mono repo with multiple apps and teams, and a decision was made to move to github actions, our platform team made a custom implementation for this issue.

We have a mono repo with multiple apps. The teams can put their app relevant github workflows into apps/<app_name>/.github/workflows/ directory. The teams can also use sub folders if they want.

A custom script is then renaming and moving all jobs to the root .github/workflows folder at commit time, checking consistency on pull requests.

We also adjusted the github UI, to have a drop-down menu for all the jobs with a user script. There is also a script checking if everything is in sync and doing a few more cleanups.

We also added support for "reusable" and "global" workflows.

I hope at some point we can drop our custom solution and this will be supported out of the box, especially on github enterprise.

[c-ameron]

cool approach! would you mind sharing how you did the drop down menu?

[keriati]

I sadly can't share code, but it's not that difficult, as the job name pattern is:

App / SomeApp / Job Name
App / SomeApp / Some Other Job
App / SomeOtherApps / Job Name 1
App / SomeOtherApps / Job Name 2

The dom can be scanned for it and then we can build a <ul> based tree and add some CSS to make it look nice.
(Yes, chatgpt can generate a user-script for this without issues.)