Auto-triage rules for Dependabot

[erinhav]

What do you think about Dependabot's new auto-triage feature?

At GitHub, we’ve been thinking deeply about how to responsibly address long-running issues around alert fatigue and false positives. Rather than over-indexing on one or two criteria like reachability or dependency scope, we believe that a responsibly-designed solution should be able to detect and reason on a rich set of complex, contextual alert metadata.

That’s why, moving forward, we’re releasing a series of ships powered by an underlying, all-new, flexible and powerful alert rules engine!

If you’ve had a chance to try out the beta feature, we'd love to hear your feedback on 1) what's working for you, 2) what needs to change, and 3) what you'd like us to tackle next!

What are auto-triage rules?

Auto-triage rules help you responsibly reduce the amount of alerts needing immediate assessment, without compromising on any potentially concerning alerts. GitHub curates suggested rulesets for everyone, which have caught millions of false positive alerts since the May 2 release. For those that need more control, custom rules are available to help you scale your security strategy based on your risk tolerance and unique contextual needs.

📖 Helpful information:

• Feature FAQ and announcement
• Dependabot alerts product documentation
• Dependabot alerts REST reference
• Dependabot alerts GraphQL reference
• Dependabot alerts webhook reference

[testworksau]

We are on GHEC, and like the ability to be able to enable Dependabot at an org level.

Will you consider adding an option to allow organisation administrators to enable auto-dismissal across all repositories, rather than having to do this on a repo-by-repo basis?

[erinhav]

Absolutely! That's currently roadmapped as the next major project to focus on, after this quarter's ship of custom configurability at the repository level.

[jeremie0]

Will enabling this feature at the repo level be available via the REST API? (at least until we can enable org-wide)

[erinhav]

@jeremie0 would you still make use of enablement via API if we had better UI-based enablement (across org-levels)?

[jeremie0]

If we could enable org-wide via the UI that would replace the need for API.

[erinhav]

Thank you!

[dduzgun-security]

Would love to see support for yarn as well to reduce alerts on devDependencies 🙏
As of now, it seems to work only with npm

[erinhav]

Noted! We're investigating this now. ❤️

[dduzgun-security]

@erinhav I was curious to know if there were any updates on the support for yarn?

[SalimBensiali]

Hi @erinhav any further update on this? Currently I am able to work around this limitation by running: npm ls <pkg-name> --prod in a Github Action that fetches Dependabot alerts for the yarn ecosystem via Graphql, but the problem with this approach is there is no way to update the dependencyScope for the alert metadata via any API if it turns out the scope is indeed DEVELOPMENT instead of RUNTIME

[SalimBensiali]

HI @erinhav 👋
Would you have more info on "custom alert rules" you mentioned in your post?

• I would love to see yarn getting the same support as npm, currently all alerts have the production scope, which renders this auto-dismissal feature unusable.
• I would love to programatically be able to add extra metadata to dependabot alerts. Currently, if Dependabot has successfully managed to raise a PR, I am able to leverage GH Actions to add metadata as tags on the associated PR, but I would love to do this at the alert level so I can also cover the alerts for which Dependabot was unable to raise a PR.

[lasselasse]

This is a great feature which would help us to reduce alert noise.

However, after I enabled the auto-dismiss feature for a private repo, I was surprised to discover after a few days that dependabot had auto-dismissed all open alerts (besides NPM also PIP dependencies which are not even supported according to the docs). I looked at a few alerts to see if the CWEs match the published low impact list but also failed to see a match there. Something clearly seems to be wrong here!

Regarding alert custom rules: It would be great to have the option to auto-dismiss alerts of certain severity levels (e.g. everything low or moderate), configurable per repository and ideally also being able to set an organization default.

[erinhav]

Fix going out now. Thanks for letting us know! ❤️

[lasselasse]

+1 All 3 are in scope. For organization defaults, would those be always on, or would you want some available to all repositories (but off by default, so not enforced initially)?

Great to hear! Any idea about an ETA? (I'm about to write a custom action to dismiss low-severity alerts and I'd gladly save myself that effort...)

Regarding settings: Ideally there would be an organizational default (off by default as you propose) that can then be overridden per repository.

[lasselasse]

Oh dear! That's not right. Have you opened a support ticket? You can also email me details at erinhav@github.com, so we can take a look.

Thanks, I'll get in touch by email for details as this concerns a private org repo.

[lasselasse]

The bugs of all open alerts having been auto-dismissed seems to be fixed.
As of today, there is only a single auto-dismissed alert in the repo in question, which is correctly classified (CWE-1333).
I haven't yet checked if there are other alerts that should have been auto-dismissed...

[Ameralameri]

Good

[Techiral]

Dependabot’s new auto-dismissal feature is a great addition to the tool. It helps to reduce the number of alerts that need immediate assessment, without compromising on any potentially concerning alerts. This feature is made possible by GitHub-curated vulnerability patterns, which take into account contextual information about how you’re using the dependency in order to substantially reduce the number of alerts that make it to your inbox. The initial release focuses on a subset of low impact development-scoped (devDependency) alerts for npm. If you’ve had a chance to try out the beta feature, GitHub would love to hear your feedback on what’s working for you, what needs to change, and which cases of vulnerabilities you’d like them to tackle next! Additionally, custom alert rules are also being introduced, allowing users to write their own logic in order to let Dependabot know which alerts should get a Dependabot pull request, and perhaps even which high-risk alerts should be auto-merged after passing built-in guardrails. This is an exciting development and I’m looking forward to seeing how it evolves! Is there anything else you would like to know? 😊

[henriquevcosta]

This is a highly desirable feature for us since dependabot has been generating a lot of noise with false positives and with it some friction with developers - however rolling this out on a repo by repo basis is problematic for us (few thousand repos) - what's the timeline to have this available as an org-level UI setting OR exposed at REST API level?

[lasselasse]

✋ I'd like to join the private beta.

[steinheber]

@erinhav while not having seen anything out there thus far, how about the API support in that regard ?
The org level enablement for auto triage will not work in the scenario of:

• having a rule that dismisses e.g. golang alert b/c of gofiles being part of the build tooling but not the final output
• however, there may be still repositories in the org that deviate from the above in that also production grade gocode is there

On top of that, administering the GH Org via IaC would also have the desperate need for REST API Support.
This for me raises the general question of any sort of API-First strategy is employed here ?

Thanks a ton for your comments

[carogalvin]

@steinheber thanks for following up! We don't have API support for this feature at this time.

[steinheber]

@carogalvin thanks a ton for your response,

Anything on the roadmap in that regard ?
And if not, where can i file a feature request ?

I do see legitimate scenarios for the topic at hand - especially in the case of having organizations w/ a diverse tech stack that need to treat dependabot alerts with varying urgency depending on the repository at hand.

Best regards,
Thomas

[carogalvin]

@steinheber we don't currently have anything on our roadmap for this; it's something that we have in our backlog as something we'd like to get to sometime. I totally agree that this is a big gap for this area, we're just pretty tightly constrained right now and can't get to all the gaps and feature improvements we'd like to do.

[PaulBernhardt]

I haven't seen how to turn this on, but from the blog post it sounds like we could maybe say certain sub dependencies, or dev vs prod, can be auto dismissed. I think that is not going to be enough. What I would like to be able to say is something like

If ALL of the root level dependencies this sub dependency was brought in by are on this list, AND the severity is not critical, dismiss it

So say my list of acceptable root dependencies is jest and eslint, and a few vulnerabilities come in:

Package A
Brought in by: jest, eslint
Severity: Medium
Action: Dismiss
A medium vulnerability that only impacts jest and eslint cannot possible influence production, close it

Package B
Brought in by: jest, webpack
Severity: Medium
Action: Keep Open
Webpack, though a dev dependency, could be including the vulnerable code in the built project. Not safe to auto dismiss

Package C
Brought in by: jest
Severity: Critical
Action: Keep Open
Maybe this is something like 'library compromised, it will email all your code to some bad guy', can't safely dismiss that.

Also, I would like to be able to set this up at the org level.

[erinhav]

That's great feedback. We're 🤏 this close to completing a major rearchitecture of our dependency graph, which will enable us to surface transitive dependency info in places like alert details and our rules engine.

While this dependency graph work is in flight, unfortunately I see the next step of exposing this data as still being months out (our supply chain team is small).

On the bright side, org-level custom rules and APIs are planned to ship this quarter. Any additional requests regarding those ships?

[lriho]

Could this functionality be enhanced for general dependency handling regardless of security implications? E.g. we keep getting dependabot alerts about latest version of @types/node (20.x.x) even though we're using Node v18 (LTS). So we would like to suppress alerts and have visibility on those suppressions. The current workflow of searching for "@dependabot ignore" in:comments leaves a lot to be desired. Also additional rules to dismiss alerts until minor or major versions are available would be handy.

[ansereb]

I would loke to have an OR operator in rule's criteria field. Something like:
cwe:400 or severity:low
Currently i have to create 2 rules: one with cwe:400 and other with severity:low
Also by some reason cwe:400,1333 is invalid construction and i have to write cwe:400 cwe:1333 .... It becomes very nasty strings with multiple CWE

[vtjballeng]

Dependabot keeps flagging older versions that are in my NuGet migration backups. We are on the current version but get these false positives from our backups. It's a nuisance warning.

[carogalvin]

OK, so to make sure I understand: you have a folder in your repository that represents a backup, and not the code that is being actively maintained used, so you would like to be able to tell Dependabot to ignore that folder for alerts and PRs, is that correct?

[vtjballeng]

@carogalvin , yes that is correct.

It would be more convenient if dependabot made this distinction on it's own. If it cannot then a manual directory exclusion is the next best option.

[carogalvin]

What would you expect Dependabot to use to tell the difference between a directory that it should and should not update, in this case?

[vtjballeng]

@carogalvin , it is certainly possible to recognize a VS repo and Migration Backup as an automatic exclusion as a user convenience. I think that methodology would help the most users seamlessly and prevent dependabot abandonment for the extra hassle from false positives it will inherently cause as-is.

Otherwise a manual user programable directory exclusion is the next best option.

[carogalvin]

Thanks for the feedback! If you have custom auto-triage rules available for your repository, you can automatically dismiss alerts for those manifests. We also have improvements to configuration for Dependabot updates on our roadmap for later this year, where we will be addressing things like telling Dependabot to ignore particular directories for updates.

[bordenit]

It doesn't seem to catch much in my Ruby or PHP based repositories. Would be good if it did more for those. But, for NuGet and .NET it works great.

[carogalvin]

@bordenit could you clarify what you mean by "it doesn't seem to catch much" - are you saying the auto-triage rules aren't dismissing when you expect it to, or that you are not receiving alerts that you are expecting?

[filips123]

I don't know if this is the right place to report this, but I have a problem that GitHub considers all dependencies as "runtime dependencies", even though they are only included as transitive dependencies of devDependencies and not used anywhere else. Because of this, I often receive Dependabot alerts for vulnerabilities in some transitive development dependencies that my project doesn't even use, even though Auto-triage rules are enabled. This happens in projects that use Yarn v1 and with normal package.json and yarn.lock files.

[carogalvin]

@filips123 thanks for your feedback! This is a limitation in our support for Yarn today, unfortunately

[filips123]

Thanks for the notice. Are there any plans to improve support for Yarn, and if so, are there any details approximately when it will be available? Also, is there any detailed explanation about which package managers (for JS and also other languages) are supported with which Dependentbot features?

[carogalvin]

We're investigating the level of effort required here but don't have a target date yet. You can see a list of supported package managers across Dependabot alerts here: https://docs.github.com/en/code-security/dependabot/dependabot-alerts/viewing-and-updating-dependabot-alerts

[makskustomer]

@carogalvin A realization that I am having is that I would like to be able to apply dependabot rules to specific repos based on property or tags. This way I can auto create dependabot PRs for only my Production tagged repos as an example or entirely close out alerts for sandbox repos.

[carogalvin]

Got it, thank you for that feedback!

[CodingSamurai]

I'd love the ability to be able to ignore certain directories from Dependabot. For instance with WordPress sites you'd want to ignore the /plugins/ directory since that code is really managed by a separate team. But you'd still want to know about warnings from code your team owns.

[carogalvin]

Thank you for that feedback!

[CodingSamurai]

@carogalvin any chance a feature like this might be added soon?

[carogalvin]

This is not currently on our planned roadmap

[voxpelli]

I would love a preset for disabling the DDoS kind of security alerts. For eg. all dev tool projects I don't want to get DDoS-related notifications.

I may have managed to solve it through this one, but there are likely more CWE-numbers than that one:

[makskustomer]

I was actually looking at something similar today where I was looking for CWE #'s + whether it was exploitable as a devdependency. Alot of CWE's do not matter when testing locally and can only be exploited in runtime. Would make for great filter if only there werent 1000's of these CWE numbers to sort through.

[isen0011]

While working on Organization-level rules, we found a limitation. If we are working on a new rule that we'd like to eventually roll out to some of our repos, we don't have a way to do that. We have to Enable it for every repo and then need to go to each repo and change the status to "disabled" for the repos where we don't want to use it. It would be very handy to have a fourth status for the rule: Available: it is available to be added to any repo in the organization, but by default is disabled in each of those repos.

[seanpclayton]

Noticed after enabling the default Dependabot auto-triage rule that critical updates were being dismissed. In this case, a prototype pollution vulnerability in node.extend. My guess here is that the rules work in an "OR" condition rather than an "AND" condition. Is there a way to enable rules with an "AND" such that we could dismiss alerts for dependencies that are both low criticality AND tagged development?