Copilot (previously code scanning) Autofix: Preview Feedback and Resources

[turbo]

Welcome to the preview for code scanning autofix!

Autofix is an AI-powered expansion of code scanning that provides users with targeted recommendations to help them fix code scanning alerts in pull requests so they can avoid introducing new security vulnerabilities. The potential fixes are generated automatically by large language models (LLMs) using data from the codebase, the pull request, and from CodeQL analysis.

Read our announcement blog here

This discussion is the place to provide feedback and ask questions about autofix.

Status

Autofix is available to all GitHub Advanced Security (GHAS) customers. Fix suggestions are available on private repositories with a working code scanning configuration.

Capabilities

Fix suggestions are currently generated for nearly all supported security queries for CodeQL. The alerts on PRs and existing alerts on the default branch are supported with the functionality.

To learn more about the capabilities, limitations, and fix generation process, please refer to our public transparency documentation.

For a more hands-on demo of autofix, take a look at this 5-minute walkthrough we've put together.

[samigt]

Love the feature, a couple of questions:

1. How can I easily find the PRs on which the bot added a comment to assess it ?
2. How can I know how many devs accepted the suggestions vs not ?

[AlonaHlobina]

Thanks for your feedback! The Security Overview tab only shows the number of autofixes generated per organisation at the moment. We are working on adding more information soon.

[quinncomendant]

When can independent open source maintainers get their hands on this lovely tool? After reading the announcement post, it seems it's intended for enterprise customers?

[AlonaHlobina]

We appreciate your interest, @quinncomendant!
You are right. The feature is available to private repositories that are part of Advanced Security. We do not plan on shipping it to open source at the moment.

[carlspring]

@AlonaHlobina ,

I think that you guys should consider creating a separate plan for security researchers / engineers, so that the pricing is more reasonable. Otherwise, for a single such engineer one would have to get an Enterprise account (with one user) and pay the GHAS per month. A lot of us are actually working for large corporations which are eager to adopt such funcitonality and we are sort of your "promoters".

[krukow]

We firmly believe that it’s highly important to be both a responsible consumer of open source software and contributor back to it, which is why open source maintainers can already take advantage of GitHub’s code scanning, secret scanning, dependency management, and private vulnerability reporting tools at no cost. Starting in September, we’re thrilled to add Copilot Autofix in pull requests to this list and offer it for free to all open source projects.

@quinncomendant via https://github.blog/news-insights/product-news/secure-code-more-than-three-times-faster-with-copilot-autofix/#securing-open-source

[nalbion]

How do you create the ````suggestion` with "Outside changed files" targeting line 16 of package.json?

[turbo]

This is part of the native integration of code scanning and autofix into the GitHub PR experience. As far as I know, this is not available as a feature for user-generated PR suggestions - that request is tracked here: https://github.com/orgs/community/discussions/9099

[ben-wilson-peak]

Hi, great to see this shipped! I hope eventually we can see autofix suggestions directly in an alert and create a PR from there?

[turbo]

Autofixes for non-PR alerts is something we're actively working on - stay tuned 👍

[krukow]

@ben-wilson-peak this is available now: https://github.blog/news-insights/product-news/secure-code-more-than-three-times-faster-with-copilot-autofix/#pay-down-your-backlog-of-security-debt

[beingminimal]

@turbo can we use it with github enterprise plan in which we will have only 1 user/seat, and if not then why you guys are blocking this? Because nowadays in this Ai era everybody is talking about one person company powered by ai and we are not able to use such ai features for us...
can you please guide me to solve this?

[NickLiffen]

Hey @beingminimal 👋

Lovely to meet you! Thanks for raising the question, if you purchase GitHub Advanced Security on the Enterprise cloud plan, you should be able to purchase it with a minimum of 1 licence, as long as you purchase following the self-serve model, laid out here: https://docs.github.com/en/enterprise-cloud@latest/billing/managing-billing-for-github-advanced-security/signing-up-for-github-advanced-security, if you go here, and you see a minimum that isn't 1, please let us know.

Have a great week.

[beingminimal]

@NickLiffen but can i buy GitHub Advanced Security 1 seat licence if i have only 1 seat in github enterprise cloud plan or we required 10 min seat in github enterprise plan to buy 1 seat of GitHub Advanced Security?

[NickLiffen]

Hey @beingminimal 👋

Thanks again for the follow up

can i buy GitHub Advanced Security 1 seat licence if i have only 1 seat in github enterprise cloud plan

Yes, you can.

There is not a minimum seat requirement on either the GitHub enterprise plan, or GitHub Advanced Secuirty

[beingminimal]

is it for sure?

[turbo]

Yes. Refer to the docs above.

[jvilimek]

Fix suggestions are currently generated for nearly all supported security queries for JavaScript/TypeScript, Java, and Python. We will be adding support for more languages soon. Only new alerts on Pull Requests are considered.

Thanks for this new cool feature!

I am (and I believe many developers among us) looking for C# support. Is this something on the roadmap already? Where to get a notification after it's released?

[turbo]

C# and Go support will be released this calendar quarter and there'll be a changelog blog post for it in the CodeQL category on github.blog

[nalbion]

Nullify provide autofix suggestions for C# @jvilimek - https://www.nullify.ai/

[turbo]

We have just shipped autofix support for C#. It's also the first language that has autofix support for nearly all rules, including the Extended suite: https://docs.github.com/en/code-security/code-scanning/managing-your-code-scanning-configuration/csharp-built-in-queries#built-in-queries-for-c-analysis

[codesafe1]

I found that when multiple vulnerabilities were introduced in PR, the comment provided by github-advanced-security would summarize multiple vulnerabilities into a link, just like the screenshot [CodeQL found more than 20 potential problems in the proposed changes. Check the Files changed tab for more details.], And it no longer provides autofix prompts for commit fixes. Is this the current function, and the automatic repair only takes effect when PR introduces a tiny alarm?

[adityasharad]

Thanks for asking. This is based on the number of alerts identified within the PR diff, and not the severity of an individual alert. When there are 20 or fewer alerts, code scanning renders those alerts as review comments within the PR review page, and autofixes will be generated for them if the alerts are supported by autofix. When there are more than 20 such alerts, the review comments are not created, and the alerts are only shown on the Files changed tab. Since there are no review comments for these alerts, autofix generation is also skipped, because the review comments are the currently-chosen location for rendering the autofix suggestions.

[Kevinf63]

This discussion is the place to provide feedback and ask questions about autofix.

Status

Autofix is available to all GitHub Advanced Security (GHAS) customers. Fix suggestions are available on private repositories with a working code scanning configuration.

Question! I'm wondering if and when this becomes available for its sister product, GitHub Advanced Security for Azure DevOps. What timeframe should we expect?

[AlonaHlobina]

Hi @Kevinf63,
There are no concrete plans to add the autofix feature to GitHub Advanced Security for Azure DevOps. Please keep tuned for updates.

[davewichers]

I have created several discussion posts related to CodeQL and AutoFix. They are here: "CodeQL XSS False Positives and XSS AutoFix incorrect location for defensive encoding" (#122802) (Now also reported here: github/codeql#16531), here: "CodeQL Findings Should be Reported in Filename Order in Pull Requests" (#123182) (Now also reported here: https://github.com/github/codeql/issues/16530), and here: "Relate Adoption of suggested AutoFixes to CodeQL Findings" (#122838). Some feedback from the GitHub team on these suggested enhancements would be appreciated.

Also, rather than creating new Discussions like this, or posting comments here, is there a better/easier way to provide specific CodeQL/AutoFix feedback to the GitHub team, rather than in a public forum? For example, I adopted an AutoFix and it created a compilation error because one of the new methods in the AutoFix throws an additional type of Exception. I want to provide feedback on that specific issue, but posting those details here seems like not the right place for feedback that is super specific like this.

[turbo]

Hi, thank you for your feedback!

This discussion is the right place to publish public feedback about autofix, please do not open separate discussions as those aren't monitored by our team.

For CodeQL issues that are not related to autofix (or where this is unclear), please open an issue in our repo at https://github.com/github/codeql

For private or confidential issues or feedback, please open a support ticket. This will be routed to our team.

We also invite any active users of autofix to book a call with our team at https://gh.io/autofix-feedback-call which is routed directly to a Product Manager working on this feature.

[AishwaryaG1802]

Hi Team,
I enabled codeQL scan and created a repo with java code. I am able to scan the repo and there are few alerts but any how i couldn't see any autofix suggested to implement.
Below are the screenshorts of the alert i am able to see . Can any one let me know if they are pre-requisites or if there is any eligibilty criteria to get auto fixes???

[davewichers]

If you carefully review this entire thread, you'll see there are 2 requirements: the repo has to be in an Enterprise (you can sign up for a 30-day free trial) AND the repo has to be PRIVATE. Once you've done this, you should see AutoFix suggestions, but only IF you have less than 20 open CodeQL findings in the pull request.

[quickchase]

Is there a way to commit multiple suggestions at the same time? There doesn't seem to be a "Add suggestion to batch" as described in the general documentation for applying a suggested change

[AlonaHlobina]

No, unfortunately, bulk actions are not supported at the moment.

[ahmadziahidary]

I'm in

[vadi2]

Would autofix be coming to public repositories as well? I'd like to try out the new C++ autofixes in my FOSS project and provide feedback.

[AlonaHlobina]

Yes, we are working on releasing autofix for our OSS users soon. Stay tuned!

[hharen]

@vadi2 Autofix is now available for all public repositories.

[github-actions[bot]]

💬 Your Product Feedback Has Been Submitted 🎉

Thank you for taking the time to share your insights with us! Your feedback is invaluable as we build a better GitHub experience for all our users.

Here's what you can expect moving forward ⏩

• Your input will be carefully reviewed and cataloged by members of our product teams. 
  • Due to the high volume of submissions, we may not always be able to provide individual responses.
  • Rest assured, your feedback will help chart our course for product improvements.
• Other users may engage with your post, sharing their own perspectives or experiences. 
• GitHub staff may reach out for further clarification or insight. 
  • We may 'Answer' your discussion if there is a current solution, workaround, or roadmap/changelog post related to the feedback.

Where to look to see what's shipping 👀

• Read the Changelog for real-time updates on the latest GitHub features, enhancements, and calls for feedback.
• Explore our Product Roadmap, which details upcoming major releases and initiatives.

What you can do in the meantime 💻

• Upvote and comment on other user feedback Discussions that resonate with you.
• Add more information at any point! Useful details include: use cases, relevant labels, desired outcomes, and any accompanying screenshots.

As a member of the GitHub community, your participation is essential. While we can't promise that every suggestion will be implemented, we want to emphasize that your feedback is instrumental in guiding our decisions and priorities.

Thank you once again for your contribution to making GitHub even better! We're grateful for your ongoing support and collaboration in shaping the future of our platform. ⭐

[kafode]

If co-pilot is going to suggest an "autofix", it would be nice if implementing the "autofix" didn't also generate a CodeQL warning with a new "autofix". I'm about four levels deep into a "URL redirection from remote source" change.

[dluc]

I just noticed some SQL query built from user-controlled sources security warnings in a C# repo. Apparently the security scanner is not taking into consideration directives such as #pragma warning disable CA2100 (https://learn.microsoft.com/en-us/dotnet/fundamentals/code-analysis/quality-rules/ca2100).
Would be nice if the scanner understood this kind of directives, to reduce the number of false positives.

[AlonaHlobina]

cc @coadaflorin

[aibaars]

@AlonaHlobina Isn't this the same as https://github.com/github/code-scanning/issues/7446 and https://github.com/github/customer-feedback/issues/3716 ? Customers can use https://github.com/advanced-security/dismiss-alerts to implement alert suppression, although native support in CodeScanning would be great.

[ebickle]

I noticed a slight UI bug - the Autofix "Beta - Give feedback" UI is appearing for alerts unrelated to CodeQL. Here's an example where it appears on an alert for a different tool that we upload results for into GitHub using SARIF files:

[AlonaHlobina]

Thanks for the feedback!

[alejandro-colomar]

Can you please stop recommending use of strncpy(3)? It's not a string-copying function. It's a very specialized function, designed to be used on utmp(5) and tar(1); nothing else.

[krukow]

cc: @andersfugmann

[marksweiss]

This rule is firing for us on names of static const Apex fields. The regex does not match the Salesforce style guide for such fields here. The rule is therefore generating false positives for this case.

Example:

[boveus]

Hi folks,

The UI for autofix suggestions that suggest code that was not changed in the original PR is confusing. Here is an example:

It is not clear what code this is alerting against - it almost makes it seem like the alert is for the code that is not part of the PR.

I am not sure what the best UI/UX would be for this case, but it might be helpful to make the autofix part distinct from the alert part.

[ChemMitch]

I clicked on the 'Generate Fix' link.
The screen flashed but nothing changed.
Where do I look for a fix or more information?

[ChemMitch]

I used the 'Generate Fix link on a different issue and a suggestion appeared on how to change the code.
Now I know how this is intended to work and it looks useful!

Request: when generating an autofix fails, can you provide some feedback?

[AlonaHlobina]

Hi @ChemMitch! Thanks for the feedback. Is it possible to provide an example or the link to the alert? We would love to take a closer look to investigate an issue

[ChemMitch]

Sure; the repo is private so here's the code
` public static boolean isInputClean(String inputToTest) {

    if( inputToTest == null || inputToTest.length() == 0){

        logger.trace("Input is null or blank");

        return true;

    }

    if( inputToTest.toLowerCase().startsWith("ftp:") || inputToTest.toLowerCase().startsWith("http:") || inputToTest.toLowerCase().startsWith("https:")) {

        logger.trace("input starts with ftp or http - not allowed!");

        return false;

    }

    Pattern urlPattern = Pattern.compile("(http|ftp|https):\\/\\/([\\w_-]+(?:(?:\\.[\\w_-]+)+))([\\w.,@?^=%&:\\/~+#-]*[\\w@?^=%&\\/~+#-])");


    if(urlPattern.matcher(inputToTest).matches()) {

        logger.warn("Input contains a URL!");

        return false;

    }

    if( inputToTest.toLowerCase().contains("<script>")) {

        logger.warn("Input contains a script");

        return false;

    }

    return true;

}

}`

The rule ID is java/polynomial-redos.
Hope that's useful!

[orhantoy]

Is the alert by any chance not in the default branch? The fix is trying to be generated against the version of the alert in the default branch so if it's not there that could explain what you're seeing. We will try to improve the UX for that particular scenario.

[chrisallenlane]

Feedback: after clicking the "Generate Fix" button, it appears to be impossible to hide the auto-generated patch diff. This is frustrating, because the diff produces a lot of visual noise that can make it difficult to reevaluate the vulnerability that was originally reported.

Suggestion: put this patch diff on a new tab on the same page. It would be convenient to be able to tab between the unadorned vulnerability report and the generated patch.

[AlonaHlobina]

Thanks for the feedback, @chrisallenlane!

[sterliakov]

Is the screenshot posted in the blog page real? If so, this is a clear indication that the tool isn't even close to being production-ready (as any other AI-based coding "assistant" I'm aware of, though: this is only about AI hype, not about some useful value). I'll copy that screenshot here for context:

Let's go through the screenshot. It identifies a prototype pollution vulnerability. Well done - though I can't verify whether the vulnerability is real based on two lines of code shown there. If that's copying a hardcoded config reference (no user input), the code is safe. Lets assume the best - the vuln is really there and should be mitigated.

Let's now see what the "intelligent" suggestion says.

1. It begins with installing an external dependency. Believe me or not, it may or may not be a good solution. It suggests this package. It has impressive 30M downloads/week as of now. Good sign. It was last released 9 years ago. Last commit was made 6 years ago. This is a red flag: the library is effectively unmaintained.
2. This causes a package.json update. Okay. package-lock.json updates aren't shown - that's suspicious. Either "autofix" introduces a broken state to be fixed manually or the repo doesn't have package-lock.json committed. Both are scaring.
3. Now... This installed package is unused. The tool severely hallucinates, introducing escape = require('escape-html'). escape is never referenced. So, we just installed a questionable package for fun, to import it, grow our bundle and never come back.
4. The fix also installs and imports whole lodash. Sorry, but _.isString is clearly not a reason to pull whole lodash in - only if it was used before. Besides, it's well established that "full" lodash import is undesired, it should have been import from lodash/isString. And it shows a high-severity vuln attributed to lodash - that means a whole security audit before depending on that library. Are we ok resolving one p-high vuln by (potentially - I didn't look up these details) introducing another one?

Now, let's look whether the reasoning is correct or adequate. Spoiler: no. After emitting two sentences, the tool rapidly confuses prototype pollution and script injection/XSS. That's why we see escape-html, I suspect. HTML sanitizer has nothing to do with prototype pollution vuln. It might be that these two lines were in fact vulnerable to XSS, but not prototype pollution - in that case some of my points should be inverted to "failed to identify the vuln correctly and applies wrong fixes to it".

Finally, !_.isString(key) check. This is NOT a safeguard against prototype pollution. Or even so: this has nothing to do with prototype pollution, to the best of my knowledge. There's nothing inherently unsafe in using, say, integers as object keys - they will be stringified, but it may well be the programmer's intent. It may be OK to do this if you're 100% certain that non-string keys should be rejected. It is not OK to blindly apply such test - it'd be safer to use key.toString() in subsequent tests instead.

The tool as shown doesn't look like something that should be even considered to be run anywhere near production code. It might be a funny toy for fans. The claim that

[...] autofixes help developers to fix vulnerabilities quickly and early in the development process

are not justified - the snippet suggested by autofix can only be thrown away and reimplemented.

Now, to wrap up my post: I'm not a JS developer, my primary language is Python, and I use JS/TS only occasionally. I'd warmly welcome a professional JS developer to join my post and confirm/disprove statements I'm trying to make.

[sj]

You are so right! We drafted that post a little while ago, and accidentally used an old screenshot 🫢. Thanks for bringing that to our attention! We've now corrected the post with an up-to-date screenshot that shows a much newer version of the Copilot Autofix feature, which proposes a correct fix to an actual XSS vulnerability (which is also easier to understand).

To be clear: Copilot Autofix can definitely fix prototype pollution too. Here's a screenshot of what that looks like:

Thanks again for bringing this to our attention. We'll definitely consider your other suggestions too for future updates. If you have any other ideas or comments: please leave them here and we'll make sure to take a look!

[ebickle]

Some quick feedback - I noticed that if an autofix is initiated from a code scanning security alert, the autofix suggestion UI removes some of the existing information from the alert.

Could this information remain visible? For example, the "show paths" link is no longer accessible when the autofix suggestion is shown. The "show paths" information is critical when understanding the data flow resulting in a potential vulnerability.

Thanks!

[AlonaHlobina]

Thank you very much for your feedback!
Currently, the "show paths" information is presented on the alert page. We are also working on improving the page layout to make it more usable and ensure easy access to all alert-related information.

[ebickle]

Thanks! I noticed the same thing afterwards - a change just have been deployed after I saw the older behavior. Page layout looks great, thanks!

[cooljeanius]

Hi, I've been trying this out lately, and one thing that's been bugging me has been how code scanning alert numbers in PR titles created by Copilot Autofix get linked to issues with that same number instead. Is there a way that code scanning alert numbers could somehow get disambiguated from issue numbers so that this kind of mixup doesn't happen?

[hharen]

Hi @cooljeanius, thank you for your feedback. The fix was deployed, for new commits, this problem shouldn't happen anymore.

[cooljeanius]

Hi @cooljeanius, thank you for your feedback. The fix was deployed, for new commits, this problem shouldn't happen anymore.

Thank you!

[ghostinhershell]

Hey everyone! 👋🏾 I'm here to announce that, not only are Security Campaigns with Copilot Autofix now in public preview, but Copilot Autofix now supports partner code scanning tools as well! 🎉

Check out the announcement post here to learn more! [Public Preview] Security Campaigns w/ Copilot Autofix 🧑‍💻