initial version of a sdk issue triage bot #264

devoncarew · 2024-05-30T22:22:39Z

Initial version of a sdk issue triage bot:

use via dart bin/triage.dart <issue>
this is expected to be triggered by a workflow file listening for newly created issues
queries github for the issue information
uses the gemini apis to summarize the issue
uses the gemini apis to classify the issue (area-vm, area-analyzer, ...)
if the issue if not already triaged (doesn't have an area- label) then the tool applies the newly determined labels to the issue

This is not quite yet ready to roll out. We are able to tune a model on ~800 existing triaged issues (see tool/create_tuning_data.dart), but are not able to call that tuned model via the current gemini apis.

A tool to triage issues from https://github.com/dart-lang/sdk.

usage: dart bin/triage.dart [options] <issue>

    --dry-run    Perform triage but don't make any actual changes to the issue.
    --force      Make changes to the issue even if it already looks triaged.
-h, --help       Print this usage information.

I’ve reviewed the contributor guide and applied the relevant portions to this PR.

Contribution guidelines:

See our contributor guide for general expectations for PRs.
Larger or significant changes should be discussed in an issue before creating a PR.
Contributions to our repos should follow the Dart style guide and use dart format.
Most changes should add an entry to the changelog and may need to rev the pubspec package version.
Changes to packages require corresponding tests.

Note that many Dart repos have a weekly cadence for reviewing PRs - please allow for some latency before initial review feedback.

devoncarew · 2024-05-30T22:23:45Z

Sample output from a --dry-run against a recent issue:

Triaging dart-lang/sdk...

## issue https://api.github.com/repos/dart-lang/sdk/issues/55880

title: Macros: Static types are broken

  I'm happy to work on fixing this myself, I just wanted to get some design feedback first. cc @jakemac53 
  
  To represent static types to macros, the following classes are used:
  ...

## gemini summary

The current design for representing static types in macros is overly complex, requiring numerous classes and leading to type erasure during serialization. This issue proposes a solution that separates data and behavior in type implementations, using a single type implementation that transforms its internal representation into a macro representation, thereby simplifying the design and addressing the type erasure problem.

## gemini classification

'''
area-language
type-enhancement
'''

This issue is about the design of static types for macros, which is a language feature. It's also clearly a feature request, as the user is proposing a change to the current implementation.

Exiting (dry run mode - not applying changes).

github-actions · 2024-05-30T22:24:01Z

PR Health

Breaking changes ✔️

Details

Package	Change	Current Version	New Version	Needed Version	Looking good?

Changelog Entry ✔️

Details

Package	Changed Files

Changes to files need to be accounted for in their respective changelogs.

Coverage ✔️

Details

File	Coverage

This check for test coverage is informational (issues shown here will not fail the PR).

API leaks ✔️

Details

The following packages contain symbols visible in the public API, but not exported by the library. Export these symbols or remove them from your publicly visible API.

Package	Leaked API symbols

License Headers ✔️

Details

// Copyright (c) 2024, the Dart project authors. Please see the AUTHORS file
// for details. All rights reserved. Use of this source code is governed by a
// BSD-style license that can be found in the LICENSE file.

Files
no missing headers

All source files should start with a license header.

Package publish validation ✔️

Details

Package	Version	Status
package:dart_flutter_team_lints	3.1.0	already published at pub.dev
package:firehose	0.9.0	already published at pub.dev

Documentation at https://github.com/dart-lang/ecosystem/wiki/Publishing-automation.

mosuem

LGTM, just some nits and ideas.

pkgs/sdk_triage_bot/lib/src/common.dart

pkgs/sdk_triage_bot/lib/src/gemini.dart

mosuem · 2024-06-04T07:46:38Z

pkgs/sdk_triage_bot/lib/src/prompts.dart

+
+Here are the descriptions of the different triage areas:
+
+area-analyzer: Use area-analyzer for Dart analyzer issues, including the analysis server and code completion.


Does it make sense to give an even longer prompt including examples?

So, two things:

The area descriptions here are taken directly from the github label descriptions. I see value in keeping the label descriptions as the source of truth - they can also help serve as docs for manual triage. We're limited to 100 chars for those descriptions however.

We tune against the most recent ~800 issues. That tuning effectively serves the same purpose as feeding lots of examples into this prompt.

I think however that we should feel free to expand this prompt. Perhaps that means including additional triage instructions (some per-area)? That could be on top of the above area- label descriptions.

Co-authored-by: Moritz <mosum@google.com>

pkgs/sdk_triage_bot/.gitignore

pkgs/sdk_triage_bot/bin/triage.dart

natebosch · 2024-06-05T21:06:00Z

pkgs/sdk_triage_bot/lib/src/gemini.dart

+  Future<List<String>> classify(String prompt);
+}
+
+class GeminiServiceImpl implements GeminiService {


[nit] I don't think we're gaining any utility from splitting into an interface and Impl - can we combine these for now?

This separation help make clear what the mocks should implement / the boundaries of the interface. Were you thinking of something like having the mocks just implement the implementations directly?

Yes - having test doubles use the interface from the implementations is the common pattern. The impl should be able to keep any non-interface members private, so from the outside the interface and test doubles won't be any different.

Thanks to the Dart paradigm of all-classes-are-interfaces (by default at least)*Impl classes are typically only used in Dart for cases where the impl needs heavyweight dependencies that a modular build can elide when only the interface is required.

OK - I did remove the extra interfaces. The code is a bit simpler (and I was able to hide the non public interface portions of the implementation classes w/ private fields).

pkgs/sdk_triage_bot/lib/src/github.dart

natebosch · 2024-06-05T21:10:06Z

pkgs/sdk_triage_bot/lib/triage.dart

+  comment += '> $summary\n';
+
+  // create github comment
+  await githubService.createComment(sdkSlug, issueNumber, comment);


Do you ever find that the model isn't able to add value beyond repeating/restating the issue input, or does it seem like it's always useful to get a summary added? I'm wondering if we'll want to find a way to let the model choose not to summarize issues that are already short.

The summary is mostly better than the user-authored title, and a very compact tldr for the issue; reading it is quicker than reading the first comment.

If after using this for a while we decide the summary's not more valuable than the noise it adds, we can remove the summary, choose to only apply it to some issues, ...

natebosch · 2024-06-05T21:11:20Z

pkgs/sdk_triage_bot/lib/triage.dart

+  required GithubService githubService,
+  required GeminiService geminiService,
+}) async {
+  print('Triaging $sdkSlug...');


As we move to productionize this and use it consistently in the long term this we'll probably want to take out the print in favor of a more log driven approach. I don't think we should block on that for now.

pkgs/sdk_triage_bot/tool/create_tuning_data.dart

Co-authored-by: Nate Bosch <nbosch@google.com>

Revisions updated by `dart tools/rev_sdk_deps.dart`. collection (https://github.com/dart-lang/collection/compare/586a5e8..141d83a): 141d83a 2024-06-07 Graciliano Monteiro Passos `CanonicalizedMap`: added constructor `fromEntries`. (dart-archive/collection#347) convert (https://github.com/dart-lang/convert/compare/302af1b..70940e3): 70940e3 2024-06-10 Alex Li ⚡ Upper-cast the return type of the decoder (dart-archive/convert#99) dartdoc (https://github.com/dart-lang/dartdoc/compare/ddb8fb4..3decf1e): 3decf1ed 2024-06-07 Sam Rawlins Restrict validation of runtime renderer files to a single, target version of (dart-lang/dartdoc#3778) ecosystem (https://github.com/dart-lang/ecosystem/compare/bc25c0c..865b2c5): 865b2c5 2024-06-07 Devon Carew update docs and metadata for package:sdk_triage_bot (dart-lang/ecosystem#266) 268b516 2024-06-06 Devon Carew initial version of a sdk issue triage bot (dart-lang/ecosystem#264) http (https://github.com/dart-lang/http/compare/a3567f8..b522000): b522000 2024-06-08 Anikate De fix inconsistent test server behavior (dart-lang/http#1227) mime (https://github.com/dart-lang/mime/compare/8d2d559..4ca2f5e): 4ca2f5e 2024-06-10 Sarah Zakarias Add `topics` to `pubspec.yaml` (dart-archive/mime#126) Change-Id: I6b6279a4ff0af5ba19cc3c4180389b949f48d623 Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/370660 Reviewed-by: Konstantin Shcheglov <scheglov@google.com> Commit-Queue: Devon Carew <devoncarew@google.com>

initial version of a sdk issue triage bot

31d9c8f

github-actions bot added the type-infra A repository infrastructure change or enhancement label May 30, 2024

devoncarew added 2 commits May 30, 2024 15:27

update help message, prompts

8538ab3

adjust prompts and diagnostic stdout output

b393e62

devoncarew requested review from mosuem and natebosch May 31, 2024 19:49

mosuem approved these changes Jun 4, 2024

View reviewed changes

devoncarew and others added 5 commits June 5, 2024 12:47

Update pkgs/sdk_triage_bot/lib/src/common.dart

6c92548

Co-authored-by: Moritz <mosum@google.com>

update the classify method type sig

610ccb9

Merge branch 'main' into triage_bot

c062fc3

regenerate the mono_repo settings

c090a5b

adjust the format of the github comment

f2c253d

natebosch approved these changes Jun 5, 2024

View reviewed changes

devoncarew and others added 4 commits June 5, 2024 14:23

Update .gitignore

b5b8d65

Apply suggestions from code review

426114c

Co-authored-by: Nate Bosch <nbosch@google.com>

rename 'getIssue' to 'fetchIssue'

753a1b5

remove extra interfaces

ec9d438

devoncarew merged commit 268b516 into main Jun 6, 2024
27 checks passed

devoncarew deleted the triage_bot branch June 6, 2024 20:04

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

initial version of a sdk issue triage bot #264

initial version of a sdk issue triage bot #264

devoncarew commented May 30, 2024

devoncarew commented May 30, 2024 •

edited

Loading

github-actions bot commented May 30, 2024

mosuem left a comment

mosuem Jun 4, 2024

devoncarew Jun 5, 2024

natebosch Jun 5, 2024

devoncarew Jun 5, 2024

natebosch Jun 6, 2024

devoncarew Jun 6, 2024

natebosch Jun 5, 2024

devoncarew Jun 5, 2024 •

edited

Loading

natebosch Jun 5, 2024

devoncarew Jun 5, 2024


		Here are the descriptions of the different triage areas:

		area-analyzer: Use area-analyzer for Dart analyzer issues, including the analysis server and code completion.

initial version of a sdk issue triage bot #264

initial version of a sdk issue triage bot #264

Conversation

devoncarew commented May 30, 2024

devoncarew commented May 30, 2024 • edited Loading

github-actions bot commented May 30, 2024

PR Health

Breaking changes ✔️

Changelog Entry ✔️

Coverage ✔️

API leaks ✔️

License Headers ✔️

Package publish validation ✔️

mosuem left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

devoncarew Jun 5, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

devoncarew commented May 30, 2024 •

edited

Loading

devoncarew Jun 5, 2024 •

edited

Loading