Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

update the prompt to include needs-info #296

Merged
merged 5 commits into from
Sep 4, 2024
Merged

update the prompt to include needs-info #296

merged 5 commits into from
Sep 4, 2024

Conversation

devoncarew
Copy link
Member

  • update the prompt to include triaging issues w/ needs-info labels
  • [breaking change] issues should have breaking-change-request labels but don't necessarily need area labels
  • add 5 'few-shot' examples in the classification prompt
  • update benchmark expectations
  • classification performance has generally improved from 56% to 65%

  • I’ve reviewed the contributor guide and applied the relevant portions to this PR.
Contribution guidelines:

Note that many Dart repos have a weekly cadence for reviewing PRs - please allow for some latency before initial review feedback.

Copy link

PR Health

Breaking changes ✔️
Package Change Current Version New Version Needed Version Looking good?
Changelog Entry ✔️
Package Changed Files

Changes to files need to be accounted for in their respective changelogs.

Coverage ✔️
File Coverage

This check for test coverage is informational (issues shown here will not fail the PR).

API leaks ✔️

The following packages contain symbols visible in the public API, but not exported by the library. Export these symbols or remove them from your publicly visible API.

Package Leaked API symbols
License Headers ✔️
// Copyright (c) 2024, the Dart project authors. Please see the AUTHORS file
// for details. All rights reserved. Use of this source code is governed by a
// BSD-style license that can be found in the LICENSE file.
Files
no missing headers

All source files should start with a license header.

Package publish validation ✔️
Package Version Status
package:dart_flutter_team_lints 3.2.0 already published at pub.dev
package:firehose 0.9.2 already published at pub.dev

Documentation at https://github.com/dart-lang/ecosystem/wiki/Publishing-automation.

@devoncarew devoncarew requested review from mosuem and lrhn September 3, 2024 15:28
pkgs/sdk_triage_bot/lib/src/gemini.dart Show resolved Hide resolved
pkgs/sdk_triage_bot/lib/src/gemini.dart Outdated Show resolved Hide resolved
@@ -43,7 +44,7 @@ class GeminiService {
/// On failures, this will throw a `GenerativeAIException`.
Future<List<String>> classify(String prompt) async {
final result = await _query(_classifyModel, prompt);
final labels = result.split(',').map((l) => l.trim()).toList();
final labels = result.split(',').map((l) => l.trim()).toList()..sort();
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

How big do we expect the result to be? (Or rather, how many commas?)

This can do an extra string allocation per part, if the result text has space after commas.

Doing

  final labels = result.trim.split(_trimCommaRE).toList()..sort();
...

final _trimCommaRE = RegExp(r'\s*,\s*');

will still trim the outer string (but it's more likely to not have leading and trailing spaces, than commas are to not have surrounding space), and then remove spaces around commas while splitting.

(The sum of many small inefficiencies is a medium inefficiency that it's hard to find the source of. But ... this uses a RegExp, so now we have two problems 😉.)

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

:)

Thanks for the feedback; this will generally have either 0 or 1 commas. I think I'd prefer to avoid a regex for this bit of code.

pkgs/sdk_triage_bot/tool/bench.dart Outdated Show resolved Hide resolved
pkgs/sdk_triage_bot/lib/src/gemini.dart Outdated Show resolved Hide resolved
@@ -45,13 +47,110 @@ If the issue is clearly a bug report, then also apply the label 'type-bug'.
If the issue is mostly a question, then also apply the label 'type-question'.
Otherwise don't apply a 'type-' label.

If the issue title starts with "[breaking change]" then apply the
`breaking-change-request` label but don't assign an area label. IMPORTANT: only
do this if the issue title starts with "[breaking change]".
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Tricky. I think we actually shouldn't add the tag.

Users can choose to write [breaking change], but that doesn't make it a breaking change request. Real breaking change requests come from the template, which should set labels too. That means that if the breaking-change-request label isn't already set, this is probably not a real breaking change request, just a user cargo-culting our process, and what they have is an enhancement request, likely one that does need an area.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think I agree. I'll change this so the bot avoids triaging [breaking change] issues; manual review can figure out if they actually need any action.

If the issue title is "Analyzer Feedback from IntelliJ", these are generally not
well qualified. For these issues, apply the 'needs-info' label but don't assign
an area label.

Return the labels as comma separated text.
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

"Returns"

(A section starting with "Returns" is the recommended way to describe the returned value, just like a section starting with "The [foo] value ..." referencing a parameter is the recommended way to document that parameter.)

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is a directive to the LLM about what it should do, not describing what it will do. As in, "You should return the ...".

pkgs/sdk_triage_bot/lib/triage.dart Outdated Show resolved Hide resolved
pkgs/sdk_triage_bot/tool/bench.dart Show resolved Hide resolved
pkgs/sdk_triage_bot/lib/src/common.dart Outdated Show resolved Hide resolved
@lrhn
Copy link
Member

lrhn commented Sep 4, 2024

(Feel free to ignore my performence comments. I just can't walk past an inefficient function. 😩)

@devoncarew devoncarew merged commit 4ba6f15 into main Sep 4, 2024
29 checks passed
@devoncarew devoncarew deleted the update_prompt branch September 4, 2024 21:22
copybara-service bot pushed a commit to dart-lang/sdk that referenced this pull request Sep 13, 2024
…l, http, lints, mockito, shelf, test, test_descriptor, typed_data, web, webkit_inspection_protocol, yaml

Revisions updated by `dart tools/rev_sdk_deps.dart`.

args (https://github.com/dart-lang/args/compare/1a24d61..e623652):
  e623652  2024-08-26  Kevin Moore  Fix angle brackets in doc comments (dart-archive/args#281)

browser_launcher (https://github.com/dart-lang/browser_launcher/compare/fa98c77..0acd188):
  0acd188  2024-08-22  Srujan Gaddam  Add CHANGELOG entry for dart-archive/browser_launcher#63 and publish 1.2.2 (dart-archive/browser_launcher#64)

collection (https://github.com/dart-lang/collection/compare/0c1f829..24b75d8):
  24b75d8  2024-08-26  Kevin Moore  Fix angle brackets in doc comments (dart-archive/collection#358)

csslib (https://github.com/dart-lang/csslib/compare/192d720..d486627):
  d486627  2024-08-26  Kevin Moore  Fix angle brackets in doc comments (dart-archive/csslib#206)

ecosystem (https://github.com/dart-lang/ecosystem/compare/8626bff..4ba6f15):
  4ba6f15  2024-09-04  Devon Carew  update the prompt to include `needs-info` (dart-lang/ecosystem#296)
  3496f54  2024-09-02  Moritz  Continue with labeling after error with issue transfer (dart-lang/ecosystem#293)
  bb45d73  2024-09-02  Moritz  Run `dart pub global run mono_repo generate` (dart-lang/ecosystem#295)
  8d5860b  2024-09-01  dependabot[bot]  Bump the github-actions group with 3 updates (dart-lang/ecosystem#297)
  6de3e8c  2024-08-29  Devon Carew  fix calculation for the labels to apply (dart-lang/ecosystem#294)
  183fdd0  2024-08-28  Devon Carew  contribute a prompt benchmarking script (dart-lang/ecosystem#292)
  f7191b7  2024-08-28  Moritz  Transfer issues in `package:trebuchet` (dart-lang/ecosystem#291)
  5da8a7a  2024-08-26  Moritz  Add `package:trebuchet` to repo. (dart-lang/ecosystem#289)
  dd8d12a  2024-08-23  Nate Bosch  Add check label to github action job name (dart-lang/ecosystem#290)

html (https://github.com/dart-lang/html/compare/0da420c..5516387):
  5516387  2024-08-28  Kevin Moore  Drop Pair util class (dart-archive/html#255)
  50837e5  2024-08-28  Nate Bosch  Fix doc comments (dart-archive/html#256)

http (https://github.com/dart-lang/http/compare/b97b8dc..dfeecf0):
  dfeecf0  2024-08-28  Brian Quinlan  Convert a `java.lang.OutOfMemoryError` into a `ClientException` (dart-lang/http#1299)
  a41141d  2024-08-28  Brian Quinlan  cupertino_http: Use `sharedDarwinSource` (dart-lang/http#1290)
  51beb4e  2024-08-28  Yaroslav Vorobev  fix(http_client_conformance_tests): multipart server (dart-lang/http#1292)
  bbfce40  2024-08-28  Brian Quinlan  Add a reference to AdapterWebSocketChannel for WebSocketChannel users (dart-lang/http#1297)
  7f21111  2024-08-21  Moritz  Add PR Health workflow (dart-lang/http#1296)

lints (https://github.com/dart-lang/lints/compare/894b500..9e4fd7d):
  9e4fd7d  2024-09-10  Lenz Paul  Fixes formatting error in the rules.md table (dart-lang/lints#206)
  6580f83  2024-09-05  Sam Rawlins  Remove avoid_null_checks_in_equality_operators from recommended (dart-lang/lints#201)

mockito (https://github.com/dart-lang/mockito/compare/eb4d1da..d0fda0c):
  d0fda0c  2024-09-11  Sam Rawlins  Bump CI tasks to use Dart 3.5.0 as stable
  2259e10  2024-09-11  Sam Rawlins  Pass a Dart language version to the Dart formatter.
  d7a8334  2024-09-10  Sam Rawlins  mockito: stop using deprecated analyzer APIs.

shelf (https://github.com/dart-lang/shelf/compare/9f2dffe..d53a8f9):
  d53a8f9  2024-09-06  Nate Bosch  Prepare to publish shelf_static (dart-lang/shelf#443)
  e725c73  2024-09-06  mx1up  Loosen mime package constraint (dart-lang/shelf#441)
  efbc566  2024-09-05  Nate Bosch  Remove links to core types with generics (dart-lang/shelf#442)

test (https://github.com/dart-lang/test/compare/cd3dbd5..9a2d155):
  9a2d155b  2024-09-11  Nate Bosch  Run package:test tests on mac (dart-lang/test#2280)
  569c6eb6  2024-09-01  dependabot[bot]  Bump the github-actions group with 2 updates (dart-lang/test#2275)
  db8cf091  2024-08-28  Nate Bosch  Remove the Pair utility (dart-lang/test#2271)
  032ef1dc  2024-08-27  Simon Binder  Support dart2wasm in node.js tests (dart-lang/test#2259)
  90481cf8  2024-08-23  Nate Bosch  Limit which health checks run (dart-lang/test#2270)
  337c7fb0  2024-08-21  Moritz  Update publish.yaml (dart-lang/test#2269)

test_descriptor (https://github.com/dart-lang/test_descriptor/compare/90743bc..3b85d38):
  3b85d38  2024-08-26  Kevin Moore  Fix angle brackets in doc comments (dart-lang/test_descriptor#70)

typed_data (https://github.com/dart-lang/typed_data/compare/365468a..2bb9e6e):
  2bb9e6e  2024-08-26  Kevin Moore  Require Dart 3.5, use TypeDataList in many places (dart-archive/typed_data#92)

web (https://github.com/dart-lang/web/compare/fb30192..933a37d):
  933a37d  2024-09-10  Srujan Gaddam  Update dart_style and use latestLanguageVersion (dart-lang/web#301)

webkit_inspection_protocol (https://github.com/google/webkit_inspection_protocol.dart/compare/119b877..e7418d7):
  e7418d7  2024-08-26  Kevin Moore  Update min SDK and a number of lints (google/webkit_inspection_protocol.dart#128)

yaml (https://github.com/dart-lang/yaml/compare/a645c39..b3d299e):
  b3d299e  2024-08-28  Kevin Moore  Drop Pair util class (dart-lang/yaml#169)

Change-Id: Ic3461659a40427c0deaafab8c79cade332f76edb
Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/385302
Reviewed-by: Konstantin Shcheglov <scheglov@google.com>
Auto-Submit: Devon Carew <devoncarew@google.com>
Commit-Queue: Konstantin Shcheglov <scheglov@google.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants