Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

improvements for transferred issues #274

Merged
merged 2 commits into from
Jun 26, 2024
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 4 additions & 2 deletions pkgs/sdk_triage_bot/lib/src/common.dart
Original file line number Diff line number Diff line change
Expand Up @@ -37,9 +37,11 @@ String get geminiKey {
return token;
}

/// Don't return more than 4k of text for an issue body.
/// Don't return more than 5k of text for an issue body.
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

(Just curious) Why the limit, if we have a huge context window available?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think this is me being defensive? I can get some data, but I suspect that 90%+ of comments fit in this limit, and the ones that don't are almost always something like a little bit of content and a huge amount of pasted stack trace.

We could keep this as is, raise it substantially so that it really is only protecting against sending in too many tokens, or only have this size limit apply when we're tuning the model (which does have an overall size limit for the tuning data).

String trimmedBody(String body) {
return body.length > 4096 ? body = body.substring(0, 4096) : body;
const textLimit = 5 * 1024;

return body.length > textLimit ? body = body.substring(0, textLimit) : body;
}

class Logger {
Expand Down
18 changes: 16 additions & 2 deletions pkgs/sdk_triage_bot/lib/src/github.dart
Original file line number Diff line number Diff line change
Expand Up @@ -19,8 +19,15 @@ class GithubService {
return result.map((item) => item.name).toList();
}

Future<Issue> fetchIssue(RepositorySlug sdkSlug, int issueNumber) async {
return await _gitHub.issues.get(sdkSlug, issueNumber);
Future<Issue> fetchIssue(RepositorySlug slug, int issueNumber) async {
return await _gitHub.issues.get(slug, issueNumber);
}

Future<List<IssueComment>> fetchIssueComments(
RepositorySlug slug, Issue issue) async {
return await _gitHub.issues
.listCommentsByIssue(slug, issue.number)
.toList();
}

Future createComment(
Expand Down Expand Up @@ -145,3 +152,10 @@ GraphQLClient _initGraphQLClient() {
link: auth.concat(HttpLink('https://api.github.com/graphql')),
);
}

extension IssueExtension on Issue {
/// Returns whether this issue has any comments.
///
/// Note that the original text for the issue is returned in the `body` field.
bool get hasComments => commentsCount > 0;
}
7 changes: 6 additions & 1 deletion pkgs/sdk_triage_bot/lib/src/prompts.dart
Original file line number Diff line number Diff line change
Expand Up @@ -5,6 +5,7 @@
String assignAreaPrompt({
required String title,
required String body,
String? lastComment,
}) {
return '''
You are a software engineer on the Dart team at Google. You are responsible for
Expand Down Expand Up @@ -36,6 +37,7 @@ area-web: Use area-web for Dart web related issues, including the DDC and dart2j
Don't make up a new area.
Don't use more than one area- label.
If it's not clear which area the issue should go in, don't apply an area- label.
Take your time when considering which area to triage the issue into.
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I would put even more pressure on the LLM ;)

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I do wonder if this phrase will make it spend more consideration on the response. I have heard that phrases like 'show your work' will make it produce more tokens, which effectively means that it's using more computation, which may then yield a better result.

I'm not sure that it'll help us here since the classification response is only ever going to yield ~1/2 dozen tokens (area-foo, type-bar).


If the issue is clearly a feature request, then also apply the label 'type-enhancement'.
If the issue is clearly a bug report, then also apply the label 'type-bug'.
Expand All @@ -48,7 +50,10 @@ Issue follows:

$title

$body''';
$body

${lastComment ?? ''}'''
.trim();
}

String summarizeIssuePrompt({
Expand Down
21 changes: 19 additions & 2 deletions pkgs/sdk_triage_bot/lib/triage.dart
Original file line number Diff line number Diff line change
Expand Up @@ -46,6 +46,22 @@ Future<void> triage(
}
logger.log('');

// If the issue has any comments, retrieve and include the last comment in the
// prompt.
String? lastComment;
if (issue.hasComments) {
final comments = await githubService.fetchIssueComments(sdkSlug, issue);
final comment = comments.last;

lastComment = '''
---

Here is the last comment on the issue (by user @${comment.user?.login}):

${trimmedBody(comment.body ?? '')}
''';
}

// decide if we should triage
final alreadyTriaged = labels.any((l) => l.startsWith('area-'));
if (alreadyTriaged && !force) {
Expand Down Expand Up @@ -76,7 +92,8 @@ Future<void> triage(
try {
// Failures here can include things like gemini safety issues, ...
classification = await geminiService.classify(
assignAreaPrompt(title: issue.title, body: bodyTrimmed),
assignAreaPrompt(
title: issue.title, body: bodyTrimmed, lastComment: lastComment),
);
} on GenerativeAIException catch (e) {
stderr.writeln('gemini: $e');
Expand Down Expand Up @@ -121,7 +138,7 @@ Future<void> triage(
logger.log('');
logger.log('---');
logger.log('');
logger.log('Triaged ${issue.htmlUrl}.');
logger.log('Triaged ${issue.htmlUrl}');
}

List<String> filterExistingLabels(
Expand Down
6 changes: 6 additions & 0 deletions pkgs/sdk_triage_bot/test/fakes.dart
Original file line number Diff line number Diff line change
Expand Up @@ -29,6 +29,12 @@ class GithubServiceMock implements GithubService {
return returnedIssue;
}

@override
Future<List<IssueComment>> fetchIssueComments(
RepositorySlug slug, Issue issue) {
return Future.value([]);
}

String? updatedComment;

@override
Expand Down