Integration tests (openhands fix issue 5076) by enyst · Pull Request #8 · enyst/playground

enyst · 2024-11-23T16:16:28Z

End-user friendly description of the problem this fixes or functionality that this introduces

Include this change in the Release Notes. If checked, you must provide an end-user friendly description for your change below

Give a summary of what the PR does, explaining any non-trivial design decisions

Link of any specific issues this addresses

github-actions · 2024-11-23T16:18:02Z

Hi! I started running the integration tests on your PR. You will receive a comment with the results shortly.

github-actions · 2024-11-23T16:24:22Z

Trigger by: Pull Request (integration-test label on PR #8)
Commit: a7ff9aa
Integration Tests Evaluation Report

You can download the full evaluation outputs here.

github-actions · 2024-11-23T16:53:00Z

Hi! I started running the integration tests on your PR. You will receive a comment with the results shortly.

github-actions · 2024-11-23T16:58:40Z

Trigger by: Pull Request (integration-test label on PR #8)
Commit: f1ac848
Integration Tests Evaluation Report

You can download the full evaluation outputs here.

github-actions · 2024-11-23T17:05:15Z

Hi! I started running the integration tests on your PR. You will receive a comment with the results shortly.

github-actions · 2024-11-23T17:09:27Z

Trigger by: Pull Request (integration-test label on PR #8)
Commit: 9ba684d
Integration Tests Evaluation Report
Success rate: 66.67% (4/6)

instance_id	success	reason
t02_add_bash_hello	True
t05_simple_browsing	False	The answer is not found in any message. Total messages: 0. Messages: []
t04_git_staging	True
t06_github_pr_browsing	False	The answer is not found in any message. Total messages: 0. Messages: []
t03_jupyter_write_file	True
t01_fix_simple_typo	True

You can download the full evaluation outputs here.

This reverts commit dcd4681.

enyst · 2024-11-25T16:10:45Z

@openhands-agent Make the integration-runner workflow work also on schedule, a nightly schedule.

github-actions · 2024-11-25T16:11:15Z

OpenHands started fixing the pr! You can monitor the progress here.

…round into int/openhands-fix-issue-5076

github-actions · 2024-11-25T20:59:41Z

Hi! I started running the integration tests on your PR. You will receive a comment with the results shortly.

github-actions · 2024-11-25T21:12:13Z

Trigger by: Pull Request (integration-test label on PR #8)
Commit: de57c25
Integration Tests Report (Haiku)
Haiku LLM Test Results:
Success rate: 100.00% (6/6)

instance_id	success	reason
t03_jupyter_write_file	True
t02_add_bash_hello	True
t01_fix_simple_typo	True
t05_simple_browsing	True
t04_git_staging	True
t06_github_pr_browsing	True

Integration Tests Report (DeepSeek)
DeepSeek LLM Test Results:
Success rate: 100.00% (6/6)

instance_id	success	reason
t03_jupyter_write_file	True
t02_add_bash_hello	True
t01_fix_simple_typo	True
t05_simple_browsing	True
t04_git_staging	True
t06_github_pr_browsing	True

Download evaluation outputs (includes both Haiku and DeepSeek results): Download

github-actions · 2024-11-25T21:26:36Z

Hi! I started running the integration tests on your PR. You will receive a comment with the results shortly.

github-actions · 2024-11-25T21:38:25Z

Hi! I started running the integration tests on your PR. You will receive a comment with the results shortly.

github-actions · 2024-11-25T21:45:02Z

Trigger by: Pull Request (integration-test label on PR #8)
Commit: 95425d3
Integration Tests Report (Haiku)
Haiku LLM Test Results:
Success rate: 100.00% (6/6)

instance_id	success	reason
t03_jupyter_write_file	True
t02_add_bash_hello	True
t01_fix_simple_typo	True
t05_simple_browsing	True
t06_github_pr_browsing	True
t04_git_staging	True

Integration Tests Report (DeepSeek)
DeepSeek LLM Test Results:
Success rate: 83.33% (5/6)

instance_id	success	reason
t03_jupyter_write_file	True
t05_simple_browsing	True
t04_git_staging	True
t02_add_bash_hello	True
t01_fix_simple_typo	True
t06_github_pr_browsing	False	The answer is not found in any message. Total messages: 4.

Download evaluation outputs (includes both Haiku and DeepSeek results): Download

github-actions · 2024-11-25T22:10:15Z

Hi! I started running the integration tests on your PR. You will receive a comment with the results shortly.

github-actions · 2024-11-25T22:16:55Z

Trigger by: Pull Request (integration-test label on PR #8)
Commit: 95425d3
Integration Tests Report (Haiku)
Haiku LLM Test Results:
Success rate: 83.33% (5/6)

instance_id	success	reason
t03_jupyter_write_file	True
t02_add_bash_hello	True
t01_fix_simple_typo	True
t05_simple_browsing	False	The answer is not found in any message. Total messages: 4.
t06_github_pr_browsing	True
t04_git_staging	True

Integration Tests Report (DeepSeek)
DeepSeek LLM Test Results:
Success rate: 83.33% (5/6)

instance_id	success	reason
t03_jupyter_write_file	True
t05_simple_browsing	True
t02_add_bash_hello	True
t04_git_staging	True
t06_github_pr_browsing	False	The answer is not found in any message. Total messages: 2.
t01_fix_simple_typo	True

Download evaluation outputs (includes both Haiku and DeepSeek results): Download

…nhands-fix-issue-5076

* Fix issue OpenHands#5076: Integration test github action * Update integration-runner.yml * Update integration-runner.yml * update variables * use haiku * use base url * fix report name * Fix pr #8: Integration tests (openhands fix issue 5076) * Revert "Fix pr #8: Integration tests (openhands fix issue 5076)" This reverts commit dcd4681. * Fix pr #8: Integration tests (openhands fix issue 5076) * use haiku explicitly, in results too * remove duplicate * Update .github/workflows/integration-runner.yml * Revert "Update .github/workflows/integration-runner.yml" This reverts commit 7e7200e. * funny space * Fix pr #8: Integration tests (openhands fix issue 5076) * artifact fix * clean up remote runtimes * clean up runtimes more aggressively - a bit unexpected though * Fix pr #8: Integration tests (openhands fix issue 5076) * fix type issue that was preventing checking results * try with waiting time * add eval notes * increase timeouts * try with CI local builds * fix eval output * set debug * fix tests! * fix outputs * keep details in logs, not github comment * tweak schedule * lint-y --------- Co-authored-by: openhands <openhands@all-hands.dev>

openhands-agent and others added 6 commits November 16, 2024 00:39

Fix issue OpenHands#5076: Integration test github action

eaf3057

Update integration-runner.yml

14fe4c6

Update integration-runner.yml

b415ad2

Merge branch 'main' into openhands-fix-issue-5076

59be01d

update variables

0fd1ddf

use haiku

bc3f136

enyst added the integration-test label Nov 23, 2024

use base url

73e8837

enyst added integration-test and removed integration-test labels Nov 23, 2024

fix report name

7af3518

enyst added integration-test and removed integration-test labels Nov 23, 2024

enyst mentioned this pull request Nov 23, 2024

[Bug]: eval-this workflow is not working OpenHands/OpenHands#5106

Closed

1 task

openhands-agent and others added 2 commits November 25, 2024 15:39

Fix pr #8: Integration tests (openhands fix issue 5076)

dcd4681

Revert "Fix pr #8: Integration tests (openhands fix issue 5076)"

1a24a94

This reverts commit dcd4681.

Repository owner deleted a comment from github-actions bot Nov 25, 2024

openhands-agent and others added 4 commits November 25, 2024 16:13

Fix pr #8: Integration tests (openhands fix issue 5076)

5e5eb0f

use haiku explicitly, in results too

1f90867

Merge branch 'int/openhands-fix-issue-5076' of github.com:enyst/playg…

4406794

…round into int/openhands-fix-issue-5076

remove duplicate

fa9e651

enyst added the integration-test label Nov 25, 2024

Repository owner deleted a comment from github-actions bot Nov 25, 2024

fix outputs

6ff6fe2

Repository owner deleted a comment from github-actions bot Nov 25, 2024

enyst added integration-test and removed integration-test labels Nov 25, 2024

enyst mentioned this pull request Nov 25, 2024

Fix issue #5076: Integration test github action OpenHands/OpenHands#5077

Merged

Repository owner deleted a comment from github-actions bot Nov 25, 2024

keep details in logs, not github comment

1956f06

enyst added integration-test and removed integration-test labels Nov 25, 2024

tweak schedule

b5c2519

lint-y

0c22181

enyst force-pushed the int/openhands-fix-issue-5076 branch from b37602a to 0c22181 Compare November 25, 2024 22:19

Merge branch 'main' of github.com:All-Hands-AI/OpenHands into int/ope…

bc4c9a2

…nhands-fix-issue-5076

add placeholder issue; add cost

f21b957

enyst added integration-test and removed integration-test labels Nov 28, 2024

enyst closed this Nov 28, 2024

coderabbitai bot mentioned this pull request Aug 26, 2025

OpenHands API Automation #94

Closed

Comments

Conversation

enyst commented Nov 23, 2024

Uh oh!

github-actions bot commented Nov 23, 2024

Uh oh!

github-actions bot commented Nov 23, 2024

Uh oh!

github-actions bot commented Nov 23, 2024

Uh oh!

github-actions bot commented Nov 23, 2024

Uh oh!

github-actions bot commented Nov 23, 2024

Uh oh!

github-actions bot commented Nov 23, 2024

Uh oh!

enyst commented Nov 25, 2024

Uh oh!

github-actions bot commented Nov 25, 2024

Uh oh!

github-actions bot commented Nov 25, 2024

Uh oh!

github-actions bot commented Nov 25, 2024

Uh oh!

github-actions bot commented Nov 25, 2024

Uh oh!

github-actions bot commented Nov 25, 2024

Uh oh!

github-actions bot commented Nov 25, 2024

Uh oh!

github-actions bot commented Nov 25, 2024

Uh oh!

github-actions bot commented Nov 25, 2024

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants