Port over integration tests

[xingyaoww]

"Integration tests" we discussed in the context of this issue should be primarily be "tests that uses LLM and runs the system end to end" - maybe we should name them regression tests?

It should consists of two types:

• Tests for "Agent behavior": using real-LLM, give the same instruction, and use a testcase to check if the agent is able to perform changes to the workspace to result in a desirable final state. We have integration tests for this at https://github.com/All-Hands-AI/OpenHands/tree/main/evaluation/integration_tests that runs daily and will comment the result to a github issue (workflow). We need to migrate it to this repository and also have it run daily with the same behavior.

• Tests for "System behavior": using real-LLM to check whether the system components is working or not. Examples would be: checking if the .reasoning_content is successfully returned by the API for changes introduced in this PR: feat: Support reasoning content in Agent SDK #139

[simonrosenberg]

@xingyaoww
integration seems to have two meanings right now: a test that covers both sdk and tools or a test that is slow and should be run daily.
To make everything clear, what do you think of this test folder organization to separate tests that should run every time vs slower tests that run daily:

tests/
├─ README.md
├─ fixtures/
├─ fast/
│ ├─ sdk/ # triggered if sdk files modified
│ ├─ tools/ # triggered if tools flles modified
│ ├─ full/ # always triggered
└─ slow/ # triggered daily │
│ ├─ sdk/
│ ├─ tools/
│ └─ full/

[xingyaoww]

Some GPT-5 feedback, @simonrosenberg what about we rename our existing integration_tests to regression_tests, and only do daily run for integration_tests?

Here’s the distinction in plain terms, no fluff:

Integration test
	•	Definition: Verifies that multiple components work together.
	•	Example here: Agent + SDK + Tools → run a prompt, expect a workspace file to be modified correctly.
	•	Goal: Catch interface/contract mismatches between parts.
	•	Run frequency: Can be fast or slow, depending on setup (LLM calls = usually slow).

Regression test
	•	Definition: Verifies that a previously working behavior still works.
	•	Example here: You fixed a bug where .reasoning_content wasn’t returned. A regression test asserts that it’s always present now.
	•	Goal: Lock in fixes/features so they never break again.
	•	Run frequency: Usually always, because regressions hurt the most.

⸻

Why the confusion

In practice, integration tests often become regression tests once you’ve debugged a bug with them. But not all regression tests are integration tests:
	•	Unit regression test: “this function should always return X when Y input.”
	•	Integration regression test: “this end-to-end flow should never break again.”

⸻

In your repo’s context

Calling these just “integration tests” is sloppy, because some are:
	•	System-level integration (LLM in the loop, end-to-end).
	•	Feature regressions (e.g., .reasoning_content is always present).

So a better naming split could be:

tests/
├── regression/   # run on every PR; must stay green
├── integration/  # larger end-to-end; run daily/nightly

That makes intent crystal clear:
	•	“Regression” = non-negotiable guardrails.
	•	“Integration” = heavier coverage, sanity of whole system.

⸻

👉 My take: “Regression” is the better name for the tests that must never break and should run on every commit/PR.
Keep “Integration” only for the big, slow end-to-end runs that prove components can still talk.

Do you want me to propose a two-level folder layout that separates regression vs integration, and also handles the fast/slow runtime distinction?

[simonrosenberg]

Some GPT-5 feedback, @simonrosenberg what about we rename our existing integration_tests to regression_tests, and only do daily run for integration_tests?

Here’s the distinction in plain terms, no fluff:

Integration test
	•	Definition: Verifies that multiple components work together.
	•	Example here: Agent + SDK + Tools → run a prompt, expect a workspace file to be modified correctly.
	•	Goal: Catch interface/contract mismatches between parts.
	•	Run frequency: Can be fast or slow, depending on setup (LLM calls = usually slow).

Regression test
	•	Definition: Verifies that a previously working behavior still works.
	•	Example here: You fixed a bug where .reasoning_content wasn’t returned. A regression test asserts that it’s always present now.
	•	Goal: Lock in fixes/features so they never break again.
	•	Run frequency: Usually always, because regressions hurt the most.

⸻

Why the confusion

In practice, integration tests often become regression tests once you’ve debugged a bug with them. But not all regression tests are integration tests:
	•	Unit regression test: “this function should always return X when Y input.”
	•	Integration regression test: “this end-to-end flow should never break again.”

⸻

In your repo’s context

Calling these just “integration tests” is sloppy, because some are:
	•	System-level integration (LLM in the loop, end-to-end).
	•	Feature regressions (e.g., .reasoning_content is always present).

So a better naming split could be:

tests/
├── regression/   # run on every PR; must stay green
├── integration/  # larger end-to-end; run daily/nightly

That makes intent crystal clear:
	•	“Regression” = non-negotiable guardrails.
	•	“Integration” = heavier coverage, sanity of whole system.

⸻

👉 My take: “Regression” is the better name for the tests that must never break and should run on every commit/PR.
Keep “Integration” only for the big, slow end-to-end runs that prove components can still talk.

Do you want me to propose a two-level folder layout that separates regression vs integration, and also handles the fast/slow runtime distinction?

Yes I agree the intent is clearer this way, we can go with

tests
├─ regression
│ ├─ sdk/
│ ├─ tools/
│ └─ full/
└─ integration
├─ sdk/
│ ├─ tools/
│ └─ full/

[xingyaoww]

@simonrosenberg i think regression test and integration test supposed to use both, so maybe we don't need full/ or sdk/ distinction -- we can always introduce this structure later if we find ourselves doing that

[simonrosenberg]

@xingyaoww
Something that can work is renaming integration -> cross.

tests/
└── cross/         # Cross-package tests
└── integration/   # Integration tests
└── sdk/           # SDK unit tests
└── tools/         # Tools unit tests

[neubig]

@simonrosenberg is there a reason why this isn't closed yet?

[xingyaoww]

BTW i think #936 isn't done yet

[all-hands-bot]

This issue is stale because it has been open for 40 days with no activity. Remove the stale label or leave a comment, otherwise it will be closed in 10 days.