Skip to content

[SYCL][E2E] Add functionality to split build and run of e2e tests #15728

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
wants to merge 26 commits into from
Closed
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
26 commits
Select commit Hold shift + click to select a range
6b70797
Add method to set build/run mode
ayylol Oct 15, 2024
08297d6
Add logic to split build/running of e2e tests
ayylol Oct 15, 2024
b679fbf
Remove added comment
ayylol Oct 15, 2024
7f7d56b
Change layout of if statement checking exclusive build mode
ayylol Oct 15, 2024
37df022
Add github workflow to only build tests
ayylol Oct 16, 2024
c8b0eb6
Change test run parameters
ayylol Oct 16, 2024
53592cf
Remove split test yaml file
ayylol Oct 17, 2024
c8588fa
Add triple selection for spir64
ayylol Oct 18, 2024
a61cb32
set hip_amd and hip_nvidia to always false for spir64
ayylol Oct 18, 2024
2eac94e
Set native_cpu to always false for spir64 triple
ayylol Oct 18, 2024
bdfd111
Merge branch 'sycl' into sycl-devops-pr/ayylol/e2e-split
ayylol Oct 23, 2024
861f17a
Add Linux build runner option to manual e2e workflow
ayylol Oct 23, 2024
e8ece44
Add test build only job to nightly
ayylol Oct 24, 2024
92522fd
Merge branch 'sycl' into sycl-devops-pr/ayylol/e2e-split
ayylol Oct 24, 2024
bd1fd50
remove commented xfail code
ayylol Oct 25, 2024
07c3c40
Add comment on exceptions list and fix formatting
ayylol Oct 25, 2024
ee764e2
Fix formatting
ayylol Oct 28, 2024
b902185
Merge branch 'sycl' into sycl-devops-pr/ayylol/e2e-split
ayylol Oct 28, 2024
563c765
rename queried_features
ayylol Oct 28, 2024
5f267ec
Format change
ayylol Oct 28, 2024
201712b
Take the union of the unsupported and requires features list
ayylol Oct 28, 2024
e19c435
Remove line filtering on `REQUIRES: run-mode` tests
ayylol Oct 29, 2024
22b7583
Split exceptions dictionary in two
ayylol Oct 29, 2024
5a8d4e4
Remove triple selection logic
ayylol Oct 30, 2024
96fe4e3
Merge branch 'sycl' into sycl-devops-pr/ayylol/e2e-split
ayylol Oct 30, 2024
146afa2
Removed nightly build job
ayylol Oct 30, 2024
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions .github/workflows/sycl-linux-run-tests.yml
Original file line number Diff line number Diff line change
Expand Up @@ -90,6 +90,7 @@ on:
- '["amdgpu"]'
- '["Linux", "arc"]'
- '["cts-cpu"]'
- '["Linux", "build"]'
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This should go to a separate PR so that we don't run needless CI jobs when this PR is updated.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think this only affects the options available for the manual e2e run. I dont think i see it affecting the pre-commit checks being ran.
That being said do you know if there is a way to not run all the checks its doing due to being a sycl-devops-pr, and make it run the checks for a normal pr?

image:
description: |
Use option ending with ":build" for AMDGPU, ":latest" for the rest.
Expand Down
49 changes: 38 additions & 11 deletions sycl/test-e2e/format.py
Original file line number Diff line number Diff line change
Expand Up @@ -154,18 +154,31 @@ def execute(self, test, litConfig):
if isinstance(script, lit.Test.Result):
return script

devices_for_test = self.select_devices_for_test(test)
if not devices_for_test:
return lit.Test.Result(
lit.Test.UNSUPPORTED, "No supported devices to run the test on"
)

substitutions = lit.TestRunner.getDefaultSubstitutions(test, tmpDir, tmpBase)
devices_for_test = []
triples = set()
for sycl_device in devices_for_test:
(backend, _) = sycl_device.split(":")
triples.add(get_triple(test, backend))
unsplit_test = False
for l in test.requires:
if "run-mode" in re.findall("[-+=._a-zA-Z0-9]+", l):
unsplit_test = True
break
if "run-mode" not in test.config.available_features:
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

maybe swap the logic so the first check isnt a not? could we check for build instead?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The issue with doing this is that the non-split mode sets both run-mode and build-mode features. what i am doing here is specifically checking for the split build only mode.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ah ok thx
build and run mode at the same time just does exactly what we do today without these changes right?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ah ok thx build and run mode at the same time just does exactly what we do today without these changes right?

Correct.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I haven't looked through the PR yet, but from previous discussions, I'd expect that we might build more tests/for more triples that are otherwise skipped in the old mode due to XFAIL/UNSUPPORTED.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

you mean for tests that compile fine but fail/unsupported at runtime, we always compile those but don't run the test? yeah that sounds cool

if unsplit_test:
return lit.Test.Result(
lit.Test.UNSUPPORTED, "Tests unsupported on split build"
)
triples = {"spir64"}
else:
devices_for_test = self.select_devices_for_test(test)
if not devices_for_test:
return lit.Test.Result(
lit.Test.UNSUPPORTED, "No supported devices to run the test on"
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

have you tried building on one system and running on a different system yet? have you have any issues with different paths to python or something like that?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, i ran some experiments to run the split side on a separate machine after running the build mode on another. I did not run into issues relating to python path, mostly issues related to the prs i've opened, and AOT tests.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ah ok cool, i expected this part might have insane nonsense to work through but im glad it seems not too bad

)

for sycl_device in devices_for_test:
(backend, _) = sycl_device.split(":")
triples.add(get_triple(test, backend))

substitutions = lit.TestRunner.getDefaultSubstitutions(test, tmpDir, tmpBase)
substitutions.append(("%{sycl_triple}", format(",".join(triples))))
# -fsycl-targets is needed for CUDA/HIP, so just use it be default so
# -that new tests by default would runnable there (unless they have
Expand Down Expand Up @@ -223,6 +236,17 @@ def get_extra_env(sycl_devices):
new_script.append(directive)
continue

# Filter commands based on split-mode
is_run_line = unsplit_test or any(
i in directive.command
for i in ["%{run}", "%{run-unfiltered-devices}", "%if run-mode"]
)

if (is_run_line and "run-mode" not in test.config.available_features) or (
not is_run_line and "build-mode" not in test.config.available_features
):
directive.command = ""

if "%{run}" not in directive.command:
new_script.append(directive)
continue
Expand Down Expand Up @@ -278,7 +302,10 @@ def get_extra_env(sycl_devices):
test, litConfig, useExternalSh, script, tmpBase
)

if len(devices_for_test) > 1:
if (
len(devices_for_test) > 1
or "run-mode" not in test.config.available_features
):
return result

# Single device - might be an XFAIL.
Expand Down
10 changes: 10 additions & 0 deletions sycl/test-e2e/lit.cfg.py
Original file line number Diff line number Diff line change
Expand Up @@ -16,6 +16,16 @@

# Configuration file for the 'lit' test runner.

# split-mode: Set if tests should run normally or with split build/run
match lit_config.params.get("split-mode", "both"):
case "run":
config.available_features.add("run-mode")
case "build":
config.available_features.add("build-mode")
case _:
config.available_features.add("run-mode")
config.available_features.add("build-mode")

# name: The name of this test suite.
config.name = "SYCL"

Expand Down
Loading