Skip to content
This repository has been archived by the owner on Dec 4, 2024. It is now read-only.

Commit

Permalink
[SPARK-50212][BUILD][3.5] Fix the conditional check for executing the…
Browse files Browse the repository at this point in the history
… `build` task in `build_and_test.yml`

### What changes were proposed in this pull request?
Comparing https://github.com/apache/spark/blob/c53dac05058c48ae1edad7912e8cc82533839ca0/.github/workflows/build_and_test.yml#L102 and https://github.com/apache/spark/blob/9d472661daad4703628e9fbf0ba9922abeed7354/.github/workflows/build_and_test.yml#L97, the master branch uses `dev/is-changed.py` to check for changes in additional modules: `variant`, `api`, `streaming-kinesis-asl`, `protobuf`, and `connect`. Among these, the `api`, `protobuf`, and `connect` modules also exist in the `branch-3.5` and should be checked as well.

Therefore, this pr includes the following changes:

1. Adds `is-changed` checks for the `api`, `protobuf`, and `connect` modules.
2. In `dev/sparktestsupport/modules.py`, adds the definition for the `api` module, aligning with the master branch.
3. In `dev/sparktestsupport/modules.py`, adds the definition for the `utils` module, aligning with the master branch. Prior to this PR, although `dev/is-changed.py` was used to check for changes in the `utils` module, its definition was missing from `dev/sparktestsupport/modules.py`.

### Why are the changes needed?
Fix the conditional check for executing the `build` task in `build_and_test.yml`

### Does this PR introduce _any_ user-facing change?
No

### How was this patch tested?
- Pass GitHub Actions

- Manually verified the effectiveness of this pull request:

Before:

Set the latest commit by executing `export APACHE_SPARK_REF=9d472661daad4703628e9fbf0ba9922abeed7354`. Then manually edit files in the `api`, `protobuf`, or `connect` modules and commit them. Run the following command:

```
./dev/is-changed.py -m "core,unsafe,kvstore,avro,utils,network-common,network-shuffle,repl,launcher,examples,sketch,graphx,catalyst,hive-thriftserver,streaming,sql-kafka-0-10,streaming-kafka-0-10,mllib-local,mllib,yarn,mesos,kubernetes,hadoop-cloud,spark-ganglia-lgpl,sql,hive,connect,protobuf,api"
```

The console will print `false`.

After:

Set the latest commit by executing `export APACHE_SPARK_REF=41446b3d98cfccf5c6f6ddb8bc3c7c6c1b1c3f54`. Then manually edit files in the `api`, `protobuf`, or `connect` modules and commit them. Run the same command as before:

```
./dev/is-changed.py -m "core,unsafe,kvstore,avro,utils,network-common,network-shuffle,repl,launcher,examples,sketch,graphx,catalyst,hive-thriftserver,streaming,sql-kafka-0-10,streaming-kafka-0-10,mllib-local,mllib,yarn,mesos,kubernetes,hadoop-cloud,spark-ganglia-lgpl,sql,hive,connect,protobuf,api"
```

The console will now print `true`.

### Was this patch authored or co-authored using generative AI tooling?
No

Closes apache#48744 from LuciferYang/is-change-3.5.

Authored-by: yangjie01 <yangjie01@baidu.com>
Signed-off-by: yangjie01 <yangjie01@baidu.com>
  • Loading branch information
LuciferYang committed Nov 5, 2024
1 parent 9b172de commit acccf53
Show file tree
Hide file tree
Showing 2 changed files with 21 additions and 5 deletions.
2 changes: 1 addition & 1 deletion .github/workflows/build_and_test.yml
Original file line number Diff line number Diff line change
Expand Up @@ -94,7 +94,7 @@ jobs:
tpcds=false
docker=false
fi
build=`./dev/is-changed.py -m "core,unsafe,kvstore,avro,utils,network-common,network-shuffle,repl,launcher,examples,sketch,graphx,catalyst,hive-thriftserver,streaming,sql-kafka-0-10,streaming-kafka-0-10,mllib-local,mllib,yarn,mesos,kubernetes,hadoop-cloud,spark-ganglia-lgpl,sql,hive"`
build=`./dev/is-changed.py -m "core,unsafe,kvstore,avro,utils,network-common,network-shuffle,repl,launcher,examples,sketch,graphx,catalyst,hive-thriftserver,streaming,sql-kafka-0-10,streaming-kafka-0-10,mllib-local,mllib,yarn,mesos,kubernetes,hadoop-cloud,spark-ganglia-lgpl,sql,hive,connect,protobuf,api"`
precondition="
{
\"build\": \"$build\",
Expand Down
24 changes: 20 additions & 4 deletions dev/sparktestsupport/modules.py
Original file line number Diff line number Diff line change
Expand Up @@ -113,6 +113,14 @@ def __hash__(self):
],
)

utils = Module(
name="utils",
dependencies=[tags],
source_file_regexes=[
"common/utils/",
],
)

kvstore = Module(
name="kvstore",
dependencies=[tags],
Expand All @@ -126,7 +134,7 @@ def __hash__(self):

network_common = Module(
name="network-common",
dependencies=[tags],
dependencies=[tags, utils],
source_file_regexes=[
"common/network-common/",
],
Expand All @@ -148,7 +156,7 @@ def __hash__(self):

unsafe = Module(
name="unsafe",
dependencies=[tags],
dependencies=[tags, utils],
source_file_regexes=[
"common/unsafe",
],
Expand Down Expand Up @@ -179,7 +187,7 @@ def __hash__(self):

core = Module(
name="core",
dependencies=[kvstore, network_common, network_shuffle, unsafe, launcher],
dependencies=[kvstore, network_common, network_shuffle, unsafe, launcher, utils],
source_file_regexes=[
"core/",
],
Expand All @@ -188,9 +196,17 @@ def __hash__(self):
],
)

api = Module(
name="api",
dependencies=[utils, unsafe],
source_file_regexes=[
"sql/api/",
],
)

catalyst = Module(
name="catalyst",
dependencies=[tags, sketch, core],
dependencies=[tags, sketch, core, api],
source_file_regexes=[
"sql/catalyst/",
],
Expand Down

0 comments on commit acccf53

Please sign in to comment.