[SPARK-40665][CONNECT] Avoid embedding Spark Connect in the Apache Spark binary release

[HyukjinKwon]

What changes were proposed in this pull request?

This PR proposes

1. Move connect to connector/connect to be consistent with Kafka and Avro.
2. Do not include this in the default Apache Spark release binary.
3. Fix the module dependency in modules.py.
4. Fix the usages in README.md with cleaning up.
5. Cleanup PySpark test structure to be consistent with other PySpark tests.

Why are the changes needed?

To make it consistent with Avro or Kafka, see also https://github.com/apache/spark/pull/37710/files#r978291019

Does this PR introduce any user-facing change?

No, this isn't released yet.

The usage of this project would be changed from:

./bin/spark-shell \
  --conf spark.plugins=org.apache.spark.sql.connect.SparkConnectPlugin

to

./bin/spark-shell \
  --packages org.apache.spark:spark-connect_2.12:3.4.0 \
  --conf spark.plugins=org.apache.spark.sql.connect.SparkConnectPlugin

How was this patch tested?

CI in the PR should verify this.

[LuciferYang]

The following 6 jars remvoed from deps file but not in shade jar, are these not required at runtime?

animal-sniffer-annotations/1.19//animal-sniffer-annotations-1.19.jar
annotations/4.1.1.4//annotations-4.1.1.4.jar
error_prone_annotations/2.10.0//error_prone_annotations-2.10.0.jar
j2objc-annotations/1.3//j2objc-annotations-1.3.jar
perfmark-api/0.25.0//perfmark-api-0.25.0.jar
proto-google-common-protos/2.0.1//proto-google-common-protos-2.0.1.jar

[INFO] --- maven-shade-plugin:3.2.4:shade (default) @ spark-connect_2.12 ---
[INFO] Including com.google.guava:guava:jar:31.0.1-jre in the shaded jar.
[INFO] Including com.google.guava:listenablefuture:jar:9999.0-empty-to-avoid-conflict-with-guava in the shaded jar.
[INFO] Including com.google.guava:failureaccess:jar:1.0.1 in the shaded jar.
[INFO] Including com.google.protobuf:protobuf-java:jar:3.21.1 in the shaded jar.
[INFO] Including io.grpc:grpc-netty-shaded:jar:1.47.0 in the shaded jar.
[INFO] Including io.grpc:grpc-core:jar:1.47.0 in the shaded jar.
[INFO] Including io.grpc:grpc-protobuf:jar:1.47.0 in the shaded jar.
[INFO] Including io.grpc:grpc-api:jar:1.47.0 in the shaded jar.
[INFO] Including io.grpc:grpc-context:jar:1.47.0 in the shaded jar.
[INFO] Including io.grpc:grpc-protobuf-lite:jar:1.47.0 in the shaded jar.
[INFO] Including io.grpc:grpc-services:jar:1.47.0 in the shaded jar.
[INFO] Including com.google.protobuf:protobuf-java-util:jar:3.19.2 in the shaded jar.
[INFO] Including io.grpc:grpc-stub:jar:1.47.0 in the shaded jar.

[LuciferYang]

spark/connect/pom.xml

Lines 154 to 158 in 1e30564

	<dependency> <!-- necessary for Java 9+ -->
	<groupId>org.apache.tomcat</groupId>
	<artifactId>annotations-api</artifactId>
	<version>${tomcat.annotations.api.version}</version>
	<scope>provided</scope>

From the above comments, when using Java 9+, annotations-api-6.0.53.jar is required. Are the corresponding startup commands different? Now annotations-api-6.0.53.jar is neither in Spark's jars directory nor shaded to connect-assembly

[HyukjinKwon]

Should probably put into connect-assembly but let's set up the tests first, and fix it together.

[LuciferYang]

ok

[HyukjinKwon]

Thanks for reviewing @LuciferYang. For #38109 (comment), let's figure it out together with adding the tests incrementally (since this is a new module)

[LuciferYang]

I checked the Spark-client/jars directory between without connect module and after this pr, their content is same. LGTM if test pass.

[HyukjinKwon]

Merged to master.

[HyukjinKwon]

Seems like we should shade a couple of more dependenices pointed out in #38109 (comment). It works with SBT because SBT assemblies and includes all dependencies whereas Maven one doesn't.

I am working on this.