Skip to content

Commit 72da6f8

Browse files
zero323HyukjinKwon
authored andcommitted
[SPARK-33002][PYTHON] Remove non-API annotations
### What changes were proposed in this pull request? This PR: - removes annotations for modules which are not part of the public API. - removes `__init__.pyi` files, if no annotations, beyond exports, are present. ### Why are the changes needed? Primarily to reduce maintenance overhead and as requested in the comments to #29591 ### Does this PR introduce _any_ user-facing change? No ### How was this patch tested? Existing tests and additional MyPy checks: ``` mypy --no-incremental --config python/mypy.ini python/pyspark MYPYPATH=python/ mypy --no-incremental --config python/mypy.ini examples/src/main/python/ml examples/src/main/python/sql examples/src/main/python/sql/streaming ``` Closes #29879 from zero323/SPARK-33002. Authored-by: zero323 <mszymkiewicz@gmail.com> Signed-off-by: HyukjinKwon <gurwls223@apache.org>
1 parent 4e1ded6 commit 72da6f8

30 files changed

+14
-1039
lines changed

python/mypy.ini

Lines changed: 4 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -32,5 +32,8 @@ ignore_missing_imports = True
3232
[mypy-pandas.*]
3333
ignore_missing_imports = True
3434

35-
[mypy-pyarrow]
35+
[mypy-pyarrow.*]
36+
ignore_missing_imports = True
37+
38+
[mypy-psutil.*]
3639
ignore_missing_imports = True

python/pyspark/_globals.pyi

Lines changed: 0 additions & 27 deletions
This file was deleted.

python/pyspark/accumulators.pyi

Lines changed: 3 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -16,7 +16,7 @@
1616
# specific language governing permissions and limitations
1717
# under the License.
1818

19-
from typing import Callable, Generic, Tuple, Type, TypeVar
19+
from typing import Callable, Dict, Generic, Tuple, Type, TypeVar
2020

2121
import socketserver.BaseRequestHandler # type: ignore
2222

@@ -27,6 +27,8 @@ U = TypeVar("U", bound=SupportsIAdd)
2727

2828
import socketserver as SocketServer
2929

30+
_accumulatorRegistry: Dict[int, Accumulator]
31+
3032
class Accumulator(Generic[T]):
3133
aid: int
3234
accum_param: AccumulatorParam[T]

python/pyspark/broadcast.pyi

Lines changed: 3 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -17,10 +17,12 @@
1717
# under the License.
1818

1919
import threading
20-
from typing import Any, Generic, Optional, TypeVar
20+
from typing import Any, Dict, Generic, Optional, TypeVar
2121

2222
T = TypeVar("T")
2323

24+
_broadcastRegistry: Dict[int, Broadcast]
25+
2426
class Broadcast(Generic[T]):
2527
def __init__(
2628
self,

python/pyspark/daemon.pyi

Lines changed: 0 additions & 29 deletions
This file was deleted.

python/pyspark/find_spark_home.pyi

Lines changed: 0 additions & 17 deletions
This file was deleted.

python/pyspark/java_gateway.pyi

Lines changed: 0 additions & 24 deletions
This file was deleted.

python/pyspark/join.pyi

Lines changed: 0 additions & 50 deletions
This file was deleted.

python/pyspark/ml/__init__.pyi

Lines changed: 0 additions & 45 deletions
This file was deleted.

python/pyspark/mllib/__init__.pyi

Lines changed: 0 additions & 32 deletions
This file was deleted.

0 commit comments

Comments
 (0)