-
Notifications
You must be signed in to change notification settings - Fork 345
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[5201] feat(client-python): Implement expressions in python client #5646
Open
SophieTech88
wants to merge
15
commits into
apache:main
Choose a base branch
from
SophieTech88:gravitino-5201
base: main
Could not load branches
Branch not found: {{ refName }}
Loading
Could not load tags
Nothing to show
Loading
Are you sure you want to change the base?
Some commits from the old base branch may be removed from the timeline,
and old review comments may become outdated.
+821
−0
Open
Changes from all commits
Commits
Show all changes
15 commits
Select commit
Hold shift + click to select a range
bade12b
[5201]Implement expressions in python client
SophieTech88 3fc7400
Update the script for named_reference and function_expression
SophieTech88 00d31e8
Update the distribution.py
SophieTech88 c21bba5
Update literals.py
SophieTech88 361acbc
Update sorts.py
SophieTech88 07a8a87
Update the transforms.py
SophieTech88 2bc8983
Fix a f string bug for distributions.py
SophieTech88 4bac456
Update the comment for Partition class and so on
SophieTech88 40a14ee
improve model structure
xunliu 2a25eb4
Update the script and fix a name bug
SophieTech88 3bc79ca
Update the Literal class to raise NotImplementedError
SophieTech88 9fe8d6c
remove Literals folder
xunliu cbb2a55
Add literals folder
xunliu 624929e
Update licenses headers and format
SophieTech88 c3d8271
Remove distributions, sorts, and transforms to separate PRs
SophieTech88 File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
16 changes: 16 additions & 0 deletions
16
clients/client-python/gravitino/api/expressions/__init__.py
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,16 @@ | ||
# Licensed to the Apache Software Foundation (ASF) under one | ||
# or more contributor license agreements. See the NOTICE file | ||
# distributed with this work for additional information | ||
# regarding copyright ownership. The ASF licenses this file | ||
# to you under the Apache License, Version 2.0 (the | ||
# "License"); you may not use this file except in compliance | ||
# with the License. You may obtain a copy of the License at | ||
# | ||
# http://www.apache.org/licenses/LICENSE-2.0 | ||
# | ||
# Unless required by applicable law or agreed to in writing, | ||
# software distributed under the License is distributed on an | ||
# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY | ||
# KIND, either express or implied. See the License for the | ||
# specific language governing permissions and limitations | ||
# under the License. |
51 changes: 51 additions & 0 deletions
51
clients/client-python/gravitino/api/expressions/expression.py
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,51 @@ | ||
# Licensed to the Apache Software Foundation (ASF) under one | ||
# or more contributor license agreements. See the NOTICE file | ||
# distributed with this work for additional information | ||
# regarding copyright ownership. The ASF licenses this file | ||
# to you under the Apache License, Version 2.0 (the | ||
# "License"); you may not use this file except in compliance | ||
# with the License. You may obtain a copy of the License at | ||
# | ||
# http://www.apache.org/licenses/LICENSE-2.0 | ||
# | ||
# Unless required by applicable law or agreed to in writing, | ||
# software distributed under the License is distributed on an | ||
# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY | ||
# KIND, either express or implied. See the License for the | ||
# specific language governing permissions and limitations | ||
# under the License. | ||
|
||
from __future__ import annotations | ||
from abc import ABC, abstractmethod | ||
from typing import List, Set, TYPE_CHECKING | ||
|
||
if TYPE_CHECKING: | ||
from gravitino.api.expressions.named_reference import NamedReference | ||
|
||
|
||
class Expression(ABC): | ||
"""Base class of the public logical expression API.""" | ||
|
||
EMPTY_EXPRESSION: List[Expression] = [] | ||
""" | ||
`EMPTY_EXPRESSION` is only used as an input when the default `children` method builds the result. | ||
""" | ||
|
||
EMPTY_NAMED_REFERENCE: List[NamedReference] = [] | ||
""" | ||
`EMPTY_NAMED_REFERENCE` is only used as an input when the default `references` method builds | ||
the result array to avoid repeatedly allocating an empty array. | ||
""" | ||
|
||
@abstractmethod | ||
def children(self) -> List[Expression]: | ||
"""Returns a list of the children of this node. Children should not change.""" | ||
pass | ||
|
||
def references(self) -> List[NamedReference]: | ||
"""Returns a list of fields or columns that are referenced by this expression.""" | ||
|
||
ref_set: Set[NamedReference] = set() | ||
for child in self.children(): | ||
ref_set.update(child.references()) | ||
return list(ref_set) |
95 changes: 95 additions & 0 deletions
95
clients/client-python/gravitino/api/expressions/function_expression.py
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,95 @@ | ||
# Licensed to the Apache Software Foundation (ASF) under one | ||
# or more contributor license agreements. See the NOTICE file | ||
# distributed with this work for additional information | ||
# regarding copyright ownership. The ASF licenses this file | ||
# to you under the Apache License, Version 2.0 (the | ||
# "License"); you may not use this file except in compliance | ||
# with the License. You may obtain a copy of the License at | ||
# | ||
# http://www.apache.org/licenses/LICENSE-2.0 | ||
# | ||
# Unless required by applicable law or agreed to in writing, | ||
# software distributed under the License is distributed on an | ||
# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY | ||
# KIND, either express or implied. See the License for the | ||
# specific language governing permissions and limitations | ||
# under the License. | ||
|
||
from __future__ import annotations | ||
from abc import abstractmethod | ||
from typing import List | ||
|
||
from gravitino.api.expressions.expression import Expression | ||
|
||
|
||
class FunctionExpression(Expression): | ||
""" | ||
The interface of a function expression. A function expression is an expression that takes a | ||
function name and a list of arguments. | ||
""" | ||
|
||
@staticmethod | ||
def of(function_name: str, *arguments: Expression) -> FuncExpressionImpl: | ||
""" | ||
Creates a new FunctionExpression with the given function name. | ||
If no arguments are provided, it uses an empty expression. | ||
:param function_name: The name of the function. | ||
:param arguments: The arguments to the function (optional). | ||
:return: The created FunctionExpression. | ||
""" | ||
arguments = list(arguments) if arguments else Expression.EMPTY_EXPRESSION | ||
return FuncExpressionImpl(function_name, arguments) | ||
|
||
@abstractmethod | ||
def function_name(self) -> str: | ||
"""Returns the function name.""" | ||
pass | ||
|
||
@abstractmethod | ||
def arguments(self) -> List[Expression]: | ||
"""Returns the arguments passed to the function.""" | ||
pass | ||
|
||
def children(self) -> List[Expression]: | ||
"""Returns the arguments as children.""" | ||
return self.arguments() | ||
|
||
|
||
class FuncExpressionImpl(FunctionExpression): | ||
""" | ||
A concrete implementation of the FunctionExpression interface. | ||
""" | ||
|
||
_function_name: str | ||
_arguments: List[Expression] | ||
|
||
def __init__(self, function_name: str, arguments: List[Expression]): | ||
super().__init__() | ||
self._function_name = function_name | ||
self._arguments = arguments | ||
|
||
def function_name(self) -> str: | ||
return self._function_name | ||
|
||
def arguments(self) -> List[Expression]: | ||
return self._arguments | ||
|
||
def __str__(self) -> str: | ||
if not self._arguments: | ||
return f"{self._function_name}()" | ||
arguments_str = ", ".join(map(str, self._arguments)) | ||
return f"{self._function_name}({arguments_str})" | ||
|
||
def __eq__(self, other: FuncExpressionImpl) -> bool: | ||
if self is other: | ||
return True | ||
if other is None or self.__class__ is not other.__class__: | ||
return False | ||
return ( | ||
self._function_name == other.function_name() | ||
and self._arguments == other.arguments() | ||
) | ||
|
||
def __hash__(self) -> int: | ||
return hash((self._function_name, tuple(self._arguments))) |
16 changes: 16 additions & 0 deletions
16
clients/client-python/gravitino/api/expressions/literals/__init__.py
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,16 @@ | ||
# Licensed to the Apache Software Foundation (ASF) under one | ||
# or more contributor license agreements. See the NOTICE file | ||
# distributed with this work for additional information | ||
# regarding copyright ownership. The ASF licenses this file | ||
# to you under the Apache License, Version 2.0 (the | ||
# "License"); you may not use this file except in compliance | ||
# with the License. You may obtain a copy of the License at | ||
# | ||
# http://www.apache.org/licenses/LICENSE-2.0 | ||
# | ||
# Unless required by applicable law or agreed to in writing, | ||
# software distributed under the License is distributed on an | ||
# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY | ||
# KIND, either express or implied. See the License for the | ||
# specific language governing permissions and limitations | ||
# under the License. |
40 changes: 40 additions & 0 deletions
40
clients/client-python/gravitino/api/expressions/literals/literal.py
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,40 @@ | ||
# Licensed to the Apache Software Foundation (ASF) under one | ||
# or more contributor license agreements. See the NOTICE file | ||
# distributed with this work for additional information | ||
# regarding copyright ownership. The ASF licenses this file | ||
# to you under the Apache License, Version 2.0 (the | ||
# "License"); you may not use this file except in compliance | ||
# with the License. You may obtain a copy of the License at | ||
# | ||
# http://www.apache.org/licenses/LICENSE-2.0 | ||
# | ||
# Unless required by applicable law or agreed to in writing, | ||
# software distributed under the License is distributed on an | ||
# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY | ||
# KIND, either express or implied. See the License for the | ||
# specific language governing permissions and limitations | ||
# under the License. | ||
|
||
from abc import abstractmethod | ||
from typing import List | ||
|
||
from gravitino.api.expressions.expression import Expression | ||
|
||
|
||
class Literal(Expression): | ||
""" | ||
Represents a constant literal value in the public expression API. | ||
""" | ||
|
||
@abstractmethod | ||
def value(self): | ||
"""The literal value.""" | ||
raise NotImplementedError("Subclasses must implement the `value` method.") | ||
|
||
@abstractmethod | ||
def data_type(self): | ||
"""The data type of the literal.""" | ||
raise NotImplementedError("Subclasses must implement the `data_type` method.") | ||
|
||
def children(self) -> List[Expression]: | ||
return Expression.EMPTY_EXPRESSION |
138 changes: 138 additions & 0 deletions
138
clients/client-python/gravitino/api/expressions/literals/literals.py
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,138 @@ | ||
# Licensed to the Apache Software Foundation (ASF) under one | ||
# or more contributor license agreements. See the NOTICE file | ||
# distributed with this work for additional information | ||
# regarding copyright ownership. The ASF licenses this file | ||
# to you under the Apache License, Version 2.0 (the | ||
# "License"); you may not use this file except in compliance | ||
# with the License. You may obtain a copy of the License at | ||
# | ||
# http://www.apache.org/licenses/LICENSE-2.0 | ||
# | ||
# Unless required by applicable law or agreed to in writing, | ||
# software distributed under the License is distributed on an | ||
# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY | ||
# KIND, either express or implied. See the License for the | ||
# specific language governing permissions and limitations | ||
# under the License. | ||
|
||
from decimal import Decimal | ||
from typing import Union | ||
from datetime import date, time, datetime | ||
|
||
from gravitino.api.expressions.literals.literal import Literal | ||
|
||
|
||
class LiteralImpl(Literal): | ||
"""Creates a literal with the given type value.""" | ||
|
||
_value: Union[int, float, str, datetime, time, date, bool, Decimal, None] | ||
_data_type: ( | ||
str # TODO: Need implement `api/src/main/java/org/apache/gravitino/rel/types` | ||
) | ||
|
||
def __init__( | ||
self, | ||
value: Union[int, float, str, datetime, time, date, bool, Decimal, None], | ||
data_type: str, | ||
): | ||
self._value = value | ||
self._data_type = data_type | ||
|
||
def value(self) -> Union[int, float, str, datetime, time, date, bool]: | ||
return self._value | ||
|
||
def data_type(self) -> str: | ||
return self._data_type | ||
|
||
def __eq__(self, other: object) -> bool: | ||
if not isinstance(other, LiteralImpl): | ||
return False | ||
return (self._value == other._value) and (self._data_type == other._data_type) | ||
|
||
def __hash__(self): | ||
return hash((self._value, self._data_type)) | ||
|
||
def __str__(self): | ||
return f"LiteralImpl(value={self._value}, data_type={self._data_type})" | ||
|
||
|
||
class Literals: | ||
"""The helper class to create literals to pass into Apache Gravitino.""" | ||
|
||
NULL = LiteralImpl(None, "NullType") | ||
|
||
@staticmethod | ||
def of(value, data_type) -> Literal: | ||
return LiteralImpl(value, data_type) | ||
|
||
@staticmethod | ||
def boolean_literal(value: bool) -> Literal: | ||
return LiteralImpl(value, "Boolean") | ||
|
||
@staticmethod | ||
def byte_literal(value: int) -> Literal: | ||
return LiteralImpl(value, "Byte") | ||
|
||
@staticmethod | ||
def unsigned_byte_literal(value: int) -> Literal: | ||
return LiteralImpl(value, "Unsigned Byte") | ||
|
||
@staticmethod | ||
def short_literal(value: int) -> Literal: | ||
return LiteralImpl(value, "Short") | ||
|
||
@staticmethod | ||
def unsigned_short_literal(value: int) -> Literal: | ||
return LiteralImpl(value, "Unsigned Short") | ||
|
||
@staticmethod | ||
def integer_literal(value: int) -> Literal: | ||
return LiteralImpl(value, "Integer") | ||
|
||
@staticmethod | ||
def unsigned_integer_literal(value: int) -> Literal: | ||
return LiteralImpl(value, "Unsigned Integer") | ||
|
||
@staticmethod | ||
def long_literal(value: int) -> Literal: | ||
return LiteralImpl(value, "Long") | ||
|
||
@staticmethod | ||
def unsigned_long_literal(value: Decimal) -> Literal: | ||
return LiteralImpl(value, "Unsigned Long") | ||
|
||
@staticmethod | ||
def float_literal(value: float) -> Literal: | ||
return LiteralImpl(value, "Float") | ||
|
||
@staticmethod | ||
def double_literal(value: float) -> Literal: | ||
return LiteralImpl(value, "Double") | ||
|
||
@staticmethod | ||
def decimal_literal(value: float) -> Literal: | ||
return LiteralImpl(value, "Decimal") | ||
|
||
@staticmethod | ||
def date_literal(value: date) -> Literal: | ||
return LiteralImpl(value, "Date") | ||
|
||
@staticmethod | ||
def time_literal(value: time) -> Literal: | ||
return LiteralImpl(value, "Time") | ||
|
||
@staticmethod | ||
def timestamp_literal(value: datetime) -> Literal: | ||
return LiteralImpl(value, "Timestamp") | ||
|
||
@staticmethod | ||
def timestamp_literal_from_string(value: str) -> Literal: | ||
return Literals.timestamp_literal(datetime.fromisoformat(value)) | ||
|
||
@staticmethod | ||
def string_literal(value: str) -> Literal: | ||
return LiteralImpl(value, "String") | ||
|
||
@staticmethod | ||
def varchar_literal(length: int, value: str) -> Literal: | ||
return LiteralImpl(value, f"Varchar({length})") |
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
you already implemented types
clients/client-python/gravitino/api/types.py
and typeclients/client-python/gravitino/api/type.py