Skip to content

[BP-3.2][FLINK-35985][transform] Correct the substring function in transform … #3702

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
leonardBang merged 1 commit into from
Nov 18, 2024
Merged

[BP-3.2][FLINK-35985][transform] Correct the substring function in transform … #3702

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 2 additions & 1 deletion docs/content.zh/docs/core-concept/transform.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -126,7 +126,8 @@ Flink CDC uses [Calcite](https://calcite.apache.org/) to parse expressions and [
| LOWER(string) | lower(string) | Returns string in lowercase. |
| TRIM(string1) | trim('BOTH',string1) | Returns a string that removes whitespaces at both sides. |
| REGEXP_REPLACE(string1, string2, string3) | regexpReplace(string1, string2, string3) | Returns a string from STRING1 with all the substrings that match a regular expression STRING2 consecutively being replaced with STRING3. E.g., 'foobar'.regexpReplace('oo\|ar', '') returns "fb". |
| SUBSTRING(string FROM integer1 [ FOR integer2 ]) | substring(string,integer1,integer2) | Returns a substring of STRING starting from position INT1 with length INT2 (to the end by default). |
| SUBSTR(string, integer1[, integer2]) | substr(string,integer1,integer2) | Returns a substring of STRING starting from position integer1 with length integer2 (to the end by default). |
| SUBSTRING(string FROM integer1 [ FOR integer2 ]) | substring(string,integer1,integer2) | Returns a substring of STRING starting from position integer1 with length integer2 (to the end by default). |
| CONCAT(string1, string2,…) | concat(string1, string2,…) | Returns a string that concatenates string1, string2, …. E.g., CONCAT('AA', 'BB', 'CC') returns 'AABBCC'. |

## Temporal Functions
Expand Down
19 changes: 10 additions & 9 deletions docs/content/docs/core-concept/transform.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -118,16 +118,17 @@ Flink CDC uses [Calcite](https://calcite.apache.org/) to parse expressions and [

## String Functions

| Function | Janino Code | Description |
| -------------------- | ------------------------ | ------------------------------------------------- |
| string1 || string2 | concat(string1, string2) | Returns the concatenation of STRING1 and STRING2. |
| CHAR_LENGTH(string) | charLength(string) | Returns the number of characters in STRING. |
| UPPER(string) | upper(string) | Returns string in uppercase. |
| LOWER(string) | lower(string) | Returns string in lowercase. |
| TRIM(string1) | trim('BOTH',string1) | Returns a string that removes whitespaces at both sides. |
| Function | Janino Code | Description |
| -------------------- | ------------------------ |---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
| string1 || string2 | concat(string1, string2) | Returns the concatenation of STRING1 and STRING2. |
| CHAR_LENGTH(string) | charLength(string) | Returns the number of characters in STRING. |
| UPPER(string) | upper(string) | Returns string in uppercase. |
| LOWER(string) | lower(string) | Returns string in lowercase. |
| TRIM(string1) | trim('BOTH',string1) | Returns a string that removes whitespaces at both sides. |
| REGEXP_REPLACE(string1, string2, string3) | regexpReplace(string1, string2, string3) | Returns a string from STRING1 with all the substrings that match a regular expression STRING2 consecutively being replaced with STRING3. E.g., 'foobar'.regexpReplace('oo\|ar', '') returns "fb". |
| SUBSTRING(string FROM integer1 [ FOR integer2 ]) | substring(string,integer1,integer2) | Returns a substring of STRING starting from position INT1 with length INT2 (to the end by default). |
| CONCAT(string1, string2,…) | concat(string1, string2,…) | Returns a string that concatenates string1, string2, …. E.g., CONCAT('AA', 'BB', 'CC') returns 'AABBCC'. |
| SUBSTR(string, integer1[, integer2]) | substr(string,integer1,integer2) | Returns a substring of STRING starting from position integer1 with length integer2 (to the end by default). |
| SUBSTRING(string FROM integer1 [ FOR integer2 ]) | substring(string,integer1,integer2) | Returns a substring of STRING starting from position integer1 with length integer2 (to the end by default). |
| CONCAT(string1, string2,…) | concat(string1, string2,…) | Returns a string that concatenates string1, string2, …. E.g., CONCAT('AA', 'BB', 'CC') returns 'AABBCC'. |

## Temporal Functions

Expand Down
57 changes: 55 additions & 2 deletions ...cdc-runtime/src/main/java/org/apache/flink/cdc/runtime/functions/SystemFunctionUtils.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -361,11 +361,64 @@ public static boolean notLike(String str, String regex) {
}

public static String substr(String str, int beginIndex) {
return str.substring(beginIndex);
return substring(str, beginIndex);
}

public static String substr(String str, int beginIndex, int length) {
return str.substring(beginIndex, beginIndex + length);
return substring(str, beginIndex, length);
}

public static String substring(String str, int beginIndex) {
return substring(str, beginIndex, Integer.MAX_VALUE);
}

public static String substring(String str, int beginIndex, int length) {
if (length < 0) {
LOG.error(
"length of 'substring(str, beginIndex, length)' must be >= 0 and Int type, but length = {}",
length);
throw new RuntimeException(
"length of 'substring(str, beginIndex, length)' must be >= 0 and Int type, but length = "
+ length);
}
if (length > Integer.MAX_VALUE || beginIndex > Integer.MAX_VALUE) {
LOG.error(
"length or start of 'substring(str, beginIndex, length)' must be Int type, but length = {}, beginIndex = {}",
beginIndex,
length);
throw new RuntimeException(
"length or start of 'substring(str, beginIndex, length)' must be Int type, but length = "
+ beginIndex
+ ", beginIndex = "
+ length);
}
if (str.isEmpty()) {
return "";
}

int startPos;
int endPos;

if (beginIndex > 0) {
startPos = beginIndex - 1;
if (startPos >= str.length()) {
return "";
}
} else if (beginIndex < 0) {
startPos = str.length() + beginIndex;
if (startPos < 0) {
return "";
}
} else {
startPos = 0;
}

if ((str.length() - startPos) < length) {
endPos = str.length();
} else {
endPos = startPos + length;
}
return str.substring(startPos, endPos);
}

public static String upper(String str) {
Expand Down
1 change: 1 addition & 0 deletions ...src/main/java/org/apache/flink/cdc/runtime/parser/metadata/TransformSqlOperatorTable.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -191,6 +191,7 @@ public void lookupOperatorOverloads(
SqlTypeFamily.INTEGER,
SqlTypeFamily.INTEGER)),
SqlFunctionCategory.STRING);
public static final SqlFunction SUBSTRING = SqlStdOperatorTable.SUBSTRING;

// ------------------
// Temporal Functions
Expand Down
19 changes: 18 additions & 1 deletion ...test/java/org/apache/flink/cdc/runtime/operators/transform/PostTransformOperatorTest.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -1499,7 +1499,24 @@ void testBuildInFunctionTransform() throws Exception {
testExpressionConditionTransform("concat('123', 'abc') = '123abc'");
testExpressionConditionTransform("upper('abc') = 'ABC'");
testExpressionConditionTransform("lower('ABC') = 'abc'");
testExpressionConditionTransform("SUBSTR('ABC', 1, 1) = 'B'");
testExpressionConditionTransform("SUBSTR('ABC', -1) = 'C'");
testExpressionConditionTransform("SUBSTR('ABC', -2, 2) = 'BC'");
testExpressionConditionTransform("SUBSTR('ABC', 0) = 'ABC'");
testExpressionConditionTransform("SUBSTR('ABC', 1) = 'ABC'");
testExpressionConditionTransform("SUBSTR('ABC', 2, 2) = 'BC'");
testExpressionConditionTransform("SUBSTR('ABC', 2, 100) = 'BC'");
testExpressionConditionTransform("SUBSTRING('ABC', -1) = 'C'");
testExpressionConditionTransform("SUBSTRING('ABC', -2, 2) = 'BC'");
testExpressionConditionTransform("SUBSTRING('ABC', 0) = 'ABC'");
testExpressionConditionTransform("SUBSTRING('ABC', 1) = 'ABC'");
testExpressionConditionTransform("SUBSTRING('ABC', 2, 2) = 'BC'");
testExpressionConditionTransform("SUBSTRING('ABC', 2, 100) = 'BC'");
testExpressionConditionTransform("SUBSTRING('ABC' FROM -1) = 'C'");
testExpressionConditionTransform("SUBSTRING('ABC' FROM -2 FOR 2) = 'BC'");
testExpressionConditionTransform("SUBSTRING('ABC' FROM 0) = 'ABC'");
testExpressionConditionTransform("SUBSTRING('ABC' FROM 1) = 'ABC'");
testExpressionConditionTransform("SUBSTRING('ABC' FROM 2 FOR 2) = 'BC'");
testExpressionConditionTransform("SUBSTRING('ABC' FROM 2 FOR 100) = 'BC'");
testExpressionConditionTransform("'ABC' like '^[a-zA-Z]'");
testExpressionConditionTransform("'123' not like '^[a-zA-Z]'");
testExpressionConditionTransform("abs(2) = 2");
Expand Down