[Feature][Connector-V2][Jira]Add Jira source connector (#3473)

* add jira source * update jira connector * Update seatunnel-e2e/seatunnel-connector-v2-e2e/connector-http-e2e/src/test/java/org/apache/seatunnel/e2e/connector/http/HttpJiraIT.java Co-authored-by: hailin0 <hailin088@gmail.com> * Update seatunnel-e2e/seatunnel-connector-v2-e2e/connector-http-e2e/src/test/java/org/apache/seatunnel/e2e/connector/http/HttpJiraIT.java Co-authored-by: hailin0 <hailin088@gmail.com> * Update seatunnel-e2e/seatunnel-connector-v2-e2e/connector-http-e2e/src/test/java/org/apache/seatunnel/e2e/connector/http/HttpJiraIT.java Co-authored-by: hailin0 <hailin088@gmail.com> * fix rv error. * Update docs/en/connector-v2/source/Jira.md Co-authored-by: hailin0 <hailin088@gmail.com> * fix ci error. * fix ci error. * fix ci error. * fix ci error Co-authored-by: hailin0 <hailin088@gmail.com>
apache · Nov 24, 2022 · fb40162 · fb40162
1 parent 705728e
commit fb40162
Show file tree

Hide file tree

Showing 13 changed files with 587 additions and 0 deletions.
diff --git a/docs/en/connector-v2/source/Jira.md b/docs/en/connector-v2/source/Jira.md
@@ -0,0 +1,160 @@
+# Jira
+
+> Jira source connector
+
+## Description
+
+Used to read data from Jira.
+
+## Key features
+
+- [x] [batch](../../concept/connector-v2-features.md)
+- [ ] [stream](../../concept/connector-v2-features.md)
+- [ ] [exactly-once](../../concept/connector-v2-features.md)
+- [x] [schema projection](../../concept/connector-v2-features.md)
+- [ ] [parallelism](../../concept/connector-v2-features.md)
+- [ ] [support user-defined split](../../concept/connector-v2-features.md)
+
+##  Options
+
+| name                        | type   | required | default value |
+| --------------------------- | ------ | -------- | ------------- |
+| url                         | String | Yes      | -             |
+| email                       | String | Yes      | -             |
+| api_token                   | String | Yes      | -             |
+| method                      | String | No       | get           |
+| schema.fields               | Config | No       | -             |
+| format                      | String | No       | json          |
+| params                      | Map    | No       | -             |
+| body                        | String | No       | -             |
+| poll_interval_ms            | int    | No       | -             |
+| retry                       | int    | No       | -             |
+| retry_backoff_multiplier_ms | int    | No       | 100           |
+| retry_backoff_max_ms        | int    | No       | 10000         |
+| common-options              | config | No       | -             |
+
+### url [String]
+
+http request url
+
+### email [String]
+
+Jira Email
+
+### api_token [String]
+
+Jira API Token
+
+https://id.atlassian.com/manage-profile/security/api-tokens
+
+### method [String]
+
+http request method, only supports GET, POST method
+
+### params [Map]
+
+http params
+
+### body [String]
+
+http body
+
+### poll_interval_ms [int]
+
+request http api interval(millis) in stream mode
+
+### retry [int]
+
+The max retry times if request http return to `IOException`
+
+### retry_backoff_multiplier_ms [int]
+
+The retry-backoff times(millis) multiplier if request http failed
+
+### retry_backoff_max_ms [int]
+
+The maximum retry-backoff times(millis) if request http failed
+
+### format [String]
+
+the format of upstream data, now only support `json` `text`, default `json`.
+
+when you assign format is `json`, you should also assign schema option, for example:
+
+upstream data is the following:
+
+```json
+
+{"code":  200, "data":  "get success", "success":  true}
+
+```
+
+you should assign schema as the following:
+
+```hocon
+
+schema {
+    fields {
+        code = int
+        data = string
+        success = boolean
+    }
+}
+
+```
+
+connector will generate data as the following:
+
+| code | data        | success |
+|------|-------------|---------|
+| 200  | get success | true    |
+
+when you assign format is `text`, connector will do nothing for upstream data, for example:
+
+upstream data is the following:
+
+```json
+
+{"code":  200, "data":  "get success", "success":  true}
+
+```
+
+connector will generate data as the following:
+
+| content |
+|---------|
+| {"code":  200, "data":  "get success", "success":  true}        |
+
+### schema [Config]
+
+#### fields [Config]
+
+the schema fields of upstream data
+
+### common options 
+
+Source plugin common parameters, please refer to [Source Common Options](common-options.md) for details
+
+## Example
+
+```hocon
+Jira {
+    url = "https://liugddx.atlassian.net/rest/api/3/search"
+    email = "test@test.com"
+    api_token = "xxx" 
+    schema {
+       fields {
+         expand = string
+         startAt = bigint
+         maxResults = int
+         total = int
+       }
+    }
+}
+```
+
+## Changelog
+
+### next version
+
+- Add Jira Source Connector
diff --git a/plugin-mapping.properties b/plugin-mapping.properties
@@ -152,4 +152,5 @@ seatunnel.source.Lemlist = connector-http-lemlist
 seatunnel.source.Klaviyo = connector-http-klaviyo
 seatunnel.sink.Slack = connector-slack
 seatunnel.source.OneSignal = connector-http-onesignal
+seatunnel.source.Jira = connector-http-jira
 seatunnel.source.Gitlab = connector-http-gitlab
diff --git a/seatunnel-connectors-v2/connector-http/connector-http-jira/pom.xml b/seatunnel-connectors-v2/connector-http/connector-http-jira/pom.xml
@@ -0,0 +1,40 @@
+<?xml version="1.0" encoding="UTF-8"?>
+<!--
+
+    Licensed to the Apache Software Foundation (ASF) under one or more
+    contributor license agreements.  See the NOTICE file distributed with
+    this work for additional information regarding copyright ownership.
+    The ASF licenses this file to You under the Apache License, Version 2.0
+    (the "License"); you may not use this file except in compliance with
+    the License.  You may obtain a copy of the License at
+
+       http://www.apache.org/licenses/LICENSE-2.0
+
+    Unless required by applicable law or agreed to in writing, software
+    distributed under the License is distributed on an "AS IS" BASIS,
+    WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+    See the License for the specific language governing permissions and
+    limitations under the License.
+
+-->
+<project xmlns="http://maven.apache.org/POM/4.0.0"
+         xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
+         xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
+    <parent>
+        <artifactId>connector-http</artifactId>
+        <groupId>org.apache.seatunnel</groupId>
+        <version>${revision}</version>
+    </parent>
+    <modelVersion>4.0.0</modelVersion>
+
+    <artifactId>connector-http-jira</artifactId>
+
+    <dependencies>
+        <dependency>
+            <groupId>org.apache.seatunnel</groupId>
+            <artifactId>connector-http-base</artifactId>
+            <version>${project.version}</version>
+        </dependency>
+    </dependencies>
+
+</project>
diff --git a/...-jira/src/main/java/org/apache/seatunnel/connectors/seatunnel/jira/source/JiraSource.java b/...-jira/src/main/java/org/apache/seatunnel/connectors/seatunnel/jira/source/JiraSource.java
@@ -0,0 +1,76 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.seatunnel.connectors.seatunnel.jira.source;
+
+import static org.apache.seatunnel.connectors.seatunnel.http.util.AuthorizationUtil.getTokenByBasicAuth;
+
+import org.apache.seatunnel.api.common.PrepareFailException;
+import org.apache.seatunnel.api.source.Boundedness;
+import org.apache.seatunnel.api.source.SeaTunnelSource;
+import org.apache.seatunnel.api.table.type.SeaTunnelRow;
+import org.apache.seatunnel.common.config.CheckConfigUtil;
+import org.apache.seatunnel.common.config.CheckResult;
+import org.apache.seatunnel.common.constants.JobMode;
+import org.apache.seatunnel.common.constants.PluginType;
+import org.apache.seatunnel.connectors.seatunnel.common.source.AbstractSingleSplitReader;
+import org.apache.seatunnel.connectors.seatunnel.common.source.SingleSplitReaderContext;
+import org.apache.seatunnel.connectors.seatunnel.http.source.HttpSource;
+import org.apache.seatunnel.connectors.seatunnel.http.source.HttpSourceReader;
+import org.apache.seatunnel.connectors.seatunnel.jira.source.config.JiraSourceConfig;
+import org.apache.seatunnel.connectors.seatunnel.jira.source.config.JiraSourceParameter;
+
+import org.apache.seatunnel.shade.com.typesafe.config.Config;
+
+import com.google.auto.service.AutoService;
+import lombok.extern.slf4j.Slf4j;
+
+@Slf4j
+@AutoService(SeaTunnelSource.class)
+public class JiraSource extends HttpSource {
+    private final JiraSourceParameter jiraSourceParameter = new JiraSourceParameter();
+
+    @Override
+    public String getPluginName() {
+        return "Jira";
+    }
+
+    @Override
+    public Boundedness getBoundedness() {
+        if (JobMode.BATCH.equals(jobContext.getJobMode())) {
+            return Boundedness.BOUNDED;
+        }
+        throw new UnsupportedOperationException("Jira source connector not support unbounded operation");
+    }
+
+    @Override
+    public void prepare(Config pluginConfig) throws PrepareFailException {
+        CheckResult result = CheckConfigUtil.checkAllExists(pluginConfig, JiraSourceConfig.URL.key(), JiraSourceConfig.EMAIL.key(), JiraSourceConfig.API_TOKEN.key());
+        if (!result.isSuccess()) {
+            throw new PrepareFailException(getPluginName(), PluginType.SOURCE, result.getMsg());
+        }
+        //get accessToken by basic auth
+        String accessToken = getTokenByBasicAuth(pluginConfig.getString(JiraSourceConfig.EMAIL.key()), pluginConfig.getString(JiraSourceConfig.API_TOKEN.key()));
+        jiraSourceParameter.buildWithConfig(pluginConfig, accessToken);
+        buildSchemaWithConfig(pluginConfig);
+    }
+
+    @Override
+    public AbstractSingleSplitReader<SeaTunnelRow> createReader(SingleSplitReaderContext readerContext) throws Exception {
+        return new HttpSourceReader(this.jiraSourceParameter, readerContext, this.deserializationSchema);
+    }
+}
diff --git a/...rc/main/java/org/apache/seatunnel/connectors/seatunnel/jira/source/JiraSourceFactory.java b/...rc/main/java/org/apache/seatunnel/connectors/seatunnel/jira/source/JiraSourceFactory.java
@@ -0,0 +1,56 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.seatunnel.connectors.seatunnel.jira.source;
+
+import org.apache.seatunnel.api.configuration.util.Condition;
+import org.apache.seatunnel.api.configuration.util.OptionRule;
+import org.apache.seatunnel.api.table.factory.Factory;
+import org.apache.seatunnel.api.table.factory.TableSourceFactory;
+import org.apache.seatunnel.connectors.seatunnel.common.schema.SeaTunnelSchema;
+import org.apache.seatunnel.connectors.seatunnel.http.config.HttpConfig;
+import org.apache.seatunnel.connectors.seatunnel.http.config.HttpRequestMethod;
+import org.apache.seatunnel.connectors.seatunnel.jira.source.config.JiraSourceConfig;
+
+import com.google.auto.service.AutoService;
+
+@AutoService(Factory.class)
+public class JiraSourceFactory implements TableSourceFactory {
+    @Override
+    public String factoryIdentifier() {
+        return "Jira";
+    }
+
+    @Override
+    public OptionRule optionRule() {
+        return OptionRule.builder()
+            .required(JiraSourceConfig.URL)
+            .required(JiraSourceConfig.EMAIL)
+            .required(JiraSourceConfig.API_TOKEN)
+            .optional(JiraSourceConfig.METHOD)
+            .optional(JiraSourceConfig.HEADERS)
+            .optional(JiraSourceConfig.PARAMS)
+            .conditional(Condition.of(HttpConfig.METHOD, HttpRequestMethod.POST), JiraSourceConfig.BODY)
+            .conditional(Condition.of(HttpConfig.FORMAT, "json"), SeaTunnelSchema.SCHEMA)
+            .optional(JiraSourceConfig.FORMAT)
+            .optional(JiraSourceConfig.POLL_INTERVAL_MILLS)
+            .optional(JiraSourceConfig.RETRY)
+            .optional(JiraSourceConfig.RETRY_BACKOFF_MAX_MS)
+            .optional(JiraSourceConfig.RETRY_BACKOFF_MULTIPLIER_MS)
+            .build();
+    }
+}
diff --git a/...n/java/org/apache/seatunnel/connectors/seatunnel/jira/source/config/JiraSourceConfig.java b/...n/java/org/apache/seatunnel/connectors/seatunnel/jira/source/config/JiraSourceConfig.java
@@ -0,0 +1,35 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.seatunnel.connectors.seatunnel.jira.source.config;
+
+import org.apache.seatunnel.api.configuration.Option;
+import org.apache.seatunnel.api.configuration.Options;
+import org.apache.seatunnel.connectors.seatunnel.http.config.HttpConfig;
+
+public class JiraSourceConfig extends HttpConfig {
+    public static final String AUTHORIZATION = "Authorization";
+    public static final Option<String> EMAIL = Options.key("email")
+            .stringType()
+            .noDefaultValue()
+            .withDescription("Jira email");
+
+    public static final Option<String> API_TOKEN = Options.key("api_token")
+        .stringType()
+        .noDefaultValue()
+        .withDescription("Jira API Token");
+}