Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Feature][Connector V2] add gitlab source connector #3408

Merged
merged 8 commits into from
Nov 23, 2022

Conversation

liugddx
Copy link
Member

@liugddx liugddx commented Nov 12, 2022

Purpose of this pull request

Check list

@liugddx liugddx closed this Nov 12, 2022
@liugddx liugddx changed the title [Feature][Connector V2] [Feature][Connector V2] expose configurable options in Iceberg [Feature][Connector V2] add gitlab source connector Nov 12, 2022
@liugddx liugddx reopened this Nov 12, 2022
@liugddx
Copy link
Member Author

liugddx commented Nov 12, 2022

#3018

@liugddx
Copy link
Member Author

liugddx commented Nov 12, 2022

test result .
image

@liugddx
Copy link
Member Author

liugddx commented Nov 12, 2022

env {
  # You can set flink configuration here
  execution.parallelism = 1
  job.mode = "STREAMING"
  #execution.checkpoint.interval = 10000
  #execution.checkpoint.data-uri = "hdfs://localhost:9000/checkpoint"
}

source {
  # This is a example source plugin **only for test and demonstrate the feature source plugin**
  Gitlab{
    url = "https://gitlab.com/api/v4/projects"
    access_token = "xxxx"
    schema {
      fields {
        id = int
        description = string
        name = string
        name_with_namespace = string
        path = string
        http_url_to_repo = string
      }
    }
  }

  # If you would like to get more information about how to configure seatunnel and see full list of source plugins,
  # please go to https://seatunnel.apache.org/docs/category/source-v2
}

transform {


  # If you would like to get more information about how to configure seatunnel and see full list of transform plugins,
  # please go to https://seatunnel.apache.org/docs/category/transform
}

sink {
  Console {
    parallelism = 1
  }

  # If you would like to get more information about how to configure seatunnel and see full list of sink plugins,
  # please go to https://seatunnel.apache.org/docs/category/sink-v2
}

@liugddx
Copy link
Member Author

liugddx commented Nov 12, 2022

@TaoZex @EricJoy2048 @TyrantLucifer @Hisoka-X PTAL thanks.

@liugddx liugddx marked this pull request as draft November 12, 2022 12:10
@TaoZex
Copy link
Contributor

TaoZex commented Nov 12, 2022

Please Add e2e test.

@liugddx
Copy link
Member Author

liugddx commented Nov 12, 2022

Please Add e2e test.
It contains sensitive information and cannot be disclosed.

@liugddx liugddx marked this pull request as ready for review November 12, 2022 12:37
@TaoZex
Copy link
Contributor

TaoZex commented Nov 12, 2022

reference
#3340

mock http server to add e2e test

@liugddx
Copy link
Member Author

liugddx commented Nov 12, 2022

reference #3340

mock http server to add e2e test

this mock test has covered the http use case.This connector is only for gitlab

@liugddx
Copy link
Member Author

liugddx commented Nov 13, 2022

The test of restful api focuses on request rules, not result parsing.Can we store keys in Actions secrets and replace them with config?
https://docs.github.com/cn/actions/security-guides/security-hardening-for-github-actions#overview

@TaoZex @EricJoy2048 @hailin0

@liugddx
Copy link
Member Author

liugddx commented Nov 14, 2022

The test of restful api focuses on request rules, not result parsing.Can we store keys in Actions secrets and replace them with config? https://docs.github.com/cn/actions/security-guides/security-hardening-for-github-actions#overview

@TaoZex @EricJoy2048 @hailin0

@ashulin @Hisoka-X @CalvinKirs @TyrantLucifer Please give us some advice.

# Conflicts:
#	plugin-mapping.properties
#	seatunnel-connectors-v2/connector-http/pom.xml
#	seatunnel-dist/pom.xml
@liugddx
Copy link
Member Author

liugddx commented Nov 21, 2022

@hailin0 @EricJoy2048 PTAL

hailin0
hailin0 previously approved these changes Nov 22, 2022

@Slf4j
@AutoService(SeaTunnelSource.class)
public class GitlabSource extends HttpSource {
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

add this method

    @Override
    public Boundedness getBoundedness() {
        if (JobMode.BATCH.equals(jobContext.getJobMode())) {
            return Boundedness.BOUNDED;
        }
        throw new UnsupportedOperationException(xxx);
    }

# Conflicts:
#	plugin-mapping.properties
#	seatunnel-connectors-v2/connector-http/pom.xml
#	seatunnel-dist/pom.xml
#	seatunnel-e2e/seatunnel-connector-v2-e2e/connector-http-e2e/pom.xml
@liugddx liugddx requested a review from hailin0 November 22, 2022 10:11
@Hisoka-X Hisoka-X merged commit 545595c into apache:dev Nov 23, 2022
@liugddx liugddx deleted the gitabsource branch November 23, 2022 10:58
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants