Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[BitSail][Connector] Support Pulsar source connector #123

Open
lichang-bd opened this issue Nov 8, 2022 · 4 comments
Open

[BitSail][Connector] Support Pulsar source connector #123

lichang-bd opened this issue Nov 8, 2022 · 4 comments
Assignees
Labels
difficulty-hard Hard difficulty to fix this issue help wanted Extra attention is needed

Comments

@lichang-bd
Copy link
Member

Is your feature request related to a problem? Please describe

A clear and concise description of what the problem is.

Example: I'm always frustrated when [...]

Describe the solution you'd like

A clear and concise description of what you want to happen.

Describe alternatives you've considered

A clear and concise description of any alternative solutions or features you've considered.

Additional context

Add any other context or screenshots about the feature request here.

@lichang-bd lichang-bd changed the title [BitSail][Connector] Support Pulsar Source connector [BitSail][Connector] Support Pulsar source connector Nov 8, 2022
@garyli1019 garyli1019 added help wanted Extra attention is needed difficulty-hard Hard difficulty to fix this issue labels Nov 9, 2022
@qidian99
Copy link

I'd like to take this issue.

@qidian99
Copy link

qidian99 commented Nov 18, 2022

A sample job config is as follows:

{
  "job": {
    "common": {
      "job_type": "STREAMING",
      "job_plugin_lib_dir": "plugin",
      "job_plugin_conf_dir": "plugin_conf",
      "enable_dynamic_loader": true,
      "instance_id": "1",
      "internal_instance_id": "1",
      "extra_properties": {
        "update-mode": "append"
      }
    },
    "reader": {
      "pulsar": {
        "source": {
          "enableAutoAcknowledgeMessage": true
        },
        "client": {
          "serviceUrl": "pulsar://lcoalhost:8080"
        },
        "admin": {
          "adminUrl": "http://localhost:6650",
          "topics": "TBW102",
          "topicMode": "list"
        },
        "consumer": {
          "startCursorMode": "latest",
          "subscriptionType": "Exclusive"
        }
      },

      "columns": [
        {
          "name": "ID",
          "type": "string"
        }
      ],
      "format_type": "json",
      "class": "com.bytedance.bitsail.connector.pulsar.source.PulsarSourceV1"
    },
    "writer": {
      "class": "com.bytedance.bitsail.connector.legacy.print.sink.PrintSink"
    }
  }
}

sample output:
image

P.S., I always get the slf4j warnings
SLF4J: Class path contains multiple SLF4J bindings., SLF4J: Actual binding is of type [org.apache.logging.slf4j.Log4jLoggerFactory] , and no log is printed which impacts the testing processing. Any help is appreciated.

qidian99 added a commit to qidian99/bitsail that referenced this issue Nov 18, 2022
Added pulsar source v1

Pulsar source connector adapt to bitsail v1 source reader.

Wired up bitsail config for Pulsar connector

Simple implementation of pollNext function in Pulsar connector
qidian99 added a commit to qidian99/bitsail that referenced this issue Nov 18, 2022
Added pulsar source v1

Pulsar source connector adapt to bitsail v1 source reader.

Wired up bitsail config for Pulsar connector

Simple implementation of pollNext function in Pulsar connector
@garyli1019
Copy link
Collaborator

@qidian99 assigned to you!

@lichang-bd
Copy link
Member Author

Reminder, is there any problem encountered in this task, we can discuss it together @qidian99

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
difficulty-hard Hard difficulty to fix this issue help wanted Extra attention is needed
Projects
None yet
Development

No branches or pull requests

3 participants