Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[FLINK-16124] YAML-ized Kinesis ingress / egresses #64

Closed
wants to merge 11 commits into from

Conversation

tzulitai
Copy link
Contributor

This is a follow-up to #61, adding the YAML-ized Kinesis ingress / egresses.


Usage

  • Ingress:
ingress:
  meta:
    type: statefun.kinesis.io/routable-protobuf-ingress
    id: com.mycomp.foo/bar
  spec:
    awsRegion:
      type: specific
      id: us-west-2
    awsCredentials:
      type: basic
      accessKeyId: my_access_key_id
      secretAccessKey: my_secret_access_key
    startupPosition:
      type: earliest
    streams:
      - stream: stream-1
        typeUrl: com.googleapis/com.mycomp.foo.MessageA
        targets:
          - com.mycomp.foo/function-1
          - com.mycomp.foo/function-2
      - stream: topic-2
        typeUrl: com.googleapis/com.mycomp.foo.MessageB
        targets:
          - com.mycomp.foo/function-2
    clientConfigProperties:
      - SocketTimeout: 9999
      - MaxConnections: 15
  • Egress:
egress:
  meta:
    type: statefun.kinesis.io/generic-egress
    id: com.mycomp.foo/bar
  spec:
    awsRegion:
      type: custom-endpoint
      endpoint: https://localhost:4567
      id: us-west-1
    awsCredentials:
      type: profile
      profileName: john-doe
      profilePath: /path/to/profile/config
    maxOutstandingRecords: 9999
    clientConfigProperties:
      - ThreadingModel: POOLED
      - ThreadPoolSize: 10

Note: by default, awsRegion, awsCredentials, startupPosition, maxOutstandingRecords, and clientConfigProperties do not need to be specified.


Changelog

Important changes are:

  • 54cf5a2 Implement runtime classes / providers for the routable protobuf Kinesis ingress

  • 001561f Binds the AutoRoutableProtobufRouter for the routable Kinesis ingress in the JsonModule

  • ad56622 Binds the new provider for the routable Kinesis ingress

  • ba46048 Implements runtime classes / providers for the generic Kinesis egress

  • 094dc6c Binds the new provider for the generic Kinesis egress

All other commits are preliminary changes for the above.

Copy link
Contributor

@igalshilman igalshilman left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM.

@tzulitai
Copy link
Contributor Author

Thanks for the review @igalshilman, merging ...

@tzulitai tzulitai closed this in 01866d5 Mar 22, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants