Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add ability to submit ingestion job using Flink #62

Merged
merged 9 commits into from
Jan 18, 2019

Conversation

pradithya
Copy link
Collaborator

@pradithya pradithya commented Jan 11, 2019

This PR adds ability to submit ingestion job to a Flink Cluster in detached mode.
To enable it, set following environment variable:

  1. JOB_RUNNER = FlinkRunner
  2. FLINK_MASTER_URL : URL to flink's job manager (e.g.: localhost:8081)
  3. FLINK_CONF_DIR: path to flink's flink-conf.yaml (e.g: /etc/flink/conf )

What's not done:

  1. Pulling metrics (Get ingestion metrics when running on Flink runner #63 )
  2. Helm chart update (Helm chart for deploying feast using Flink as runner #64 )

Fix #21

@pradithya
Copy link
Collaborator Author

/assign @zhilingc
/assign @woop
/assign @tims

Copy link
Collaborator

@zhilingc zhilingc left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Could you add the licenses to the files?

@zhilingc
Copy link
Collaborator

/approve

@feast-ci-bot
Copy link
Collaborator

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: zhilingc

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@woop
Copy link
Member

woop commented Jan 12, 2019

/unassign woop

Think @tims should review this instead of me. It looks fine to me personally.

throw new IllegalStateException("Unable to initialize DataflowJobManager", e);
}
case FLINK:
org.apache.flink.configuration.Configuration configuration =
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Minor suggestion.. I think this construction code should be moved to a factory style method of the FlinkJobManager.

Similarly for the DataflowJobManager.

@tims
Copy link
Contributor

tims commented Jan 18, 2019

/lgtm

@feast-ci-bot feast-ci-bot merged commit c525845 into feast-dev:master Jan 18, 2019
Yanson pushed a commit to Yanson/feast that referenced this pull request Jul 29, 2020
…_redis_password

Redis security (added password to redis)
dmartinol pushed a commit to dmartinol/feast that referenced this pull request Aug 5, 2024
…ent-interceptor-auth

Created the grpc client auth header interceptor and removed the manua…
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

5 participants