You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Currently the sources data model in Feast Core is hardcoded specifically to KafkaSource.
Possible Solution
Make sure none of the columns in our data model is specific to a single source. The pattern we have followed thus far is to use Map<String,String> for Source, Runner, Store configurations. One way to solve this problem is to have source configuration stored as a serialized JSON map in a single column in the Source table.
The text was updated successfully, but these errors were encountered:
woop
changed the title
Sources should be generic
Sources should be generalized
Apr 18, 2020
While I get the utility and convenience especially for deploying Feast with more distributed ownership in the organization, we ignore / work around a lot of the Source and Store configuration kept in the registry RDBMS, in favor of service discovery.
I also think that asking data scientists and data engineers to be concerned with operational infra configuration like Kafka broker addresses when registering feature sets is not an elegant separation of concerns.
Maybe we can take this as an opportunity to consider design alternatives as well.
Expected Behavior
Sources are generic and extensible.
Current Behavior
Currently the sources data model in Feast Core is hardcoded specifically to KafkaSource.
Possible Solution
Make sure none of the columns in our data model is specific to a single source. The pattern we have followed thus far is to use
Map<String,String>
for Source, Runner, Store configurations. One way to solve this problem is to havesource
configuration stored as a serialized JSON map in a single column in the Source table.The text was updated successfully, but these errors were encountered: