Source data model in Feast Core should be generalized

[woop]

Expected Behavior

Sources are generic and extensible.

Current Behavior

Currently the sources data model in Feast Core is hardcoded specifically to KafkaSource.

Possible Solution

Make sure none of the columns in our data model is specific to a single source. The pattern we have followed thus far is to use Map<String,String> for Source, Runner, Store configurations. One way to solve this problem is to have source configuration stored as a serialized JSON map in a single column in the Source table.

[ches]

While I get the utility and convenience especially for deploying Feast with more distributed ownership in the organization, we ignore / work around a lot of the Source and Store configuration kept in the registry RDBMS, in favor of service discovery.

I also think that asking data scientists and data engineers to be concerned with operational infra configuration like Kafka broker addresses when registering feature sets is not an elegant separation of concerns.

Maybe we can take this as an opportunity to consider design alternatives as well.

[woop]

Hi @ches, would love to continue the discussion. I've created an issue specifically for the topic over here: #633

That allows this card to stay on track separately since I don't see us making fundamental changes to sources in 0.5