Draft - Issue 989 big data support scaling vuu how to abstract vuu interfaces to allow different implementations of key classes by adding features on construction

[chrisjstevo]

First example of a plugin structure for project.

[netlify]

✅ Deploy Preview for papaya-valkyrie-395400 canceled.

Name	Link
🔨 Latest commit	6a550ff
🔍 Latest deploy log	https://app.netlify.com/sites/papaya-valkyrie-395400/deploys/65ae751267b00e0008938968

[rumakt]

Do we want to have 'features' depend on particular 'use-cases' like 'order' or 'price' ?

[rumakt]

I like the 'feature' and 'plugin' idea. This is very good 1st step towards making vuu a composable project

[naleeha]

I think this is good for understanding what it could potentially look like with the plugin concept & bootstraping relevant features etc.
I'm not sure about the feature concepts and having factory for each at that level.
I wonder we can make the construction of main concepts a bit simpler by making it less flexible - e.g. you always create a VirtualViewPort from a VirtualDataTable which knows to take in VirtualSort function etc
Ensure we resolve the right implementation at creation using dependency injection

We may want to re-use the implementation in multiple plugin as well. E.g. VirtualDataTable for both Ignite and Hadoop.
If thats the case, would it be better to tie the table creation by table type rather than plugin?
For registering and resolving new types when you add a plugin, maybe we can have a simple DI Container, similar to what you have for features in this
registry.register[DataTableFactory](VirtualDataTableFactory)
registry.resolve[DataTableFactory](type = VirtualDataTable.type)

.addTable(
    TableDef(
      name = BasketTradingTable,
      type = VirtualDataTable,
      columns = Columns.fromNames(InstanceId.string(), BT.BasketId.string())
    ),
    (table, registry) => new BasketTradingProvider(table, registry.Resolve[DataSource](type = IgniteDataSource)),
  )

I think there is still a lot of context im not aware of & normally find things i didnt think of at implementation time. Having a second implementation would help a lot in visualizing the right solution. Would it make sense as next step to have a spike of implementing the Virtual/BigData vuu solution using the ignite & once we have the second example, try and come up with the exact interfaces and how we can wire them up correctly?

[rumakt]

Wondering if 'RollingCache' is the most fortunate name for this trait ? The contract does not suggest anything 'rolling' here ? i.e. it can but does it have to ?

[chrisjstevo]

Agreed, renamed from Rolling to Windowed

[rumakt]

Good step forward, lets keep the ball rolling.

[linux-foundation-easycla]

• ✅ login: rumakt / name: Tomasz Rumak (514a531, 4e03230, 9226e33, 61c7090, f9e8f95, fed77e4, c4d2090)
• ✅ login: chrisjstevo (70e79ad, f04c077, cef0089, b09e9c6, e8c5a7f, 1e9ca7b, f6480a4, 80c2f1a, b037baa, 8688125, 58a261d, 18bf20f, 857a681, 913510b, 3f31f83, a1bed3d, 6a550ff)
• ❌ The email address for the commit (90b6a5a) is not linked to the GitHub account, preventing the EasyCLA check. Consult this Help Article and GitHub Help to resolve. (To view the commit's email address, add .patch at the end of this PR page's URL.) For further assistance with EasyCLA, please submit a support request ticket.