Support required to enable single sourcing solution

[arun1205]

Context
Workspace module is being planned to be relocated from Sunbird ED (Consumption portal) to Sunbird coKreat (creation portal) to ease the management of workspace related data and processes

As a better practice, it is being planned that the Workspace module should have its own database, and is decoupled so that the data used is not updated/edited by any other building block/service.

In order to implement the above agenda, Co-kreate team suggests the replication of data from Sunbird ED (Consumption portal) to Sunbird coKreat (creation portal), so that the data at Sunbird coKreat serves as the single source of truth and henceforth Sunbird ED portals syncs itself with Sunbird coKreat data to keep itself updated with latest data

Refer to the detailed PRD and context here https://project-sunbird.atlassian.net/wiki/spaces/SingleSource/pages/2932637697/Workspace+in+Sourcing+Portal

Problem Statement

Currently there are two front-end tools to create and manage assets and content.

1. A “workspace” that is part of common portal (both for consumption and creation). This has been the front-end UI for creating managing assets since the beginning of sunbird. This is used to create content by internal creators of the sourcing organization. Content created through this tool resides in the same repository as that is used by consumption.

2. A Vidyadaan /creation portal (sourcing and contributor) that is used for sourcing content from contributors (external or internal) by creating sourcing projects. Content created through this tool resides in sourcing repository. Once the content is reviewed and approved by the sourcing organization reviewers, the content is published in consumption repository.

Users have to go through two different tools which is not an efficient way and causes confusion. The two front-end tools also use different versions of software and hence maintenance also becomes difficult.

Solution

The solution is to:

1. Build a common front-end that can cater to all sourcing activities described above
2. The single front-end solution will take care of generalization and externalization so that it can easily be extended to support new object types and categories.
3. Have separate repositories for sourcing and consumption. Only published assets are pushed to consumption repository and available for consumption

Current Behaviour

As of now workspace module is a part of consumption portal and consumes and updates the data at consumption portal

Expected Behaviour

Workspace module should be a part of creation portal and the its data should be decoupled from Consumption portal and only read access to be provided on Consumption data
The consumption portal is expected to sync itself with the creation DB on a daily basis

Diagrammatic Representation

Today…

1. Workspace inside Sunbird ED uses the Channel, Categories, Framework, and License data from consumption graph/database.
2. There is/are sync job(s) that sync the channel, categories, framework, and license between consumption and creation graph databases.
3. There are variation in schema across these two graphs. For example, creation graph/db has attributes for storing Reviewer comments whereas the consumption graph/db does not have that attribute.

Possible Solution
As we brainstorm on solution, we need to keep in mind not just the needs of a large-scale adopter but also needs of a small/mid-scale adopter. Not every adopter may want two separate graphs. Some may prefer a single graph for sourcing and creation both depending on their scale.

The reason for enabling separate sourcing and creation graph is to support huge volume of creation and even larger volumes of consumption (read) transactions.

ACTION ITEM	COMMENTS[CO-CREATE TEAM]	COMMENTS[KNOWLEDGE TEAM]
Schema finalization for creation & consumption portals as to which properties needs to be migrated to creation DB and which ones are only Consumption portal specific	Knowledge and Co-create teams need to collaborate and finalize on the schema.
Validation of Meta data when data is migrated between Creation and Consumption DBs	Needs to be Done
Validation of Telemetry when content / collection is created and migrated between Creation and Consumption DBs	Needs to be Done
API config (ON/OFF) .In case we need to activate apis that want to update the consumption DB at any point of time because of some explainable reason	It is already in place
Channel Data	Copy of Consumption portal’s Channel data
Schema(Content/ Collection/ Question)	Copy of Consumption portal’s Schema(Content/ Collection/ Question)
Categories Data	Creation portal  already has Categories data  in prod env. Need to decide between consumption and creation's data as to which one to be treated as  single source of truthKnowledge and Co-create teams need to collaborate and finalize this
Framework Data	Copy of Consumption Framework data
License Data	Copy of Consumption License data
Configurable databases for Creation & Consumption Portal (In case, in future we are to revert creation db and consumption DB as one again)	Knowledge and Co-create teams needs to collaborate and finalize on the this
Auto creation job to sync the data b/w creation & consumption(any object)	Sync job required that runs on a daily basisNeeds to be done
Use same identifiers for Creation & Consumption -> Backward compatibility	Knowledge team to finalize on the this including:DesignImplementationExisting Data migration
Multiple projects for single collection contributionsSame collection used for multiple projects for contribution	QUES: If project A and project B share the same collection and if project A's contributor edits project A , what should happen to project B’'s collection?ANSWER: Project A will have its own copy of collection that would not be shared with project B and any edit on Project A won't have effect on Project B

@vinukumar-vs @surendrasinghs @pallakartheekreddy @vaivk369

[surendrasinghs]

@Sunbird-Knowlg/sunbirdknowlg_workinggroup Any further thoughts? This is an important aspect to solve.

[rhwarrier]

@vinukumar-vs could we set up an initial meeting with co-kreat to discuss some of these action items above? We'll have to factor these into the roadmap based on the agreed priorities.

[surendrasinghs]

Sharing meeting notes here for everyone's reference. We discussed and agreed upon the following:

1. Workspace reference app should be decoupled from Sunbird Ed and made pluggable. coKreat and/or Ed can use it as relevant.

Assuming that an adopter deploys both Sunbird Ed and coKreat - where both of them a have a Knowlg instance running in them. Following should be supported

2. SB Knowlg should have the functionality for configuring certain objects as read-only. For example, when Sunbird Ed and coKreat are deployed together, the Content Service in SB ED should be made read-only for that instance. In this case, SB coKreat will be used to create, update, modify, and delete Content. Published content from coKreat will be imported into Ed repo and read operations at scale can be supported through that.
3. SB Knowlg should have the functionality for referencing objects from another instance of Knowlg. For example, when coKreat and Ed are deployed together, certain objects such as taxonomy, category can be referenced in Ed directly from coKreat repo.
4. SB Knowlg should provide a configurable IMPORT API so that adopter can configure which objects to import/sync between two instances of Knowlg.

For standalone coKreat instances,

5. For scenarios where coKreat is used independently, we will not need 2, 3, 4, & 5 from above. We will just need to make Workspace reference app pluggable so that it can be used in coKreat.

Check out the visual diagrams used in our discussion here https://docs.google.com/presentation/d/1bMVhS8aolV-6-Bs21ELtUrZp3jwPPPJrFU8I53GPjXY/edit?usp=sharing

@vrayulu @pallakartheekreddy @vinukumar-vs @arun1205 @vaivk369 @rhwarrier Please add if anything.

[rhwarrier]

@vinukumar-vs there's a few items here that will need attention. Will need to figure out if it fits into the decoupling efforts, and also the when/ how of it - including design discussions.

@surendrasinghs how about initiating a tasks log + calendar plan for this effort, so we ensure none of the items are missed, and they're all planned and done in the right order/ timeframe as they should be? @vrayulu @SanthoshVasabhaktula