[FEAT]: Support redis hosting for 4 database on cloud

[lwaekfjlk]

Description

with real backend

Additional Information

No response

[shreypandey]

Hi @lwaekfjlk,
For redis support, a multi container app can be created using docker-compose with the support of local development. The following 3 container will be there:

1. backend
2. frontend
3. redis

A bash script can be created which will abstract the logic of running of the multi container app from the users. Something like this:

# set env variables and config
docker compose --env-file "$env_file" build
docker compose --env-file "$env_file" up

Users just need to run the following command to run the project in their local. This will make easier for users to run and develop the project.

bash run.sh

For handling the scenario where the user want to run the example sdk using a python script(as mentioned in running example of readme.md), support of on-memory and redis database client can be provided, picking up the appropriate database using config/env variable.
This is an pseudo implementation of handling redis as well as currently supported on-memory storage.

class DatabaseClient(ABC):
    @abstractmethod 
    def add(self, data: T) -> None:
        
    @abstractmethod
    def update(self, pk: str, updates: Dict[str, Any]) -> bool:
        
    @abstractmethod
    def delete(self, pk: str) -> bool:
        
    @abstractmethod
    def get(self, **conditions: Union[str, int, float, List[int], None]) -> List[T]:

    @abstractmethod
    def search(self, query_embedding: np.ndarray, **conditions: Union[str, int, float, List[int], None]):    # Types can be revised later
        
class LocalDatabaseClient(DatabaseClient):
    # Concrete implementation of the LocalDatabaseProvider. 
    # In this case, it will mostly be replicated from existing BaseDB class 

class RedisDatabaseClient():
    __redis_client__: redis.Redis = None # For singleton connection to redis
    
    @classmethod
    def create_client(cls):
        if cls.__redis_client__ is None:
            cls.__redis_client__ = cls.create_redis_client() # Initialize redis SDK here
    
    # Concrete implementation of abstract methods according to redis client SDK

DATABASE_REGISTRY: Dict[str, Type[DatabaseClient]] = {
    "redis": RedisDatabaseClient,
    "local": LocalDatabaseClient
}

class DatabaseClientHandler:
    __database_client__: DatabaseClient = None # For singleton client
     
    @classmethod
    def get_instance(cls) -> DatabaseClient:
        if cls.__database_client__ is None:
            database_type = os.getenv("database_type", "local") # This can be from config also
            if not database_type in DATABASE_REGISTRY:
                raise Exception("Invalid database type")
            cls.__database_client__ = DATABASE_REGISTRY.get(database_type)()
            return cls.__database_client__

The BaseDB will look something like this:

class BaseDB(Generic[T]):
    def __init__(
        self, data_class: Type[T], load_file_path: Optional[str] = None
    ) -> None:
        self.project_name: Optional[str] = None
        self.data_class = data_class
        # data, embeddings and load_from_json should be handled in appropriate DatabaseClient
        self.data_base_client = DatabaseClientHandler.get_instance()
        
    def set_project_name(self, project_name: str) -> None:
        self.project_name = project_name

    def add(self, data: T) -> None:
        if self.project_name is not None:
            data.project_name = self.project_name
        self.data_base_client.add(data)
        logger.info(
            f"Creating instance of '{data.__class__.__name__}': '{data.model_dump()}'"
        )
.
.
.

Let me know your thoughts on this.

[lwaekfjlk]

that's perfect! We should implement this. I can try to make one initial PR asap. Thanks a lot for your suggestions.

[lwaekfjlk]

@shreypandey feel free to create a PR to share any progress so that we can work together to make this feature.

[shreypandey]

@lwaekfjlk I will create a PR on this. You can assign this issue to me if you want.

[lwaekfjlk]

that's awesome!!!

[lwaekfjlk]

looking forward to this.