Talawa api multitenancy (#776)

* Initial setup (#726) * Initial setup * simplified creation * added readme.md file * added connection manager and id helper functions * Error handling & security updates (#744) * [Feature Request] New Features for Talawa-admin added (#725) * check authentication feature added for talawa admin * tests for check authentication added * minor fixes * typo fixes * minor fixes * npm err fixes * google recaptcha configured * new dependency added * mailer, forgotPassword endpoint and otp endpoint is configured * graphql endpoints updated * constants updated * Documentation updated * tests for changes added * hard coded value error fixes * lint fixes * lint fixes * failed test cases fixes * Failed tests fixes * minor fixes * marked-teminal dependency updated * Tests fixes * tests updated * tests fixes * tests updated * new resolvers added and schema updated * graphql mutations updated * tests updated for changes * tests updated * tests updated * tests updated * minor fixes * [Feature Request] : Plugin Architecture for Server (#730) * Test : lib/resolvers/group_chat_query/groupChatMessages.js * Add Test : Added Valid JSON Check * Create README.md * Create README.md * Update is-auth.js * AddedDocs Docs from withfilter pending * Update index.js * completed * sample * update/index.js * update/index.js * update/is-auth.js * Update/readme.md * update/image-readme * Add/query-schema * Update/Plugin-graphQL-schema * Add/Plugin-MongoDB-Model * Add/createPlugin-and-refractoring * Add/getPlugins * Add/Mutation/UpdatePluginStatus * Add/Mutation/updateTempPluginInstalledOrgs * Add/Mutation/Schema * Add/import/mutations * Add/plugins * Fix/path-err * Fix/Erros * update/lockfile * Add/plugin-model * Update is-auth.js * Removed extra queries * Documentation added for plugin queries and models * Fix/`delelte cr in prettier/prettier` * updated : mutation * Add/test `getPlugins` * Fix/`lint error` * test `updatePluginInstalledOrgs` * Test/ `updateInstallStatus` * Fix/Erros-1 * Fix/Erros-2 * Fix/Erros-3 * Test/`Queries.js` * Test/`Mutation.js` * remove extra `console.log` * Delete admin-plugin-query.js * Delete super-admin-plugin-query.js * Fix/Package-lock * added delimiter to helper functions * error handling for connection manager * added readme.md for connection manager * error handling for Database module Co-authored-by: Asmit Kumar Sirohi <63240102+asmitsirohi@users.noreply.github.com> Co-authored-by: Siddhesh Bhupendra Kuakde <65951872+SiddheshKukade@users.noreply.github.com> * Create new tenant on createOrganization * Custom schema functionality * Added the context function * modify createPost resolver * fix createPost, createOrganization resolvers * Initial setup (#726) * Initial setup * simplified creation * added readme.md file * added connection manager and id helper functions * Error handling & security updates (#744) * [Feature Request] New Features for Talawa-admin added (#725) * check authentication feature added for talawa admin * tests for check authentication added * minor fixes * typo fixes * minor fixes * npm err fixes * google recaptcha configured * new dependency added * mailer, forgotPassword endpoint and otp endpoint is configured * graphql endpoints updated * constants updated * Documentation updated * tests for changes added * hard coded value error fixes * lint fixes * lint fixes * failed test cases fixes * Failed tests fixes * minor fixes * marked-teminal dependency updated * Tests fixes * tests updated * tests fixes * tests updated * new resolvers added and schema updated * graphql mutations updated * tests updated for changes * tests updated * tests updated * tests updated * minor fixes * [Feature Request] : Plugin Architecture for Server (#730) * Test : lib/resolvers/group_chat_query/groupChatMessages.js * Add Test : Added Valid JSON Check * Create README.md * Create README.md * Update is-auth.js * AddedDocs Docs from withfilter pending * Update index.js * completed * sample * update/index.js * update/index.js * update/is-auth.js * Update/readme.md * update/image-readme * Add/query-schema * Update/Plugin-graphQL-schema * Add/Plugin-MongoDB-Model * Add/createPlugin-and-refractoring * Add/getPlugins * Add/Mutation/UpdatePluginStatus * Add/Mutation/updateTempPluginInstalledOrgs * Add/Mutation/Schema * Add/import/mutations * Add/plugins * Fix/path-err * Fix/Erros * update/lockfile * Add/plugin-model * Update is-auth.js * Removed extra queries * Documentation added for plugin queries and models * Fix/`delelte cr in prettier/prettier` * updated : mutation * Add/test `getPlugins` * Fix/`lint error` * test `updatePluginInstalledOrgs` * Test/ `updateInstallStatus` * Fix/Erros-1 * Fix/Erros-2 * Fix/Erros-3 * Test/`Queries.js` * Test/`Mutation.js` * remove extra `console.log` * Delete admin-plugin-query.js * Delete super-admin-plugin-query.js * Fix/Package-lock * added delimiter to helper functions * error handling for connection manager * added readme.md for connection manager * error handling for Database module Co-authored-by: Asmit Kumar Sirohi <63240102+asmitsirohi@users.noreply.github.com> Co-authored-by: Siddhesh Bhupendra Kuakde <65951872+SiddheshKukade@users.noreply.github.com> * dynamic functionality additions * update tenant schema folder * modify all posts-comments resolvers * resolve failing tests for posts modifications * modify events-tasks-projects related resolvers * resolve failing tests * modified user queries * modify longin mutation to return events properly * modify org mutations and resolve failing tests * diconnect properly from the database * if not connection for org search in the database for it * group chats and direct chats mutations * remove console logs * remove unnecessary comments * fix drop database Co-authored-by: Mickey <31347338+JamaicanFriedChicken@users.noreply.github.com> Co-authored-by: Asmit Kumar Sirohi <63240102+asmitsirohi@users.noreply.github.com> Co-authored-by: Siddhesh Bhupendra Kuakde <65951872+SiddheshKukade@users.noreply.github.com> Co-authored-by: JamaicanFriedChicken <mikey_lue777@hotmail.com>
PalisadoesFoundation · Nov 1, 2022 · 7f12160 · 7f12160
1 parent 3b958de
commit 7f12160
Show file tree

Hide file tree

Showing 109 changed files with 2,445 additions and 273 deletions.
diff --git a/constants.js b/constants.js
@@ -31,6 +31,8 @@ const ORGANIZATION_NOT_FOUND_MESSAGE = 'organization.notFound';
 const ORGANIZATION_NOT_FOUND_CODE = 'organization.notFound';
 const ORGANIZATION_NOT_FOUND_PARAM = 'organization';
 
+const CONNECTION_NOT_FOUND = 'Connection not found';
+
 const ORGANIZATION_MEMBER_NOT_FOUND = "Organization's user is not a member";
 const ORGANIZATION_MEMBER_NOT_FOUND_MESSAGE = 'organization.member.notFound';
 const ORGANIZATION_MEMBER_NOT_FOUND_CODE = 'organization.member.notFound';
@@ -85,9 +87,12 @@ const POST_NOT_FOUND_MESSAGE = 'post.notFound';
 const POST_NOT_FOUND_CODE = 'post.notFound';
 const POST_NOT_FOUND_PARAM = 'post';
 
+const DATABASE_CONNECTION_FAIL = 'Failed to connect to database';
+
 const STATUS_ACTIVE = 'ACTIVE';
 
 const IN_PRODUCTION = process.env.NODE_ENV === 'production';
+const IN_TEST = process.env.NODE_ENV === 'test';
 
 if (process.env.NODE_ENV === 'test') {
   URL = 'http://localhost:4000/graphql';
@@ -98,6 +103,7 @@ module.exports = {
 
   URL,
   IN_PRODUCTION,
+  IN_TEST,
 
   USER_NOT_AUTHORIZED,
   USER_NOT_AUTHORIZED_MESSAGE,
@@ -114,6 +120,8 @@ module.exports = {
   ORGANIZATION_NOT_FOUND_CODE,
   ORGANIZATION_NOT_FOUND_PARAM,
 
+  CONNECTION_NOT_FOUND,
+
   EVENT_PROJECT_NOT_FOUND,
   EVENT_PROJECT_NOT_FOUND_CODE,
   EVENT_PROJECT_NOT_FOUND_MESSAGE,
@@ -182,4 +190,6 @@ module.exports = {
   POST_NOT_FOUND_MESSAGE,
   POST_NOT_FOUND_CODE,
   POST_NOT_FOUND_PARAM,
+
+  DATABASE_CONNECTION_FAIL,
 };
diff --git a/db.js b/db.js
@@ -1,5 +1,6 @@
 const mongoose = require('mongoose');
 const logger = require('logger');
+const destroyConnections = require('./lib/ConnectionManager/destroyConnections');
 
 const connect = async () => {
   try {
@@ -16,6 +17,7 @@ const connect = async () => {
 };
 
 const disconnect = async () => {
+  await destroyConnections();
   await mongoose.connection.close();
 };
 module.exports = { connect, disconnect };
diff --git a/index.js b/index.js
@@ -1,5 +1,6 @@
 require('dotenv').config(); // pull env variables from .env file
 
+const connectionManager = require('./lib/ConnectionManager');
 const depthLimit = require('graphql-depth-limit');
 const { ApolloServer, PubSub } = require('apollo-server-express');
 const http = require('http');
@@ -113,7 +114,7 @@ const apolloServer = new ApolloServer({
     auth: AuthenticationDirective,
     role: RoleAuthorizationDirective,
   },
-  context: ({ req, res, connection }) => {
+  context: async ({ req, res, connection }) => {
     if (connection) {
       return {
         ...connection,
@@ -172,6 +173,7 @@ apolloServer.installSubscriptionHandlers(httpServer);
 const serverStart = async () => {
   try {
     await database.connect();
+    await connectionManager.initTenants();
     httpServer.listen(process.env.PORT || 4000, () => {
       logger.info(
         `🚀 Server ready at http://localhost:${process.env.PORT || 4000}${

diff --git a/lib/ConnectionManager/README.md b/lib/ConnectionManager/README.md
@@ -0,0 +1,48 @@
+# Connection Manager Module
+
+This is the module that is responsible for connecting/disconnecting to each tenant's database.
+
+### Prototype :
+
+`addTenantConnection(organizationId)`
+
+`getTenantConnection(organizationId)`
+
+`initTenants()`
+
+`destroyConnections()`
+
+`addTenantConnection` is responsible for connecting to the database of a specific organization `organizationId`
+and returning a reference to the connection object if the organization is not found it will throw an organization not found error.
+
+`getTenantConnection` is responsible for retrivnig a connection to `organizationId` which already is in memory if it's not the function will throw a connection not found error.
+
+`initTenants` is usually used on the startup of the api which connects to all the available tenants that are stored.
+
+`destoryConnections` is used to close the connections and deleting them from memory.
+
+### Basic usage :
+
+This is mostly how it's going to be used:
+
+```javascript
+const connection = getTenantConnection(organizationId);
+const Post = connection.Post;
+
+// posts is an array of the posts stored in that specific tenant.
+const posts = await Post.find({});
+// to close connections:
+await destroyConnections();
+// would throw an error:
+const con = getTenantConnection(organizationId);
+```
+
+### This is the main way to manage the databases that are devided into 2 types:
+
+##### main database:
+
+which holds connections, organizations, users, files data (shared data).
+
+##### tenant databases;
+
+which holds organization specific data such as posts, events, comments...
diff --git a/lib/ConnectionManager/addTenantConnection.js b/lib/ConnectionManager/addTenantConnection.js
@@ -0,0 +1,31 @@
+const Tenant = require('../models/Tenant');
+const Database = require('../Database/index');
+const { setConnection } = require('./connections');
+const { NotFoundError } = require('errors');
+const requestContext = require('talawa-request-context');
+const {
+  ORGANIZATION_NOT_FOUND,
+  ORGANIZATION_NOT_FOUND_MESSAGE,
+  ORGANIZATION_NOT_FOUND_PARAM,
+  ORGANIZATION_NOT_FOUND_CODE,
+  IN_PRODUCTION,
+} = require('../../constants');
+
+module.exports = async (organizationId) => {
+  const [tenant] = await Tenant.find({ organization: organizationId });
+  if (!tenant || !tenant.url) {
+    throw new NotFoundError(
+      !IN_PRODUCTION
+        ? ORGANIZATION_NOT_FOUND
+        : requestContext.translate(ORGANIZATION_NOT_FOUND_MESSAGE),
+      ORGANIZATION_NOT_FOUND_CODE,
+      ORGANIZATION_NOT_FOUND_PARAM
+    );
+  }
+  let connection = new Database(tenant.url, {
+    schema: !tenant.scheme || tenant.scheme.length === 0 ? [] : tenant.scheme,
+  });
+  await connection.connect();
+  setConnection(tenant.organization, connection);
+  return connection;
+};
diff --git a/lib/ConnectionManager/connections.js b/lib/ConnectionManager/connections.js
@@ -0,0 +1,57 @@
+const Tenant = require('../../lib/models/Tenant');
+const Database = require('../Database/index');
+
+const connections = {};
+const MainDB = require('../../lib/models');
+
+const setConnection = (orgId, connection) => {
+  connections[`${orgId}`] = connection;
+};
+
+const getConnection = async (orgId) => {
+  if (!connections[orgId]) {
+    const [tenant] = await Tenant.find({ organization: orgId });
+    if (!tenant || !tenant.url) {
+      return MainDB;
+    }
+    let connection = new Database(tenant.url, {
+      schema: !tenant.scheme || tenant.scheme.length === 0 ? [] : tenant.scheme,
+    });
+    await connection.connect();
+    setConnection(tenant.organization, connection);
+    return connection;
+  }
+  return connections[orgId];
+};
+
+const getAllConnections = () => {
+  return connections;
+};
+
+const orgHasTenant = (orgId) => {
+  if (connections[orgId]) return true;
+  return false;
+};
+
+const destroy = async () => {
+  for (let conn in connections) {
+    await connections[conn].disconnect();
+    delete connections[conn];
+  }
+};
+
+const destroyOneConnection = async (orgId) => {
+  if (!connections[orgId]) return false;
+  await connections[orgId].disconnect();
+  delete connections[orgId];
+  return true;
+};
+
+module.exports = {
+  setConnection,
+  getConnection,
+  destroy,
+  getAllConnections,
+  orgHasTenant,
+  destroyOneConnection,
+};
diff --git a/lib/ConnectionManager/destroyConnections.js b/lib/ConnectionManager/destroyConnections.js
@@ -0,0 +1,5 @@
+const connections = require('./connections');
+
+module.exports = async () => {
+  await connections.destroy();
+};
diff --git a/lib/ConnectionManager/destroyOneConnection.js b/lib/ConnectionManager/destroyOneConnection.js
@@ -0,0 +1,5 @@
+const connections = require('./connections');
+
+module.exports = async (orgId) => {
+  return await connections.destroyOneConnection(orgId);
+};
diff --git a/lib/ConnectionManager/getTenantConnection.js b/lib/ConnectionManager/getTenantConnection.js
@@ -0,0 +1,5 @@
+const { getConnection } = require('./connections');
+
+module.exports = async (organizationId) => {
+  return await getConnection(organizationId);
+};
diff --git a/lib/ConnectionManager/index.js b/lib/ConnectionManager/index.js
@@ -0,0 +1,13 @@
+const addTenantConnection = require('./addTenantConnection');
+const getTenantConnection = require('./getTenantConnection');
+const initTenants = require('./initTenants');
+const destroyConnections = require('./destroyConnections');
+const destroyOneConnection = require('./destroyOneConnection');
+
+module.exports = {
+  addTenantConnection,
+  getTenantConnection,
+  initTenants,
+  destroyConnections,
+  destroyOneConnection,
+};
diff --git a/lib/ConnectionManager/initTenants.js b/lib/ConnectionManager/initTenants.js
@@ -0,0 +1,17 @@
+const Tenant = require('../models/Tenant');
+const logger = require('logger');
+const Database = require('../Database/index');
+const { setConnection } = require('./connections');
+
+module.exports = async () => {
+  try {
+    const databases = await Tenant.find();
+    for (let db of databases) {
+      let connection = new Database(db.url, { schema: db.scheme });
+      await connection.connect();
+      setConnection(db.organization, connection);
+    }
+  } catch (e) {
+    logger.error('Error while connecting to mongo database', e);
+  }
+};
diff --git a/lib/Database/MongoImplementation/index.js b/lib/Database/MongoImplementation/index.js
@@ -0,0 +1,86 @@
+const MongooseInstance = require('mongoose').Mongoose;
+const { DATABASE_CONNECTION_FAIL } = require('../../../constants');
+const MessageChat = require('./schema/Chat');
+const Comment = require('./schema/Comment');
+const DirectChat = require('./schema/DirectChat');
+const DirectChatMessage = require('./schema/DirectChatMessage');
+const Event = require('./schema/Event');
+const EventProject = require('./schema/EventProject');
+const File = require('./schema/File');
+const Group = require('./schema/Group');
+const GroupChat = require('./schema/GroupChat');
+const GroupChatMessage = require('./schema/GroupChatMessage');
+const ImageHash = require('./schema/ImageHash');
+const Language = require('./schema/Language');
+const MembershipRequest = require('./schema/MembershipRequest');
+const Message = require('./schema/Message');
+const Organization = require('./schema/Organization');
+const Plugin = require('./schema/Plugin');
+const PluginField = require('./schema/PluginField');
+const Post = require('./schema/Post');
+const Task = require('./schema/Task');
+const User = require('./schema/User');
+
+const defaultSchema = {
+  MessageChat,
+  Comment,
+  DirectChat,
+  DirectChatMessage,
+  Event,
+  EventProject,
+  File,
+  Group,
+  GroupChat,
+  GroupChatMessage,
+  ImageHash,
+  Language,
+  MembershipRequest,
+  Message,
+  Organization,
+  Plugin,
+  PluginField,
+  Post,
+  Task,
+  User,
+};
+
+function MongoDB(url, schemaObjects) {
+  this.mongo = new MongooseInstance();
+  //  this.schema = schemaObjects ? schemaObjects : defaultSchema;
+  if (!schemaObjects) {
+    let schemaObj = {};
+    for (let obj of schemaObjects) {
+      if (defaultSchema[obj]) {
+        schemaObj[obj] = defaultSchema[obj];
+      }
+    }
+    this.schema = schemaObj;
+  } else {
+    this.schema = defaultSchema;
+  }
+
+  this.type = 'Mongoose';
+  this.Native = this.mongo.connection.db;
+  this.connect = async () => {
+    try {
+      await this.mongo.connect(url, {
+        useCreateIndex: true,
+        useUnifiedTopology: true,
+        useFindAndModify: false,
+        useNewUrlParser: true,
+      });
+      this.Native = this.mongo.connection.db;
+    } catch (error) {
+      throw new Error(DATABASE_CONNECTION_FAIL);
+    }
+  };
+  this.disconnect = async () => {
+    await this.mongo.connection.close();
+  };
+  // creating the schema
+  for (let key in this.schema) {
+    this[key] = this.mongo.model(key, this.schema[key]);
+  }
+}
+
+module.exports = MongoDB;
diff --git a/lib/Database/MongoImplementation/schema/Chat.js b/lib/Database/MongoImplementation/schema/Chat.js
@@ -0,0 +1,30 @@
+const mongoose = require('mongoose');
+const Schema = mongoose.Schema;
+
+const chatSchema = new Schema({
+  message: {
+    type: String,
+    required: true,
+  },
+  languageBarrier: {
+    type: Boolean,
+    required: false,
+    default: false,
+  },
+  sender: {
+    type: Schema.Types.ObjectId,
+    ref: 'User',
+    required: true,
+  },
+  receiver: {
+    type: Schema.Types.ObjectId,
+    ref: 'User',
+    required: true,
+  },
+  createdAt: {
+    type: Date,
+    required: true,
+    default: Date.now,
+  },
+});
+module.exports = chatSchema;