Gat 1771

.old.migrations/README.md

@@ -0,0 +1,40 @@
+# HDR UK GATEWAY - Data Migrations
+
+The primary data source used by the Gateway Project is the noSQL solution provided by MongoDb.  Data migration strategy is a fundamental part of software development and release cycles for a data intensive web application.  The project team have chosen the NPM package Migrate-Mongoose - https://www.npmjs.com/package/migrate-mongoose to assist in the management of data migration scripts.  This package allows developers to write versioned, reversible data migration scripts using the Mongoose library.
+
+For more information on what migration scripts are and their purpose, please see sample background reading here - https://www.red-gate.com/simple-talk/sql/database-administration/using-migration-scripts-in-database-deployments/
+
+### Using migration scrips
+
+To create a data migration script, follow these steps:
+
+#### Step 1
+
+Ensure your terminal's working directory is the Gateway API and that node packages have been installed using 'npm i'.
+
+#### Step 2
+
+Run the command below, replacing 'my_new_migration_script' with the name of the script you want to create.  The name does not need to be unique, as it will be prefixed automatically with a timestamp, but it should be easily recognisable and relate strongly to the database change that will take place if the script is executed.
+
+./node_modules/.bin/migrate create my_new_migration_script
+
+#### Step 3
+
+Your new migration scripts should now be available in './migrations/', which you can now modify.  You can import the required Mongoose models as normal to interact with the MongoDb database.  The migration scripts that run locally will use the connection string taken from your .env file against the variable 'MIGRATE_dbConnectionUri'.
+
+Complete the scripts required for the UP process, and if possible, the DOWN process.  For awareness, the UP scripts run automatically as part of our CI/CD pipeline, and the DOWN scripts exist to reverse database changes if necessary, this is a manual process.
+
+#### Step 4
+
+With the scripts written, the functions can be tested by running the following command, replacing 'my_new_migration_script' with the name of the script you want to execute without the time stamp so for example 
+node -r esm migrations/migrate.js up add_globals 
+
+node -r esm migrations/migrate.js up my_new_migration_script
+
+When this process is completed, the connected database will have a new document representing your migration scripts inside the 'migrations' collection, which tracks the state of the migration.  If you need to run your scripts multiple times for test purposes, you can change the state of the migration to 'Down'.  
+
+During this process, please ensure you are using a personal database.
+
+#### Step 5
+
+Commit the code to the relevant git branch and raise a pull request.  The migration script will run automatically as the code moves through each environment.

migrations/migrate.js → .old.migrations/migrate.js

@@ -1,7 +1,7 @@
 import cli from 'migrate-mongoose/src/cli'; //lgtm [js/unused-local-variable]
 import mongoose from 'mongoose';
 
-mongoose.connect(process.env.MIGRATE_dbConnectionUri, {
+mongoose.connect(`${process.env.MIGRATE_dbConnectionUri}/${process.env.database}/?retryWrites=true&w=majority`, {
 	useNewUrlParser: true,
 	useFindAndModify: false,
 	useUnifiedTopology: true,

google_analytics.json

@@ -0,0 +1,3 @@
+{
+
+}

migrate-mongo-config.js

@@ -0,0 +1,46 @@
+// In this file you can configure migrate-mongo
+
+// Have to call this as this is pre-app start, thus env hasn't
+// been populated yet
+require('dotenv').config();
+
+const config = {
+  mongodb: {
+    // TODO Change (or review) the url to your MongoDB:
+    url: 'mongodb+srv://' +
+      process.env.user +
+      ':' +
+      process.env.password +
+      '@' +
+      process.env.cluster +
+      '?ssl=true&retryWrites=true&w=majority',
+
+    // TODO Change this to your database name:
+    databaseName: process.env.database,
+
+    options: {
+      useNewUrlParser: true, // removes a deprecation warning when connecting
+      useUnifiedTopology: true, // removes a deprecating warning when connecting
+      //   connectTimeoutMS: 3600000, // increase connection timeout to 1 hour
+      //   socketTimeoutMS: 3600000, // increase socket timeout to 1 hour
+    }
+  },
+
+  // The migrations dir, can be an relative or absolute path. Only edit this when really necessary.
+  migrationsDir: "migrations",
+
+  // The mongodb collection where the applied changes are stored. Only edit this when really necessary.
+  changelogCollectionName: "changelog",
+
+  // The file extension to create migrations and search for in migration dir 
+  migrationFileExtension: ".js",
+
+  // Enable the algorithm to create a checksum of the file contents and use that in the comparison to determine
+  // if the file should be run.  Requires that scripts are coded to be run multiple times.
+  useFileHash: false,
+
+  // Don't change this, unless you know what you're doing
+  moduleSystem: 'commonjs',
+};
+
+module.exports = config;

migrations/20221122095337-add_published_flag_to_data_requests.js

@@ -0,0 +1,30 @@
+module.exports = {
+  async up(db, client) {
+    // TODO write your migration here.
+    // See https://github.com/seppevs/migrate-mongo/#creating-a-new-migration-script
+
+    /**
+     * Update DAR to include an overriding published field to determine the published
+     * state of a DAR edit form publication by a custodian
+     */
+    // await db.collection('data_requests').updateMany({
+    //   $set: { "published_form": false },
+    // }); 
+
+    await db.collection('data_requests').updateMany({},
+      {
+        $set: { "publishedForm": false }
+      }
+    );
+  },
+
+  async down(db, client) {
+    // TODO write the statements to rollback your migration (if possible)
+
+    await db.collection('data_requests').updateMany({},
+      {
+        $unset: { "publishedForm": false }
+      }
+    );
+  }
+};

migrations/README.md

@@ -1,40 +1,62 @@
 # HDR UK GATEWAY - Data Migrations
 
-The primary data source used by the Gateway Project is the noSQL solution provided by MongoDb.  Data migration strategy is a fundamental part of software development and release cycles for a data intensive web application.  The project team have chosen the NPM package Migrate-Mongoose - https://www.npmjs.com/package/migrate-mongoose to assist in the management of data migration scripts.  This package allows developers to write versioned, reversible data migration scripts using the Mongoose library.
+The primary data source used by the Gateway Project is the noSQL solution provided by MongoDb.
+Data migration strategy is a fundemental part of software development and release cycles for a
+data intensive web application. The project team have chosen the NPM package Migrate-Mongo - https://www.npmjs.com/package/migrate-mongo
+to assist in the management of data migration scripts. This package allows developers to write versioned,
+reversible data migration scripts using the Mongoose library.
 
-For more information on what migration scripts are and their purpose, please see sample background reading here - https://www.red-gate.com/simple-talk/sql/database-administration/using-migration-scripts-in-database-deployments/
+For more information on what migration scripts are and their purpose, please see sample 
+background reading here - https://www.red-gate.com/simple-talk/sql/database-administration/using-migration-scripts-in-database-deployments/
 
 ### Using migration scrips
 
 To create a data migration script, follow these steps:
 
 #### Step 1
 
-Ensure your terminal's working directory is the Gateway API and that node packages have been installed using 'npm i'.
+Ensure your terminal's working directory is the Gateway API and that node packages have
+been installed using 'npm i'.
 
 #### Step 2
 
-Run the command below, replacing 'my_new_migration_script' with the name of the script you want to create.  The name does not need to be unique, as it will be prefixed automatically with a timestamp, but it should be easily recognisable and relate strongly to the database change that will take place if the script is executed.
+Run the command below, replacing 'my_new_migration_script' with the name of the script
+you want to create.  The name does not need to be unique, as it will be prefixed automatically
+with a timestamp, but it should be easily recognisable and relate strongly to the database
+change that will take place if the script is executed.
 
-./node_modules/.bin/migrate create my_new_migration_script
+./node_modules/.bin/migrate-mongo create my_new_migration_script
 
 #### Step 3
 
-Your new migration scripts should now be available in './migrations/', which you can now modify.  You can import the required Mongoose models as normal to interact with the MongoDb database.  The migration scripts that run locally will use the connection string taken from your .env file against the variable 'MIGRATE_dbConnectionUri'.
+Your new migration scripts should now be available in './migrations/', which you can now modify.
+You can interact directly with the database. The migration scripts that run locally will use the
+connection string config taken from your .env file against the variables: database, user, password and cluster.
 
-Complete the scripts required for the UP process, and if possible, the DOWN process.  For awareness, the UP scripts run automatically as part of our CI/CD pipeline, and the DOWN scripts exist to reverse database changes if necessary, this is a manual process.
+Complete the scripts required for the UP process, and if possible, the DOWN process.  For awareness, the UP
+scripts run automatically as part of our CI/CD pipeline, and the DOWN scripts exist to reverse
+database changes if necessary, this is a manual process.
 
 #### Step 4
 
-With the scripts written, the functions can be tested by running the following command, replacing 'my_new_migration_script' with the name of the script you want to execute without the time stamp so for example 
-node -r esm migrations/migrate.js up add_globals 
+With the scripts written, the functions can be tested by running the following command,
+replacing 'my_new_migration_script' with the name of the script you want to execute without
+the time stamp so for example
 
-node -r esm migrations/migrate.js up my_new_migration_script
+./node_modules/.bin/migrate-mongo up (to run all migration updates)
+./node_modules/.bin/migrate-mongo down (to rollback migration updates)
+./node_modules/.bin/migrate-mongo up my_new_migration_script (to run a single migration update)
+./node_modules/.bin/migrate-mongo down my_new_migration_script (to rollback a single migration update)
+./node_modules/.bin/migrate-mongo status (to list any pending migrations yet to be run)
 
-When this process is completed, the connected database will have a new document representing your migration scripts inside the 'migrations' collection, which tracks the state of the migration.  If you need to run your scripts multiple times for test purposes, you can change the state of the migration to 'Down'.  
+When this process is completed, the connected database will have a new document representing your
+migration scripts inside the 'migrations' collection, which tracks the state of the migration.
+If you need to run your scripts multiple times for test purposes, you can change the state of
+the migration to 'Down'.  
 
 During this process, please ensure you are using a personal database.
 
 #### Step 5
 
-Commit the code to the relevant git branch and raise a pull request.  The migration script will run automatically as the code moves through each environment.
+Commit the code to the relevant git branch and raise a pull request.  The migration script
+will run automatically as the code moves through each environment.

package.json

@@ -47,7 +47,7 @@
 		"keygrip": "^1.1.0",
 		"lodash": "^4.17.19",
 		"mailchimp-api-v3": "^1.15.0",
-		"migrate-mongoose": "^4.0.0",
+		"migrate-mongo": "^9.0.0",
 		"moment": "^2.29.3",
 		"mongoose": "^5.12.7",
 		"morgan": "^1.10.0",
@@ -87,6 +87,7 @@
 		"supertest": "^4.0.2"
 	},
 	"scripts": {
+		"start-with-migrate": "./node_modules/.bin/migrate up && node index.js",
 		"start": "node index.js",
 		"server": "nodemon --ignore 'src/**/*.json' index.js",
 		"debug": "nodemon --inspect=0.0.0.0:3001 index.js",

src/resources/datarequest/datarequest.model.js

@@ -103,6 +103,7 @@ const DataRequestSchema = new Schema(
 		originId: { type: Schema.Types.ObjectId, ref: 'data_request' },
 		versionTree: { type: Object, default: {} },
 		isShared: { type: Boolean, default: false },
+		publishedForm: { type: Boolean, default: false },
 	},
 	{
 		timestamps: true,