Merge #5372

5372: Rework stock_movement_status calculation r=jmcameron a=jniles

This is a full rewrite of the `stock_movement_status` table and associated stored procedures.  The goal of this rewrite is to improve clarity, remove user-defined variables that may potentially collide across MySQL connections, and improve the performance of the `GetCMM()` calculation.


Closes #5073.
Closes #5144.
Closes #5332.
Closes #5360.


----
![image](https://user-images.githubusercontent.com/896472/108206217-6741e600-7126-11eb-86c1-ada342009708.png)


![image](https://user-images.githubusercontent.com/896472/108207993-c1dc4180-7128-11eb-9bf7-c611e20ec500.png)


![heidisql_0CKbDzKdzl](https://user-images.githubusercontent.com/896472/108201428-17602080-7120-11eb-81c5-723e8cc122e9.png)



This PR contains many far reaching changes.  Here is a lot of the notable ones:

 1. I've removed the `avg_consumption` column from the inventory table.  It was never used, as far as I can tell, to do any reporting.  It was used in the stock import feature, but we don't use the `avg_consumption` to report on, so I just removed that code.
 2. I've removed the `stock_consumption` table and associated procedures.  It is used in several reports and they are now broken.  However, I would argue they were always broken since the `stock_consumption` did not record accurate information about stock consumption.
 3.  I've entirely changed the structure behind the `stock_movement_status` table for the following reasons:
     1. The [previous structure](https://github.com/IMA-WorldHealth/bhima/blob/e455a8011db69ad48839afc35977d3dd041d3241/server/models/schema.sql#L2016) was very hard to audit.  For example, it had `start_date`, `end_date`, and `quantity`.  Is that the quantity at the start?  The quantity at the end?  If I wanted to find the value on date X, I needed to do a BETWEEN query.
     2. The table did not distinguish between consumption information and exit information.  This made the `GetCMM()` calculation have to look up additional information from the `stock_movement` table.  A user would need to as well.
     3. In theory, we'll be calling `GetCMM()` more than we will be moving stock.  It makes sense to make the performance tradeoff to increase the speed of `GetCMM()` and decrease the speed of inserting stock.
 5. I've renamed the `GetCMM()` call to `GetAMC()` for clarity.  It also takes in a single date and uses the depot to find the correct value of the date range.  This prevents us from having inconsistent results because a developer didn't properly compute the date range.  If we need to support a custom date range, we'll just write another SP.
 6. I've slightly changed the API of `GetAMC()`:
     1. It returns `head_days` and `tail_days` instead of `days_before_stock_consumption`.  That is because `days_before_stock_consumption` is the real number of days before the first consumption.  Instead, it the days between the start of the window and the next record.  Similarly, `tail_days` gives you the number of days between the last consumption record and the end of the window.
     2. It returns the `quantity_in_stock` so that we can perform stock out date calculations.
     3. `first_inventory_movement_date` -> `min_date`
     4. `last_inventory_movement_date` -> `max_date`

    
------

It isn't all good.  In fact, initializing this is terrible.  On my machine (4GB of RAM):
```mysql
mysql> use vanga;
mysql> CALL zRecomputeStockMovementStatus();
Query OK, 0 rows affected (22 min 4.81 sec)

mysql> use imck;
mysql> CALL zRecomputeStockMovementStatus();
Query OK, 0 rows affected (4 min 20.31 sec)
```

This is _horrible_.  I did get a lot faster speeds when I was just testing on MySQL 8, but the syntax didn't work on MySQL 5.7 (#5364).  When I fixed it on MySQL5.7, I ran into issues on MySQL 8.  Basically, we got the worst performance of both worlds by trying to get it to work correctly on both worlds.

However, I still think it is worth merging because the performance of any individual transaction isn't bad.  It is only when rebuilding the database that things go horribly long.

----

### How to test

Download this PR and run the tests!  (`yarn test:integration` and `yarn test:integration:stock`).  If that works, try with a production database.  Note, as above, you'll need to `CALL zRecomputeStockMovementStatus()` to build the table.

Co-authored-by: Jonathan Cameron <jmcameron@gmail.com>
Co-authored-by: dependabot-preview[bot] <27856297+dependabot-preview[bot]@users.noreply.github.com>
Co-authored-by: mbayopanda <mbayopanda@gmail.com>
Co-authored-by: Jonathan Niles <jonathanwniles@gmail.com>

.semaphore/semaphore.yml

@@ -70,6 +70,7 @@ blocks:
             - env_var: MYSQL_VERSION
               values:
                 - '5.7'
+                - '8'
             - env_var: NODEJS_VERSION
               values:
                 - '12'

client/src/i18n/en/stock.json

@@ -91,7 +91,8 @@
       "DOWNLOAD_TEMPLATE_HERE" : "Download the template here",
       "DESCRIPTION" : "Please download and then fill in the template file appropriately with your stock information. Then select the completed file via the input box below",
       "PLEASE_SELECT_DEPOT" : "Please select a depot first",
-      "UPLOAD_SUCCESS" : "Stock imported successfully"
+      "UPLOAD_SUCCESS" : "Stock imported successfully",
+      "UPLOAD_DATE_REQUIRED" : "Define the importation date please"
     },
     "INVENTORY_UNIT"  : "Unit",
     "INPUT"           : "Input",
@@ -118,13 +119,13 @@
     "MONTHLY_CONSUMPTION" : {
       "AVERAGE_MONTHLY_CONSUMPTION"  : "Average Monthly Consumption",
       "ALGO_1"          : "Algorithm 1",
-      "ALGO_1_COMMENT"  : "The average monthly consumption is obtained by dividing the quantity consumed during the period by the number of days with stock for the period, and by multiplying the result by 30.5.",
+      "ALGO_1_COMMENT"  : "The average monthly consumption is obtained by dividing the quantity consumed during the period by the number of days with stock during the period, and by multiplying the result by 30.5.",
       "ALGO_2"          : "Algorithm 2",
       "ALGO_2_COMMENT"  : "The average consumption is obtained by dividing the quantity consumed during the period by the number of days of consumption for the period, and by multiplying the result by 30.5.",
       "ALGO_3"          : "Algorithm 3",
       "ALGO_3_COMMENT"  : "The average consumption is obtained by dividing the quantity consumed during the period by the number of days in the period, and by multiplying the result obtained by 30.5.",
       "ALGO_4"          : "Algorithm 4 (MSH)",
-      "ALGO_4_COMMENT"  : "The average consumption is obtained by dividing the quantity consumed during the period by the difference of the number of months in the period minus the total number of days of stock out in the period. The MSH algorithm is recommended by the Management Sciences for Health organization (https://www.msh.org).",
+      "ALGO_4_COMMENT"  : "The average consumption is obtained by dividing the quantity consumed during the period by number of months of stock (found by subtracting the number of days of stock outs divided by 30.5 from the number of months in the period). The MSH algorithm is recommended by the Management Sciences for Health organization (https://www.msh.org).",
       "FIRST_INVENTORY_MOVEMENT_DATE" : "First Recorded Movement",
       "LAST_INVENTORY_MOVEMENT_DATE" : "Last Recorded Movement",
       "DAYS_BEFORE_CONSUMPTION" : "Days before First Consumption",

client/src/i18n/fr/stock.json

@@ -92,7 +92,8 @@
       "DOWNLOAD_TEMPLATE_HERE" : "Télécharger le modèle ici",
       "DESCRIPTION" : "Veuillez télécharger puis remplir le fichier modèle convenablement avec les informations de votre stock puis séléctionner le fichier rempli grâce à la zone de saisie ci-dessous",
       "PLEASE_SELECT_DEPOT" : "Veuillez séléctionner un dépot d'abord",
-      "UPLOAD_SUCCESS" : "Stock importé avec succès"
+      "UPLOAD_SUCCESS" : "Stock importé avec succès",
+      "UPLOAD_DATE_REQUIRED" : "Veuillez indiquer la date pour cette importation"
     },
     "INVENTORY_UNIT"  : "Unité",
     "IO"              : "Entrée/Sortie",
@@ -130,11 +131,11 @@
       "LAST_INVENTORY_MOVEMENT_DATE" : "Dernier mouvement enregistré ",
       "DAYS_BEFORE_CONSUMPTION" : "Jours avant la première consommation ",
       "NUMBER_OF_MONTHS" : "Nombre de mois",
-      "SUM_CONSUMPTION_DAY" : "Consommation de stock ",
+      "SUM_CONSUMPTION_DAY" : "Jours de consommation de stock ",
       "SUM_CONSUMED_QUANTITY" :"Quantité consommée",
       "SUM_DAYS" : "Jours au total",
       "SUM_STOCK_DAY" : "Jours avec stock disponible en dépôt",
-      "SUM_STOCK_OUT_DAYS" : "Rupture de stock",
+      "SUM_STOCK_OUT_DAYS" : "Jours de rupture de stock",
       "NEXT_STOCK_OUT" : "Prochaine rupture de stock",
       "TITLE"  : "Consommation Mensuelle Moyenne",
       "DAYS" : "Jours",

client/src/modules/stock/import/stockImport.html

@@ -42,7 +42,11 @@ <h3 class="alert alert-success" ng-show="StockCtrl.depot.uuid">
             <span translate>STOCK.IMPORT.PLEASE_SELECT_DEPOT</span>
           </p>
 
-          <br>
+          <!-- date -->
+          <bh-date-editor
+            date-value="StockCtrl.operationDate"
+            on-change="StockCtrl.onDateChange(date)">
+          </bh-date-editor>
 
           <div>
             <span translate>STOCK.IMPORT.TEMPLATE_FILE_DESCRIPTION</span>

client/src/modules/stock/import/stockImport.js

@@ -29,21 +29,30 @@ function StockImportController(
     vm.depot = depot;
   };
 
+  vm.onDateChange = date => {
+    vm.operationDate = date;
+  };
+
   vm.submit = () => {
     // send data only when a file is selected
     if (!vm.depot.uuid || !vm.file) {
       vm.noSelectedFile = true;
       return null;
     }
 
+    if (!vm.operationDate) {
+      Notify.warn('STOCK.IMPORT.UPLOAD_DATE_REQUIRED');
+      return null;
+    }
+
     return uploadFile(vm.file);
   };
 
   /** upload the file to server */
   function uploadFile(file) {
     const parameters = {
       url : '/stock/import/',
-      data : { file, depot_uuid : vm.depot.uuid },
+      data : { file, depot_uuid : vm.depot.uuid, date : vm.operationDate },
     };
 
     // upload the file to the server

client/src/modules/stock/inventories/modals/amc.modal.html

@@ -53,10 +53,10 @@
           </dd>
 
           <dt translate>STOCK.MONTHLY_CONSUMPTION.FIRST_INVENTORY_MOVEMENT_DATE</dt>
-          <dd>{{ModalCtrl.data.first_inventory_movement_date | date }}</dd>
+          <dd>{{ModalCtrl.data.min_date | date:ModalCtrl.DATE_FORMAT }}</dd>
 
           <dt translate>STOCK.MONTHLY_CONSUMPTION.LAST_INVENTORY_MOVEMENT_DATE</dt>
-          <dd>{{ModalCtrl.data.last_inventory_movement_date | date }}</dd>
+          <dd>{{ModalCtrl.data.max_date | date:ModalCtrl.DATE_FORMAT }}</dd>
 
           <dt translate>STOCK.MONTHLY_CONSUMPTION.SUM_CONSUMED_QUANTITY</dt>
           <dd class="text-primary">{{ModalCtrl.data.sum_consumed_quantity }} {{ModalCtrl.inventory.unit}}</dd>

client/src/modules/stock/inventories/modals/amc.modal.js

@@ -12,32 +12,36 @@ function StockAMCModalController(Stock, Notify, Instance, data, moment, Constant
 
   vm.DATE_FORMAT = Constants.dates.format;
 
-  Stock.inventories.loadAMCForInventory(data.inventory_uuid, data.depot_uuid)
-    .then(items => {
-      vm.data = items;
-
-      vm.settings = vm.data.settings;
-      vm.inventory = items.inventory;
-      vm.depot = items.depot;
-
-      // FIXME(@jniles) - make this use the quantity from the getCMM() algorithm which
-      // is currently returning incorrect data.
-      vm.data.quantity_in_stock = data.quantity;
-
-      vm.data.avg_consumption = vm.data[vm.settings.average_consumption_algo];
-
-      // calculate date when the stock will run out at current consumption rate
-      const monthsOfStockLeft = vm.data.quantity_in_stock / vm.data.avg_consumption;
-      const stockOutDate = moment().add(monthsOfStockLeft, 'months').toDate();
-
-      // provide this information to the view.
-      vm.data.stock_out_date = stockOutDate;
-
-      // nicer aliases to use in the HTML
-      vm.isAlgo1 = vm.settings.average_consumption_algo === 'algo1';
-      vm.isAlgo2 = vm.settings.average_consumption_algo === 'algo2';
-      vm.isAlgo3 = vm.settings.average_consumption_algo === 'algo3';
-      vm.isAlgo4 = vm.settings.average_consumption_algo === 'algo_msh';
-    })
-    .catch(Notify.handleError);
+  function startup() {
+    vm.loading = true;
+
+    Stock.inventories.loadAMCForInventory(data.inventory_uuid, data.depot_uuid)
+      .then(items => {
+        vm.data = items;
+
+        vm.settings = vm.data.settings;
+        vm.inventory = items.inventory;
+        vm.depot = items.depot;
+
+        vm.data.avg_consumption = vm.data[vm.settings.average_consumption_algo];
+
+        // calculate date when the stock will run out at current consumption rate
+        // NOTE(@jniles): momentjs does not accept decimals as of 2.12.0.  We need to use days, not
+        // months
+        const daysOfStockLeft = (vm.data.quantity_in_stock / vm.data.avg_consumption) * 30.5;
+        vm.data.stock_out_date = moment().add(daysOfStockLeft, 'days').toDate();
+
+        // nicer aliases to use in the HTML
+        vm.isAlgo1 = vm.settings.average_consumption_algo === 'algo1';
+        vm.isAlgo2 = vm.settings.average_consumption_algo === 'algo2';
+        vm.isAlgo3 = vm.settings.average_consumption_algo === 'algo3';
+        vm.isAlgo4 = vm.settings.average_consumption_algo === 'algo_msh';
+      })
+      .catch(Notify.handleError)
+      .finally(() => {
+        vm.loading = false;
+      });
+  }
+
+  startup();
 }

client/src/modules/stock/inventories/registry.js

@@ -302,7 +302,7 @@ function StockInventoriesController(
   vm.viewAMCCalculations = viewAMCCalculations;
 
   function viewAMCCalculations(item) {
-    return Modal.openAMCCalculationModal(item)
+    Modal.openAMCCalculationModal(item)
       .catch(angular.noop);
   }
 

client/src/modules/stock/stock.service.js

@@ -244,10 +244,12 @@ function StockModalService(Modal) {
 
   function openAMCCalculationModal(request) {
     const templateUrl = 'modules/stock/inventories/modals/amc.modal.html';
-    const controller = 'StockAMCModalController as ModalCtrl';
+    const controller = 'StockAMCModalController';
+
     const params = angular.extend(modalParameters, {
       templateUrl,
       controller,
+      controllerAs : 'ModalCtrl',
       size         : 'lg',
       resolve      : { data : () => request },
     });

package.json

@@ -78,7 +78,7 @@
     "jeremielodi"
   ],
   "dependencies": {
-    "@ima-worldhealth/coral": "^2.4.0",
+    "@ima-worldhealth/coral": "^2.5.1",
     "@ima-worldhealth/tree": "^2.3.0",
     "@types/angular": "^1.8.0",
     "@uirouter/angularjs": "^1.0.29",
@@ -100,7 +100,7 @@
     "bootstrap": "^3.3.0",
     "chart.js": "^2.9.4",
     "chartjs-plugin-datalabels": "^0.7.0",
-    "connect-redis": "^5.0.0",
+    "connect-redis": "^5.1.0",
     "cron": "^1.7.1",
     "cropper": "^4.0.0",
     "cross-env": "^7.0.3",
@@ -115,7 +115,7 @@
     "font-awesome": "^4.7.0",
     "helmet": "^4.4.1",
     "inline-source": "^7.1.0",
-    "ioredis": "^4.19.4",
+    "ioredis": "^4.22.0",
     "jaro-winkler": "^0.2.8",
     "jquery": "^3.5.1",
     "jsbarcode": "^3.11.3",
@@ -129,7 +129,7 @@
     "mysql": "^2.16.0",
     "ng-file-upload": "^12.2.13",
     "ngstorage": "^0.3.11",
-    "p-retry": "^4.3.0",
+    "p-retry": "^4.4.0",
     "q": "^1.5.1",
     "stream-to-promise": "^3.0.0",
     "tempy": "^1.0.0",
@@ -144,15 +144,15 @@
   "devDependencies": {
     "@ima-worldhealth/rewire": "^4.1.0",
     "angular-mocks": "^1.8.2",
-    "chai": "^4.2.0",
+    "chai": "^4.3.0",
     "chai-as-promised": "^7.1.1",
-    "chai-datetime": "^1.7.0",
+    "chai-datetime": "^1.8.0",
     "chai-http": "^4.3.0",
     "chai-spies": "^1.0.0",
     "chai-spies-next": "^0.9.3",
     "cssnano": "^4.1.10",
     "del": "^6.0.0",
-    "eslint": "^7.19.0",
+    "eslint": "^7.20.0",
     "eslint-config-airbnb-base": "^14.2.1",
     "eslint-plugin-import": "^2.22.1",
     "gulp": "^4.0.0",
@@ -167,22 +167,22 @@
     "gulp-template": "^5.0.0",
     "gulp-typescript": "^5.0.0",
     "gulp-uglify": "^3.0.1",
-    "karma": "^6.0.4",
+    "karma": "^6.1.1",
     "karma-chai": "^0.1.0",
     "karma-chai-dom": "^1.1.0",
     "karma-chai-spies": "^0.1.4",
     "karma-chrome-launcher": "^3.0.0",
     "karma-mocha": "^2.0.1",
     "karma-ng-html2js-preprocessor": "^1.0.0",
     "merge-stream": "^2.0.0",
-    "mocha": "^8.2.1",
+    "mocha": "^8.3.0",
     "protractor": "^7.0.0",
     "protractor-console-plugin": "^0.1.1",
     "qs": "^6.9.6",
-    "release-it": "^14.2.2",
+    "release-it": "^14.4.0",
     "sinon": "^9.2.4",
     "standard-version": "^9.1.0",
-    "typescript": "^4.1.3"
+    "typescript": "^4.1.5"
   },
   "homepage": "https://docs.bhi.ma",
   "engines": {

server/controllers/inventory/depots/extra.js

@@ -86,11 +86,9 @@ async function getUsers(req, res, next) {
 async function getInventoryAverageMonthlyConsumption(req, res, next) {
   const { uuid, inventoryUuid } = req.params;
   try {
-
-    const monthAvgConsumption = req.session.stock_settings.month_average_consumption;
     const [[averageMonthlyConsumption]] = await db.exec(
-      'CALL getCMM(DATE_SUB(NOW(), INTERVAL ? MONTH), NOW(), ?, ?);',
-      [monthAvgConsumption, db.bid(inventoryUuid), db.bid(uuid)],
+      'CALL GetAMC(DATE(NOW()), ?, ?);',
+      [db.bid(uuid), db.bid(inventoryUuid)],
     );
 
     const sql = `SELECT

server/controllers/inventory/inventory/core.js

@@ -259,7 +259,7 @@ function getItemsMetadataById(uid) {
       it.text AS type, ig.name AS groupName, BUID(ig.uuid) AS group_uuid,
       ig.unique_item, i.consumable, i.locked, i.stock_min, i.sellable,
       i.stock_max, i.created_at AS timestamp, i.type_id, i.unit_id, i.unit_weight, i.unit_volume,
-      ig.sales_account, i.default_quantity, i.avg_consumption, i.delay, i.purchase_interval,
+      ig.sales_account, i.default_quantity, i.delay, i.purchase_interval,
       i.last_purchase, i.num_purchase, ig.tracking_consumption, ig.tracking_expiration,
       i.importance
     FROM inventory AS i JOIN inventory_type AS it

server/controllers/stock/core.js

@@ -235,7 +235,7 @@ async function getLotsDepot(depotUuid, params, finalClause) {
       DATEDIFF(l.expiration_date, CURRENT_DATE()) AS lifetime,
       BUID(l.inventory_uuid) AS inventory_uuid, BUID(l.origin_uuid) AS origin_uuid,
       i.code, i.text, BUID(m.depot_uuid) AS depot_uuid,
-      m.date AS entry_date, i.avg_consumption, i.purchase_interval, i.delay,
+      m.date AS entry_date, i.purchase_interval, i.delay,
       iu.text AS unit_type,
       ig.name AS group_name, ig.tracking_expiration, ig.tracking_consumption,
       dm.text AS documentReference, t.name AS tag_name, t.color
@@ -308,13 +308,13 @@ async function getBulkInventoryCMM(lots, monthAverageConsumption, averageConsump
   // create a list of unique depot/inventory_uuid combinations to avoid querying the server multiple
   // times for the same inventory item.
   const params = _.chain(lots)
-    .map(row => ([monthAverageConsumption, row.inventory_uuid, row.depot_uuid]))
+    .map(row => ([row.depot_uuid, row.inventory_uuid]))
     .uniqBy(row => row.toString())
     .value();
 
   // query the server
   const cmms = await Promise.all(
-    params.map(row => db.exec(`CALL getCMM(DATE_SUB(NOW(), INTERVAL ? MONTH), NOW(), HUID(?), HUID(?))`, row)
+    params.map(row => db.exec(`CALL GetAMC(DATE(NOW()), HUID(?), HUID(?))`, row)
       .then(values => values[0][0])),
   );
 
@@ -650,7 +650,7 @@ async function getInventoryQuantityAndConsumption(params) {
       DATEDIFF(l.expiration_date, CURRENT_DATE()) AS lifetime,
       BUID(l.inventory_uuid) AS inventory_uuid, BUID(l.origin_uuid) AS origin_uuid,
       l.entry_date, BUID(i.uuid) AS inventory_uuid, i.code, i.text, BUID(m.depot_uuid) AS depot_uuid,
-      i.avg_consumption, i.purchase_interval, i.delay, MAX(m.created_at) AS last_movement_date,
+      i.purchase_interval, i.delay, MAX(m.created_at) AS last_movement_date,
       iu.text AS unit_type, ig.tracking_consumption, ig.tracking_expiration,
       BUID(ig.uuid) AS group_uuid, ig.name AS group_name,
       dm.text AS documentReference, d.enterprise_id
@@ -829,7 +829,7 @@ function getInventoryMovements(params) {
       m.quantity, m.is_exit, m.date,
       BUID(l.inventory_uuid) AS inventory_uuid, BUID(l.origin_uuid) AS origin_uuid,
       l.entry_date, i.code, i.text, BUID(m.depot_uuid) AS depot_uuid,
-      i.avg_consumption, i.purchase_interval, i.delay, iu.text AS unit_type,
+      i.purchase_interval, i.delay, iu.text AS unit_type,
       dm.text AS documentReference, flux.label as flux
     FROM stock_movement m
       JOIN lot l ON l.uuid = m.lot_uuid

server/controllers/stock/import.js

@@ -37,6 +37,7 @@ function downloadTemplate(req, res, next) {
 async function importStock(req, res, next) {
   let queryParams;
 
+  const operationDate = new Date(req.body.date);
   const filePath = req.files[0].path;
   const depotUuid = db.bid(req.body.depot_uuid);
   const documentUuid = db.bid(util.uuid());
@@ -46,7 +47,7 @@ async function importStock(req, res, next) {
     await db.one('SELECT uuid FROM depot WHERE uuid = ?', depotUuid);
 
     // get the fiscal year period information
-    const period = await Fiscal.lookupFiscalYearByDate(new Date());
+    const period = await Fiscal.lookupFiscalYearByDate(operationDate);
 
     // read the csv file
     const data = await util.formatCsvToJson(filePath);
@@ -55,10 +56,11 @@ async function importStock(req, res, next) {
     checkDataFormat(data);
 
     const transaction = db.transaction();
-    const query = 'CALL ImportStock(?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?);';
+    const query = 'CALL ImportStock(?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?);';
 
     data.forEach(item => {
       queryParams = [
+        moment(operationDate).format('YYYY-MM-DD'),
         req.session.enterprise.id,
         req.session.project.id,
         req.session.user.id,