Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

bug(Stock Value report crashes server with too much data) #4556

Merged
merged 6 commits into from
Jun 3, 2020
Merged

bug(Stock Value report crashes server with too much data) #4556

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
6 commits
Select commit Hold shift + click to select a range
1bf9347
Saving changes
lomamech May 22, 2020
0f656cb
bug(Stock Value report crashes server with too much data)
lomamech May 27, 2020
a4c475b
Remove not necessary migration-hev_mai_2020.sql in Master
lomamech May 28, 2020
a118113
Save Change
lomamech May 28, 2020
299cb15
improve(Stock Value Report)
lomamech Jun 2, 2020
e87b689
Sanitaze code
lomamech Jun 2, 2020
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
81 changes: 73 additions & 8 deletions server/controllers/stock/reports/stock/value.js
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -20,7 +20,6 @@ function stockValue(req, res, next) {
}

async function reporting(_options, session) {

const data = {};
const enterpriseId = session.enterprise.id;

Expand All @@ -30,21 +29,87 @@ async function reporting(_options, session) {

const report = new ReportManager(STOCK_VALUE_REPORT_TEMPLATE, session, optionReport);


const options = (typeof (_options.params) === 'string') ? JSON.parse(_options.params) : _options.params;
data.dateTo = options.dateTo;
data.depot = await db.one('SELECT * FROM depot WHERE uuid=?', [db.bid(options.depot_uuid)]);
const stockValues = await db.exec('CALL stockValue(?,?,?);', [
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If we are not longer using the stockValue() procedure, let's remove it from the stored procedures.

db.bid(options.depot_uuid), options.dateTo, options.currency_id,
]);

// Get inventories movemented
const sqlGetInventories = `
SELECT DISTINCT(BUID(mov.inventory_uuid)) AS inventory_uuid, mov.text AS inventory_name
FROM(
SELECT inv.uuid AS inventory_uuid, inv.text, sm.date
FROM stock_movement AS sm
JOIN lot AS l ON l.uuid = sm.lot_uuid
JOIN inventory AS inv ON inv.uuid = l.inventory_uuid
WHERE sm.depot_uuid = ? AND DATE(sm.date) <= DATE(?)
) AS mov
ORDER BY mov.text ASC;
`;

/*
* Here we first search for all the products that have
* been stored in stock in a warehouse,
* then we collect all the movements of stocks linked to a warehouse,
* then we calculate the unit cost weighted average for each product
*/
const stockValues = await db.exec(sqlGetInventories, [db.bid(options.depot_uuid), options.dateTo]);
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sp, if I am understanding this correctly, this code will generate 1 + N SQL queries in a transaction, where N is the number of inventory items in a depot. At Vanga, the total number of inventory items is currently 784. At IMCK, the total number is 1302. So we could potentially get hundreds of SQL queries launched from a single query here.

... I'm not sure how optimal this is in low-end hardware like a Raspberry Pi.

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

the way I understood the logic of the stock value report came down to retrieving the last line of the stock card report, my idea was to launch several requests on the stock movement table in order to calculate for each unit cost weight and the stock value after each transaction

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I had given up on the idea of using an aggregation table because of the risk of having backdated stock movements


const sqlGetMovementByDepot = `
SELECT sm.document_uuid, sm.depot_uuid, sm.lot_uuid, sm.quantity, sm.unit_cost, sm.date, sm.is_exit,
sm.created_at, BUID(inv.uuid) AS inventory_uuid, inv.text AS inventory_text, map.text AS docRef
FROM stock_movement AS sm
JOIN lot AS l ON l.uuid = sm.lot_uuid
JOIN inventory AS inv ON inv.uuid = l.inventory_uuid
JOIN document_map AS map ON map.uuid = sm.document_uuid
WHERE sm.depot_uuid = ? AND DATE(sm.date) <= DATE(?)
ORDER BY inv.text, DATE(sm.date), sm.created_at ASC
`;

const allMovements = await db.exec(sqlGetMovementByDepot, [db.bid(options.depot_uuid), options.dateTo]);

stockValues.forEach(stock => {
stock.movements = allMovements.filter(movement => (movement.inventory_uuid === stock.inventory_uuid));
});

let stockTotal = 0;
const exchangeRate = await Exchange.getExchangeRate(enterpriseId, options.currency_id, new Date());
const rate = exchangeRate.rate || 1;

stockValues.forEach(stock => {
let quantityInStock = 0;
let weightedAverageUnitCost = 0;

stock.movements.forEach(item => {
const isExit = item.is_exit ? (-1) : 1;

if (!item.is_exit && (quantityInStock > 0)) {
weightedAverageUnitCost = ((quantityInStock * weightedAverageUnitCost) + (item.quantity * item.unit_cost))
/ (item.quantity + quantityInStock);
} else if (!item.is_exit && (quantityInStock === 0)) {
weightedAverageUnitCost = item.unit_cost;
}

quantityInStock += (item.quantity * isExit);
item.quantityInStock = quantityInStock;
item.weightedAverageUnitCost = weightedAverageUnitCost;
});

stock.stockQtt = quantityInStock;
stock.stockUnitCost = weightedAverageUnitCost * rate;
stock.stockValue = (quantityInStock * weightedAverageUnitCost) * rate;
stockTotal += (stock.stockValue * rate);
});

const stockValueElements = options.exclude_zero_value
? stockValues[0].filter(item => item.stockValue > 0) : stockValues[0];
? stockValues.filter(item => item.stockValue > 0) : stockValues;
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is there anyway we can apply this filter higher in logic? Preferably in the SQL. It would probably improve performance.

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

it will be difficult seeing that the products we see the stock shortage of a product after on the last line, after having noted all the entries and exits from Stock



data.stockValues = stockValueElements || [];
const stokTolal = stockValues[1][0] || {};
data.stocktotal = stokTolal.total;

data.stocktotal = stockTotal;
data.emptyResult = data.stockValues.length === 0;
data.rate = Exchange.getExchangeRate(enterpriseId, options.currency_id, new Date());

data.currency_id = options.currency_id;
return report.render(data);
}
Expand Down
2 changes: 1 addition & 1 deletion server/controllers/stock/reports/stock_sheet.report.handlebars
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -55,7 +55,7 @@
{{#each rows}}
<tr>
<td style="border-left: 1px solid #000;">{{reference}}</td>
<td style="border-right: 1px solid #000;">{{date date}}</td>
<td style="border-right: 1px solid #000;">{{timestamp date}}</td>
{{!-- entry --}}
<td class="text-right">{{#if entry.quantity}}{{entry.quantity}}{{/if}}</td> <td class="text-right">{{#if entry.unit_cost}}{{precision entry.unit_cost 4}}{{/if}}</td>
<td class="text-right" style="border-right: 1px solid #000;">{{#if entry.value}}{{currency entry.value ../metadata.enterprise.currency_id}}{{/if}}</td>
Expand Down
98 changes: 45 additions & 53 deletions server/controllers/stock/reports/stock_value.report.handlebars
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -4,61 +4,53 @@

<div class="container">
{{> header}}

<!-- body -->
<div class="row">
<div class="col-xs-12">

<!-- page title -->
<h2 class="text-center text-uppercase">
{{translate 'REPORT.STOCK_VALUE.TITLE'}}
</h2>

<h3 class="text-center">
{{#if depot.text}}{{depot.text}}{{/if}}
</h3>

<h4 class="text-center">
{{date dateTo}}
</h4>

<br>

<!-- list of data -->
<table class="table table-condensed table-report">
<thead>
<tr style="background-color:#ddd;">
<th class="text-center">{{translate 'FORM.LABELS.INVENTORY'}}</th>
<th class="text-center">{{translate 'FORM.LABELS.QUANTITY'}}</th>
<th class="text-center">{{translate 'STOCK.UNIT_COST'}}</th>
<th class="text-center">{{translate 'FORM.LABELS.VALUE'}}</th>
</tr>
</thead>
<tbody>
{{#each stockValues}}
<tr>
<td style="width:50%">{{inventory_name}}</td>
<td class="text-right">{{stockQtt}}</td>
<td class="text-right">{{stockUnitCost}}</td>
<td class="text-right">{{currency stockValue ../currency_id}}</td>
<!-- body -->
<div class="row">
<div class="col-xs-12">

<!-- page title -->
<h2 class="text-center text-uppercase">
{{translate 'REPORT.STOCK_VALUE.TITLE'}}
</h2>

<h3 class="text-center">
{{#if depot.text}}{{depot.text}}{{/if}}
</h3>

<h4 class="text-center">
{{date dateTo}}
</h4>

<br>

<!-- list of data -->
<table class="table table-condensed table-report">
<thead>
<tr style="background-color:#ddd;">
<th class="text-center">{{translate 'FORM.LABELS.INVENTORY'}}</th>
<th class="text-center">{{translate 'FORM.LABELS.QUANTITY'}}</th>
<th class="text-center">{{translate 'STOCK.UNIT_COST'}}</th>
<th class="text-center">{{translate 'FORM.LABELS.VALUE'}}</th>
</tr>
{{/each}}
<!-- no data -->
{{#if emptyResult}}
<tr><th class="text-center" colspan="4">{{translate 'STOCK.NO_DATA'}}</th></tr>
</thead>
<tbody>
{{#each stockValues}}
<tr>
<td style="width:50%">{{inventory_name}}</td>
<td class="text-right">{{stockQtt}}</td>
<td class="text-right">{{currency stockUnitCost ../currency_id}}</td>
<td class="text-right">{{currency stockValue ../currency_id}}</td>
</tr>
{{else}}
<tr>
<th colspan="3" class="text-right">{{ translate "FORM.LABELS.TOTAL"}}</th>
<th class="text-right">{{ currency stocktotal ./currency_id }}</th>
</tr>
{{/if}}


</tbody>

</table>
{{> emptyTable columns=4}}
{{/each}}
<tr>
<th colspan="3" class="text-right">{{ translate "FORM.LABELS.TOTAL"}}</th>
<th class="text-right">{{ currency stocktotal currency_id }}</th>
</tr>
</tbody>
</table>
</div>
</div>
</div>

</div>
</body>
139 changes: 0 additions & 139 deletions server/models/procedures/stock.sql
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -274,143 +274,4 @@ BEGIN
VALUES (HUID(UUID()), documentUuid, depotUuid, lotUuid, fluxId, CURRENT_DATE(), stockLotQuantity, inventoryUnitCost, 0, userId, periodId);
END $$

DROP PROCEDURE IF EXISTS `stockValue`$$
CREATE PROCEDURE `stockValue`(
IN depotUuid BINARY(16),
IN dateTo DATE,
IN currencyId INT
)
BEGIN
DECLARE done BOOLEAN;
DECLARE mvtIsExit tinyint(1);
DECLARE mvtQtt, stockQtt, newQuantity INT(11);
DECLARE mvtUnitCost, mvtValue, newValue, newCost, exchangeRate, stockUnitCost, stockValue DECIMAL(19, 4);

DECLARE _documentReference VARCHAR(100);
DECLARE _date DATETIME;
DECLARE _inventoryUuid BINARY(16);
DECLARE _iteration, _newStock, _enterpriseId INT;


DECLARE curs1 CURSOR FOR
SELECT i.uuid, m.is_exit, l.unit_cost, m.quantity, m.date, dm.text AS documentReference
FROM stock_movement m
JOIN lot l ON l.uuid = m.lot_uuid
JOIN inventory i ON i.uuid = l.inventory_uuid
JOIN inventory_unit iu ON iu.id = i.unit_id
JOIN depot d ON d.uuid = m.depot_uuid
LEFT JOIN document_map dm ON dm.uuid = m.document_uuid
WHERE m.depot_uuid = depotUuid AND DATE(m.date) <= dateTo
ORDER BY i.text, m.created_at ASC;

DECLARE CONTINUE HANDLER FOR NOT FOUND SET done = TRUE;

DROP TEMPORARY TABLE IF EXISTS stage_movement;
CREATE TEMPORARY TABLE stage_movement(
inventory_uuid BINARY(16),
isExit TINYINT(1),
qtt INT(11),
unit_cost DECIMAL(19, 4),
VALUE DECIMAL(19, 4),
DATE DATETIME,
reference VARCHAR(100),
stockQtt INT(11),
stockUnitCost DECIMAL(19, 4),
stockValue DECIMAL(19, 4),
iteration INT
);

SET _enterpriseId = (SELECT enterprise_id FROM depot WHERE uuid= depotUuid);
SET exchangeRate = IFNULL(GetExchangeRate(_enterpriseId,currencyId ,dateTo), 1);

OPEN curs1;
read_loop: LOOP

FETCH curs1 INTO _inventoryUuid, mvtIsExit, mvtUnitCost, mvtQtt, _date, _documentReference;
IF done THEN
LEAVE read_loop;
END IF;

SELECT COUNT(inventory_uuid) INTO _newStock FROM stage_movement WHERE inventory_uuid = _inventoryUuid;

-- initialize stock qtt, value and unit cost for a new inventory
IF _newStock = 0 THEN
SET _iteration = 0;

SET stockQtt= 0;
SET stockUnitCost = 0;
SET stockValue = 0;

SET mvtValue = 0;
SET newQuantity = 0;
SET newValue = 0;
SET newCost = 0;
END IF;

SET mvtUnitCost = mvtUnitCost * (exchangeRate);

-- stock exit movement, the stock quantity decreases
IF mvtIsExit = 1 THEN
SET stockQtt = stockQtt - mvtQtt;
SET stockValue = stockQtt * stockUnitCost;
-- ignore negative stock value
IF stockValue < 0 THEN
SET stockValue = 0;
END IF;
ELSE
-- stock entry movement, the stock quantity increases
SET newQuantity = mvtQtt + stockQtt;

-- ignore negative stock value
IF stockValue < 0 THEN
SET newValue = mvtUnitCost * mvtQtt;
ELSE
SET newValue = (mvtUnitCost * mvtQtt) + stockValue;
END IF;

-- don't use cumulated quantity when stock quantity < 0
-- in this case use movement quantity only
IF stockQtt < 0 THEN
SET newCost = newValue / IF(mvtQtt = 0, 1, mvtQtt);
ELSE
SET newCost = newValue / IF(newQuantity = 0, 1, newQuantity);
END IF;

SET stockQtt = newQuantity;
SET stockUnitCost = newCost;
SET stockValue = newValue;
END IF;

INSERT INTO stage_movement VALUES (
_inventoryUuid, mvtIsExit, mvtQtt, stockQtt, mvtQtt * mvtUnitCost, _date, _documentReference, stockQtt, stockUnitCost, stockValue, _iteration
);
SET _iteration = _iteration + 1;
END LOOP;
CLOSE curs1;

DROP TEMPORARY TABLE IF EXISTS stage_movement_copy;
CREATE TEMPORARY TABLE stage_movement_copy AS SELECT * FROM stage_movement;

-- inventory stock
SELECT BUID(sm.inventory_uuid) AS inventory_uuid, i.text as inventory_name, sm.stockQtt, sm.stockUnitCost, sm.stockValue
FROM stage_movement sm
JOIN inventory i ON i.uuid = sm.inventory_uuid
INNER JOIN (
SELECT inventory_uuid, MAX(iteration) as max_iteration
FROM stage_movement_copy
GROUP BY inventory_uuid
)x ON x.inventory_uuid = sm.inventory_uuid AND x.max_iteration = sm.iteration
ORDER BY i.text ASC;

-- total in stock
SELECT SUM(sm.stockValue) as total
FROM stage_movement as sm
INNER JOIN (
SELECT inventory_uuid, MAX(iteration) as max_iteration
FROM stage_movement_copy
GROUP BY inventory_uuid
)x ON x.inventory_uuid = sm.inventory_uuid AND x.max_iteration = sm.iteration;

END $$

DELIMITER ;