bug(Stock Value report crashes server with too much data) #4556

lomamech · 2020-05-27T15:10:23Z

Improvement and optimization of the stock value by deposit report

lomamech · 2020-05-27T15:40:00Z

@jniles @mbayopanda I managed to optimize the report, please do the review

jniles

I tried to test this, but it still crashes for me. My virtual machine only has 4GB of RAM, but that should be more than enough. After 2 minutes, the request times out and retries. It also pegs my MySQL CPU usage at over 300%.

I'm not sure this solves the problem at all. I still think, as mentioned in our previous discussion, that something like period_total is the way to go. That way, we are doing very little computation in the actual report itself.

jniles · 2020-05-28T08:11:44Z

migration-hev_mai_2020.sql

+SET character_set_database = 'utf8mb4';
+SET collation_database = 'utf8mb4_unicode_ci';
+SET CHARACTER SET utf8mb4, CHARACTER_SET_CONNECTION = utf8mb4;
+SET collation_connection = 'utf8mb4_unicode_ci';


Is this really necessary? The SQL dump should be exporting with these defaults. Is that not the case?

Is this really necessary? The SQL dump should be exporting with these defaults. Is that not the case?

This portion of code was added when I rebase with master, I haven't added it personally

jniles · 2020-05-28T08:15:45Z

server/controllers/stock/reports/stock/value.js

+   * but also the cumulative value of entries *quantityCum* and exits *cumOut*
+   * and the quantity in stock after each movement of stock *quantityStock*
+  */
+  const stockValues = await db.exec(sqlGetInventories, [db.bid(options.depot_uuid), options.dateTo]);


Sp, if I am understanding this correctly, this code will generate 1 + N SQL queries in a transaction, where N is the number of inventory items in a depot. At Vanga, the total number of inventory items is currently 784. At IMCK, the total number is 1302. So we could potentially get hundreds of SQL queries launched from a single query here.

... I'm not sure how optimal this is in low-end hardware like a Raspberry Pi.

the way I understood the logic of the stock value report came down to retrieving the last line of the stock card report, my idea was to launch several requests on the stock movement table in order to calculate for each unit cost weight and the stock value after each transaction

I had given up on the idea of using an aggregation table because of the risk of having backdated stock movements

jniles · 2020-05-28T08:17:07Z

server/controllers/stock/reports/stock/value.js

+    });
+  });
+
+  let stockTolal = 0;


This looks like a typo - it should be "stockTotal"

jniles · 2020-05-28T08:25:08Z

server/controllers/stock/reports/stock_value.report.handlebars

+              <tr>
+                <td style="width:50%">{{inventory_name}}</td>
+                <td class="text-right">{{stockQtt}}</td>
+                <td class="text-right">{{currency (multiply stockUnitCost ../rate) ../currency_id}}</td>


Can you do this multiplication before putting it in the template?

jniles · 2020-05-28T08:25:22Z

server/controllers/stock/reports/stock_value.report.handlebars

+              </tr>
+            {{/each}}
+            <!-- no data -->
+            {{#if emptyResult}}


There is no need for this "emptyResult" check. Just use {{#each}}{{else}}{{/each}} as discussed in WhatsApp.

So, it would look like this:

{{#each stockValues}} {{! ... etc ... }} {{else}} <tr><th class="text-center" colspan="4">{{translate 'STOCK.NO_DATA'}}</th></tr> {{/each}}

jniles · 2020-05-28T08:40:50Z

server/controllers/stock/reports/stock/value.js

  const options = (typeof (_options.params) === 'string') ? JSON.parse(_options.params) : _options.params;
  data.dateTo = options.dateTo;
  data.depot = await db.one('SELECT * FROM depot WHERE uuid=?', [db.bid(options.depot_uuid)]);
-  const stockValues = await db.exec('CALL stockValue(?,?,?);', [


If we are not longer using the stockValue() procedure, let's remove it from the stored procedures.

jniles · 2020-05-28T08:50:41Z

server/controllers/stock/reports/stock/value.js


  const stockValueElements = options.exclude_zero_value
-    ? stockValues[0].filter(item => item.stockValue > 0) : stockValues[0];
+    ? stockValues.filter(item => item.stockValue > 0) : stockValues;


Is there anyway we can apply this filter higher in logic? Preferably in the SQL. It would probably improve performance.

it will be difficult seeing that the products we see the stock shortage of a product after on the last line, after having noted all the entries and exits from Stock

lomamech · 2020-05-28T09:42:36Z

I tried to test this, but it still crashes for me. My virtual machine only has 4GB of RAM, but that should be more than enough. After 2 minutes, the request times out and retries. It also pegs my MySQL CPU usage at over 300%.

I'm not sure this solves the problem at all. I still think, as mentioned in our previous discussion, that something like period_total is the way to go. That way, we are doing very little computation in the actual report itself.

With me it works really well

jniles · 2020-05-28T10:00:05Z

That's because you have a machine with 16GB of RAM that cost many, many dollars. We can't expect poor hospitals to buy these kinds of computers.

lomamech · 2020-05-28T12:24:10Z

That's because you have a machine with 16GB of RAM that cost many, many dollars. We can't expect poor hospitals to buy these kinds of computers.

I think the problem stems from the way in which stock managers calculate the stock value of a product, I fear that even if we only had computers with large capacities, even after 10 or 15 years. use, that the stock value report then crashes out again

mbayopanda

it seems like stock_sheet report is not available

- Improvement and optimization of the stock value by deposit report closes Third-Culture-Software#4116

- Improvement and optimization of the stock value report, reduction of operations on the MYSQL side

lomamech · 2020-06-02T14:01:55Z

@jniles i need a review

jniles

This is much better for performance. The query takes 10 seconds on my machine (22 seconds total time). It also doesn't peg MySQL at 300% CPU usage. See below.

I still think we can do better via period_total style table, but this fixes the bug in production to an acceptable level where the report is usable. Great work!

bors r+

bors · 2020-06-03T08:13:37Z

Build succeeded:

continuous-integration/appveyor/branch
continuous-integration/travis-ci/push

lomamech requested a review from jniles May 27, 2020 15:10

kwilu added size: XL Bug performance labels May 27, 2020

lomamech requested a review from mbayopanda May 27, 2020 15:10

jniles mentioned this pull request May 28, 2020

Do not sent empty search params to /depots/search/name #4557

Closed

jniles reviewed May 28, 2020

View reviewed changes

mbayopanda reviewed Jun 1, 2020

View reviewed changes

lomamech closed this Jun 1, 2020

lomamech added 5 commits June 2, 2020 06:29

Saving changes

1bf9347

bug(Stock Value report crashes server with too much data)

0f656cb

- Improvement and optimization of the stock value by deposit report closes Third-Culture-Software#4116

Remove not necessary migration-hev_mai_2020.sql in Master

a4c475b

Save Change

a118113

improve(Stock Value Report)

299cb15

- Improvement and optimization of the stock value report, reduction of operations on the MYSQL side

lomamech reopened this Jun 2, 2020

lomamech force-pushed the fixRemakeStockValueReport branch from b1d7df6 to 299cb15 Compare June 2, 2020 13:02

Sanitaze code

e87b689

jniles approved these changes Jun 3, 2020

View reviewed changes

bors bot merged commit 4d04833 into Third-Culture-Software:master Jun 3, 2020

lomamech deleted the fixRemakeStockValueReport branch October 2, 2020 07:59

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

bug(Stock Value report crashes server with too much data) #4556

bug(Stock Value report crashes server with too much data) #4556

lomamech commented May 27, 2020

lomamech commented May 27, 2020

jniles left a comment

jniles May 28, 2020

lomamech May 28, 2020

jniles May 28, 2020

lomamech May 28, 2020

lomamech May 28, 2020

jniles May 28, 2020

jniles May 28, 2020

jniles May 28, 2020

jniles May 28, 2020

jniles May 28, 2020

lomamech May 28, 2020

lomamech commented May 28, 2020

jniles commented May 28, 2020 •

edited

Loading

lomamech commented May 28, 2020

mbayopanda left a comment

lomamech commented Jun 2, 2020

jniles left a comment

bors bot commented Jun 3, 2020

bug(Stock Value report crashes server with too much data) #4556

bug(Stock Value report crashes server with too much data) #4556

Conversation

lomamech commented May 27, 2020

lomamech commented May 27, 2020

jniles left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

lomamech commented May 28, 2020

jniles commented May 28, 2020 • edited Loading

lomamech commented May 28, 2020

mbayopanda left a comment

Choose a reason for hiding this comment

lomamech commented Jun 2, 2020

jniles left a comment

Choose a reason for hiding this comment

bors bot commented Jun 3, 2020

jniles commented May 28, 2020 •

edited

Loading