refactor: supercharge Bulk actions

[ruthra-kumar]

Bulk Processing feature was poorly designed, due to which its performance deteriorates as more and more actions are taken in a single day. With the redesign, we've improved the normal processing speed significantly and removed the performance degradation completely.

Details:

1. Bulk Transaction Log has been converted to a virtual doctype and has simple success and fail count, and UI button to retry for failed documents.
2. Bulk Transaction Log Detail has been converted from child to a normal doctype.
3. update_logger method, which updated the entire Bulk Transaction Log document for every document, which directly caused the poor performance, has been removed completely. It has been replaced with create_log which is just a simple method to create Bulk Transaction Log Detail

Performance

Compared to the 26.668 s from the previous refactor, this 6.74 s is a 74% reduction in runtime for processing the same volume of documents.

In [1]: sinvs=frappe.db.get_all("Sales Invoice", filters={"status":"Overdue"}, limit=100)

In [2]: from erpnext.utilities.bulk_transaction import job

In [3]: %time job(sinvs, "Sales Invoice", "Payment Entry")
CPU times: user 3.44 s, sys: 200 ms, total: 3.64 s
Wall time: 6.74 s

Demo

bulk_refactor.mp4

Internal Ref: 5034, 5626

[codecov]

Codecov Report

Merging #38038 (a393a6b) into develop (922fffd) will decrease coverage by 0.16%.
Report is 25 commits behind head on develop.
The diff coverage is 0.00%.

@@             Coverage Diff             @@
##           develop   #38038      +/-   ##
===========================================
- Coverage    67.39%   67.23%   -0.16%     
===========================================
  Files          757      757              
  Lines        60371    60441      +70     
===========================================
- Hits         40686    40637      -49     
- Misses       19685    19804     +119     

Files	Coverage Δ
erpnext/utilities/bulk_transaction.py	0.00% <0.00%> (-69.90%)	⬇️
...ctype/bulk_transaction_log/bulk_transaction_log.py	0.00% <0.00%> (-31.04%)	⬇️

... and 8 files with indirect coverage changes