if config ResendRequestChunkSize arg. is didn't work，probably oom #778

wajncn · 2024-03-18T11:03:29Z

Describe the bug
If the amount of data is particularly large 。

ResetOnLogon=N
ResetOnLogout=N
ResetOnDisconnect=N
ResetOnError=N

in quickfixj-initiator delete local quickfixj file。 restart quickfixj-acceptor OOM （If there is no excessive memory）

need config

ResendRequestChunkSize=1000

Can solve this problem

system information:

OS: Win10
Java version JDK8
QFJ Version FIX.5.0SP2

#777

The text was updated successfully, but these errors were encountered:

wajncn · 2024-03-18T11:41:00Z

ResetOnLogon=N
ResetOnLogout=N
ResetOnDisconnect=N
ResetOnError=N
DisconnectOnError=N
ContinueInitializationOnError=N
ResendRequestChunkSize=3

INFO quickfix.Session - load beginSeqNo1,endSeqNo:3  beginString：FIXT.1.1
INFO quickfix.Session - load beginSeqNo4,endSeqNo:6  beginString：FIXT.1.1
INFO quickfix.Session - load beginSeqNo7,endSeqNo:9  beginString：FIXT.1.1
INFO quickfix.Session - load beginSeqNo10,endSeqNo:12  beginString：FIXT.1.1
INFO quickfix.Session - load beginSeqNo12,endSeqNo:14  beginString：FIXT.1.1
INFO quickfix.Session - load beginSeqNo15,endSeqNo:17  beginString：FIXT.1.1
INFO quickfix.Session - load beginSeqNo18,endSeqNo:20  beginString：FIXT.1.1
INFO quickfix.Session - load beginSeqNo21,endSeqNo:23  beginString：FIXT.1.1
INFO quickfix.Session - load beginSeqNo24,endSeqNo:26  beginString：FIXT.1.1
INFO quickfix.Session - load beginSeqNo27,endSeqNo:29  beginString：FIXT.1.1
INFO quickfix.Session - load beginSeqNo30,endSeqNo:32  beginString：FIXT.1.1
INFO quickfix.Session - load beginSeqNo33,endSeqNo:35  beginString：FIXT.1.1
INFO quickfix.Session - load beginSeqNo36,endSeqNo:38  beginString：FIXT.1.1
INFO quickfix.Session - load beginSeqNo39,endSeqNo:41  beginString：FIXT.1.1
INFO quickfix.Session - load beginSeqNo42,endSeqNo:44  beginString：FIXT.1.1
INFO quickfix.Session - load beginSeqNo45,endSeqNo:47  beginString：FIXT.1.1
INFO quickfix.Session - load beginSeqNo48,endSeqNo:50  beginString：FIXT.1.1
INFO quickfix.Session - load beginSeqNo51,endSeqNo:53  beginString：FIXT.1.1
INFO quickfix.Session - load beginSeqNo54,endSeqNo:56  beginString：FIXT.1.1
INFO quickfix.Session - load beginSeqNo57,endSeqNo:59  beginString：FIXT.1.1
INFO quickfix.Session - load beginSeqNo60,endSeqNo:62  beginString：FIXT.1.1
INFO quickfix.Session - load beginSeqNo63,endSeqNo:65  beginString：FIXT.1.1
INFO quickfix.Session - load beginSeqNo66,endSeqNo:68  beginString：FIXT.1.1
INFO quickfix.Session - load beginSeqNo69,endSeqNo:71  beginString：FIXT.1.1
INFO quickfix.Session - load beginSeqNo72,endSeqNo:74  beginString：FIXT.1.1
INFO quickfix.Session - load beginSeqNo75,endSeqNo:77  beginString：FIXT.1.1
INFO quickfix.Session - load beginSeqNo78,endSeqNo:80  beginString：FIXT.1.1
INFO quickfix.Session - load beginSeqNo81,endSeqNo:83  beginString：FIXT.1.1
INFO quickfix.Session - load beginSeqNo84,endSeqNo:86  beginString：FIXT.1.1
INFO quickfix.Session - load beginSeqNo87,endSeqNo:89  beginString：FIXT.1.1
INFO quickfix.Session - load beginSeqNo90,endSeqNo:92  beginString：FIXT.1.1
INFO quickfix.Session - load beginSeqNo93,endSeqNo:95  beginString：FIXT.1.1
INFO quickfix.Session - load beginSeqNo95,endSeqNo:97  beginString：FIXT.1.1
INFO quickfix.Session - load beginSeqNo98,endSeqNo:100  beginString：FIXT.1.1
INFO quickfix.Session - load beginSeqNo101,endSeqNo:103  beginString：FIXT.1.1
INFO quickfix.Session - load beginSeqNo104,endSeqNo:106  beginString：FIXT.1.1
INFO quickfix.Session - load beginSeqNo107,endSeqNo:109  beginString：FIXT.1.1
INFO quickfix.Session - load beginSeqNo110,endSeqNo:112  beginString：FIXT.1.1
INFO quickfix.Session - load beginSeqNo113,endSeqNo:115  beginString：FIXT.1.1
INFO quickfix.Session - load beginSeqNo116,endSeqNo:118  beginString：FIXT.1.1
INFO quickfix.Session - load beginSeqNo119,endSeqNo:121  beginString：FIXT.1.1
INFO quickfix.Session - load beginSeqNo122,endSeqNo:124  beginString：FIXT.1.1
INFO quickfix.Session - load beginSeqNo125,endSeqNo:127  beginString：FIXT.1.1
INFO quickfix.Session - load beginSeqNo128,endSeqNo:130  beginString：FIXT.1.1
INFO quickfix.Session - load beginSeqNo131,endSeqNo:133  beginString：FIXT.1.1
INFO quickfix.Session - load beginSeqNo134,endSeqNo:136  beginString：FIXT.1.1
INFO quickfix.Session - load beginSeqNo137,endSeqNo:139  beginString：FIXT.1.1
INFO quickfix.Session - load beginSeqNo140,endSeqNo:142  beginString：FIXT.1.1
INFO quickfix.Session - load beginSeqNo142,endSeqNo:144  beginString：FIXT.1.1
INFO quickfix.Session - load beginSeqNo145,endSeqNo:147  beginString：FIXT.1.1
INFO quickfix.Session - load beginSeqNo148,endSeqNo:150  beginString：FIXT.1.1
INFO quickfix.Session - load beginSeqNo151,endSeqNo:153  beginString：FIXT.1.1
INFO quickfix.Session - load beginSeqNo154,endSeqNo:156  beginString：FIXT.1.1
INFO quickfix.Session - load beginSeqNo157,endSeqNo:159  beginString：FIXT.1.1
INFO quickfix.Session - load beginSeqNo160,endSeqNo:162  beginString：FIXT.1.1
INFO quickfix.Session - load beginSeqNo163,endSeqNo:165  beginString：FIXT.1.1
INFO quickfix.Session - load beginSeqNo166,endSeqNo:168  beginString：FIXT.1.1
INFO quickfix.Session - load beginSeqNo169,endSeqNo:171  beginString：FIXT.1.1
INFO quickfix.Session - load beginSeqNo172,endSeqNo:174  beginString：FIXT.1.1

chrjohn · 2024-03-18T15:20:22Z

Hi @wajncn thanks but I don't fully understand the issue and the related PR.
You can set chunk size in the config, but why do we need to change the code for this?

wajncn · 2024-03-18T16:11:07Z

Hi @wajncn thanks but I don't fully understand the issue and the related PR. You can set chunk size in the config, but why do we need to change the code for this?

If you don't change the code， endSeqNo is Max SeqNo。 FIleStore all the data is loaded。
Although the parameters are configured 【ResendRequestChunkSize=3】。

So, change the code。

wajncn · 2024-03-18T16:17:56Z

Hi @wajncn thanks but I don't fully understand the issue and the related PR. You can set chunk size in the config, but why do we need to change the code for this?

Suppose the store file is 3GB，100000rows。 java max heap is 2G。
acceptor and initiator config【ResendRequestChunkSize=1000】。
acceptor online。 initiator offline。 then start initiator after。 acceptor need load 3GBdata to memory，bug memory limit 2G。 then OOM exception～。

if merge this issues，there wouldn't be this problem。

chrjohn · 2024-03-18T17:46:28Z

OK, starting to understand. Are you able to supply a unit test?

wajncn · 2024-03-19T03:36:25Z

OK, starting to understand. Are you able to supply a unit test?

I wrote it locally acceptor and initiator demo。 It meets my expectations

wajncn · 2024-03-19T05:34:34Z

OK, starting to understand. Are you able to supply a unit test?
I submit a unit test as you requested 。 you can debug 【Session#2293 line https://github.com/wajncn/quickfixj/blob/e0c1d9ef730b524fe11953b459bebb629823680a/quickfixj-core/src/main/java/quickfix/Session.java#L2293】 and 【quickfix.Session#resendMessages】【https://github.com/wajncn/quickfixj/blob/e0c1d9ef730b524fe11953b459bebb629823680a/quickfixj-core/src/main/java/quickfix/Session.java#L2335】

wajncn · 2024-03-20T13:29:07Z

OK, starting to understand. Are you able to supply a unit test?

hi ，Please help me look at this pr。

chrjohn · 2024-03-20T18:52:48Z

Thanks again for the PR. I'm looking at the issues as time permits, please be patient. :)

wajncn · 2024-03-22T14:33:19Z

Thanks again for the PR. I'm looking at the issues as time permits, please be patient. :)

how test passed ？

wajncn · 2024-04-02T13:16:57Z

??

chrjohn · 2024-04-02T14:46:17Z

#778 (comment)

:)

wajncn · 2024-05-23T02:52:39Z

oh....

wajncn · 2024-05-23T02:52:49Z

merge ？

chrjohn · 2024-05-23T03:20:44Z

Did you see my comment?
#777 (comment)

wajncn added the bug label Mar 18, 2024

wajncn changed the title ~~if config ResendRequestChunkSize arg. is didn't work~~ if config ResendRequestChunkSize arg. is didn't work，probably oom Mar 18, 2024

wajncn mentioned this issue Mar 20, 2024

During a resend-request the request for messages from storage allows for unbounded memory usage #639

Open

chrjohn linked a pull request Mar 25, 2024 that will close this issue

fix oom. if config resendRequestChunkSize. Loading data is batch #777

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

if config ResendRequestChunkSize arg. is didn't work，probably oom #778

if config ResendRequestChunkSize arg. is didn't work，probably oom #778

wajncn commented Mar 18, 2024

wajncn commented Mar 18, 2024 •

edited

Loading

chrjohn commented Mar 18, 2024

wajncn commented Mar 18, 2024

wajncn commented Mar 18, 2024

chrjohn commented Mar 18, 2024

wajncn commented Mar 19, 2024 •

edited

Loading

wajncn commented Mar 19, 2024 •

edited

Loading

wajncn commented Mar 20, 2024

chrjohn commented Mar 20, 2024

wajncn commented Mar 22, 2024 •

edited

Loading

wajncn commented Apr 2, 2024

chrjohn commented Apr 2, 2024

wajncn commented May 23, 2024

wajncn commented May 23, 2024

chrjohn commented May 23, 2024

if config ResendRequestChunkSize arg. is didn't work，probably oom #778

if config ResendRequestChunkSize arg. is didn't work，probably oom #778

Comments

wajncn commented Mar 18, 2024

wajncn commented Mar 18, 2024 • edited Loading

chrjohn commented Mar 18, 2024

wajncn commented Mar 18, 2024

wajncn commented Mar 18, 2024

chrjohn commented Mar 18, 2024

wajncn commented Mar 19, 2024 • edited Loading

wajncn commented Mar 19, 2024 • edited Loading

wajncn commented Mar 20, 2024

chrjohn commented Mar 20, 2024

wajncn commented Mar 22, 2024 • edited Loading

wajncn commented Apr 2, 2024

chrjohn commented Apr 2, 2024

wajncn commented May 23, 2024

wajncn commented May 23, 2024

chrjohn commented May 23, 2024

wajncn commented Mar 18, 2024 •

edited

Loading

wajncn commented Mar 19, 2024 •

edited

Loading

wajncn commented Mar 19, 2024 •

edited

Loading

wajncn commented Mar 22, 2024 •

edited

Loading