Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
chore: Batch archiver requests (#10442)
Instead of downloading every block and keeping it in memory, download in batches of (configurable) 100 blocks and store each batch as it is retrieved. Also downloads blocks in parallel within the batch once their logs have been retrieved. Following is an `info` level logging for the archiver, with a batch size of 5 (for testing) and an artifical delay of 1s for fetching logs. Notice the timestamps: ``` [11:10:38.855] INFO: archiver Starting archiver sync to rollup contract 0x2b09a171300b3820c14f2faea559acc9270787de from L1 block 20 to current L1 block 265 [11:10:41.907] INFO: archiver Downloaded L2 block 2 blockHash: "0x0d088ff05560afa67b8a1a9d7099310ba3268426dd4a488bcfdff6fdef59607f" blockNumber: 2 [11:10:41.907] INFO: archiver Downloaded L2 block 1 blockHash: "0x150e8994ed59731daaed645e9e3d1687d7eb3648d08eca027f3dacf2deafdd71" blockNumber: 1 [11:10:43.957] INFO: archiver Downloaded L2 block 3 blockHash: "0x1b0fe9562164a3e081dede1a47765be7f7f1d53842af58b4240686dad4270878" blockNumber: 3 [11:10:43.958] INFO: archiver Downloaded L2 block 5 blockHash: "0x27be6c799c01cd97c203c5141deae0f9a1eb694fc5aaf57a06a90bff9d910916" blockNumber: 5 [11:10:43.959] INFO: archiver Downloaded L2 block 4 blockHash: "0x22904cda410acba1ed66b180bbd0057b2a2cf61564267ac87ba77a237f961986" blockNumber: 4 [11:10:43.959] INFO: archiver Downloaded L2 block 6 blockHash: "0x12857e4319f643ad58f7e2a95470b482b166897038d23263a97c277dbeb82b54" blockNumber: 6 [11:10:44.991] INFO: archiver Downloaded L2 block 7 blockHash: "0x2e82579f660cae4a1c38c80f781d4dc03adf97aa8098b2d6e4157b56c79e415b" blockNumber: 7 [11:10:44.991] INFO: archiver Downloaded L2 block 9 blockHash: "0x01848df8c72f6c51e23e5aa898bdae14a381ff4de7a28e592b0529e1a114a3af" blockNumber: 9 [11:10:44.991] INFO: archiver Downloaded L2 block 8 blockHash: "0x29ca2d4898d1a540d46f45e9a20c029d6765167e031ff7d51701be109189f8ae" blockNumber: 8 [11:10:44.992] INFO: archiver Downloaded L2 block 10 blockHash: "0x0b286a06ffef588628974f94c376829a51f36aa59668f766c1bc4896f0690e75" blockNumber: 10 [11:10:46.027] INFO: archiver Downloaded L2 block 11 blockHash: "0x2a05ceb6fd3c9400a73480b4bf796dbdd8f46cbb35e37e1165e50beb514bb3a4" blockNumber: 11 [11:10:46.027] INFO: archiver Downloaded L2 block 14 blockHash: "0x179ad2642bce681dabbf4a277ef2cd5e1f7b6812bd9fac9358a6346187b81eb6" blockNumber: 14 [11:10:46.027] INFO: archiver Downloaded L2 block 12 blockHash: "0x2242c7eaa8133409069ca6c6f466b08672f0c3308a0ce89c4306908f1ebd2f4d" blockNumber: 12 [11:10:46.028] INFO: archiver Downloaded L2 block 13 blockHash: "0x19070934599ca22124e158f29e3476fd69575773dc987cd93aa3dc25341c2ce0" blockNumber: 13 [11:10:47.060] INFO: archiver Downloaded L2 block 18 blockHash: "0x0b3b73690622ab3110b633d306e77e7ba17dfc52727e47177f274211e85faa5c" blockNumber: 18 [11:10:47.060] INFO: archiver Downloaded L2 block 15 blockHash: "0x1641ce6f8bf798f9e9adb91b74fc0f1a13b822156309e5c9b4139b705ca65854" blockNumber: 15 [11:10:47.060] INFO: archiver Downloaded L2 block 16 blockHash: "0x11d802bf4c814458d463479420dc88fc5fa0ccd114bab910f23a8e028d6af907" blockNumber: 16 [11:10:47.061] INFO: archiver Downloaded L2 block 17 blockHash: "0x2f16f9fe977fd0202b46b494f35a313353eb90eb91402925110422419ada04f3" blockNumber: 17 [11:10:48.094] INFO: archiver Downloaded L2 block 21 blockHash: "0x2bd6a5e4433f87eb5290c0d9daef97ab0819137e67a1b83a2d232c95f887bac5" blockNumber: 21 [11:10:48.094] INFO: archiver Downloaded L2 block 22 blockHash: "0x0fba866a869b819e181f95743047998e6f9d516fc4c40247e72ca76ecf6cafe3" blockNumber: 22 [11:10:48.094] INFO: archiver Downloaded L2 block 20 blockHash: "0x168ca8474eef62fc291eb9c1e73973ffc2b820961d9285b00df3666d617b8f8e" blockNumber: 20 [11:10:48.095] INFO: archiver Downloaded L2 block 19 blockHash: "0x1aa0fdd5d089aae60efb7c1f27d4dff86ba6c0d0d5021e2cec30e94d957df26d" blockNumber: 19 [11:10:49.127] INFO: archiver Downloaded L2 block 25 blockHash: "0x00d65386773e5786f39883d9ce9905a17ab77abe30d1567bec168793ba5af1b8" blockNumber: 25 [11:10:49.128] INFO: archiver Downloaded L2 block 23 blockHash: "0x26fe1e7196f76d6b4ae0342b878ec21525e44ab6b9038d8014039807decf6ab5" blockNumber: 23 [11:10:49.128] INFO: archiver Downloaded L2 block 26 blockHash: "0x052ba994594f33ffa0a3801547351d8af8e1657888ff175381180b801041452f" blockNumber: 26 [11:10:49.128] INFO: archiver Downloaded L2 block 24 blockHash: "0x086bacb80c17ca96234eb727e931195916f6fb5fb5b145d8f96559394dde9e56" blockNumber: 24 [11:10:49.129] INFO: archiver Downloaded L2 block 22 blockHash: "0x0fba866a869b819e181f95743047998e6f9d516fc4c40247e72ca76ecf6cafe3" blockNumber: 22 [11:10:50.172] INFO: archiver Downloaded L2 block 30 blockHash: "0x091c1f8bcaf206eae58921ff2859320ed9e7822a82a95ff99fcc60fca0614a42" blockNumber: 30 [11:10:50.173] INFO: archiver Downloaded L2 block 29 blockHash: "0x2614dc5fc6408580f564b057fac6c3afa2d0e8fd2be428a3481d12ae89ade7e9" blockNumber: 29 [11:10:50.173] INFO: archiver Downloaded L2 block 27 blockHash: "0x140a16edfa828dc786246cb3a834c3103538a19c79d21f6298c4090c49fa8e0e" blockNumber: 27 [11:10:50.173] INFO: archiver Downloaded L2 block 28 blockHash: "0x1d7793e2df27dfad0f8a41bbb4d13e1701be133ecc117b2c8a2c02abb51dd513" blockNumber: 28 [11:10:50.173] INFO: archiver Downloaded L2 block 26 blockHash: "0x052ba994594f33ffa0a3801547351d8af8e1657888ff175381180b801041452f" blockNumber: 26 [11:10:51.207] INFO: archiver Downloaded L2 block 30 blockHash: "0x091c1f8bcaf206eae58921ff2859320ed9e7822a82a95ff99fcc60fca0614a42" blockNumber: 30 [11:10:51.207] INFO: archiver Downloaded L2 block 32 blockHash: "0x204264d3fe9f82b61bb1405bb02cfc59299178dfdebeba75e0691597fd59c000" blockNumber: 32 [11:10:51.208] INFO: archiver Downloaded L2 block 34 blockHash: "0x164d62ba59c01e9a29740d4c83368e15de8bb5a5b8621e8bdd1d0e0978e99d79" blockNumber: 34 [11:10:51.208] INFO: archiver Downloaded L2 block 33 blockHash: "0x007dcb5015c13006cabda887a0e023899e5c327a340179dbd3ec56edc58e146a" blockNumber: 33 [11:10:51.208] INFO: archiver Downloaded L2 block 31 blockHash: "0x00ad45cc4b2b7f0916bd86c11d12628321715b43e204662d15c2c174285ff608" blockNumber: 31 [11:10:53.242] INFO: archiver Downloaded L2 block 34 blockHash: "0x164d62ba59c01e9a29740d4c83368e15de8bb5a5b8621e8bdd1d0e0978e99d79" blockNumber: 34 [11:10:53.243] INFO: archiver Downloaded L2 block 35 blockHash: "0x19252e14d4fd8af5e47651cf31abaeb310a6a59c9754d642fba30ac6853fd076" blockNumber: 35 [11:10:53.244] INFO: archiver Downloaded L2 block 37 blockHash: "0x17ee8d4ba85eb5b2f6a0bcb70115bdd94c90c1dc8fc89f1c9d2418ab944395ac" blockNumber: 37 [11:10:53.244] INFO: archiver Downloaded L2 block 36 blockHash: "0x0b0b0f4694fe81717cb30f59f5dc5d6dc5b632c25832ac424eea074d0c3c19a7" blockNumber: 36 [11:10:55.285] INFO: archiver Downloaded L2 block 41 blockHash: "0x2f7f979d0bae806fcd0bfaed001c1286ae569ecf85aa4063b0906f2097dd5c4c" blockNumber: 41 [11:10:55.285] INFO: archiver Downloaded L2 block 38 blockHash: "0x016418f770c61aa2e649433ab59c2a124acb8e10ac395bb3e1f8dbf84f405039" blockNumber: 38 [11:10:55.286] INFO: archiver Downloaded L2 block 39 blockHash: "0x1588ad911757f334ad2e969588f7fe149bde4a49c763c636af0c6ffb4f475660" blockNumber: 39 [11:10:55.287] INFO: archiver Downloaded L2 block 40 blockHash: "0x06e69f6ba327f6acb88de656058f68eb4998dbee0fec04cf891aabf3b047b071" blockNumber: 40 [11:10:57.327] INFO: archiver Downloaded L2 block 42 blockHash: "0x1ea461af6e875e1bb91dd729f196245ac54b2652799172bba10f77fa2a2fb248" blockNumber: 42 [11:10:57.328] INFO: archiver Downloaded L2 block 45 blockHash: "0x1f7e13643d2286e77dc8de2e37c73d230b77f4cbd7e11fdb9eb7b1c136fe180a" blockNumber: 45 [11:10:57.328] INFO: archiver Downloaded L2 block 43 blockHash: "0x0d89250055f022310144c869ecfdf8da7099fd9bd5e5680687a64b8ad7e588ce" blockNumber: 43 [11:10:57.329] INFO: archiver Downloaded L2 block 44 blockHash: "0x2212d3ae031d4549789c39a403f91f4f06e4e031244700dbb56b381d7730c203" blockNumber: 44 [11:10:59.363] INFO: archiver Downloaded L2 block 46 blockHash: "0x04028bca36e3e6644ceef2425c636ba594e73f4e50d673766e6d4ce8e8b256fb" blockNumber: 46 [11:10:59.364] INFO: archiver Downloaded L2 block 49 blockHash: "0x035b4fc20e48c41bc850e3a4c3c17f4d2004ef3704c660275b63c04925a7e139" blockNumber: 49 [11:10:59.364] INFO: archiver Downloaded L2 block 48 blockHash: "0x2993838786ed7f76d994e9e86f8801657666857b2e46ba9c874c9512a701d112" blockNumber: 48 [11:10:59.365] INFO: archiver Downloaded L2 block 47 blockHash: "0x075078e4ad9c68625585d02bc501bf5645c7cd6601c91090daa88915c8c97a8d" blockNumber: 47 [11:11:00.405] INFO: archiver Downloaded L2 block 53 blockHash: "0x17bd954fbe9ed7c3c376fb340c302dd97297284ebc5c3a5c65f1693a5b26fa0a" blockNumber: 53 [11:11:00.405] INFO: archiver Downloaded L2 block 50 blockHash: "0x0c95eeb66fa8e7704940d81faa1f6d084bc38dc7ffd7a697f1344c50507088dc" blockNumber: 50 [11:11:00.406] INFO: archiver Downloaded L2 block 52 blockHash: "0x2841bc4f270bbc36e31336bbb1e22d90de24134beb16568eda8352b80d364b3b" blockNumber: 52 [11:11:00.406] INFO: archiver Downloaded L2 block 51 blockHash: "0x2dd14899c2e30294e35e795d267a36890f5971d2e265e59645ed5df9acf1b336" blockNumber: 51 [11:11:01.460] INFO: archiver Downloaded L2 block 57 blockHash: "0x1b55c1c5375a29d7f5793c19af6fbf0a5b8183eab97695298e0b1eafd61693b5" blockNumber: 57 [11:11:01.461] INFO: archiver Downloaded L2 block 55 blockHash: "0x19635ee907e378546db1a15810db50b5e77f526d68a19040c482d5ccefb0127e" blockNumber: 55 [11:11:01.461] INFO: archiver Downloaded L2 block 56 blockHash: "0x0847cee0bce17dface027efbb368ac35773f372fef20d9a4afa0947f216ac20d" blockNumber: 56 [11:11:01.463] INFO: archiver Downloaded L2 block 54 blockHash: "0x29043b57317ba3f06994a6638d2185b88b0b1f868e4ccbaf510671a951a3f424" blockNumber: 54 [11:11:02.517] INFO: archiver Downloaded L2 block 58 blockHash: "0x00e95202a2906628cba2c0061519b064757b7a7884e6a8a584b08cfc3baf3825" blockNumber: 58 [11:11:02.518] INFO: archiver Downloaded L2 block 61 blockHash: "0x1b3425cb6f5e199f6f5e0848d3f741a8980fef5fb5fff3a142163c460726dfa9" blockNumber: 61 [11:11:02.518] INFO: archiver Downloaded L2 block 59 blockHash: "0x1e80799672dbfd1fad11e0607cc063e9c0749e8f8b82474f0d579ceb66d5db91" blockNumber: 59 [11:11:02.519] INFO: archiver Downloaded L2 block 60 blockHash: "0x2212c79084a2793836b5fdf446bdd2c4def00d68ec64ebae90b64a16664c6b30" blockNumber: 60 [11:11:03.572] INFO: archiver Downloaded L2 block 62 blockHash: "0x0fd061e3ee9e44a697d4fe81160897cda6df121937b385db7d0b24900ef62a23" blockNumber: 62 [11:11:03.573] INFO: archiver Downloaded L2 block 63 blockHash: "0x2dffde0e911c1f41427d1122cbf6dd28823560c826381094c7f6f26c01f97271" blockNumber: 63 [11:11:03.574] INFO: archiver Downloaded L2 block 65 blockHash: "0x228f48d523eafdc673152ae8e9510eaf7bb856390a0c3d43d8cca97eda0ed217" blockNumber: 65 [11:11:03.574] INFO: archiver Downloaded L2 block 64 blockHash: "0x2bcd0a71d39f8d2e5a290608d8c719e20b264e62227c68152fcd5541f753e25e" blockNumber: 64 [11:11:04.638] INFO: archiver Downloaded L2 block 69 blockHash: "0x1cb98870e6ffbab13afb58add2b024ef29cf9345cced66cd8f0f8f1e0e675ac1" blockNumber: 69 [11:11:04.639] INFO: archiver Downloaded L2 block 67 blockHash: "0x03c0de0563d63f8365e58d34f3ff55f50c85cafdbf97d31b4e698be2720483b9" blockNumber: 67 [11:11:04.640] INFO: archiver Downloaded L2 block 66 blockHash: "0x164d1b0a7ff82e959330bb7a9af22e9f3bf9aa85351891a108cd87586fba048d" blockNumber: 66 [11:11:04.640] INFO: archiver Downloaded L2 block 68 blockHash: "0x0aa4502055e2ee2f44548ece76c8829fdf79b5a514bf40ae238ff62685f8a23b" blockNumber: 68 [11:11:04.641] INFO: archiver Downloaded L2 block 65 blockHash: "0x228f48d523eafdc673152ae8e9510eaf7bb856390a0c3d43d8cca97eda0ed217" blockNumber: 65 [11:11:05.672] INFO: archiver Downloaded L2 block 71 blockHash: "0x2b86465312009ffea5dbc436e97b47e11817068520b3d7f04c1117fe6e969630" blockNumber: 71 [11:11:05.672] INFO: archiver Downloaded L2 block 70 blockHash: "0x2951e898e112dcde4d91d1ca32ef23522baed0aa064a53864f7ea87770b44b32" blockNumber: 70 [11:11:05.672] INFO: archiver Downloaded L2 block 69 blockHash: "0x1cb98870e6ffbab13afb58add2b024ef29cf9345cced66cd8f0f8f1e0e675ac1" blockNumber: 69 [11:11:05.673] INFO: archiver Downloaded L2 block 73 blockHash: "0x2c95db4f54323cb8ef5e9668570752a237b3b176c9d298e3cae650909593514d" blockNumber: 73 [11:11:05.673] INFO: archiver Downloaded L2 block 72 blockHash: "0x210e66af85dc45471729d1ed625eef96ece7a15f54a0dfb18aac0ee35721ec1c" blockNumber: 72 [11:11:06.721] INFO: archiver Downloaded L2 block 77 blockHash: "0x15f71bdd862b0d0016d1001a70b912cf32321e7347e02c3827a64f1bee139922" blockNumber: 77 [11:11:06.721] INFO: archiver Downloaded L2 block 73 blockHash: "0x2c95db4f54323cb8ef5e9668570752a237b3b176c9d298e3cae650909593514d" blockNumber: 73 [11:11:06.722] INFO: archiver Downloaded L2 block 74 blockHash: "0x29fda4905834eb581d02541456bb48653d29e89790ab1203a348cb0d663d4636" blockNumber: 74 [11:11:06.722] INFO: archiver Downloaded L2 block 75 blockHash: "0x1f4118d232a014a3c7776cac40fd1dc1baf219e811726002f715e9abe1120f09" blockNumber: 75 [11:11:06.722] INFO: archiver Downloaded L2 block 76 blockHash: "0x0342ab801e16f9146b109adb704b3d442b45877bf4cf0f38ecbe65e53e9ff447" blockNumber: 76 [11:11:07.767] INFO: archiver Downloaded L2 block 77 blockHash: "0x15f71bdd862b0d0016d1001a70b912cf32321e7347e02c3827a64f1bee139922" blockNumber: 77 [11:11:07.767] INFO: archiver Downloaded L2 block 79 blockHash: "0x1897627bbe054c2dbf1c19062c50b4bab839fc4f119d0683af625c9ab6eef20a" blockNumber: 79 [11:11:07.767] INFO: archiver Downloaded L2 block 81 blockHash: "0x0f502ab8cdb5ab5dcc5dd194e53775c4e512ffdd0645ea363ab52921072c2003" blockNumber: 81 [11:11:07.768] INFO: archiver Downloaded L2 block 78 blockHash: "0x263a83bf5c2410135a08fb48b0323fad959742c5d3d0376b11c55009e0961003" blockNumber: 78 [11:11:07.768] INFO: archiver Downloaded L2 block 80 blockHash: "0x15669498212e68da4e015a4eb90fe1b9404f7e33fa922b242b2c3982c2ac975a" blockNumber: 80 [11:11:08.806] INFO: archiver Downloaded L2 block 81 blockHash: "0x0f502ab8cdb5ab5dcc5dd194e53775c4e512ffdd0645ea363ab52921072c2003" blockNumber: 81 [11:11:08.807] INFO: archiver Downloaded L2 block 82 blockHash: "0x24f6959cf5f228e97ff2934de0b68ee6bdd4b0bc9cef9f6fb60da7e60b354dfb" blockNumber: 82 [11:11:08.811] INFO: archiver Updated proven chain to block 81 (epoch 7) provenBlockNumber: 81 provenEpochNumber: 7 [11:11:08.813] INFO: archiver Initial archiver sync to L1 block 265 complete. ``` Fixes #10428 Fixes #10242
- Loading branch information