Delta update generation broken on Windows x64 #18696

mihaiplesa · 2021-10-11T17:33:03Z

Windows x64 build is broken when generating delta update. Chromium 95 related.

18:30:26 FAILED: chrome.7z setup.ex_ gen/chrome/installer/mini_installer/mini_installer/packed_files.rc chrome.packed.7z 18:30:26 C:/jenkins/x64-nightly/src/brave/vendor/depot_tools/bootstrap-2@3_8_10_chromium_23_bin/python3/bin/python3.exe ../../chrome/tools/build/win/create_installer_archive.py --build_dir . --staging_dir gen/chrome/installer/mini_installer/mini_installer --input_file ../../chrome/installer/mini_installer/chrome.release --resource_file_path gen/chrome/installer/mini_installer/mini_installer/packed_files.rc --target_arch=x64 --distribution=_brave --output_dir . --chrome_runtime_deps gen/chrome_component.runtime_deps --setup_runtime_deps gen/setup.runtime_deps --setup_exe_format=DIFF --diff_algorithm=COURGETTE --last_chrome_installer delta --enable_hidpi=1 --depfile gen/chrome/installer/mini_installer/archive.d 18:30:26 b'Done Adding Additional Store\r\n' 18:30:26 b'Successfully signed: gen/chrome/installer/mini_installer/mini_installer\\temp_installer_archive\\Chrome-bin\\brave.exe\r\r\n' 18:30:26 b'Done Adding Additional Store\r\n' 18:30:26 b'Successfully signed: gen/chrome/installer/mini_installer/mini_installer\\temp_installer_archive\\Chrome-bin\\chrome_proxy.exe\r\r\n' 18:30:26 b'Done Adding Additional Store\r\n' 18:30:26 b'Successfully signed: gen/chrome/installer/mini_installer/mini_installer\\temp_installer_archive\\Chrome-bin\\97.1.35.25\\chrome.dll\r\r\n' 18:30:26 b'Done Adding Additional Store\r\n' 18:30:26 b'Successfully signed: gen/chrome/installer/mini_installer/mini_installer\\temp_installer_archive\\Chrome-bin\\97.1.36.26\\chrome.dll\r\r\n' 18:30:26 b'Done Adding Additional Store\r\n' 18:30:26 b'Successfully signed: gen/chrome/installer/mini_installer/mini_installer\\temp_installer_archive\\Chrome-bin\\97.1.36.26\\chrome_elf.dll\r\r\n' 18:30:26 b'Done Adding Additional Store\r\n' 18:30:26 b'Successfully signed: gen/chrome/installer/mini_installer/mini_installer\\temp_installer_archive\\Chrome-bin\\97.1.36.26\\chrome_pwa_launcher.exe\r\r\n' 18:30:26 b'Done Adding Additional Store\r\n' 18:30:26 b'Successfully signed: gen/chrome/installer/mini_installer/mini_installer\\temp_installer_archive\\Chrome-bin\\97.1.36.26\\d3dcompiler_47.dll\r\r\n' 18:30:26 b'Done Adding Additional Store\r\n' 18:30:26 b'Successfully signed: gen/chrome/installer/mini_installer/mini_installer\\temp_installer_archive\\Chrome-bin\\97.1.36.26\\eventlog_provider.dll\r\r\n' 18:30:26 b'Done Adding Additional Store\r\n' 18:30:26 b'Successfully signed: gen/chrome/installer/mini_installer/mini_installer\\temp_installer_archive\\Chrome-bin\\97.1.36.26\\libEGL.dll\r\r\n' 18:30:26 b'Done Adding Additional Store\r\n' 18:30:26 b'Successfully signed: gen/chrome/installer/mini_installer/mini_installer\\temp_installer_archive\\Chrome-bin\\97.1.36.26\\libGLESv2.dll\r\r\n' 18:30:26 b'Done Adding Additional Store\r\n' 18:30:26 b'Successfully signed: gen/chrome/installer/mini_installer/mini_installer\\temp_installer_archive\\Chrome-bin\\97.1.36.26\\mojo_core.dll\r\r\n' 18:30:26 b'Done Adding Additional Store\r\n' 18:30:26 b'Successfully signed: gen/chrome/installer/mini_installer/mini_installer\\temp_installer_archive\\Chrome-bin\\97.1.36.26\\notification_helper.exe\r\r\n' 18:30:26 b'Done Adding Additional Store\r\n' 18:30:26 b'Successfully signed: gen/chrome/installer/mini_installer/mini_installer\\temp_installer_archive\\Chrome-bin\\97.1.36.26\\vk_swiftshader.dll\r\r\n' 18:30:26 b'Done Adding Additional Store\r\n' 18:30:26 b'Successfully signed: gen/chrome/installer/mini_installer/mini_installer\\temp_installer_archive\\Chrome-bin\\97.1.36.26\\vulkan-1.dll\r\r\n' 18:30:26 b'Done Adding Additional Store\r\n' 18:30:26 b'Successfully signed: gen/chrome/installer/mini_installer/mini_installer\\temp_installer_archive\\Chrome-bin\\97.1.36.26\\swiftshader\\libEGL.dll\r\r\n' 18:30:26 b'Done Adding Additional Store\r\n' 18:30:26 b'Successfully signed: gen/chrome/installer/mini_installer/mini_installer\\temp_installer_archive\\Chrome-bin\\97.1.36.26\\swiftshader\\libGLESv2.dll\r\r\n' 18:30:26 b'Done Adding Additional Store\r\n' 18:30:26 b'Successfully signed: .\\setup.exe\r\r\n' 18:30:26 Traceback (most recent call last): 18:30:26 File "../../chrome/tools/build/win/create_installer_archive.py", line 228, in RunSystemCommand 18:30:26 output = subprocess.check_output(cmd, stderr=subprocess.STDOUT) 18:30:26 File "C:\jenkins\x64-nightly\src\brave\vendor\depot_tools\bootstrap-2@3_8_10_chromium_23_bin\python3\bin\lib\subprocess.py", line 415, in check_output 18:30:26 return run(*popenargs, stdout=PIPE, timeout=timeout, check=True, 18:30:26 File "C:\jenkins\x64-nightly\src\brave\vendor\depot_tools\bootstrap-2@3_8_10_chromium_23_bin\python3\bin\lib\subprocess.py", line 516, in run 18:30:26 raise CalledProcessError(retcode, process.args, 18:30:26 subprocess.CalledProcessError: Command 'delta\courgette.exe -gen "delta\chrome.7z" ".\chrome.7z" ".\chrome_patch.diff"' returned non-zero exit status 1. 18:30:26 18:30:26 During handling of the above exception, another exception occurred: 18:30:26 18:30:26 Traceback (most recent call last): 18:30:26 File "../../chrome/tools/build/win/create_installer_archive.py", line 672, in <module> 18:30:26 sys.exit(main(options)) 18:30:26 File "../../chrome/tools/build/win/create_installer_archive.py", line 568, in main 18:30:26 archive_file = CreateArchiveFile(options, staging_dir, 18:30:26 File "../../chrome/tools/build/win/create_installer_archive.py", line 319, in CreateArchiveFile 18:30:26 GenerateDiffPatch(options, prev_archive_file, archive_file, patch_file) 18:30:26 File "../../chrome/tools/build/win/create_installer_archive.py", line 170, in GenerateDiffPatch 18:30:26 RunSystemCommand(cmd, options.verbose) 18:30:26 File "../../chrome/tools/build/win/create_installer_archive.py", line 232, in RunSystemCommand 18:30:26 raise Exception("Error while running cmd: %s\n" 18:30:26 Exception: Error while running cmd: delta\courgette.exe -gen "delta\chrome.7z" ".\chrome.7z" ".\chrome_patch.diff" 18:30:26 Exit code: 1 18:30:26 Command output: 18:30:26 b'[0110/230611.003:VERBOSE1:ensemble_create.cc(237)] start GenerateEnsemblePatch\r\n[0110/230612.794:VERBOSE1:ensemble_create.cc(138)] done FindEmbeddedElements 1.79131\r\n[0110/230612.794:VERBOSE1:ensemble_create.cc(144)] old has 15 elements\r\n[0110/230612.794:VERBOSE1:ensemble_create.cc(145)] new has 16 elements\r\n[0110/230643.844:VERBOSE1:ensemble_create.cc(155)] done make bases 31.0499s\r\n[0110/230653.789:VERBOSE1:ensemble_create.cc(189)] Compare old(4,17577978,174435840) to new(4,32,174435840) --> 4 in 9.94443s\r\n[0110/230653.789:VERBOSE1:ensemble_create.cc(208)] Matched old(4,17577978,174435840) to new(4,32,174435840) --> 4\r\n[0110/230705.965:VERBOSE1:ensemble_create.cc(189)] Compare old(4,17577978,174435840) to new(4,192550797,189615616) --> 24319129 in 12.1764s\r\n[0110/230705.965:VERBOSE1:ensemble_create.cc(208)] Matched old(4,17577978,174435840) to new(4,192550797,189615616) --> 24319129\r\n[0110/230706.029:VERBOSE1:ensemble_create.cc(189)] Compare old(4,193559076,1080320) to new(4,383711947,1176064) --> 255016 in 0.063621s\r\n[0110/230706.096:VERBOSE1:ensemble_create.cc(189)] Compare old(4,194645996,1462784) to new(4,383711947,1176064) --> 303029 in 0.067149s\r\n[0110/230706.129:VERBOSE1:ensemble_create.cc(189)] Compare old(4,200637820,5632) to new(4,383711947,1176064) --> 1093208 in 0.032998s\r\n[0110/230706.182:VERBOSE1:ensemble_create.cc(189)] Compare old(4,210867527,407040) to new(4,383711947,1176064) --> 537095 in 0.052866s\r\n[0110/230706.236:VERBOSE1:ensemble_create.cc(189)] Compare old(4,242633920,1336320) to new(4,383711947,1176064) --> 267311 in 0.054171s\r\n[0110/230706.301:VERBOSE1:ensemble_create.cc(189)] Compare old(4,243976840,1087488) to new(4,383711947,1176064) --> 272349 in 0.064423s\r\n[0110/230706.355:VERBOSE1:ensemble_create.cc(189)] Compare old(4,253831689,417792) to new(4,383711947,1176064) --> 530137 in 0.054377s\r\n[0110/230706.438:VERBOSE1:ensemble_create.cc(189)] Compare old(4,254256081,2725376) to new(4,383711947,1176064) --> 390534 in 0.082645s\r\n[0110/230706.491:VERBOSE1:ensemble_create.cc(189)] Compare old(4,261062051,705024) to new(4,383711947,1176064) --> 495468 in 0.053444s\r\n[0110/230706.572:VERBOSE1:ensemble_create.cc(189)] Compare old(4,261773675,2387456) to new(4,383711947,1176064) --> 259586 in 0.080255s\r\n[0110/230706.629:VERBOSE1:ensemble_create.cc(189)] Compare old(4,264167731,876544) to new(4,383711947,1176064) --> 311796 in 0.057407s\r\n[0110/230706.629:VERBOSE1:ensemble_create.cc(208)] Matched old(4,193559076,1080320) to new(4,383711947,1176064) --> 255016\r\n[0110/230706.701:VERBOSE1:ensemble_create.cc(189)] Compare old(4,193559076,1080320) to new(4,384894611,1536512) --> 568414 in 0.072018s\r\n[0110/230706.766:VERBOSE1:ensemble_create.cc(189)] Compare old(4,194645996,1462784) to new(4,384894611,1536512) --> 219218 in 0.064675s\r\n[0110/230706.806:VERBOSE1:ensemble_create.cc(189)] Compare old(4,200637820,5632) to new(4,384894611,1536512) --> 1446332 in 0.0396s\r\n[0110/230706.865:VERBOSE1:ensemble_create.cc(189)] Compare old(4,210867527,407040) to new(4,384894611,1536512) --> 877848 in 0.059134s\r\n[0110/230706.925:VERBOSE1:ensemble_create.cc(189)] Compare old(4,242633920,1336320) to new(4,384894611,1536512) --> 557779 in 0.060389s\r\n[0110/230706.996:VERBOSE1:ensemble_create.cc(189)] Compare old(4,243976840,1087488) to new(4,384894611,1536512) --> 562567 in 0.070743s\r\n[0110/230707.056:VERBOSE1:ensemble_create.cc(189)] Compare old(4,253831689,417792) to new(4,384894611,1536512) --> 874022 in 0.060144s\r\n[0110/230707.144:VERBOSE1:ensemble_create.cc(189)] Compare old(4,254256081,2725376) to new(4,384894611,1536512) --> 673030 in 0.087585s\r\n[0110/230707.202:VERBOSE1:ensemble_create.cc(189)] Compare old(4,261062051,705024) to new(4,384894611,1536512) --> 801530 in 0.057876s\r\n[0110/230707.283:VERBOSE1:ensemble_create.cc(189)] Compare old(4,261773675,2387456) to new(4,384894611,1536512) --> 298608 in 0.080866s\r\n[0110/230707.345:VERBOSE1:ensemble_create.cc(189)] Compare old(4,264167731,876544) to new(4,384894611,1536512) --> 578909 in 0.062405s\r\n[0110/230707.345:VERBOSE1:ensemble_create.cc(208)] Matched old(4,194645996,1462784) to new(4,384894611,1536512) --> 219218\r\n[0110/230707.547:VERBOSE1:ensemble_create.cc(189)] Compare old(4,196115380,4515840) to new(4,386437723,4515840) --> 6 in 0.201534s\r\n[0110/230707.889:VERBOSE1:ensemble_create.cc(189)] Compare old(4,211281167,6210048) to new(4,386437723,4515840) --> 1859302 in 0.34277s\r\n[0110/230708.180:VERBOSE1:ensemble_create.cc(189)] Compare old(4,254256081,2725376) to new(4,386437723,4515840) --> 2085082 in 0.290962s\r\n[0110/230708.423:VERBOSE1:ensemble_create.cc(189)] Compare old(4,257186673,3868672) to new(4,386437723,4515840) --> 2029767 in 0.242239s\r\n[0110/230708.723:VERBOSE1:ensemble_create.cc(189)] Compare old(4,261773675,2387456) to new(4,386437723,4515840) --> 2143960 in 0.299676s\r\n[0110/230708.723:VERBOSE1:ensemble_create.cc(208)] Matched old(4,196115380,4515840) to new(4,386437723,4515840) --> 6\r\n[0110/230708.723:VERBOSE1:ensemble_create.cc(189)] Compare old(4,193559076,1080320) to new(4,390960163,5632) --> 289 in 0.000217s\r\n[0110/230708.723:VERBOSE1:ensemble_create.cc(189)] Compare old(4,194645996,1462784) to new(4,390960163,5632) --> 274 in 0.000214s\r\n[0110/230708.723:VERBOSE1:ensemble_create.cc(189)] Compare old(4,200637820,5632) to new(4,390960163,5632) --> 129 in 9e-05s\r\n[0110/230708.723:VERBOSE1:ensemble_create.cc(189)] Compare old(4,210867527,407040) to new(4,390960163,5632) --> 521 in 0.000202s\r\n[0110/230708.724:VERBOSE1:ensemble_create.cc(189)] Compare old(4,242633920,1336320) to new(4,390960163,5632) --> 299 in 0.000196s\r\n[0110/230708.724:VERBOSE1:ensemble_create.cc(189)] Compare old(4,243976840,1087488) to new(4,390960163,5632) --> 293 in 0.000216s\r\n[0110/230708.724:VERBOSE1:ensemble_create.cc(189)] Compare old(4,253831689,417792) to new(4,390960163,5632) --> 513 in 0.000197s\r\n[0110/230708.724:VERBOSE1:ensemble_create.cc(189)] Compare old(4,261062051,705024) to new(4,390960163,5632) --> 522 in 0.000208s\r\n[0110/230708.724:VERBOSE1:ensemble_create.cc(189)] Compare old(4,264167731,876544) to new(4,390960163,5632) --> 324 in 0.000222s\r\n[0110/230708.724:VERBOSE1:ensemble_create.cc(208)] Matched old(4,200637820,5632) to new(4,390960163,5632) --> 129\r\n[0110/230708.750:VERBOSE1:ensemble_create.cc(189)] Compare old(4,193559076,1080320) to new(4,401189870,429568) --> 88462 in 0.025857s\r\n[0110/230708.776:VERBOSE1:ensemble_create.cc(189)] Compare old(4,194645996,1462784) to new(4,401189870,429568) --> 93207 in 0.025946s\r\n[0110/230708.788:VERBOSE1:ensemble_create.cc(189)] Compare old(4,200637820,5632) to new(4,401189870,429568) --> 386270 in 0.011683s\r\n[0110/230708.807:VERBOSE1:ensemble_create.cc(189)] Compare old(4,210867527,407040) to new(4,401189870,429568) --> 83878 in 0.018464s\r\n[0110/230708.830:VERBOSE1:ensemble_create.cc(189)] Compare old(4,242633920,1336320) to new(4,401189870,429568) --> 86199 in 0.022986s\r\n[0110/230708.855:VERBOSE1:ensemble_create.cc(189)] Compare old(4,243976840,1087488) to new(4,401189870,429568) --> 91638 in 0.025597s\r\n[0110/230708.875:VERBOSE1:ensemble_create.cc(189)] Compare old(4,253831689,417792) to new(4,401189870,429568) --> 97396 in 0.019591s\r\n[0110/230708.896:VERBOSE1:ensemble_create.cc(189)] Compare old(4,261062051,705024) to new(4,401189870,429568) --> 93314 in 0.020733s\r\n[0110/230708.926:VERBOSE1:ensemble_create.cc(189)] Compare old(4,261773675,2387456) to new(4,401189870,429568) --> 91539 in 0.030801s\r\n[0110/230708.950:VERBOSE1:ensemble_create.cc(189)] Compare old(4,264167731,876544) to new(4,401189870,429568) --> 93727 in 0.023156s\r\n[0110/230708.950:VERBOSE1:ensemble_create.cc(208)] Matched old(4,210867527,407040) to new(4,401189870,429568) --> 83878\r\n[0110/230709.358:VERBOSE1:ensemble_create.cc(189)] Compare old(4,196115380,4515840) to new(4,401626038,6939648) --> 3409088 in 0.408216s\r\n[0110/230709.772:VERBOSE1:ensemble_create.cc(189)] Compare old(4,211281167,6210048) to new(4,401626038,6939648) --> 1224519 in 0.414127s\r\n[0110/230710.088:VERBOSE1:ensemble_create.cc(189)] Compare old(4,257186673,3868672) to new(4,401626038,6939648) --> 2360125 in 0.315964s\r\n[0110/230710.088:VERBOSE1:ensemble_create.cc(208)] Matched old(4,211281167,6210048) to new(4,401626038,6939648) --> 1224519\r\n[0110/230710.173:VERBOSE1:ensemble_create.cc(189)] Compare old(4,193559076,1080320) to new(4,434145338,1518592) --> 407208 in 0.085123s\r\n[0110/230710.261:VERBOSE1:ensemble_create.cc(189)] Compare old(4,194645996,1462784) to new(4,434145338,1518592) --> 437712 in 0.087614s\r\n[0110/230710.304:VERBOSE1:ensemble_create.cc(189)] Compare old(4,200637820,5632) to new(4,434145338,1518592) --> 1431734 in 0.043088s\r\n[0110/230710.373:VERBOSE1:ensemble_create.cc(189)] Compare old(4,210867527,407040) to new(4,434145338,1518592) --> 727134 in 0.068958s\r\n[0110/230710.442:VERBOSE1:ensemble_create.cc(189)] Compare old(4,242633920,1336320) to new(4,434145338,1518592) --> 354314 in 0.068481s\r\n[0110/230710.526:VERBOSE1:ensemble_create.cc(189)] Compare old(4,243976840,1087488) to new(4,434145338,1518592) --> 410071 in 0.084755s\r\n[0110/230710.597:VERBOSE1:ensemble_create.cc(189)] Compare old(4,253831689,417792) to new(4,434145338,1518592) --> 713089 in 0.071005s\r\n[0110/230710.703:VERBOSE1:ensemble_create.cc(189)] Compare old(4,254256081,2725376) to new(4,434145338,1518592) --> 499735 in 0.105871s\r\n[0110/230710.772:VERBOSE1:ensemble_create.cc(189)] Compare old(4,261062051,705024) to new(4,434145338,1518592) --> 666074 in 0.068508s\r\n[0110/230710.877:VERBOSE1:ensemble_create.cc(189)] Compare old(4,261773675,2387456) to new(4,434145338,1518592) --> 361787 in 0.10492s\r\n[0110/230710.952:VERBOSE1:ensemble_create.cc(189)] Compare old(4,264167731,876544) to new(4,434145338,1518592) --> 445491 in 0.074903s\r\n[0110/230710.952:VERBOSE1:ensemble_create.cc(208)] Matched old(4,242633920,1336320) to new(4,434145338,1518592) --> 354314\r\n[0110/230711.018:VERBOSE1:ensemble_create.cc(189)] Compare old(4,193559076,1080320) to new(4,435670530,1179648) --> 272807 in 0.066099s\r\n[0110/230711.085:VERBOSE1:ensemble_create.cc(189)] Compare old(4,194645996,1462784) to new(4,435670530,1179648) --> 297209 in 0.067112s\r\n[0110/230711.118:VERBOSE1:ensemble_create.cc(189)] Compare old(4,200637820,5632) to new(4,435670530,1179648) --> 1099798 in 0.033088s\r\n[0110/230711.171:VERBOSE1:ensemble_create.cc(189)] Compare old(4,210867527,407040) to new(4,435670530,1179648) --> 540128 in 0.053136s\r\n[0110/230711.226:VERBOSE1:ensemble_create.cc(189)] Compare old(4,242633920,1336320) to new(4,435670530,1179648) --> 271409 in 0.054349s\r\n[0110/230711.289:VERBOSE1:ensemble_create.cc(189)] Compare old(4,243976840,1087488) to new(4,435670530,1179648) --> 255709 in 0.063677s\r\n[0110/230711.344:VERBOSE1:ensemble_create.cc(189)] Compare old(4,253831689,417792) to new(4,435670530,1179648) --> 534142 in 0.054604s\r\n[0110/230711.427:VERBOSE1:ensemble_create.cc(189)] Compare old(4,254256081,2725376) to new(4,435670530,1179648) --> 395336 in 0.08293s\r\n[0110/230711.481:VERBOSE1:ensemble_create.cc(189)] Compare old(4,261062051,705024) to new(4,435670530,1179648) --> 498702 in 0.053635s\r\n[0110/230711.561:VERBOSE1:ensemble_create.cc(189)] Compare old(4,261773675,2387456) to new(4,435670530,1179648) --> 254063 in 0.080526s\r\n[0110/230711.619:VERBOSE1:ensemble_create.cc(189)] Compare old(4,264167731,876544) to new(4,435670530,1179648) --> 306826 in 0.057274s\r\n[0110/230711.619:VERBOSE1:ensemble_create.cc(208)] Matched old(4,261773675,2387456) to new(4,435670530,1179648) --> 254063\r\n[0110/230711.646:VERBOSE1:ensemble_create.cc(189)] Compare old(4,193559076,1080320) to new(4,445622107,443392) --> 94571 in 0.027247s\r\n[0110/230711.673:VERBOSE1:ensemble_create.cc(189)] Compare old(4,194645996,1462784) to new(4,445622107,443392) --> 100299 in 0.027364s\r\n[0110/230711.685:VERBOSE1:ensemble_create.cc(189)] Compare old(4,200637820,5632) to new(4,445622107,443392) --> 397707 in 0.012069s\r\n[0110/230711.705:VERBOSE1:ensemble_create.cc(189)] Compare old(4,210867527,407040) to new(4,445622107,443392) --> 105682 in 0.019722s\r\n[0110/230711.729:VERBOSE1:ensemble_create.cc(189)] Compare old(4,242633920,1336320) to new(4,445622107,443392) --> 91963 in 0.023812s\r\n[0110/230711.756:VERBOSE1:ensemble_create.cc(189)] Compare old(4,243976840,1087488) to new(4,445622107,443392) --> 98631 in 0.026712s\r\n[0110/230711.775:VERBOSE1:ensemble_create.cc(189)] Compare old(4,253831689,417792) to new(4,445622107,443392) --> 88264 in 0.019677s\r\n[0110/230711.797:VERBOSE1:ensemble_create.cc(189)] Compare old(4,261062051,705024) to new(4,445622107,443392) --> 100411 in 0.021568s\r\n[0110/230711.829:VERBOSE1:ensemble_create.cc(189)] Compare old(4,261773675,2387456) to new(4,445622107,443392) --> 96717 in 0.032282s\r\n[0110/230711.854:VERBOSE1:ensemble_create.cc(189)] Compare old(4,264167731,876544) to new(4,445622107,443392) --> 101147 in 0.024221s\r\n[0110/230711.854:VERBOSE1:ensemble_create.cc(208)] Matched old(4,253831689,417792) to new(4,445622107,443392) --> 88264\r\n[0110/230712.029:VERBOSE1:ensemble_create.cc(189)] Compare old(4,194645996,1462784) to new(4,446072099,3182080) --> 1359536 in 0.17534s\r\n[0110/230712.237:VERBOSE1:ensemble_create.cc(189)] Compare old(4,196115380,4515840) to new(4,446072099,3182080) --> 1641917 in 0.207498s\r\n[0110/230712.464:VERBOSE1:ensemble_create.cc(189)] Compare old(4,211281167,6210048) to new(4,446072099,3182080) --> 883967 in 0.22783s\r\n[0110/230712.599:VERBOSE1:ensemble_create.cc(189)] Compare old(4,242633920,1336320) to new(4,446072099,3182080) --> 1263230 in 0.134771s\r\n[0110/230712.772:VERBOSE1:ensemble_create.cc(189)] Compare old(4,243976840,1087488) to new(4,446072099,3182080) --> 1326227 in 0.172618s\r\n[0110/230712.960:VERBOSE1:ensemble_create.cc(189)] Compare old(4,254256081,2725376) to new(4,446072099,3182080) --> 725327 in 0.188447s\r\n[0110/230713.115:VERBOSE1:ensemble_create.cc(189)] Compare old(4,257186673,3868672) to new(4,446072099,3182080) --> 887540 in 0.154608s\r\n[0110/230713.321:VERBOSE1:ensemble_create.cc(189)] Compare old(4,261773675,2387456) to new(4,446072099,3182080) --> 1134683 in 0.20629s\r\n[0110/230713.322:VERBOSE1:ensemble_create.cc(208)] Matched old(4,254256081,2725376) to new(4,446072099,3182080) --> 725327\r\n[0110/230713.595:VERBOSE1:ensemble_create.cc(189)] Compare old(4,196115380,4515840) to new(4,449459395,4459008) --> 2093058 in 0.273766s\r\n[0110/230713.893:VERBOSE1:ensemble_create.cc(189)] Compare old(4,211281167,6210048) to new(4,449459395,4459008) --> 1117331 in 0.298037s\r\n[0110/230714.151:VERBOSE1:ensemble_create.cc(189)] Compare old(4,254256081,2725376) to new(4,449459395,4459008) --> 1201160 in 0.257482s\r\n[0110/230714.345:VERBOSE1:ensemble_create.cc(189)] Compare old(4,257186673,3868672) to new(4,449459395,4459008) --> 865816 in 0.193568s\r\n[0110/230714.616:VERBOSE1:ensemble_create.cc(189)] Compare old(4,261773675,2387456) to new(4,449459395,4459008) --> 1386177 in 0.271327s\r\n[0110/230714.616:VERBOSE1:ensemble_create.cc(208)] Matched old(4,257186673,3868672) to new(4,449459395,4459008) --> 865816\r\n[0110/230714.659:VERBOSE1:ensemble_create.cc(189)] Compare old(4,193559076,1080320) to new(4,453925109,733696) --> 224759 in 0.043016s\r\n[0110/230714.702:VERBOSE1:ensemble_create.cc(189)] Compare old(4,194645996,1462784) to new(4,453925109,733696) --> 228302 in 0.043318s\r\n[0110/230714.722:VERBOSE1:ensemble_create.cc(189)] Compare old(4,200637820,5632) to new(4,453925109,733696) --> 676096 in 0.019909s\r\n[0110/230714.754:VERBOSE1:ensemble_create.cc(189)] Compare old(4,210867527,407040) to new(4,453925109,733696) --> 274013 in 0.031985s\r\n[0110/230714.792:VERBOSE1:ensemble_create.cc(189)] Compare old(4,242633920,1336320) to new(4,453925109,733696) --> 219592 in 0.037374s\r\n[0110/230714.834:VERBOSE1:ensemble_create.cc(189)] Compare old(4,243976840,1087488) to new(4,453925109,733696) --> 228139 in 0.042623s\r\n[0110/230714.867:VERBOSE1:ensemble_create.cc(189)] Compare old(4,253831689,417792) to new(4,453925109,733696) --> 274145 in 0.032784s\r\n[0110/230714.916:VERBOSE1:ensemble_create.cc(189)] Compare old(4,254256081,2725376) to new(4,453925109,733696) --> 202972 in 0.048772s\r\n[0110/230714.946:VERBOSE1:ensemble_create.cc(189)] Compare old(4,261062051,705024) to new(4,453925109,733696) --> 131970 in 0.029991s\r\n[0110/230714.998:VERBOSE1:ensemble_create.cc(189)] Compare old(4,261773675,2387456) to new(4,453925109,733696) --> 209133 in 0.051665s\r\n[0110/230715.036:VERBOSE1:ensemble_create.cc(189)] Compare old(4,264167731,876544) to new(4,453925109,733696) --> 238986 in 0.038122s\r\n[0110/230715.036:VERBOSE1:ensemble_create.cc(208)] Matched old(4,261062051,705024) to new(4,453925109,733696) --> 131970\r\n[0110/230715.178:VERBOSE1:ensemble_create.cc(189)] Compare old(4,193559076,1080320) to new(4,454665405,2615296) --> 1056771 in 0.142306s\r\n[0110/230715.320:VERBOSE1:ensemble_create.cc(189)] Compare old(4,194645996,1462784) to new(4,454665405,2615296) --> 945955 in 0.141135s\r\n[0110/230715.502:VERBOSE1:ensemble_create.cc(189)] Compare old(4,196115380,4515840) to new(4,454665405,2615296) --> 1410371 in 0.182144s\r\n[0110/230715.616:VERBOSE1:ensemble_create.cc(189)] Compare old(4,242633920,1336320) to new(4,454665405,2615296) --> 1029193 in 0.114252s\r\n[0110/230715.756:VERBOSE1:ensemble_create.cc(189)] Compare old(4,243976840,1087488) to new(4,454665405,2615296) --> 1053780 in 0.139527s\r\n[0110/230715.929:VERBOSE1:ensemble_create.cc(189)] Compare old(4,254256081,2725376) to new(4,454665405,2615296) --> 1123375 in 0.173449s\r\n[0110/230716.070:VERBOSE1:ensemble_create.cc(189)] Compare old(4,257186673,3868672) to new(4,454665405,2615296) --> 1081520 in 0.141291s\r\n[0110/230716.179:VERBOSE1:ensemble_create.cc(189)] Compare old(4,261062051,705024) to new(4,454665405,2615296) --> 1407840 in 0.108779s\r\n[0110/230716.333:VERBOSE1:ensemble_create.cc(189)] Compare old(4,261773675,2387456) to new(4,454665405,2615296) --> 476253 in 0.153475s\r\n[0110/230716.457:VERBOSE1:ensemble_create.cc(189)] Compare old(4,264167731,876544) to new(4,454665405,2615296) --> 1114500 in 0.123909s\r\n[0110/230716.457:VERBOSE1:ensemble_create.cc(208)] Matched old(4,261773675,2387456) to new(4,454665405,2615296) --> 476253\r\n[0110/230716.510:VERBOSE1:ensemble_create.cc(189)] Compare old(4,193559076,1080320) to new(4,457287301,948736) --> 208742 in 0.053277s\r\n[0110/230716.564:VERBOSE1:ensemble_create.cc(189)] Compare old(4,194645996,1462784) to new(4,457287301,948736) --> 209811 in 0.053606s\r\n[0110/230716.590:VERBOSE1:ensemble_create.cc(189)] Compare old(4,200637820,5632) to new(4,457287301,948736) --> 885171 in 0.026558s\r\n[0110/230716.633:VERBOSE1:ensemble_create.cc(189)] Compare old(4,210867527,407040) to new(4,457287301,948736) --> 419462 in 0.042773s\r\n[0110/230716.678:VERBOSE1:ensemble_create.cc(189)] Compare old(4,242633920,1336320) to new(4,457287301,948736) --> 199227 in 0.044269s\r\n[0110/230716.730:VERBOSE1:ensemble_create.cc(189)] Compare old(4,243976840,1087488) to new(4,457287301,948736) --> 201938 in 0.052294s\r\n[0110/230716.774:VERBOSE1:ensemble_create.cc(189)] Compare old(4,253831689,417792) to new(4,457287301,948736) --> 415915 in 0.044011s\r\n[0110/230716.841:VERBOSE1:ensemble_create.cc(189)] Compare old(4,254256081,2725376) to new(4,457287301,948736) --> 316154 in 0.066887s\r\n[0110/230716.885:VERBOSE1:ensemble_create.cc(189)] Compare old(4,261062051,705024) to new(4,457287301,948736) --> 389652 in 0.043992s\r\n[0110/230716.950:VERBOSE1:ensemble_create.cc(189)] Compare old(4,261773675,2387456) to new(4,457287301,948736) --> 194142 in 0.065175s\r\n[0110/230716.996:VERBOSE1:ensemble_create.cc(189)] Compare old(4,264167731,876544) to new(4,457287301,948736) --> 203252 in 0.045355s\r\n[0110/230716.996:VERBOSE1:ensemble_create.cc(208)] Matched old(4,261773675,2387456) to new(4,457287301,948736) --> 194142\r\n[0110/230716.996:VERBOSE1:ensemble_create.cc(218)] done FindGenerators found 16 in 65.993s\r\n[0110/230720.204:VERBOSE1:simple_delta.cc(35)] GenerateSimpleDelta 31 31\r\n[0110/230720.204:VERBOSE1:bsdiff_create.cc(96)] Start bsdiff\r\n[0110/230720.204:VERBOSE1:bsdiff_create.cc(123)] done divsufsort 0.000364\r\n[0110/230720.204:VERBOSE1:bsdiff_create.cc(338)] Control tuples: 1 copy bytes: 31 mistakes: 0 (skips: 1) extra bytes: 0\r\nUncompressed bsdiff patch size 34\r\nEnd bsdiff 0.000442\r\n[0110/230732.364:VERBOSE1:adjustment_method_2.cc(1238)] Adjuster::Adjust\r\n[0110/230806.681:VERBOSE1:adjustment_method_2.cc(1279)] Adjuster::Solve 31.9742\r\n[0110/231530.317:VERBOSE1:adjustment_method_2.cc(1279)] Adjuster::Solve 443.319\r\n[0110/231533.500:VERBOSE1:label_manager.cc(34)] 259533 of 275077 labels pre-assigned.\r\n[0110/231533.500:VERBOSE1:label_manager.cc(55)] fill forward 15544\r\n[0110/231533.500:VERBOSE1:label_manager.cc(77)] fill backward 0\r\n[0110/231533.501:VERBOSE1:label_manager.cc(93)] infill 0\r\n[0110/231533.505:VERBOSE1:label_manager.cc(34)] 1276915 of 1307436 labels pre-assigned.\r\n[0110/231533.507:VERBOSE1:label_manager.cc(55)] fill forward 30521\r\n[0110/231533.508:VERBOSE1:label_manager.cc(77)] fill backward 0\r\n[0110/231533.509:VERBOSE1:label_manager.cc(93)] infill 0\r\n[0110/231551.502:VERBOSE1:adjustment_method_2.cc(1238)] Adjuster::Adjust\r\n[0110/231625.106:VERBOSE1:adjustment_method_2.cc(1279)] Adjuster::Solve 30.8354\r\n[0110/232443.468:VERBOSE1:adjustment_method_2.cc(1279)] Adjuster::Solve 497.986\r\n[0110/232448.517:VERBOSE1:label_manager.cc(34)] 223765 of 278861 labels pre-assigned.\r\n[0110/232448.517:VERBOSE1:label_manager.cc(55)] fill forward 23025\r\n[0110/232448.518:VERBOSE1:label_manager.cc(77)] fill backward 6866\r\n[0110/232448.519:VERBOSE1:label_manager.cc(93)] infill 25205\r\n[0110/232448.527:VERBOSE1:label_manager.cc(34)] 590715 of 1911492 labels pre-assigned.\r\n[0110/232448.537:VERBOSE1:label_manager.cc(55)] fill forward 291096\r\n[0110/232448.543:VERBOSE1:label_manager.cc(77)] fill backward 112447\r\n[0110/232448.551:VERBOSE1:label_manager.cc(93)] infill 917234\r\n[0110/232454.889:VERBOSE1:adjustment_method_2.cc(1238)] Adjuster::Adjust\r\n[0110/232455.029:VERBOSE1:adjustment_method_2.cc(1279)] Adjuster::Solve 0.130938\r\n[0110/232456.717:VERBOSE1:adjustment_method_2.cc(1279)] Adjuster::Solve 1.68757\r\n[0110/232456.734:VERBOSE1:label_manager.cc(34)] 1843 of 2173 labels pre-assigned.\r\n[0110/232456.734:VERBOSE1:label_manager.cc(55)] fill forward 267\r\n[0110/232456.734:VERBOSE1:label_manager.cc(77)] fill backward 5\r\n[0110/232456.734:VERBOSE1:label_manager.cc(93)] infill 58\r\n[0110/232456.734:VERBOSE1:label_manager.cc(34)] 6511 of 12426 labels pre-assigned.\r\n[0110/232456.734:VERBOSE1:label_manager.cc(55)] fill forward 961\r\n[0110/232456.734:VERBOSE1:label_manager.cc(77)] fill backward 344\r\n[0110/232456.734:VERBOSE1:label_manager.cc(93)] infill 4610\r\n[0110/232456.841:VERBOSE1:adjustment_method_2.cc(1238)] Adjuster::Adjust\r\n[0110/232456.974:VERBOSE1:adjustment_method_2.cc(1279)] Adjuster::Solve 0.124842\r\n[0110/232458.346:VERBOSE1:adjustment_method_2.cc(1279)] Adjuster::Solve 1.37125\r\n[0110/232458.359:VERBOSE1:label_manager.cc(34)] 1886 of 1920 labels pre-assigned.\r\n[0110/232458.359:VERBOSE1:label_manager.cc(55)] fill forward 11\r\n[0110/232458.359:VERBOSE1:label_manager.cc(77)] fill backward 0\r\n[0110/232458.359:VERBOSE1:label_manager.cc(93)] infill 23\r\n[0110/232458.359:VERBOSE1:label_manager.cc(34)] 6074 of 11011 labels pre-assigned.\r\n[0110/232458.359:VERBOSE1:label_manager.cc(55)] fill forward 879\r\n[0110/232458.359:VERBOSE1:label_manager.cc(77)] fill backward 280\r\n[0110/232458.360:VERBOSE1:label_manager.cc(93)] infill 3778\r\n[0110/232458.653:VERBOSE1:adjustment_method_2.cc(1238)] Adjuster::Adjust\r\n[0110/232459.310:VERBOSE1:adjustment_method_2.cc(1279)] Adjuster::Solve 0.62568\r\n[0110/232504.956:VERBOSE1:adjustment_method_2.cc(1279)] Adjuster::Solve 5.63969\r\n[0110/232505.005:VERBOSE1:label_manager.cc(34)] 7555 of 7903 labels pre-assigned.\r\n[0110/232505.005:VERBOSE1:label_manager.cc(55)] fill forward 348\r\n[0110/232505.005:VERBOSE1:label_manager.cc(77)] fill backward 0\r\n[0110/232505.005:VERBOSE1:label_manager.cc(93)] infill 0\r\n[0110/232505.005:VERBOSE1:label_manager.cc(34)] 30344 of 30760 labels pre-assigned.\r\n[0110/232505.005:VERBOSE1:label_manager.cc(55)] fill forward 416\r\n[0110/232505.005:VERBOSE1:label_manager.cc(77)] fill backward 0\r\n[0110/232505.006:VERBOSE1:label_manager.cc(93)] infill 0\r\n[0110/232505.143:VERBOSE1:adjustment_method_2.cc(1238)] Adjuster::Adjust\r\n[0110/232505.143:VERBOSE1:adjustment_method_2.cc(1279)] Adjuster::Solve 1.9e-05\r\n[0110/232505.143:VERBOSE1:adjustment_method_2.cc(1279)] Adjuster::Solve 2e-06\r\n[0110/232505.143:VERBOSE1:label_manager.cc(34)] 0 of 6 labels pre-assigned.\r\n[0110/232505.143:VERBOSE1:label_manager.cc(55)] fill forward 6\r\n[0110/232505.143:VERBOSE1:label_manager.cc(77)] fill backward 0\r\n[0110/232505.143:VERBOSE1:label_manager.cc(93)] infill 0\r\n[0110/232505.143:VERBOSE1:label_manager.cc(34)] 0 of 0 labels pre-assigned.\r\n[0110/232505.143:VERBOSE1:label_manager.cc(55)] fill forward 0\r\n[0110/232505.143:VERBOSE1:label_manager.cc(77)] fill backward 0\r\n[0110/232505.143:VERBOSE1:label_manager.cc(93)] infill 0\r\n[0110/232505.167:VERBOSE1:adjustment_method_2.cc(1238)] Adjuster::Adjust\r\n[0110/232505.245:VERBOSE1:adjustment_method_2.cc(1279)] Adjuster::Solve 0.075668\r\n[0110/232505.699:VERBOSE1:adjustment_method_2.cc(1279)] Adjuster::Solve 0.453243\r\n[0110/232505.703:VERBOSE1:label_manager.cc(34)] 1126 of 1130 labels pre-assigned.\r\n[0110/232505.703:VERBOSE1:label_manager.cc(55)] fill forward 0\r\n[0110/232505.703:VERBOSE1:label_manager.cc(77)] fill backward 0\r\n[0110/232505.703:VERBOSE1:label_manager.cc(93)] infill 4\r\n[0110/232505.703:VERBOSE1:label_manager.cc(34)] 2679 of 3917 labels pre-assigned.\r\n[0110/232505.703:VERBOSE1:label_manager.cc(55)] fill forward 316\r\n[0110/232505.703:VERBOSE1:label_manager.cc(77)] fill backward 37\r\n[0110/232505.703:VERBOSE1:label_manager.cc(93)] infill 885\r\n[0110/232506.104:VERBOSE1:adjustment_method_2.cc(1238)] Adjuster::Adjust\r\n[0110/232507.342:VERBOSE1:adjustment_method_2.cc(1279)] Adjuster::Solve 1.18198\r\n[0110/232517.153:VERBOSE1:adjustment_method_2.cc(1279)] Adjuster::Solve 9.79962\r\n[0110/232517.259:VERBOSE1:label_manager.cc(34)] 11336 of 12326 labels pre-assigned.\r\n[0110/232517.260:VERBOSE1:label_manager.cc(55)] fill forward 834\r\n[0110/232517.260:VERBOSE1:label_manager.cc(77)] fill backward 88\r\n[0110/232517.260:VERBOSE1:label_manager.cc(93)] infill 68\r\n[0110/232517.260:VERBOSE1:label_manager.cc(34)] 34786 of 49971 labels pre-assigned.\r\n[0110/232517.260:VERBOSE1:label_manager.cc(55)] fill forward 5625\r\n[0110/232517.260:VERBOSE1:label_manager.cc(77)] fill backward 1583\r\n[0110/232517.260:VERBOSE1:label_manager.cc(93)] infill 7977\r\n[0110/232517.528:VERBOSE1:adjustment_method_2.cc(1238)] Adjuster::Adjust\r\n[0110/232517.703:VERBOSE1:adjustment_method_2.cc(1279)] Adjuster::Solve 0.162049\r\n[0110/232520.118:VERBOSE1:adjustment_method_2.cc(1279)] Adjuster::Solve 2.41315\r\n[0110/232520.143:VERBOSE1:label_manager.cc(34)] 2176 of 2320 labels pre-assigned.\r\n[0110/232520.143:VERBOSE1:label_manager.cc(55)] fill forward 25\r\n[0110/232520.143:VERBOSE1:label_manager.cc(77)] fill backward 22\r\n[0110/232520.143:VERBOSE1:label_manager.cc(93)] infill 97\r\n[0110/232520.143:VERBOSE1:label_manager.cc(34)] 7576 of 18504 labels pre-assigned.\r\n[0110/232520.143:VERBOSE1:label_manager.cc(55)] fill forward 1532\r\n[0110/232520.144:VERBOSE1:label_manager.cc(77)] fill backward 692\r\n[0110/232520.144:VERBOSE1:label_manager.cc(93)] infill 8704\r\n[0110/232520.299:VERBOSE1:adjustment_method_2.cc(1238)] Adjuster::Adjust\r\n[0110/232520.437:VERBOSE1:adjustment_method_2.cc(1279)] Adjuster::Solve 0.124808\r\n[0110/232522.900:VERBOSE1:adjustment_method_2.cc(1279)] Adjuster::Solve 2.46218\r\n[0110/232522.927:VERBOSE1:label_manager.cc(34)] 1658 of 2130 labels pre-assigned.\r\n[0110/232522.927:VERBOSE1:label_manager.cc(55)] fill forward 112\r\n[0110/232522.927:VERBOSE1:label_manager.cc(77)] fill backward 88\r\n[0110/232522.927:VERBOSE1:label_manager.cc(93)] infill 272\r\n[0110/232522.927:VERBOSE1:label_manager.cc(34)] 7032 of 12668 labels pre-assigned.\r\n[0110/232522.927:VERBOSE1:label_manager.cc(55)] fill forward 1809\r\n[0110/232522.927:VERBOSE1:label_manager.cc(77)] fill backward 723\r\n[0110/232522.927:VERBOSE1:label_manager.cc(93)] infill 3104\r\n[0110/232522.985:VERBOSE1:adjustment_method_2.cc(1238)] Adjuster::Adjust\r\n[0110/232523.075:VERBOSE1:adjustment_method_2.cc(1279)] Adjuster::Solve 0.087164\r\n[0110/232523.553:VERBOSE1:adjustment_method_2.cc(1279)] Adjuster::Solve 0.477105\r\n[0110/232523.557:VERBOSE1:label_manager.cc(34)] 1270 of 1274 labels pre-assigned.\r\n[0110/232523.557:VERBOSE1:label_manager.cc(55)] fill forward 0\r\n[0110/232523.557:VERBOSE1:label_manager.cc(77)] fill backward 0\r\n[0110/232523.557:VERBOSE1:label_manager.cc(93)] infill 4\r\n[0110/232523.557:VERBOSE1:label_manager.cc(34)] 2677 of 3988 labels pre-assigned.\r\n[0110/232523.557:VERBOSE1:label_manager.cc(55)] fill forward 308\r\n[0110/232523.557:VERBOSE1:label_manager.cc(77)] fill backward 45\r\n[0110/232523.557:VERBOSE1:label_manager.cc(93)] infill 958\r\n[0110/232523.748:VERBOSE1:adjustment_method_2.cc(1238)] Adjuster::Adjust\r\n[0110/232523.974:VERBOSE1:adjustment_method_2.cc(1279)] Adjuster::Solve 0.204205\r\n[0110/232527.639:VERBOSE1:adjustment_method_2.cc(1279)] Adjuster::Solve 3.66371\r\n[0110/232527.681:VERBOSE1:label_manager.cc(34)] 2411 of 2824 labels pre-assigned.\r\n[0110/232527.681:VERBOSE1:label_manager.cc(55)] fill forward 325\r\n[0110/232527.681:VERBOSE1:label_manager.cc(77)] fill backward 52\r\n[0110/232527.681:VERBOSE1:label_manager.cc(93)] infill 36\r\n[0110/232527.681:VERBOSE1:label_manager.cc(34)] 13146 of 21243 labels pre-assigned.\r\n[0110/232527.681:VERBOSE1:label_manager.cc(55)] fill forward 2180\r\n[0110/232527.682:VERBOSE1:label_manager.cc(77)] fill backward 731\r\n[0110/232527.682:VERBOSE1:label_manager.cc(93)] infill 5186\r\n[0110/232528.010:VERBOSE1:adjustment_method_2.cc(1238)] Adjuster::Adjust\r\n[0110/232528.396:VERBOSE1:adjustment_method_2.cc(1279)] Adjuster::Solve 0.352781\r\n[0110/232534.176:VERBOSE1:adjustment_method_2.cc(1279)] Adjuster::Solve 5.77762\r\n[0110/232534.242:VERBOSE1:label_manager.cc(34)] 4376 of 4627 labels pre-assigned.\r\n[0110/232534.242:VERBOSE1:label_manager.cc(55)] fill forward 188\r\n[0110/232534.242:VERBOSE1:label_manager.cc(77)] fill backward 12\r\n[0110/232534.242:VERBOSE1:label_manager.cc(93)] infill 51\r\n[0110/232534.242:VERBOSE1:label_manager.cc(34)] 17717 of 30654 labels pre-assigned.\r\n[0110/232534.243:VERBOSE1:label_manager.cc(55)] fill forward 3297\r\n[0110/232534.243:VERBOSE1:label_manager.cc(77)] fill backward 1288\r\n[0110/232534.243:VERBOSE1:label_manager.cc(93)] infill 8352\r\n[0110/232534.398:VERBOSE1:adjustment_method_2.cc(1238)] Adjuster::Adjust\r\n[0110/232534.487:VERBOSE1:adjustment_method_2.cc(1279)] Adjuster::Solve 0.082827\r\n[0110/232535.451:VERBOSE1:adjustment_method_2.cc(1279)] Adjuster::Solve 0.962758\r\n[0110/232535.459:VERBOSE1:label_manager.cc(34)] 1217 of 1221 labels pre-assigned.\r\n[0110/232535.459:VERBOSE1:label_manager.cc(55)] fill forward 0\r\n[0110/232535.459:VERBOSE1:label_manager.cc(77)] fill backward 0\r\n[0110/232535.459:VERBOSE1:label_manager.cc(93)] infill 4\r\n[0110/232535.460:VERBOSE1:label_manager.cc(34)] 6975 of 8430 labels pre-assigned.\r\n[0110/232535.460:VERBOSE1:label_manager.cc(55)] fill forward 422\r\n[0110/232535.460:VERBOSE1:label_manager.cc(77)] fill backward 53\r\n[0110/232535.460:VERBOSE1:label_manager.cc(93)] infill 980\r\n[0110/232535.620:VERBOSE1:adjustment_method_2.cc(1238)] Adjuster::Adjust\r\n[0110/232535.824:VERBOSE1:adjustment_method_2.cc(1279)] Adjuster::Solve 0.186286\r\n[0110/232539.404:VERBOSE1:adjustment_method_2.cc(1279)] Adjuster::Solve 3.57838\r\n[0110/232539.442:VERBOSE1:label_manager.cc(34)] 2176 of 3062 labels pre-assigned.\r\n[0110/232539.442:VERBOSE1:label_manager.cc(55)] fill forward 65\r\n[0110/232539.442:VERBOSE1:label_manager.cc(77)] fill backward 107\r\n[0110/232539.442:VERBOSE1:label_manager.cc(93)] infill 714\r\n[0110/232539.443:VERBOSE1:label_manager.cc(34)] 10992 of 23787 labels pre-assigned.\r\n[0110/232539.443:VERBOSE1:label_manager.cc(55)] fill forward 1885\r\n[0110/232539.443:VERBOSE1:label_manager.cc(77)] fill backward 901\r\n[0110/232539.443:VERBOSE1:label_manager.cc(93)] infill 10009\r\n[0110/232539.619:VERBOSE1:adjustment_method_2.cc(1238)] Adjuster::Adjust\r\n[0110/232539.744:VERBOSE1:adjustment_method_2.cc(1279)] Adjuster::Solve 0.113667\r\n[0110/232541.725:VERBOSE1:adjustment_method_2.cc(1279)] Adjuster::Solve 1.97993\r\n[0110/232541.747:VERBOSE1:label_manager.cc(34)] 1476 of 1796 labels pre-assigned.\r\n[0110/232541.747:VERBOSE1:label_manager.cc(55)] fill forward 79\r\n[0110/232541.747:VERBOSE1:label_manager.cc(77)] fill backward 81\r\n[0110/232541.747:VERBOSE1:label_manager.cc(93)] infill 160\r\n[0110/232541.747:VERBOSE1:label_manager.cc(34)] 5673 of 9801 labels pre-assigned.\r\n[0110/232541.747:VERBOSE1:label_manager.cc(55)] fill forward 1438\r\n[0110/232541.747:VERBOSE1:label_manager.cc(77)] fill backward 622\r\n[0110/232541.747:VERBOSE1:label_manager.cc(93)] infill 2068\r\n[0110/232542.437:VERBOSE1:simple_delta.cc(35)] GenerateSimpleDelta 395024062 414164933\r\n[0110/232542.437:VERBOSE1:bsdiff_create.cc(96)] Start bsdiff\r\n[0110/232722.028:VERBOSE1:bsdiff_create.cc(123)] done divsufsort 99.5824\r\n[0110/233018.789:VERBOSE1:bsdiff_create.cc(338)] Control tuples: 5166153 copy bytes: 364423206 mistakes: 4412181 (skips: 4692989) extra bytes: 49741727\r\nUncompressed bsdiff patch size 90695777\r\nEnd bsdiff 276.352\r\n[0110/233023.369:VERBOSE1:simple_delta.cc(35)] GenerateSimpleDelta 658789123 458245733\r\n[0110/233023.369:VERBOSE1:bsdiff_create.cc(96)] Start bsdiff\r\n[0110/233023.389:ERROR:bsdiff_create.cc(115)] Could not allocate I[], 2635156496 bytes\r\n-gen failed.\r\n'

The text was updated successfully, but these errors were encountered:

simonhong · 2021-10-19T07:03:53Z

On my local Win10, I got success for delta installer between 1.33.13 and 1.33.18.
brave_installer_95_1_33_13.exe - 87MB
brave_installer_95_1_33_18.exe - 6.6MB

Below is my steps for generating delta installer for 1.33.18

Run npm run create_dist Release -- --skip_signing with 1.33.13 repo and copied out/Release/chrome.7z to out/Release/priv_installer dir
Run npm run create_dist Release -- --skip_signing --build_delta_installer --last_chrome_installer=priv_installer with 1.33.18 repo
Got 6.6MB sized brave_installer_95_1_33_18.exe and update works well from 1.33.13 to 1.33.18

Did large diff test between 1.30.89 and 1.33.18.

Installed current stable version 1.30.89 and copied its chrome.7z to priv_installer
Got 27MB brave_installer_95_1_33_18.exe
Updated from 1.30.89 to 1.33.18

simonhong · 2021-10-19T07:53:14Z

@mihaiplesa Curious what makes different result?

mihaiplesa · 2021-10-19T11:49:42Z

@simonhong how much RAM do you have?

simonhong · 2021-10-19T11:59:13Z

@mihaiplesa 64GB is installed.

mihaiplesa · 2021-11-01T15:20:39Z

@mherrmann will be looking at rebasing the Omaha client and I will re-test after.

mherrmann · 2022-01-10T17:43:22Z

@mihaiplesa the traceback you posted above is cut off. It ends with Command output:. I'd be interested in the lines following this one. You don't happen to have those any more, do you?

mihaiplesa · 2022-01-10T23:37:17Z

@mherrmann have updated the description with the expanded log.

mherrmann · 2022-01-11T12:13:13Z

Thanks! The last sentence appears interesting:

[0110/233023.389:ERROR:bsdiff_create.cc(115)] Could not allocate I[], 2635156496 bytes

Also looking at bsdiff_create.cc, it seems it is unable to allocate 2635156496 bytes (=2.45 GB) of RAM.

My hypothesis is that the build is running into some RAM limit here. It could be a limit on the subprocess, the main build process, or the build node. Since it works on @simonhong's machine, my suspicion is that it's the build node.

@mihaiplesa could it be that the build node runs out of RAM in this affected step?

mherrmann · 2022-01-11T20:08:19Z

@linhkikuchi @mihaiplesa told me you had already done some research into this issue. He also told me that every time you tried to reproduce it, the problem did not occur. Can you tell me what exactly you tried and what you learned?

Here's my current status: The bottom-level command that fails is

delta\courgette.exe -gen "delta\chrome.7z" ".\chrome.7z" ".\chrome_patch.diff"

It gets called by a build step that calls

python3.exe ../../chrome/tools/build/win/create_installer_archive.py --build_dir . --staging_dir gen/chrome/installer/mini_installer/mini_installer --input_file ../../chrome/installer/mini_installer/chrome.release --resource_file_path gen/chrome/installer/mini_installer/mini_installer/packed_files.rc --target_arch=x64 --distribution=_brave --output_dir . --chrome_runtime_deps gen/chrome_component.runtime_deps --setup_runtime_deps gen/setup.runtime_deps --skip_signing --setup_exe_format=DIFF --diff_algorithm=COURGETTE --last_chrome_installer delta --enable_hidpi=1 --depfile gen/chrome/installer/mini_installer/archive.d

So there's essentially a stack of command lines that start in Jenkins and end at courgett.exe.

My approach now would be to connect to a build node via RPM and try to invoke the above commands until one fails. So I'd start with courgette.exe then the python3.exe one, all the way to the top build command or (if necessary) Jenkins. At some point it must fail.

Have you tried this (or something similar) already? One thing that makes it hard is that the build node gets deallocated pretty quickly after the build is finished.

mherrmann · 2022-01-12T09:11:11Z

The problem is easily reproducible on a build node with the command

delta\courgette.exe -gen "delta\chrome.7z" ".\chrome.7z" ".\chrome_patch.diff"

Output

C:\ws\src\out\Release>delta\courgette.exe -gen "delta\chrome.7z" ".\chrome.7z" ".\chrome_patch.diff"
[0112/085047.034:VERBOSE1:ensemble_create.cc(237)] start GenerateEnsemblePatch
[0112/085048.879:VERBOSE1:ensemble_create.cc(138)] done FindEmbeddedElements 1.83882
[0112/085048.879:VERBOSE1:ensemble_create.cc(144)] old has 15 elements
[0112/085048.879:VERBOSE1:ensemble_create.cc(145)] new has 16 elements
[0112/085118.362:VERBOSE1:ensemble_create.cc(155)] done make bases 29.4778s
[0112/085132.113:VERBOSE1:ensemble_create.cc(189)] Compare old(4,18103493,174511104) to new(4,32,174511104) --> 0 in 13.752s
[0112/085132.113:VERBOSE1:ensemble_create.cc(195)] Skip new(4,32,174511104) - identical to old(4,18103493,174511104)
[0112/085146.280:VERBOSE1:ensemble_create.cc(189)] Compare old(4,18103493,174511104) to new(4,192591795,174523904) --> 14247458 in 14.1674s
[0112/085146.280:VERBOSE1:ensemble_create.cc(208)] Matched old(4,18103493,174511104) to new(4,192591795,174523904) --> 14247458
[0112/085146.343:VERBOSE1:ensemble_create.cc(189)] Compare old(4,194160131,1080832) to new(4,368653215,1080832) --> 33 in 0.059038s
[0112/085146.405:VERBOSE1:ensemble_create.cc(189)] Compare old(4,195247563,1463296) to new(4,368653215,1080832) --> 191491 in 0.063958s
[0112/085146.436:VERBOSE1:ensemble_create.cc(189)] Compare old(4,201239899,5632) to new(4,368653215,1080832) --> 1002094 in 0.030919s
[0112/085146.483:VERBOSE1:ensemble_create.cc(189)] Compare old(4,211469606,407040) to new(4,368653215,1080832) --> 461473 in 0.052169s
[0112/085146.546:VERBOSE1:ensemble_create.cc(189)] Compare old(4,243663816,1337344) to new(4,368653215,1080832) --> 160504 in 0.057085s
[0112/085146.608:VERBOSE1:ensemble_create.cc(189)] Compare old(4,245007760,1088000) to new(4,368653215,1080832) --> 144923 in 0.061863s
[0112/085146.655:VERBOSE1:ensemble_create.cc(189)] Compare old(4,254867627,417792) to new(4,368653215,1080832) --> 457146 in 0.053631s
[0112/085146.733:VERBOSE1:ensemble_create.cc(189)] Compare old(4,255292019,2725376) to new(4,368653215,1080832) --> 336205 in 0.080434s
[0112/085146.796:VERBOSE1:ensemble_create.cc(189)] Compare old(4,262097989,705024) to new(4,368653215,1080832) --> 424053 in 0.054804s
[0112/085146.874:VERBOSE1:ensemble_create.cc(189)] Compare old(4,262809613,2388480) to new(4,368653215,1080832) --> 188594 in 0.079114s
[0112/085146.936:VERBOSE1:ensemble_create.cc(189)] Compare old(4,265204693,876544) to new(4,368653215,1080832) --> 197438 in 0.057813s
[0112/085146.936:VERBOSE1:ensemble_create.cc(208)] Matched old(4,194160131,1080832) to new(4,368653215,1080832) --> 33
[0112/085146.999:VERBOSE1:ensemble_create.cc(189)] Compare old(4,194160131,1080832) to new(4,369734047,1463296) --> 471099 in 0.073594s
[0112/085147.061:VERBOSE1:ensemble_create.cc(189)] Compare old(4,195247563,1463296) to new(4,369734047,1463296) --> 41 in 0.06151s
[0112/085147.108:VERBOSE1:ensemble_create.cc(189)] Compare old(4,201239899,5632) to new(4,369734047,1463296) --> 1376021 in 0.038407s
[0112/085147.171:VERBOSE1:ensemble_create.cc(189)] Compare old(4,211469606,407040) to new(4,369734047,1463296) --> 816281 in 0.059696s
[0112/085147.233:VERBOSE1:ensemble_create.cc(189)] Compare old(4,243663816,1337344) to new(4,369734047,1463296) --> 470793 in 0.065019s
[0112/085147.296:VERBOSE1:ensemble_create.cc(189)] Compare old(4,245007760,1088000) to new(4,369734047,1463296) --> 462932 in 0.071283s
[0112/085147.358:VERBOSE1:ensemble_create.cc(189)] Compare old(4,254867627,417792) to new(4,369734047,1463296) --> 814545 in 0.060635s
[0112/085147.452:VERBOSE1:ensemble_create.cc(189)] Compare old(4,255292019,2725376) to new(4,369734047,1463296) --> 629458 in 0.087231s
[0112/085147.514:VERBOSE1:ensemble_create.cc(189)] Compare old(4,262097989,705024) to new(4,369734047,1463296) --> 745754 in 0.061212s
[0112/085147.593:VERBOSE1:ensemble_create.cc(189)] Compare old(4,262809613,2388480) to new(4,369734047,1463296) --> 240562 in 0.08255s
[0112/085147.655:VERBOSE1:ensemble_create.cc(189)] Compare old(4,265204693,876544) to new(4,369734047,1463296) --> 470864 in 0.064053s
[0112/085147.655:VERBOSE1:ensemble_create.cc(208)] Matched old(4,195247563,1463296) to new(4,369734047,1463296) --> 41
[0112/085147.906:VERBOSE1:ensemble_create.cc(189)] Compare old(4,196717459,4515840) to new(4,371197343,4515840) --> 9 in 0.249805s
[0112/085148.250:VERBOSE1:ensemble_create.cc(189)] Compare old(4,211883246,6210560) to new(4,371197343,4515840) --> 1860603 in 0.340529s
[0112/085148.531:VERBOSE1:ensemble_create.cc(189)] Compare old(4,255292019,2725376) to new(4,371197343,4515840) --> 2085086 in 0.281598s
[0112/085148.781:VERBOSE1:ensemble_create.cc(189)] Compare old(4,258222611,3868672) to new(4,371197343,4515840) --> 2029770 in 0.248003s
[0112/085149.063:VERBOSE1:ensemble_create.cc(189)] Compare old(4,262809613,2388480) to new(4,371197343,4515840) --> 2145192 in 0.288521s
[0112/085149.063:VERBOSE1:ensemble_create.cc(208)] Matched old(4,196717459,4515840) to new(4,371197343,4515840) --> 9
[0112/085149.063:VERBOSE1:ensemble_create.cc(189)] Compare old(4,194160131,1080832) to new(4,375721839,5632) --> 213 in 0.000276s
[0112/085149.063:VERBOSE1:ensemble_create.cc(189)] Compare old(4,195247563,1463296) to new(4,375721839,5632) --> 206 in 0.000263s
[0112/085149.063:VERBOSE1:ensemble_create.cc(189)] Compare old(4,201239899,5632) to new(4,375721839,5632) --> 36 in 8.7e-05s
[0112/085149.063:VERBOSE1:ensemble_create.cc(189)] Compare old(4,211469606,407040) to new(4,375721839,5632) --> 502 in 0.000249s
[0112/085149.078:VERBOSE1:ensemble_create.cc(189)] Compare old(4,243663816,1337344) to new(4,375721839,5632) --> 229 in 0.000249s
[0112/085149.078:VERBOSE1:ensemble_create.cc(189)] Compare old(4,245007760,1088000) to new(4,375721839,5632) --> 227 in 0.000261s
[0112/085149.078:VERBOSE1:ensemble_create.cc(189)] Compare old(4,254867627,417792) to new(4,375721839,5632) --> 492 in 0.000244s
[0112/085149.078:VERBOSE1:ensemble_create.cc(189)] Compare old(4,262097989,705024) to new(4,375721839,5632) --> 501 in 0.000252s
[0112/085149.078:VERBOSE1:ensemble_create.cc(189)] Compare old(4,265204693,876544) to new(4,375721839,5632) --> 247 in 0.000266s
[0112/085149.078:VERBOSE1:ensemble_create.cc(208)] Matched old(4,201239899,5632) to new(4,375721839,5632) --> 36
[0112/085149.094:VERBOSE1:ensemble_create.cc(189)] Compare old(4,194160131,1080832) to new(4,385944946,407040) --> 65423 in 0.028275s
[0112/085149.125:VERBOSE1:ensemble_create.cc(189)] Compare old(4,195247563,1463296) to new(4,385944946,407040) --> 69814 in 0.027475s
[0112/085149.141:VERBOSE1:ensemble_create.cc(189)] Compare old(4,201239899,5632) to new(4,385944946,407040) --> 365772 in 0.011339s
[0112/085149.156:VERBOSE1:ensemble_create.cc(189)] Compare old(4,211469606,407040) to new(4,385944946,407040) --> 12 in 0.018012s
[0112/085149.188:VERBOSE1:ensemble_create.cc(189)] Compare old(4,243663816,1337344) to new(4,385944946,407040) --> 64638 in 0.026799s
[0112/085149.219:VERBOSE1:ensemble_create.cc(189)] Compare old(4,245007760,1088000) to new(4,385944946,407040) --> 68764 in 0.026983s
[0112/085149.235:VERBOSE1:ensemble_create.cc(189)] Compare old(4,254867627,417792) to new(4,385944946,407040) --> 67134 in 0.020439s
[0112/085149.250:VERBOSE1:ensemble_create.cc(189)] Compare old(4,262097989,705024) to new(4,385944946,407040) --> 67764 in 0.023345s
[0112/085149.297:VERBOSE1:ensemble_create.cc(189)] Compare old(4,262809613,2388480) to new(4,385944946,407040) --> 76225 in 0.033702s
[0112/085149.313:VERBOSE1:ensemble_create.cc(189)] Compare old(4,265204693,876544) to new(4,385944946,407040) --> 69188 in 0.025011s
[0112/085149.313:VERBOSE1:ensemble_create.cc(208)] Matched old(4,211469606,407040) to new(4,385944946,407040) --> 12
[0112/085149.703:VERBOSE1:ensemble_create.cc(189)] Compare old(4,196717459,4515840) to new(4,386351986,6210560) --> 3073987 in 0.38233s
[0112/085150.047:VERBOSE1:ensemble_create.cc(189)] Compare old(4,211883246,6210560) to new(4,386351986,6210560) --> 105655 in 0.348597s
[0112/085150.345:VERBOSE1:ensemble_create.cc(189)] Compare old(4,258222611,3868672) to new(4,386351986,6210560) --> 2070104 in 0.30335s
[0112/085150.345:VERBOSE1:ensemble_create.cc(208)] Matched old(4,211883246,6210560) to new(4,386351986,6210560) --> 105655
[0112/085150.438:VERBOSE1:ensemble_create.cc(189)] Compare old(4,194160131,1080832) to new(4,418158925,1337344) --> 248010 in 0.08503s
[0112/085150.516:VERBOSE1:ensemble_create.cc(189)] Compare old(4,195247563,1463296) to new(4,418158925,1337344) --> 281861 in 0.079894s
[0112/085150.548:VERBOSE1:ensemble_create.cc(189)] Compare old(4,201239899,5632) to new(4,418158925,1337344) --> 1251382 in 0.03849s
[0112/085150.626:VERBOSE1:ensemble_create.cc(189)] Compare old(4,211469606,407040) to new(4,418158925,1337344) --> 596202 in 0.064894s
[0112/085150.689:VERBOSE1:ensemble_create.cc(189)] Compare old(4,243663816,1337344) to new(4,418158925,1337344) --> 35 in 0.062464s
[0112/085150.767:VERBOSE1:ensemble_create.cc(189)] Compare old(4,245007760,1088000) to new(4,418158925,1337344) --> 249671 in 0.07885s
[0112/085150.830:VERBOSE1:ensemble_create.cc(189)] Compare old(4,254867627,417792) to new(4,418158925,1337344) --> 586002 in 0.066225s
[0112/085150.923:VERBOSE1:ensemble_create.cc(189)] Compare old(4,255292019,2725376) to new(4,418158925,1337344) --> 419533 in 0.099272s
[0112/085151.001:VERBOSE1:ensemble_create.cc(189)] Compare old(4,262097989,705024) to new(4,418158925,1337344) --> 546125 in 0.067544s
[0112/085151.095:VERBOSE1:ensemble_create.cc(189)] Compare old(4,262809613,2388480) to new(4,418158925,1337344) --> 255618 in 0.097641s
[0112/085151.158:VERBOSE1:ensemble_create.cc(189)] Compare old(4,265204693,876544) to new(4,418158925,1337344) --> 285175 in 0.071474s
[0112/085151.158:VERBOSE1:ensemble_create.cc(208)] Matched old(4,243663816,1337344) to new(4,418158925,1337344) --> 35
[0112/085151.236:VERBOSE1:ensemble_create.cc(189)] Compare old(4,194160131,1080832) to new(4,419496269,1088000) --> 145166 in 0.064224s
[0112/085151.298:VERBOSE1:ensemble_create.cc(189)] Compare old(4,195247563,1463296) to new(4,419496269,1088000) --> 183987 in 0.064397s
[0112/085151.330:VERBOSE1:ensemble_create.cc(189)] Compare old(4,201239899,5632) to new(4,419496269,1088000) --> 1011733 in 0.031253s
[0112/085151.376:VERBOSE1:ensemble_create.cc(189)] Compare old(4,211469606,407040) to new(4,419496269,1088000) --> 467470 in 0.052701s
[0112/085151.439:VERBOSE1:ensemble_create.cc(189)] Compare old(4,243663816,1337344) to new(4,419496269,1088000) --> 163965 in 0.057592s
[0112/085151.486:VERBOSE1:ensemble_create.cc(189)] Compare old(4,245007760,1088000) to new(4,419496269,1088000) --> 34 in 0.056752s
[0112/085151.548:VERBOSE1:ensemble_create.cc(189)] Compare old(4,254867627,417792) to new(4,419496269,1088000) --> 462821 in 0.05377s
[0112/085151.626:VERBOSE1:ensemble_create.cc(189)] Compare old(4,255292019,2725376) to new(4,419496269,1088000) --> 343052 in 0.081959s
[0112/085151.689:VERBOSE1:ensemble_create.cc(189)] Compare old(4,262097989,705024) to new(4,419496269,1088000) --> 429856 in 0.055476s
[0112/085151.767:VERBOSE1:ensemble_create.cc(189)] Compare old(4,262809613,2388480) to new(4,419496269,1088000) --> 183618 in 0.081309s
[0112/085151.830:VERBOSE1:ensemble_create.cc(189)] Compare old(4,265204693,876544) to new(4,419496269,1088000) --> 187307 in 0.057248s
[0112/085151.830:VERBOSE1:ensemble_create.cc(208)] Matched old(4,245007760,1088000) to new(4,419496269,1088000) --> 34
[0112/085151.845:VERBOSE1:ensemble_create.cc(189)] Compare old(4,194160131,1080832) to new(4,429349580,417792) --> 71603 in 0.0286s
[0112/085151.876:VERBOSE1:ensemble_create.cc(189)] Compare old(4,195247563,1463296) to new(4,429349580,417792) --> 76947 in 0.028986s
[0112/085151.892:VERBOSE1:ensemble_create.cc(189)] Compare old(4,201239899,5632) to new(4,429349580,417792) --> 375119 in 0.011694s
[0112/085151.908:VERBOSE1:ensemble_create.cc(189)] Compare old(4,211469606,407040) to new(4,429349580,417792) --> 72888 in 0.020718s
[0112/085151.939:VERBOSE1:ensemble_create.cc(189)] Compare old(4,243663816,1337344) to new(4,429349580,417792) --> 70069 in 0.027102s
[0112/085151.970:VERBOSE1:ensemble_create.cc(189)] Compare old(4,245007760,1088000) to new(4,429349580,417792) --> 74976 in 0.028384s
[0112/085151.986:VERBOSE1:ensemble_create.cc(189)] Compare old(4,254867627,417792) to new(4,429349580,417792) --> 12 in 0.01878s
[0112/085152.017:VERBOSE1:ensemble_create.cc(189)] Compare old(4,262097989,705024) to new(4,429349580,417792) --> 73132 in 0.023996s
[0112/085152.048:VERBOSE1:ensemble_create.cc(189)] Compare old(4,262809613,2388480) to new(4,429349580,417792) --> 81538 in 0.035292s
[0112/085152.080:VERBOSE1:ensemble_create.cc(189)] Compare old(4,265204693,876544) to new(4,429349580,417792) --> 76809 in 0.02613s
[0112/085152.080:VERBOSE1:ensemble_create.cc(208)] Matched old(4,254867627,417792) to new(4,429349580,417792) --> 12
[0112/085152.236:VERBOSE1:ensemble_create.cc(189)] Compare old(4,194160131,1080832) to new(4,429767372,2725376) --> 1054584 in 0.165386s
[0112/085152.392:VERBOSE1:ensemble_create.cc(189)] Compare old(4,195247563,1463296) to new(4,429767372,2725376) --> 1093279 in 0.158075s
[0112/085152.580:VERBOSE1:ensemble_create.cc(189)] Compare old(4,196717459,4515840) to new(4,429767372,2725376) --> 1388415 in 0.186922s
[0112/085152.721:VERBOSE1:ensemble_create.cc(189)] Compare old(4,243663816,1337344) to new(4,429767372,2725376) --> 1011829 in 0.129801s
[0112/085152.877:VERBOSE1:ensemble_create.cc(189)] Compare old(4,245007760,1088000) to new(4,429767372,2725376) --> 1058805 in 0.156811s
[0112/085153.018:VERBOSE1:ensemble_create.cc(189)] Compare old(4,255292019,2725376) to new(4,429767372,2725376) --> 12 in 0.146639s
[0112/085153.159:VERBOSE1:ensemble_create.cc(189)] Compare old(4,258222611,3868672) to new(4,429767372,2725376) --> 643251 in 0.146762s
[0112/085153.284:VERBOSE1:ensemble_create.cc(189)] Compare old(4,262097989,705024) to new(4,429767372,2725376) --> 1227375 in 0.124533s
[0112/085153.471:VERBOSE1:ensemble_create.cc(189)] Compare old(4,262809613,2388480) to new(4,429767372,2725376) --> 906743 in 0.180496s
[0112/085153.612:VERBOSE1:ensemble_create.cc(189)] Compare old(4,265204693,876544) to new(4,429767372,2725376) --> 1129767 in 0.141529s
[0112/085153.612:VERBOSE1:ensemble_create.cc(208)] Matched old(4,255292019,2725376) to new(4,429767372,2725376) --> 12
[0112/085153.862:VERBOSE1:ensemble_create.cc(189)] Compare old(4,196717459,4515840) to new(4,432691364,3868672) --> 1776579 in 0.252983s
[0112/085154.128:VERBOSE1:ensemble_create.cc(189)] Compare old(4,211883246,6210560) to new(4,432691364,3868672) --> 875385 in 0.265904s
[0112/085154.362:VERBOSE1:ensemble_create.cc(189)] Compare old(4,255292019,2725376) to new(4,432691364,3868672) --> 889850 in 0.224446s
[0112/085154.534:VERBOSE1:ensemble_create.cc(189)] Compare old(4,258222611,3868672) to new(4,432691364,3868672) --> 12 in 0.169791s
[0112/085154.768:VERBOSE1:ensemble_create.cc(189)] Compare old(4,262809613,2388480) to new(4,432691364,3868672) --> 1139354 in 0.233873s
[0112/085154.768:VERBOSE1:ensemble_create.cc(208)] Matched old(4,258222611,3868672) to new(4,432691364,3868672) --> 12
[0112/085154.815:VERBOSE1:ensemble_create.cc(189)] Compare old(4,194160131,1080832) to new(4,436560142,705024) --> 198894 in 0.04757s
[0112/085154.862:VERBOSE1:ensemble_create.cc(189)] Compare old(4,195247563,1463296) to new(4,436560142,705024) --> 201580 in 0.044873s
[0112/085154.878:VERBOSE1:ensemble_create.cc(189)] Compare old(4,201239899,5632) to new(4,436560142,705024) --> 650364 in 0.019654s
[0112/085154.909:VERBOSE1:ensemble_create.cc(189)] Compare old(4,211469606,407040) to new(4,436560142,705024) --> 240201 in 0.033573s
[0112/085154.956:VERBOSE1:ensemble_create.cc(189)] Compare old(4,243663816,1337344) to new(4,436560142,705024) --> 195322 in 0.042171s
[0112/085155.003:VERBOSE1:ensemble_create.cc(189)] Compare old(4,245007760,1088000) to new(4,436560142,705024) --> 202886 in 0.044485s
[0112/085155.035:VERBOSE1:ensemble_create.cc(189)] Compare old(4,254867627,417792) to new(4,436560142,705024) --> 240372 in 0.034125s
[0112/085155.081:VERBOSE1:ensemble_create.cc(189)] Compare old(4,255292019,2725376) to new(4,436560142,705024) --> 180070 in 0.052452s
[0112/085155.113:VERBOSE1:ensemble_create.cc(189)] Compare old(4,262097989,705024) to new(4,436560142,705024) --> 12 in 0.030529s
[0112/085155.175:VERBOSE1:ensemble_create.cc(189)] Compare old(4,262809613,2388480) to new(4,436560142,705024) --> 192001 in 0.056092s
[0112/085155.206:VERBOSE1:ensemble_create.cc(189)] Compare old(4,265204693,876544) to new(4,436560142,705024) --> 212333 in 0.040657s
[0112/085155.206:VERBOSE1:ensemble_create.cc(208)] Matched old(4,262097989,705024) to new(4,436560142,705024) --> 12
[0112/085155.348:VERBOSE1:ensemble_create.cc(189)] Compare old(4,194160131,1080832) to new(4,437265166,2388480) --> 916952 in 0.142086s
[0112/085155.488:VERBOSE1:ensemble_create.cc(189)] Compare old(4,195247563,1463296) to new(4,437265166,2388480) --> 804618 in 0.134415s
[0112/085155.660:VERBOSE1:ensemble_create.cc(189)] Compare old(4,196717459,4515840) to new(4,437265166,2388480) --> 1294345 in 0.171235s
[0112/085155.770:VERBOSE1:ensemble_create.cc(189)] Compare old(4,211469606,407040) to new(4,437265166,2388480) --> 1362673 in 0.111941s
[0112/085155.879:VERBOSE1:ensemble_create.cc(189)] Compare old(4,243663816,1337344) to new(4,437265166,2388480) --> 893623 in 0.113639s
[0112/085156.020:VERBOSE1:ensemble_create.cc(189)] Compare old(4,245007760,1088000) to new(4,437265166,2388480) --> 912110 in 0.134968s
[0112/085156.129:VERBOSE1:ensemble_create.cc(189)] Compare old(4,254867627,417792) to new(4,437265166,2388480) --> 1346722 in 0.11377s
[0112/085156.301:VERBOSE1:ensemble_create.cc(189)] Compare old(4,255292019,2725376) to new(4,437265166,2388480) --> 1027873 in 0.161482s
[0112/085156.442:VERBOSE1:ensemble_create.cc(189)] Compare old(4,258222611,3868672) to new(4,437265166,2388480) --> 992021 in 0.141845s
[0112/085156.551:VERBOSE1:ensemble_create.cc(189)] Compare old(4,262097989,705024) to new(4,437265166,2388480) --> 1280650 in 0.11269s
[0112/085156.676:VERBOSE1:ensemble_create.cc(189)] Compare old(4,262809613,2388480) to new(4,437265166,2388480) --> 113 in 0.129562s
[0112/085156.801:VERBOSE1:ensemble_create.cc(189)] Compare old(4,265204693,876544) to new(4,437265166,2388480) --> 975001 in 0.122961s
[0112/085156.801:VERBOSE1:ensemble_create.cc(208)] Matched old(4,262809613,2388480) to new(4,437265166,2388480) --> 113
[0112/085156.863:VERBOSE1:ensemble_create.cc(189)] Compare old(4,194160131,1080832) to new(4,439653646,876544) --> 113444 in 0.053229s
[0112/085156.910:VERBOSE1:ensemble_create.cc(189)] Compare old(4,195247563,1463296) to new(4,439653646,876544) --> 106318 in 0.051688s
[0112/085156.942:VERBOSE1:ensemble_create.cc(189)] Compare old(4,201239899,5632) to new(4,439653646,876544) --> 815457 in 0.024977s
[0112/085156.973:VERBOSE1:ensemble_create.cc(189)] Compare old(4,211469606,407040) to new(4,439653646,876544) --> 359368 in 0.042346s
[0112/085157.020:VERBOSE1:ensemble_create.cc(189)] Compare old(4,243663816,1337344) to new(4,439653646,876544) --> 110953 in 0.048885s
[0112/085157.082:VERBOSE1:ensemble_create.cc(189)] Compare old(4,245007760,1088000) to new(4,439653646,876544) --> 102957 in 0.050778s
[0112/085157.129:VERBOSE1:ensemble_create.cc(189)] Compare old(4,254867627,417792) to new(4,439653646,876544) --> 357474 in 0.043383s
[0112/085157.192:VERBOSE1:ensemble_create.cc(189)] Compare old(4,255292019,2725376) to new(4,439653646,876544) --> 274046 in 0.066673s
[0112/085157.238:VERBOSE1:ensemble_create.cc(189)] Compare old(4,262097989,705024) to new(4,439653646,876544) --> 334565 in 0.045525s
[0112/085157.301:VERBOSE1:ensemble_create.cc(189)] Compare old(4,262809613,2388480) to new(4,439653646,876544) --> 139371 in 0.065888s
[0112/085157.348:VERBOSE1:ensemble_create.cc(189)] Compare old(4,265204693,876544) to new(4,439653646,876544) --> 49 in 0.042251s
[0112/085157.348:VERBOSE1:ensemble_create.cc(208)] Matched old(4,265204693,876544) to new(4,439653646,876544) --> 49
[0112/085157.348:VERBOSE1:ensemble_create.cc(218)] done FindGenerators found 15 in 70.3042s
[0112/085200.382:VERBOSE1:simple_delta.cc(35)] GenerateSimpleDelta 30 30
[0112/085200.382:VERBOSE1:bsdiff_create.cc(96)] Start bsdiff
[0112/085200.382:VERBOSE1:bsdiff_create.cc(123)]  done divsufsort 0.000405
[0112/085200.382:VERBOSE1:bsdiff_create.cc(338)] Control tuples: 1  copy bytes: 30  mistakes: 0  (skips: 1)  extra bytes: 0
Uncompressed bsdiff patch size 34
End bsdiff 0.00147
[0112/085211.747:VERBOSE1:adjustment_method_2.cc(1238)] Adjuster::Adjust
[0112/085243.589:VERBOSE1:adjustment_method_2.cc(1279)]  Adjuster::Solve 29.6034
[0112/085928.718:VERBOSE1:adjustment_method_2.cc(1279)]  Adjuster::Solve 404.829
[0112/085931.754:VERBOSE1:label_manager.cc(34)] 259283 of 275297 labels pre-assigned.
[0112/085931.754:VERBOSE1:label_manager.cc(55)]   fill forward 15775
[0112/085931.754:VERBOSE1:label_manager.cc(77)]   fill backward 79
[0112/085931.754:VERBOSE1:label_manager.cc(93)]   infill 160
[0112/085931.754:VERBOSE1:label_manager.cc(34)] 1271710 of 1307837 labels pre-assigned.
[0112/085931.754:VERBOSE1:label_manager.cc(55)]   fill forward 30840
[0112/085931.754:VERBOSE1:label_manager.cc(77)]   fill backward 2014
[0112/085931.770:VERBOSE1:label_manager.cc(93)]   infill 3273
[0112/085936.649:VERBOSE1:adjustment_method_2.cc(1238)] Adjuster::Adjust
[0112/085936.805:VERBOSE1:adjustment_method_2.cc(1279)]  Adjuster::Solve 0.13896
[0112/085938.091:VERBOSE1:adjustment_method_2.cc(1279)]  Adjuster::Solve 1.28634
[0112/085938.091:VERBOSE1:label_manager.cc(34)] 2149 of 2149 labels pre-assigned.
[0112/085938.091:VERBOSE1:label_manager.cc(55)]   fill forward 0
[0112/085938.091:VERBOSE1:label_manager.cc(77)]   fill backward 0
[0112/085938.106:VERBOSE1:label_manager.cc(93)]   infill 0
[0112/085938.106:VERBOSE1:label_manager.cc(34)] 8581 of 8707 labels pre-assigned.
[0112/085938.106:VERBOSE1:label_manager.cc(55)]   fill forward 126
[0112/085938.106:VERBOSE1:label_manager.cc(77)]   fill backward 0
[0112/085938.106:VERBOSE1:label_manager.cc(93)]   infill 0
[0112/085938.200:VERBOSE1:adjustment_method_2.cc(1238)] Adjuster::Adjust
[0112/085938.320:VERBOSE1:adjustment_method_2.cc(1279)]  Adjuster::Solve 0.118929
[0112/085939.408:VERBOSE1:adjustment_method_2.cc(1279)]  Adjuster::Solve 1.0809
[0112/085939.408:VERBOSE1:label_manager.cc(34)] 1909 of 1909 labels pre-assigned.
[0112/085939.408:VERBOSE1:label_manager.cc(55)]   fill forward 0
[0112/085939.408:VERBOSE1:label_manager.cc(77)]   fill backward 0
[0112/085939.408:VERBOSE1:label_manager.cc(93)]   infill 0
[0112/085939.408:VERBOSE1:label_manager.cc(34)] 7805 of 7918 labels pre-assigned.
[0112/085939.408:VERBOSE1:label_manager.cc(55)]   fill forward 113
[0112/085939.408:VERBOSE1:label_manager.cc(77)]   fill backward 0
[0112/085939.408:VERBOSE1:label_manager.cc(93)]   infill 0
[0112/085939.684:VERBOSE1:adjustment_method_2.cc(1238)] Adjuster::Adjust
[0112/085940.309:VERBOSE1:adjustment_method_2.cc(1279)]  Adjuster::Solve 0.590196
[0112/085945.361:VERBOSE1:adjustment_method_2.cc(1279)]  Adjuster::Solve 5.05168
[0112/085945.392:VERBOSE1:label_manager.cc(34)] 7555 of 7903 labels pre-assigned.
[0112/085945.392:VERBOSE1:label_manager.cc(55)]   fill forward 348
[0112/085945.408:VERBOSE1:label_manager.cc(77)]   fill backward 0
[0112/085945.408:VERBOSE1:label_manager.cc(93)]   infill 0
[0112/085945.408:VERBOSE1:label_manager.cc(34)] 30344 of 30760 labels pre-assigned.
[0112/085945.408:VERBOSE1:label_manager.cc(55)]   fill forward 416
[0112/085945.408:VERBOSE1:label_manager.cc(77)]   fill backward 0
[0112/085945.408:VERBOSE1:label_manager.cc(93)]   infill 0
[0112/085945.501:VERBOSE1:adjustment_method_2.cc(1238)] Adjuster::Adjust
[0112/085945.501:VERBOSE1:adjustment_method_2.cc(1279)]  Adjuster::Solve 1.7e-05
[0112/085945.501:VERBOSE1:adjustment_method_2.cc(1279)]  Adjuster::Solve 3e-06
[0112/085945.501:VERBOSE1:label_manager.cc(34)] 0 of 6 labels pre-assigned.
[0112/085945.501:VERBOSE1:label_manager.cc(55)]   fill forward 6
[0112/085945.501:VERBOSE1:label_manager.cc(77)]   fill backward 0
[0112/085945.501:VERBOSE1:label_manager.cc(93)]   infill 0
[0112/085945.501:VERBOSE1:label_manager.cc(34)] 0 of 0 labels pre-assigned.
[0112/085945.501:VERBOSE1:label_manager.cc(55)]   fill forward 0
[0112/085945.501:VERBOSE1:label_manager.cc(77)]   fill backward 0
[0112/085945.501:VERBOSE1:label_manager.cc(93)]   infill 0
[0112/085945.533:VERBOSE1:adjustment_method_2.cc(1238)] Adjuster::Adjust
[0112/085945.611:VERBOSE1:adjustment_method_2.cc(1279)]  Adjuster::Solve 0.070751
[0112/085946.017:VERBOSE1:adjustment_method_2.cc(1279)]  Adjuster::Solve 0.408159
[0112/085946.017:VERBOSE1:label_manager.cc(34)] 1126 of 1126 labels pre-assigned.
[0112/085946.017:VERBOSE1:label_manager.cc(55)]   fill forward 0
[0112/085946.017:VERBOSE1:label_manager.cc(77)]   fill backward 0
[0112/085946.017:VERBOSE1:label_manager.cc(93)]   infill 0
[0112/085946.017:VERBOSE1:label_manager.cc(34)] 3085 of 3168 labels pre-assigned.
[0112/085946.017:VERBOSE1:label_manager.cc(55)]   fill forward 83
[0112/085946.017:VERBOSE1:label_manager.cc(77)]   fill backward 0
[0112/085946.017:VERBOSE1:label_manager.cc(93)]   infill 0
[0112/085946.377:VERBOSE1:adjustment_method_2.cc(1238)] Adjuster::Adjust
[0112/085947.548:VERBOSE1:adjustment_method_2.cc(1279)]  Adjuster::Solve 1.12772
[0112/085955.632:VERBOSE1:adjustment_method_2.cc(1279)]  Adjuster::Solve 8.07576
[0112/085955.711:VERBOSE1:label_manager.cc(34)] 12168 of 12318 labels pre-assigned.
[0112/085955.711:VERBOSE1:label_manager.cc(55)]   fill forward 150
[0112/085955.711:VERBOSE1:label_manager.cc(77)]   fill backward 0
[0112/085955.711:VERBOSE1:label_manager.cc(93)]   infill 0
[0112/085955.711:VERBOSE1:label_manager.cc(34)] 44094 of 46380 labels pre-assigned.
[0112/085955.711:VERBOSE1:label_manager.cc(55)]   fill forward 2285
[0112/085955.711:VERBOSE1:label_manager.cc(77)]   fill backward 0
[0112/085955.711:VERBOSE1:label_manager.cc(93)]   infill 1
[0112/085955.946:VERBOSE1:adjustment_method_2.cc(1238)] Adjuster::Adjust
[0112/085956.102:VERBOSE1:adjustment_method_2.cc(1279)]  Adjuster::Solve 0.146064
[0112/085957.888:VERBOSE1:adjustment_method_2.cc(1279)]  Adjuster::Solve 1.77674
[0112/085957.888:VERBOSE1:label_manager.cc(34)] 2259 of 2259 labels pre-assigned.
[0112/085957.888:VERBOSE1:label_manager.cc(55)]   fill forward 0
[0112/085957.888:VERBOSE1:label_manager.cc(77)]   fill backward 0
[0112/085957.888:VERBOSE1:label_manager.cc(93)]   infill 0
[0112/085957.888:VERBOSE1:label_manager.cc(34)] 10941 of 11118 labels pre-assigned.
[0112/085957.888:VERBOSE1:label_manager.cc(55)]   fill forward 177
[0112/085957.888:VERBOSE1:label_manager.cc(77)]   fill backward 0
[0112/085957.888:VERBOSE1:label_manager.cc(93)]   infill 0
[0112/085957.981:VERBOSE1:adjustment_method_2.cc(1238)] Adjuster::Adjust
[0112/085958.138:VERBOSE1:adjustment_method_2.cc(1279)]  Adjuster::Solve 0.137212
[0112/085959.436:VERBOSE1:adjustment_method_2.cc(1279)]  Adjuster::Solve 1.30545
[0112/085959.451:VERBOSE1:label_manager.cc(34)] 2106 of 2106 labels pre-assigned.
[0112/085959.451:VERBOSE1:label_manager.cc(55)]   fill forward 0
[0112/085959.451:VERBOSE1:label_manager.cc(77)]   fill backward 0
[0112/085959.451:VERBOSE1:label_manager.cc(93)]   infill 0
[0112/085959.451:VERBOSE1:label_manager.cc(34)] 8723 of 8844 labels pre-assigned.
[0112/085959.451:VERBOSE1:label_manager.cc(55)]   fill forward 121
[0112/085959.451:VERBOSE1:label_manager.cc(77)]   fill backward 0
[0112/085959.451:VERBOSE1:label_manager.cc(93)]   infill 0
[0112/085959.498:VERBOSE1:adjustment_method_2.cc(1238)] Adjuster::Adjust
[0112/085959.576:VERBOSE1:adjustment_method_2.cc(1279)]  Adjuster::Solve 0.082041
[0112/085959.998:VERBOSE1:adjustment_method_2.cc(1279)]  Adjuster::Solve 0.416222
[0112/085959.998:VERBOSE1:label_manager.cc(34)] 1271 of 1271 labels pre-assigned.
[0112/085959.998:VERBOSE1:label_manager.cc(55)]   fill forward 0
[0112/085959.998:VERBOSE1:label_manager.cc(77)]   fill backward 0
[0112/085959.998:VERBOSE1:label_manager.cc(93)]   infill 0
[0112/085959.998:VERBOSE1:label_manager.cc(34)] 3098 of 3181 labels pre-assigned.
[0112/085959.998:VERBOSE1:label_manager.cc(55)]   fill forward 83
[0112/085959.998:VERBOSE1:label_manager.cc(77)]   fill backward 0
[0112/085959.998:VERBOSE1:label_manager.cc(93)]   infill 0
[0112/090000.170:VERBOSE1:adjustment_method_2.cc(1238)] Adjuster::Adjust
[0112/090000.389:VERBOSE1:adjustment_method_2.cc(1279)]  Adjuster::Solve 0.199454
[0112/090003.470:VERBOSE1:adjustment_method_2.cc(1279)]  Adjuster::Solve 3.08888
[0112/090003.502:VERBOSE1:label_manager.cc(34)] 2698 of 2835 labels pre-assigned.
[0112/090003.502:VERBOSE1:label_manager.cc(55)]   fill forward 137
[0112/090003.502:VERBOSE1:label_manager.cc(77)]   fill backward 0
[0112/090003.502:VERBOSE1:label_manager.cc(93)]   infill 0
[0112/090003.502:VERBOSE1:label_manager.cc(34)] 16967 of 17800 labels pre-assigned.
[0112/090003.502:VERBOSE1:label_manager.cc(55)]   fill forward 833
[0112/090003.502:VERBOSE1:label_manager.cc(77)]   fill backward 0
[0112/090003.502:VERBOSE1:label_manager.cc(93)]   infill 0
[0112/090003.783:VERBOSE1:adjustment_method_2.cc(1238)] Adjuster::Adjust
[0112/090004.126:VERBOSE1:adjustment_method_2.cc(1279)]  Adjuster::Solve 0.313607
[0112/090008.750:VERBOSE1:adjustment_method_2.cc(1279)]  Adjuster::Solve 4.61335
[0112/090008.782:VERBOSE1:label_manager.cc(34)] 4438 of 4612 labels pre-assigned.
[0112/090008.782:VERBOSE1:label_manager.cc(55)]   fill forward 174
[0112/090008.797:VERBOSE1:label_manager.cc(77)]   fill backward 0
[0112/090008.797:VERBOSE1:label_manager.cc(93)]   infill 0
[0112/090008.797:VERBOSE1:label_manager.cc(34)] 23913 of 25055 labels pre-assigned.
[0112/090008.797:VERBOSE1:label_manager.cc(55)]   fill forward 1142
[0112/090008.797:VERBOSE1:label_manager.cc(77)]   fill backward 0
[0112/090008.797:VERBOSE1:label_manager.cc(93)]   infill 0
[0112/090008.922:VERBOSE1:adjustment_method_2.cc(1238)] Adjuster::Adjust
[0112/090009.000:VERBOSE1:adjustment_method_2.cc(1279)]  Adjuster::Solve 0.077676
[0112/090009.860:VERBOSE1:adjustment_method_2.cc(1279)]  Adjuster::Solve 0.860437
[0112/090009.876:VERBOSE1:label_manager.cc(34)] 1217 of 1217 labels pre-assigned.
[0112/090009.876:VERBOSE1:label_manager.cc(55)]   fill forward 0
[0112/090009.876:VERBOSE1:label_manager.cc(77)]   fill backward 0
[0112/090009.876:VERBOSE1:label_manager.cc(93)]   infill 0
[0112/090009.876:VERBOSE1:label_manager.cc(34)] 7509 of 7620 labels pre-assigned.
[0112/090009.876:VERBOSE1:label_manager.cc(55)]   fill forward 111
[0112/090009.876:VERBOSE1:label_manager.cc(77)]   fill backward 0
[0112/090009.876:VERBOSE1:label_manager.cc(93)]   infill 0
[0112/090010.017:VERBOSE1:adjustment_method_2.cc(1238)] Adjuster::Adjust
[0112/090010.236:VERBOSE1:adjustment_method_2.cc(1279)]  Adjuster::Solve 0.211341
[0112/090012.847:VERBOSE1:adjustment_method_2.cc(1279)]  Adjuster::Solve 2.61203
[0112/090012.878:VERBOSE1:label_manager.cc(34)] 2993 of 2999 labels pre-assigned.
[0112/090012.878:VERBOSE1:label_manager.cc(55)]   fill forward 6
[0112/090012.878:VERBOSE1:label_manager.cc(77)]   fill backward 0
[0112/090012.878:VERBOSE1:label_manager.cc(93)]   infill 0
[0112/090012.878:VERBOSE1:label_manager.cc(34)] 15792 of 16001 labels pre-assigned.
[0112/090012.878:VERBOSE1:label_manager.cc(55)]   fill forward 209
[0112/090012.878:VERBOSE1:label_manager.cc(77)]   fill backward 0
[0112/090012.878:VERBOSE1:label_manager.cc(93)]   infill 0
[0112/090012.972:VERBOSE1:adjustment_method_2.cc(1238)] Adjuster::Adjust
[0112/090013.097:VERBOSE1:adjustment_method_2.cc(1279)]  Adjuster::Solve 0.111536
[0112/090014.035:VERBOSE1:adjustment_method_2.cc(1279)]  Adjuster::Solve 0.938756
[0112/090014.035:VERBOSE1:label_manager.cc(34)] 1782 of 1782 labels pre-assigned.
[0112/090014.035:VERBOSE1:label_manager.cc(55)]   fill forward 0
[0112/090014.035:VERBOSE1:label_manager.cc(77)]   fill backward 0
[0112/090014.035:VERBOSE1:label_manager.cc(93)]   infill 0
[0112/090014.035:VERBOSE1:label_manager.cc(34)] 6688 of 6797 labels pre-assigned.
[0112/090014.035:VERBOSE1:label_manager.cc(55)]   fill forward 109
[0112/090014.035:VERBOSE1:label_manager.cc(77)]   fill backward 0
[0112/090014.035:VERBOSE1:label_manager.cc(93)]   infill 0
[0112/090014.347:VERBOSE1:simple_delta.cc(35)] GenerateSimpleDelta 210030234 210087628
[0112/090014.347:VERBOSE1:bsdiff_create.cc(96)] Start bsdiff
[0112/090037.969:VERBOSE1:bsdiff_create.cc(123)]  done divsufsort 23.6118
[0112/090045.365:VERBOSE1:bsdiff_create.cc(338)] Control tuples: 29400  copy bytes: 205294392  mistakes: 657397  (skips: 770993)  extra bytes: 4793236
Uncompressed bsdiff patch size 6340311
End bsdiff 31.0145
[0112/090047.460:VERBOSE1:simple_delta.cc(35)] GenerateSimpleDelta 467705248 440533223
[0112/090047.460:VERBOSE1:bsdiff_create.cc(96)] Start bsdiff
[0112/090047.491:ERROR:bsdiff_create.cc(115)] Could not allocate I[], 1870820996 bytes
-gen failed.

mherrmann · 2022-01-12T15:18:11Z

The problem appears to be related to address space fragmentation. At least, it can be fixed by reducing kPagedArrayDefaultPageLogSize from 18 (which stands for a page size of 1 MB) to 14 (which should be a page size of 1/16th MB) in courgette/third_party/bsdiff/paged_array.h. I am able to reproduce the problem not just on CI VMs, but also on my Win 10 machine (with 64 GB of RAM and > 1 TB free disk space). So far, all attempts to fix the problem by increasing the Page File size in Windows have failed.

mherrmann · 2022-01-13T09:47:28Z

@simonhong was so kind to re-test on his machine. He now also gets the error above.

mherrmann · 2022-01-13T10:16:46Z

This also reproduces when I take courgette.exe from stock Chromium 98.0.4735.1.

linhkikuchi · 2022-01-14T00:47:49Z

Previously I was able to run create_dist without error when I rdp into the windows node, but now I also have the same error , my windows node has 96GB RAM.

mihaiplesa · 2022-02-01T17:10:16Z

Re-opening as it still doesn't seem to work.

mherrmann · 2022-02-10T16:22:01Z

Just tested this with Nightly v1.37.40 updating from 1.37.36. Successfully received and applied the delta, as confirmed by logs :-)

kjozwiak · 2022-02-16T19:41:06Z

We'll run through this and label it was QA/Yes once we start running through the 1.35.x - Release #2 release call.

mihaiplesa added OS/Windows OS/Desktop labels Oct 11, 2021

simonhong self-assigned this Oct 15, 2021

rebron assigned mihaiplesa and unassigned simonhong Nov 2, 2021

mherrmann self-assigned this Jan 10, 2022

mihaiplesa added the priority/P3 The next thing for us to work on. It'll ride the trains. label Jan 10, 2022

mihaiplesa added priority/P2 A bad problem. We might uplift this to the next planned release. and removed priority/P3 The next thing for us to work on. It'll ride the trains. labels Jan 11, 2022

mherrmann mentioned this issue Jan 11, 2022

Fix generation of Windows delta update files brave/brave-core#11808

Merged

25 tasks

mherrmann closed this as completed in brave/brave-core#11808 Jan 17, 2022

kjozwiak added this to the 1.34.x - Release #2 milestone Jan 19, 2022

mherrmann added the release-notes/include label Jan 19, 2022

kjozwiak added QA/No release-notes/exclude and removed QA/Yes release-notes/include labels Jan 20, 2022

mihaiplesa removed this from the 1.34.x - Release #2 milestone Feb 1, 2022

mihaiplesa reopened this Feb 1, 2022

mihaiplesa mentioned this issue Feb 1, 2022

Release checklist for 1.35.x #20812

Closed

36 tasks

This was referenced Feb 1, 2022

Fix windows delta updates brave/brave-core#12098

Merged

npm run create_dist on macOS produces excessive output with repeated lines #16417

Closed

Test devops branch for Windows delta updates (do not merge) brave/brave-core#12129

Closed

mihaiplesa added this to the 1.37.x - Nightly milestone Feb 4, 2022

mherrmann closed this as completed in brave/brave-core#12098 Feb 9, 2022

mherrmann added QA/Yes release-notes/include and removed QA/No OS/Desktop release-notes/exclude labels Feb 9, 2022

brave-builds mentioned this issue Feb 10, 2022

Fix windows delta updates (uplift to 1.36.x) brave/brave-core#12210

Merged

7 tasks

mihaiplesa mentioned this issue Feb 10, 2022

Fix windows delta updates (uplift to 1.35.x) brave/brave-core#12211

Merged

7 tasks

kjozwiak modified the milestones: 1.37.x - Nightly, 1.35.x - Release #3 Feb 10, 2022

LaurenWags added the OS/Desktop label Feb 16, 2022

LaurenWags mentioned this issue Feb 16, 2022

[Desktop] Release notes for 1.35.x - Release #3 #21142

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Delta update generation broken on Windows x64 #18696

Delta update generation broken on Windows x64 #18696

mihaiplesa commented Oct 11, 2021 •

edited

Loading

simonhong commented Oct 19, 2021 •

edited

Loading

simonhong commented Oct 19, 2021 •

edited

Loading

mihaiplesa commented Oct 19, 2021

simonhong commented Oct 19, 2021

mihaiplesa commented Nov 1, 2021

mherrmann commented Jan 10, 2022 •

edited

Loading

mihaiplesa commented Jan 10, 2022

mherrmann commented Jan 11, 2022

mherrmann commented Jan 11, 2022

mherrmann commented Jan 12, 2022 •

edited

Loading

mherrmann commented Jan 12, 2022

mherrmann commented Jan 13, 2022

mherrmann commented Jan 13, 2022

linhkikuchi commented Jan 14, 2022

mihaiplesa commented Feb 1, 2022

mherrmann commented Feb 10, 2022

kjozwiak commented Feb 16, 2022

Delta update generation broken on Windows x64 #18696

Delta update generation broken on Windows x64 #18696

Comments

mihaiplesa commented Oct 11, 2021 • edited Loading

simonhong commented Oct 19, 2021 • edited Loading

simonhong commented Oct 19, 2021 • edited Loading

mihaiplesa commented Oct 19, 2021

simonhong commented Oct 19, 2021

mihaiplesa commented Nov 1, 2021

mherrmann commented Jan 10, 2022 • edited Loading

mihaiplesa commented Jan 10, 2022

mherrmann commented Jan 11, 2022

mherrmann commented Jan 11, 2022

mherrmann commented Jan 12, 2022 • edited Loading

mherrmann commented Jan 12, 2022

mherrmann commented Jan 13, 2022

mherrmann commented Jan 13, 2022

linhkikuchi commented Jan 14, 2022

mihaiplesa commented Feb 1, 2022

mherrmann commented Feb 10, 2022

kjozwiak commented Feb 16, 2022

mihaiplesa commented Oct 11, 2021 •

edited

Loading

simonhong commented Oct 19, 2021 •

edited

Loading

simonhong commented Oct 19, 2021 •

edited

Loading

mherrmann commented Jan 10, 2022 •

edited

Loading

mherrmann commented Jan 12, 2022 •

edited

Loading