xdelta3 encoding is too slow when generating a diff between two files which are more 10Gb of size

[nachammaichidambaram]

No description provided.

[nachammaichidambaram]

xdelta3 encoding is too slow when generating a diff between two files which are more 10Gb of size. I tried all the options like increasing the bytes size using -B changing the compression level -1 and modifying -S djw . Using fastest machines with gpu support.

[mgrinzPlayer]

xdelta3 does not utilize GPU. It uses only one CPU thread. The most CPU time consuming are: secondary compression, then internal compression. If you are planning compressing diff files with external compressor like rar,7z,xz,etc better disable secondary compression with -S none.

On my old test machine, first line takes 60 seconds, second line 190 sec, pak file size around 10GB:

xdelta3.exe -v -e -0 -S none -I 0 -B 1073741824 -s file.pak file.pak.NEW fileTEST1.patch
xdelta3.exe -v -e -9 -S none -I 0 -B 1073741824 -s file.pak file.pak.NEW fileTEST2.patch

TEST1.patch size = ~90MB, compressed with 7z(mx9,dict512MB, 1thread) = 35MB
TEST2.patch size = ~38MB, compressed with 7z(mx9,dict512MB, 1thread) = 24.5MB

[nachammaichidambaram]

Hello Jmacd, Thanks for your reply Iam using xdelta3 in linux machine trying to diff tar files I tried the xdelta3 command with -S none option but still it uses lzma for secondary compression can you help on this please Also let me know any command line options to be used for decoding Thanks, Nachammai

…

On Thu, 10 Aug 2023 at 8:43 PM, mgrinzPlayer ***@***.***> wrote: xdelta3 does not utilize GPU. It uses only one CPU thread. The most CPU time consuming are: secondary compression, then internal compression. If you are planning compressing diff files with external compressor like rar,7z,xz,etc better disable secondary compression with -S none. On my old test machine, first line takes 60 seconds, second line 190 sec, pak file size around 10GB: xdelta3.exe -v -e -0 -S none -I 0 -B 1073741824 -s file.pak file.pak.NEW fileTEST1.patch xdelta3.exe -v -e -9 -S none -I 0 -B 1073741824 -s file.pak file.pak.NEW fileTEST2.patch TEST1.patch size = ~90MB, compressed with 7z(mx9,dict512MB, 1thread) = 35MB TEST2.patch size = ~38MB, compressed with 7z(mx9,dict512MB, 1thread) = 24.5MB — Reply to this email directly, view it on GitHub <#269 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/A2LNBJFMLGLJF2BOLSDLY7TXUT3CFANCNFSM6AAAAAA3KFFPPY> . You are receiving this because you authored the thread.Message ID: ***@***.***>

[nachammaichidambaram]

Currently decoding gives non-seekable source in decode: XD3_INTERNAL command which I used for decoding Xdelta3 -S none -d -f -R -s source.tar patch target.tar The version which Iam using is 3.1.0 If there is a latest version available with a fix for this , please let me know Thanks, Nachammai On Thu, 10 Aug 2023 at 9:08 PM, Nachammai Chidambaram < ***@***.***> wrote:

…

Hello Jmacd, Thanks for your reply Iam using xdelta3 in linux machine trying to diff tar files I tried the xdelta3 command with -S none option but still it uses lzma for secondary compression can you help on this please Also let me know any command line options to be used for decoding Thanks, Nachammai On Thu, 10 Aug 2023 at 8:43 PM, mgrinzPlayer ***@***.***> wrote: > xdelta3 does not utilize GPU. It uses only one CPU thread. The most CPU > time consuming are: secondary compression, then internal compression. If > you are planning compressing diff files with external compressor like > rar,7z,xz,etc better disable secondary compression with -S none. > > On my old test machine, first line takes 60 seconds, second line 190 sec, > pak file size around 10GB: > > xdelta3.exe -v -e -0 -S none -I 0 -B 1073741824 -s file.pak file.pak.NEW fileTEST1.patch > xdelta3.exe -v -e -9 -S none -I 0 -B 1073741824 -s file.pak file.pak.NEW fileTEST2.patch > > TEST1.patch size = ~90MB, compressed with 7z(mx9,dict512MB, 1thread) = > 35MB > TEST2.patch size = ~38MB, compressed with 7z(mx9,dict512MB, 1thread) = > 24.5MB > > — > Reply to this email directly, view it on GitHub > <#269 (comment)>, or > unsubscribe > <https://github.com/notifications/unsubscribe-auth/A2LNBJFMLGLJF2BOLSDLY7TXUT3CFANCNFSM6AAAAAA3KFFPPY> > . > You are receiving this because you authored the thread.Message ID: > ***@***.***> >

[mgrinzPlayer]

I'm not jmacd.

The version which Iam using is 3.1.0

From where? Did you compile it? In this repo, since this commit e1a0a25, LZMA is not the default secondary compression.

anyway, decoding, first line should be good for most files, second line for bigger files (from "--help" "-B bytes source window size", also bigger buffers, sometimes it can speed up patching process)

xdelta3.exe -v -d -s file.pak fileTEST1.patch file.pak.NEW
xdelta3.exe -v -d -B 1073741824 -s file.pak fileTEST1.patch file.pak.NEW

edit:
Suggestion. Try getting older xdelta versions, I would try 3.0.7 and 3.0.8. Maybe those versions will be enough for your needs.