Stop bundling external libraries

[pete4abw]

@ppisar at @RedHatOfficial offered a suggestion to stop bundling zpaq and lzma libraries with lrzip-next. The main reason was security and secondarily maintenance.

Back in the day, lzma was not a part of many distributions and zpaq is still not. While small changes have been made to both sets of sources, such a project is worth looking at. The same goes with bzip3, which is not a part of all distros.

Comments welcome.

[kspalaiologos]

You could always check if the libraries are present on the system, and if not, use a copy from a Git submodule.

[pete4abw]

If git is available. For example, neither lzma or xz is on github. Will probably leave it to developers or downstream packagers to satisfy requirements. This may be a big job anyway, especially for lzma.

[pete4abw]

Heard back from Lasse Collin, author of XZ ( @tukaani ) and the lzma library. Unfortunately, the library is not current with the SDK and its features lack multi threading for the match finder and no assembler decoder. Also, the file output would be incompatible with past lrzip-next versions. Block header changes, etc.

[pete4abw]

In addition, no debugging would be possible when external libraries are used unless rebuilt locally. This is especially important with lzma and zpaq where large chunks of memory are used and large blocks of data passed. So for now, this is not going to happen.

[ppisar]

A good start is separate lrzip-next changes and the files copied from external libraries.

E.g. move "/////////////////////////// lrzip functions //////////////////" line and subsequent lines from src/lib/libzpaq/libzpaq.h to a separate header file and #include <libzpaq.h> from the separate file. Then if you need different types in libzpaq.h API, wrap the original ZPAQ functions into your new API functions. It will help you maintaining the code when a new ZPAQ version comes. It will also help others review your changes in the bundled files. And it will also help you to format the needed patched for ZPAQ and sending them to ZPAQ author to merge them into ZPAQ.

[pete4abw]

That may work. For the lzma files, it will take a little more effort since the ASM files were rewritten for nasm/yasm compatibility and I removed a lot of Windows cruft. However, since ZPAQ has gone 6 years wtthout an update, it's doubtful it will be revised although there is at least one active fork.

[kspalaiologos]

Maybe consider using .patch files?

…

On Mon, 21 Nov 2022, 10:15 Petr Pisar, ***@***.***> wrote: A good start is separate lrzip-next changes and the files copied from external libraries. E.g. move "/////////////////////////// lrzip functions //////////////////" line and subsequent lines from src/lib/libzpaq/libzpaq.h to a separate header file and #include <libzpaq.h> from the separate file. Then if you need different types in libzpaq.h API, wrap the original ZPAQ functions into your new API functions. It will help you maintaining the code when a new ZPAQ version comes. It will also help others review your changes in the bundled files. And it will also help you to format the needed patched for ZPAQ and sending them to ZPAQ author to merge them into ZPAQ. — Reply to this email directly, view it on GitHub <#94 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AGTTDFO5USMBR44QUXCGSHLWJM4U3ANCNFSM6AAAAAASENJJP4> . You are receiving this because you commented.Message ID: ***@***.***>

[pete4abw]

I trink what ppsiar is about is not to alter the source files from the external source. We'll see. The same could go with bzip3 where I commented out one function.