Skip to content

Add Lzma library files to the repository#124003

Draft
rzikm wants to merge 19 commits intodotnet:mainfrom
rzikm:lzma
Draft

Add Lzma library files to the repository#124003
rzikm wants to merge 19 commits intodotnet:mainfrom
rzikm:lzma

Conversation

@rzikm
Copy link
Member

@rzikm rzikm commented Feb 4, 2026

Preparatory PR for #1542

Copilot AI review requested due to automatic review settings February 4, 2026 13:22
Copy link
Contributor

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR adds the XZ Utils library (version 5.8.2) to the dotnet/runtime repository as preparatory work for issue #1542. The addition includes the complete LZMA compression library source code, build configurations, documentation, and integration scripts.

Changes:

  • Adds XZ Utils 5.8.2 library files from the tukaani-project/xz repository
  • Includes CMake and Autotools build system configurations
  • Adds documentation, examples, and internationalization support files
  • Configures the library for static linking with multithreading disabled

Reviewed changes

Copilot reviewed 75 out of 497 changed files in this pull request and generated no comments.

File Description
src/native/external/xz/* Complete XZ Utils 5.8.2 library source code including build scripts, documentation, examples, and localization files
src/native/external/xz.cmake CMake configuration to integrate XZ Utils into dotnet/runtime build system with specific feature flags
src/native/external/xz.version Version tracking file indicating XZ Utils v5.8.2 from tukaani-project
src/native/external/cgmanifest.json Component governance manifest entry for XZ Utils dependency

@rzikm rzikm marked this pull request as draft February 4, 2026 13:23
@dotnet-policy-service
Copy link
Contributor

Tagging subscribers to this area: @karelz, @dotnet/area-system-io-compression
See info in area-owners.md if you want to be subscribed.

Copilot AI review requested due to automatic review settings February 9, 2026 12:14
Copy link
Contributor

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 79 out of 284 changed files in this pull request and generated 7 comments.

#include "tuklib_common.h"
TUKLIB_DECLS_BEGIN

#define tuklib_open_stdxx TUKLIB_SYMBOL(tuklib_open_stdxxx)
Copy link

Copilot AI Feb 9, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The macro name tuklib_open_stdxx looks like a typo/inconsistency with the header/implementation name tuklib_open_stdxxx. Consider renaming the macro to tuklib_open_stdxxx to avoid confusing API consumers and accidental use of the wrong identifier.

Suggested change
#define tuklib_open_stdxx TUKLIB_SYMBOL(tuklib_open_stdxxx)
#define tuklib_open_stdxxx TUKLIB_SYMBOL(tuklib_open_stdxxx)

Copilot uses AI. Check for mistakes.
@rzikm
Copy link
Member Author

rzikm commented Feb 9, 2026

I am getting compile errors on linux:

  ERROR: /source/dotnet/runtime/src/native/libs/System.IO.Compression.Native/entrypoints.c file did not match entries exported from /source/dotnet/runtime/artifacts/obj/native/net11.0-linux-Debug-x64/System.IO.Compression.Native/libSystem.IO.Compression.Native.so
  DIFFERENCES FOUND: 
  lzma_stream_decoder

Seems like there is something wrong with the exports specification:

System.IO.Compression.Native$ nm libSystem.IO.Compression.Native.so | grep " lzma_stream_decoder"
00000000003c51c0 t lzma_stream_decoder
00000000003c4410 t lzma_stream_decoder_init

All the other exported entries have T in the nm output, e.g.

000000000019dd90 T BrotliDecoderDecompress

But the exports file seems to be generated correctly:

        ZSTD_CCtx_setPledgedSrcSize;
        ZDICT_trainFromBuffer;
        lzma_stream_decoder;
    local: *;
};

I am not sure what is going on here, maybe there is a problem because the library functions are declared as extern?

extern LZMA_API(lzma_ret) lzma_alone_decoder(
lzma_stream *strm, uint64_t memlimit)
lzma_nothrow lzma_attr_warn_unused_result;

I will need to investigate a bit more

Copilot AI review requested due to automatic review settings February 10, 2026 17:24
Copy link
Contributor

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 79 out of 284 changed files in this pull request and generated 2 comments.

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
Copilot AI review requested due to automatic review settings February 10, 2026 17:38
@rzikm
Copy link
Member Author

rzikm commented Feb 10, 2026

I will need to investigate a bit more

Should be working now

Copy link
Contributor

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 79 out of 284 changed files in this pull request and generated 3 comments.

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
Copilot AI review requested due to automatic review settings February 10, 2026 17:49
Copy link
Contributor

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 79 out of 284 changed files in this pull request and generated no new comments.

Copilot AI review requested due to automatic review settings February 11, 2026 12:27
Copy link
Contributor

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 82 out of 287 changed files in this pull request and generated 3 comments.

<NativeSystemLibrary Include="z" Condition="'$(UseSystemZlib)' == 'true'" />
<NativeSystemLibrary Include="brotlienc;brotlidec;brotlicommon" Condition="'$(UseSystemBrotli)' == 'true'" />
<NativeSystemLibrary Include="zstd" Condition="'$(UseSystemZstd)' == 'true'" />
<NativeSystemLibrary Include="lzma" Condition="'$(UseSystemXz)' == 'true'" />
Copy link

Copilot AI Feb 11, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The condition uses UseSystemXz, but this PR introduces UseSystemLzma (and uses it elsewhere in the same file). This makes the selection logic inconsistent and can cause -llzma to be omitted/added incorrectly; update the condition to reference UseSystemLzma.

Suggested change
<NativeSystemLibrary Include="lzma" Condition="'$(UseSystemXz)' == 'true'" />
<NativeSystemLibrary Include="lzma" Condition="'$(UseSystemLzma)' == 'true'" />

Copilot uses AI. Check for mistakes.
<NativeLibrary Include="$(IlcSdkPath)brotlienc$(LibFileExt)" />
<NativeLibrary Include="$(IlcSdkPath)brotlidec$(LibFileExt)" />
<NativeLibrary Include="$(IlcSdkPath)zstd_static$(LibFileExt)" />
<NativeLibrary Include="$(IlcSdkPath)lzma$(LibFileExt)" />
Copy link

Copilot AI Feb 11, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The library name here is lzma$(LibFileExt) (e.g., lzma.lib), but the packaging manifest in this PR adds lzma_static.lib. This mismatch is likely to cause build/package failures on Windows unless an import/static lib named lzma.lib is produced; consider aligning this to lzma_static$(LibFileExt) (or whichever artifact is actually built).

Suggested change
<NativeLibrary Include="$(IlcSdkPath)lzma$(LibFileExt)" />
<NativeLibrary Include="$(IlcSdkPath)lzma_static$(LibFileExt)" />

Copilot uses AI. Check for mistakes.
@rzikm
Copy link
Member Author

rzikm commented Feb 11, 2026

More to address on 32-bit Windows build

2026-02-11T12:50:08.2419461Z   D:\a\_work\1\s\src\native\external\xz\src\liblzma\common\index.c(910,4): error C4244: 'function': conversion from 'const lzma_vli' to 'size_t', possible loss of data [D:\a\_work\1\s\artifacts\obj\coreclr\windows.x86.Release\ide\_deps\xz-build\liblzma.vcxproj]
2026-02-11T12:50:08.2541597Z   D:\a\_work\1\s\src\native\external\xz\src\liblzma\common\index.c(921,24): error C4242: '=': conversion from 'const lzma_vli' to 'size_t', possible loss of data [D:\a\_work\1\s\artifacts\obj\coreclr\windows.x86.Release\ide\_deps\xz-build\liblzma.vcxproj]
2026-02-11T12:50:08.3122411Z   D:\a\_work\1\s\src\native\external\xz\src\liblzma\common\index.c(922,34): error C4244: '=': conversion from 'const lzma_vli' to 'size_t', possible loss of data [D:\a\_work\1\s\artifacts\obj\coreclr\windows.x86.Release\ide\_deps\xz-build\liblzma.vcxproj]

Copilot AI review requested due to automatic review settings February 12, 2026 14:58
Copy link
Contributor

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 82 out of 288 changed files in this pull request and generated 2 comments.

Comment on lines +45 to +49
list(APPEND CMAKE_REQUIRED_DEFINITIONS
-D_GNU_SOURCE
-D_NETBSD_SOURCE
-D_OPENBSD_SOURCE
-D__EXTENSIONS__
Copy link

Copilot AI Feb 12, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

add_compile_definitions sets _OPENBSD_SOURCE (with a leading underscore) but CMAKE_REQUIRED_DEFINITIONS appends -D_OPENBSD_SOURCE (missing underscore). This mismatch can cause CMake feature checks to run with different feature-test macros than the actual build. Update the required definition to -D_OPENBSD_SOURCE with the leading underscore to match.

Copilot uses AI. Check for mistakes.
Comment on lines +110 to +113
if(HAVE_SYS_PARAM_H)
set(TUKLIB_CPUCORES_DEFINITIONS
"HAVE_PARAM_H;TUKLIB_CPUCORES_SYSCTL"
CACHE INTERNAL "")
Copy link

Copilot AI Feb 12, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Same macro inconsistency as in tuklib_physmem.cmake: the checks are gated on HAVE_SYS_PARAM_H but the propagated compile definitions use HAVE_PARAM_H. Align this to HAVE_SYS_PARAM_H so the C sources see the expected macro.

Copilot uses AI. Check for mistakes.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants