Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

we need a winlibs subset #57

Closed
shawnlaffan opened this issue Feb 27, 2023 · 11 comments
Closed

we need a winlibs subset #57

shawnlaffan opened this issue Feb 27, 2023 · 11 comments
Labels

Comments

@shawnlaffan
Copy link
Contributor

The winlibs gcc11 system we are currently using has a large number of extra files in it. We need to sort out which ones to retain.

A quick diff of the bin dir filenames between the strawberry perl 8.3 set and the winlibs gcc11 set is below. I filtered out all lines with clang or LLVM assuming we don't need them.

Many of the DLLs we have our own versions of via the extlibs builds (e.g. zlib1.dll) but others might be useful (ninja.exe?).

Others we need to get from elsewhere.

gmake is one example. We do already have it at https://strawberryperl.com/package/kmx/64_tools/ although that's a 2014 build. Maybe we can grab one from msys2?

--- strawberry.txt	2023-02-27 20:21:04.902942100 +1100
+++ winlibs.txt	2023-02-27 20:20:58.262130700 +1100
@@ -1,13 +1,43 @@
+FileCheck.exe
 addr2line.exe
+analyze-build
 ar.exe
 as.exe
+bugpoint.exe
 c++.exe
 c++filt.exe
+count.exe
 cpp.exe
+diagtool.exe
 dlltool.exe
 dllwrap.exe
-dwp.exe
+dos2unix.exe
+doxygen.exe
+dsymutil.exe
 elfedit.exe
+find-all-symbols.exe
 g++.exe
 gcc-ar.exe
 gcc-nm.exe
@@ -18,40 +48,362 @@
 gcov.exe
 gdb-add-index
 gdb.exe
-gdborig.exe
 gdbserver.exe
+gdc.exe
 gendef.exe
 genidl.exe
-genpeimg.exe
 gfortran.exe
-gmake.exe
 gprof.exe
+hmaptool
+iconv.exe
+intercept-build
+jwasm.exe
 ld.bfd.exe
 ld.exe
-ld.gold.exe
+ld.lld.exe
+ld64.lld.darwinnew.exe
+ld64.lld.darwinold.exe
+ld64.lld.exe
+libLTO.dll
+libRemarks.dll
 libatomic-1.dll
+libdl.dll
+libexpat-1.dll
+libffi-8.dll
+libfindAllSymbols.dll
 libgcc_s_seh-1.dll
+libgccjit-0.dll
+libgfortran-5.dll
+libgmp-10.dll
 libgomp-1.dll
+libgomp-plugin-nvptx-1.dll
+libiconv-2.dll
+libintl-8.dll
+libisl-23.dll
+liblldCOFF.dll
+liblldCommon.dll
+liblldCore.dll
+liblldDriver.dll
+liblldELF.dll
+liblldMachO.dll
+liblldMachO2.dll
+liblldMinGW.dll
+liblldReaderWriter.dll
+liblldWasm.dll
+liblldYAML.dll
+liblldb.dll
+liblzma-5.dll
+libmpc-3.dll
+libmpfr-6.dll
+libncursesw6.dll
 libquadmath-0.dll
+libsource-highlight-4.dll
 libssp-0.dll
 libstdc++-6.dll
 libwinpthread-1.dll
+libxml2-2.dll
+libz3.dll
+libzstd.dll
+llc.exe
+lld-link.exe
+lld.exe
+lldb-argdumper.exe
+lldb-instr.exe
+lldb.exe
+lli-child-target.exe
+lli.exe
+lto-dump.exe
+lua54.dll
+mac2unix.exe
+mingw32-make.exe
+mman-win.dll
+modularize.exe
+nasm.exe
+ndisasm.exe
+ninja.exe
 nm.exe
+not.exe
+obj2yaml.exe
 objcopy.exe
 objdump.exe
+opt.exe
+pexports.exe
+pp-trace.exe
 ranlib.exe
 readelf.exe
+sancov.exe
+sanstats.exe
+scan-build
+scan-build-py
+scan-build.bat
+scan-view
 size.exe
+split-file.exe
 strings.exe
 strip.exe
-widl.exe
+unix2dos.exe
+unix2mac.exe
+verify-uselistorder.exe
+vsyasm.exe
+wasm-ld.exe
 windmc.exe
 windres.exe
 x86_64-w64-mingw32-c++.exe
 x86_64-w64-mingw32-g++.exe
-x86_64-w64-mingw32-gcc-8.3.0.exe
+x86_64-w64-mingw32-gcc-11.2.0.exe
 x86_64-w64-mingw32-gcc-ar.exe
 x86_64-w64-mingw32-gcc-nm.exe
 x86_64-w64-mingw32-gcc-ranlib.exe
 x86_64-w64-mingw32-gcc.exe
+x86_64-w64-mingw32-gdc.exe
 x86_64-w64-mingw32-gfortran.exe
+xxhash.dll
+yaml-bench.exe
+yaml2obj.exe
+yasm.exe
+ytasm.exe
+zlib1.dll
@shawnlaffan
Copy link
Contributor Author

Now I look at the winlibs site, I see it has versions without LLVM.

I'll keep this open as we still need to clean up the extra dlls, and sub in a more recent version of patch.exe (noted in #56 (comment)).

@shawnlaffan shawnlaffan changed the title we need a gcc 11 subset we need a winlibs subset Apr 23, 2023
@shawnlaffan
Copy link
Contributor Author

An audit of the dll dependencies of executables in the c\bin dir gives the data below (see code at end of comment).

The "clashes" are DLLs for which there exists both a strawberry perl version and one from winlibs.

The "Users' list is the set of DLLs in the c\bin dir that are depended on by executables in the same dir.

The short summary is that we cannot easily remove the "clashing" DLLs from the distribution. This might have downstream effects on CPAN modules.

(There are no clashes for libgmp, libmpfr and libmpc as we currently only build static versions).

Clashes:
{
    libexpat-1__.dll   "libexpat-1.dll",
    libiconv-2__.dll   "libiconv-2.dll",
    liblzma-5__.dll    "liblzma-5.dll",
    zlib1__.dll        "zlib1.dll"
}
Users:
{
    libexpat-1.dll              [
        [0] "gdb.exe"
    ],
    libgmp-10.dll               [
        [0] "lto-dump.exe"
    ],
    libiconv-2.dll              [
        [0]  "c++.exe",
        [1]  "cpp.exe",
        [2]  "doxygen.exe",
        [3]  "g++.exe",
        [4]  "gcc.exe",
        [5]  "gcov-dump.exe",
        [6]  "gcov-tool.exe",
        [7]  "gcov.exe",
        [8]  "gdb.exe",
        [9]  "gfortran.exe",
        [10] "lto-dump.exe"
    ],
    libintl-8.dll               [
        [0]  "addr2line.exe",
        [1]  "ar.exe",
        [2]  "as.exe",
        [3]  "c++filt.exe",
        [4]  "dlltool.exe",
        [5]  "dllwrap.exe",
        [6]  "elfedit.exe",
        [7]  "gmake.exe",
        [8]  "gprof.exe",
        [9]  "ld.exe",
        [10] "mingw32-make.exe",
        [11] "nm.exe",
        [12] "objcopy.exe",
        [13] "objdump.exe",
        [14] "ranlib.exe",
        [15] "readelf.exe",
        [16] "size.exe",
        [17] "strings.exe",
        [18] "strip.exe",
        [19] "windmc.exe",
        [20] "windres.exe"
    ],
    libisl-23.dll               [
        [0] "lto-dump.exe"
    ],
    liblzma-5.dll               [
        [0] "gdb.exe"
    ],
    libmpc-3.dll                [
        [0] "lto-dump.exe"
    ],
    libmpfr-6.dll               [
        [0] "gdb.exe",
        [1] "lto-dump.exe"
    ],
    libncursesw6.dll            [
        [0] "gdb.exe"
    ],
    libsource-highlight-4.dll   [
        [0] "gdb.exe"
    ],
    libzstd.dll                 [
        [0] "ccache.exe",
        [1] "lto-dump.exe"
    ],
    mman-win.dll                [
        [0] "ccache.exe",
        [1] "gdb.exe"
    ],
    xxhash.dll                  [
        [0] "gdb.exe"
    ],
    zlib1.dll                   [
        [0]  "addr2line.exe",
        [1]  "ar.exe",
        [2]  "as.exe",
        [3]  "c++filt.exe",
        [4]  "dlltool.exe",
        [5]  "gcov.exe",
        [6]  "gdb.exe",
        [7]  "gprof.exe",
        [8]  "ld.exe",
        [9]  "lto-dump.exe",
        [10] "nm.exe",
        [11] "objcopy.exe",
        [12] "objdump.exe",
        [13] "ranlib.exe",
        [14] "readelf.exe",
        [15] "size.exe",
        [16] "strings.exe",
        [17] "strip.exe",
        [18] "windmc.exe",
        [19] "windres.exe"
    ]
}

Code:

use 5.020;
use strict;
use warnings;
use Data::Printer;
use Path::Tiny '0.018';

my $bindir = $ARGV[0] // 'c/bin';

my @files = grep {$_ !~ /x86_64-w64|nvptx/} Path::Tiny->cwd->child($bindir)->children (qr /.exe$/);

my %uses;

my $exclude = qr /__|stdc++|winpthread|x86_64|libgcc_s_seh|msvcrt|api-ms-win-crt/;

foreach my $file (@files) {
    #$file = "$file";
    
    my @deps =
        grep {!-e "/mnt/c/Windows/System32/$_"} 
        map {s/\s+DLL\sName\:\s*(.+)\n/$1/r} 
        grep {$_ !~ $exclude} 
        grep {/DLL Name/} 
        `objdump -x $file`;
    foreach my $dep (@deps) {
        push @{$uses{$dep}}, $file->basename;
    }
}

my %clash;
foreach my $name (sort keys %uses) {
    my $alt_name = ($name =~ s/.dll/__.dll/r);
    if (path($bindir)->child($alt_name)->exists) {
        $clash{$alt_name} = $name;
    }
}

say STDERR "Clashes:";
p %clash;
say STDERR "Users:";
p %uses;

@sisyphus
Copy link

Clashes:
{
libexpat-1__.dll "libexpat-1.dll",
libiconv-2__.dll "libiconv-2.dll",
liblzma-5__.dll "liblzma-5.dll",
zlib1__.dll "zlib1.dll"
}

It's not apparent to me why this would be a problem.
Obviously, the winlibs executables that look for (eg) a libexpat dll will load libexpat-1.dll.
And my expectation is that the vendor modules that look for (eg) a libexpat dll will have been built such that they load libexpat-1__.dll.
I can't see any clash.

Can you provide a demo case that would make this issue clearer to me.

Cheers,
Rob

@shawnlaffan
Copy link
Contributor Author

It's not apparent to me why this would be a problem.
Obviously, the winlibs executables that look for (eg) a libexpat dll will load libexpat-1.dll.
And my expectation is that the vendor modules that look for (eg) a libexpat dll will have been built such that they load libexpat-1__.dll.

It might well be that there are no issues and that I'm just overthinking things.

The modules under vendor\lib do all link to the DLLs with underscores (let me know if you want the script I used).

But that still leaves user-installed modules under site\lib. One scenario is that a user builds a module from CPAN, and that module links to zlib1.dll instead of zlib1__.dll. If some other software package is then installed such that the Strawberry perl c\bin dir is now later in the path, and that new package has a zlib1.dll that is incompatible, then the CPAN module will fail when it is used. Maybe that's not an issue for the set of DLLs listed under the "clash" label, though.

I assume clashes like this are a reason why Strawberry perl has always appended one or two underscores to the DLL names. Or maybe the reasons are completely different but now lost in time.

@sisyphus
Copy link

There should be no need for concern.
Staying with the "libexpat" example, any module with a dependency on libexpat will be given a -lexpat link, and Strawberry's c/lib/libexpat.a import lib will direct it to libexpat-1__.dll.
There would only be problem if c/lib/libexpat.a specified libexpat-1.dll.

There's probably an objdump command that will tell you the name of the dll specified by the import lib.
But I just open the import lib and do a text search.
c/lib/libexpat.a contains a match for libexpat-1__.dll, but no match for libexpat-1.dll.
(Hardly proves anything ... but good enough for me ;-)

Similarly for the other 3 "clashes", the import lib specifies only the double-underscore variant.

There are 3 libraries in c/lib that specify "zlib1__.dll" - namely libz.a, libzlib.a and libzdll.a.
None of those 3 libs specify "zlib1.dll".

In providing those underscore variants, you ensure that it's strawberry's dlls that are being loaded.
If there was only one libexpat dll in c/bin, and it was named libexpat-1.dll, then you couldn't be confident that the user has not done something that will load a different libexpat-1.dll (found earlier in the path).

Cheers,
Rob

@shawnlaffan
Copy link
Contributor Author

Alien modules that link to dynamic libs might have issues, though. I maintain a few that don't do static linking and go looking for DLLs. I'll run some checks with FFI::CheckLib.

@shawnlaffan
Copy link
Contributor Author

FFI::CheckLib finds the winlibs versions first.

perl -MFFI::CheckLib -E"say join qq{\n}, find_lib( lib => [ qw/iconv expat zlib1 lzma/ ] );"
C:\sp536_20230420a\c\bin\libexpat-1.dll
C:\sp536_20230420a\c\bin\libiconv-2.dll
C:\sp536_20230420a\c\bin\liblzma-5.dll
C:\sp536_20230420a\c\bin\zlib1.dll
perl -MFFI::CheckLib=where -E"say join qq{\n}, where ('iconv')"
C:\sp536_20230420a\c\bin\libiconv-2.dll
C:\sp536_20230420a\c\bin\libiconv-2__.dll

An Alien::libtiff share build generates a libtiff that links to the Strawberry perl versions so maybe this is not so much of an issue. I'll work my way up to an Alien::gdal install as it uses FFI::CheckLib in its alienfile as a tougher test.

@sisyphus
Copy link

Alien modules that link to dynamic libs might have issues, though. I maintain a few that don't do static linking and go looking for DLLs.

If the module expects to access third party dlls, then the ".a" file it links to needs to be an import library.
If that ".a" file turns out to be a static library then you'll end up with a long list of unresolved dynamic symbols.
In such cases I've generally found that it's the ".dll.a" (not the ".a" file) that needs to be linked in.

(Not entirely sure this is relevant to your issue ;-)

Cheers,
Rob

@shawnlaffan
Copy link
Contributor Author

shawnlaffan commented Apr 24, 2023

I'll work my way up to an Alien::gdal install as it uses FFI::CheckLib in its alienfile as a tougher test.

libgdal under Alien::gdal links to the underscored DLLs so it looks like we're ok on this front.

@shawnlaffan
Copy link
Contributor Author

The gcc-13 winlibs distribution includes ~60MB of python files. It would be nice to not need these, as much as anything to reduce download sizes. The only exe or dll file that depends on this is gdb.exe.

I have added code to the build-extlibs repo (not yet pushed to a PR) to build a gdb that depends on other extlibs deps. It will be included in the next dev release.

I'll close this once the release is out and the code has been pushed.

(And FWIW, the python files have already been removed from the last gcc-13 based dev release, so its gdb does not work).

@shawnlaffan
Copy link
Contributor Author

The latest dev release includes our own gdb.exe.
https://github.com/StrawberryPerl/Perl-Dist-Strawberry/releases/tag/dev_5.36.1_20230528_gcc13

I'll close this issue now.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

No branches or pull requests

2 participants