update config patch files for 5.34 and 5.36? #56
Comments
|
If it turns out there are not many changes needed then maybe we can switch to using command line arguments? It seems to work for @sisyphus in #39 (comment) |
|
@xenu Do you have any thoughts here? |
|
I prefer the idea of configuration stuff being as lightweight as possible: command-line args in scripts being preferred over patches, supplying a config file after that, etc. If nothing else, that makes it likelier to survive changes in Perl going forward. And any patches might suggest changes needed in core Perl in any case, which could be fed back in, and kept working by making Perl's own CI exercise it. In a perfect world, the methods used to build at least the main Perl binaries/modules should be native Perl building, and Strawberry would only need to concern itself with "extra" stuff. |
@shawnlaffan Just want to mention that the same command line args (as used when building) must also be passed when testing ( |
|
@hakonhagland - yes, that would make life more complex. Another option is to edit the files in place rather than overwriting them completely. It would be simple enough to do with a subroutine passed a data structure of arg names with values, or undef to comment the line out. |
|
Here is the filtered list of changes to config_H.gc to show only the defines. This should make it easier to get a handle on what is being changed, in this case for perl 5.36. The -#define HAS_LSTAT /**/
-#define HAS_READLINK /**/
+#define HAS_SETLOCALE /**/
-#define HAS_SYMLINK /**/
+#define I_DBM /**/
+#define I_GDBM /**/
-#define BYTEORDER 0x12345678 /* large digits for MSB */
+#define BYTEORDER 0x1234 /* large digits for MSB */
-#define PTRSIZE 8 /**/
+#define PTRSIZE 4 /**/
-#define SSize_t long long /* signed count of bytes */
+#define SSize_t int /* signed count of bytes */
-#define OSVERS "10.0.19043.1237" /**/
+#define OSVERS "6.1" /**/
-#define LONG_DOUBLESIZE 16 /**/
+#define LONG_DOUBLESIZE 12 /**/
+#define HAS_MKSTEMP
-#define HAS_SOCKADDR_STORAGE /**/
+#define I_NDBM /**/
-#define HAS_ATOLL /**/
-#define HAS_NON_INT_BITFIELDS /**/
+#define HAS_BUILTIN_EXPECT /* we have gcc-8.3 */
+#define HAS_BUILTIN_CHOOSE_EXPR /* we have gcc-8.3 */
-#define HAS_MODFL_PROTO /**/
-#define HAS_SETLOCALE /**/
-#define HAS_STRTOLL /**/
-#define HAS_STRTOULL /**/
-#define LONGDBLINFBYTES 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x80, 0xff, 0x7f, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00 /**/
-#define LONGDBLNANBYTES 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0xc0, 0xff, 0xff, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00 /**/
+#define LONGDBLINFBYTES 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x80, 0xff, 0x7f, 0x00, 0x00 /**/
+#define LONGDBLNANBYTES 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0xc0, 0xff, 0xff, 0x00, 0x00 /**/
-#define IVTYPE long long /**/
-#define UVTYPE unsigned long long /**/
+#define IVTYPE long /**/
+#define UVTYPE unsigned long /**/
-#define IVSIZE 8 /**/
-#define UVSIZE 8 /**/
+#define IVSIZE 4 /**/
+#define UVSIZE 4 /**/
-#define NV_PRESERVES_UV_BITS 53
-#define NV_OVERFLOWS_INTEGERS_AT (256.0*256.0*256.0*256.0*256.0*256.0*2.0*2.0*2.0*2.0*2.0)
+#define NV_PRESERVES_UV
+#define NV_PRESERVES_UV_BITS 32
+#define NV_OVERFLOWS_INTEGERS_AT 256.0*256.0*256.0*256.0*256.0*256.0*2.0*2.0*2.0*2.0*2.0
-#define IVdf "I64d" /**/
-#define UVuf "I64u" /**/
-#define UVof "I64o" /**/
-#define UVxf "I64x" /**/
-#define UVXf "I64X" /**/
+#define IVdf "ld" /**/
+#define UVuf "lu" /**/
+#define UVof "lo" /**/
+#define UVxf "lx" /**/
+#define UVXf "lX" /**/
-#define ST_DEV_SIGN 1 /* st_dev sign */
-#define ST_DEV_SIZE 4 /* st_dev size */
-#define ST_INO_SIZE 8 /* st_ino size */
+#define ST_INO_SIZE 4 /* st_ino size */
-#define USE_64_BIT_INT /**/
-#define USE_LARGE_FILES /**/
-#define ARCHNAME "MSWin32-x64-perlio" /**/
+#define ARCHNAME "MSWin32-x86-perlio" /**/
-#define GETENV_PRESERVES_OTHER_THREAD /**/
+#define USE_THREADS /* until src is revised*/
-#define Off_t long long /* <offset> type */
-#define LSEEKSIZE 8 /* <offset> size */
-#define Off_t_size 8 /* <offset> size */
+#define Off_t long /* <offset> type */
+#define LSEEKSIZE 4 /* <offset> size */
+#define Off_t_size 4 /* <offset> size */
-#define Size_t_size 8 /**/
+#define Size_t_size 4 /**/ |
|
For 5.34 (currently what we're working on with the Dockerfile in spbuild):
Since we're using 64-bit compilers on a 64-bit system, should we not leave all of the size definitions as-they-are? ContainerAdministrator@93b8bbba23dc MINGW64 /c/spbuild
# cat foo.c
#include <stdio.h>
#include <sys/types.h>
int main(void) {
off_t blah;
size_t bloop;
return 0;
}
ContainerAdministrator@93b8bbba23dc MINGW64 /c/spbuild
# gcc -E foo.c | grep size_t
__extension__ typedef unsigned long long size_t;
__extension__ typedef long long ssize_t;
typedef size_t rsize_t;
ContainerAdministrator@93b8bbba23dc MINGW64 /c/spbuild
# gcc -E foo.c | grep off_
# 1 "z:\\mingw64\\x86_64-w64-mingw32\\include\\_mingw_off_t.h" 1 3
typedef long _off_t;
# 26 "z:\\mingw64\\x86_64-w64-mingw32\\include\\_mingw_off_t.h" 3
typedef off32_t off_t;
static __attribute__ ((__unused__)) __inline__ __attribute__((__cdecl__)) int fseeko(FILE *_File, _off_t _Offset, int
_Origin) {
static __attribute__ ((__unused__)) __inline__ __attribute__((__cdecl__)) _off_t ftello(FILE *_File) {
off_t blah;
This part we should likely leave, but alter the comments and the OSVERS as well? PS C:\spbuild> [environment]::OSVersion.Version
Major Minor Build Revision
----- ----- ----- --------
10 0 17763 0 |
|
I've been running 5.36 in my experiments on the assumption we can come back to 5.34 once that's done. I should also note that I kept the build config simple and went with the gcc 8.3 toolchain. That saved editing the locations of all the libraries (I have work in progress to reduce hard coding when run under docker). It also resulted in a range of perl core test failures but those might just be due to the perl config used. I'll shift to gcc 11 once we have the perl config patches worked out. |
|
My initial goal was to cut/release 5.34 with the gcc-11.2 setup and then go with gcc-12.2 for 5.36. That was also back in a time when I assumed this whole thing would be much easier. :) |
|
My current optimism is that the two builds will be pretty similar as they will use the same external libs and gcc toolchain. If we can clean up the patches then we might even be able to refactor the build configs as all that will change is the perl version. Maybe. Possibly. |
If you're getting a 64-bit perl as a result, it's because those specs are being ignored/overwritten by gmake.
As I've sort of just implied - I'm aware of no such changes that need to be made. And if Strawberry developers come across %Config settings that could be portably altered to provide enhancement, then please report them as perl5 issues. Cheers, |
|
So a starting point might be to leave most settings as-is in the config files but update these (based on my incomplete understanding of such things):
What of the other settings near to -#define HAS_ATOLL /**/
-#define HAS_NON_INT_BITFIELDS /**/
+#define HAS_BUILTIN_EXPECT /* we have gcc-8.3 */
+#define HAS_BUILTIN_CHOOSE_EXPR /* we have gcc-8.3 */
-#define HAS_MODFL_PROTO /**/
-#define HAS_SETLOCALE /**/
-#define HAS_STRTOLL /**/
-#define HAS_STRTOULLWRT printing formats, running this one liner for a 5.32 install gives these settings. Some are defined in the config file, others are not in the diffs. Should all "I64o" cases be changed to "lo", for example? perl -MData::Printer -MConfig -E"say np %Config" | findstr /i format | findstr /v d_ |
|
So I did a test run with the "uncleaned" GCC 11 toolchain, without any patching of the config files. It results in a large number of errors before failure. A sample is below but they are all essentially the same (let me know if the full set is needed). I assume I've missed something obvious or there are issues with not cleaning up the toolchain files (see #57). For comparison, running with the GCC 8 toolchain from https://strawberryperl.com/package/kmx/64_gcctoolchain/ builds perl and is now running tests. The good news is that it passes several tests it failed when the configs were being patched ( GCC 11 errors: |
|
IIRC (unverified) these issues have been fixed in the perl-5.36.0 source, but not the perl-5.34.0 source. Update: Duh - I knew I had seen those errors before .... but then I attributed them to the wrong issue. Cheers, |
|
I spoke with @xenu on IRC and it looks like we are going to have some issues because we're trying to use the UCRT version of the winlibs build. Perl isn't yet able to work there. We will need to try the MSVCRT build instead for now. @shawnlaffan Can you try changing your checked out Dockerfile to use https://github.com/brechtsanders/winlibs_mingw/releases/download/10.4.0-10.0.0-msvcrt-r1/winlibs-x86_64-posix-seh-gcc-10.3.0-mingw-w64-9.0.0-r2.zip instead and test with that? That'd take us down to gcc v10 as well. |
|
Will give that a go. I assume it will need a full rebuild of the extlibs? |
Yes, unfortunately, everything will have to be built with that new winlibs selection. |
Just FYI, there's a minor typo in the link, Edit: although that's the msvcrt. More thinking needed. Edit 2: This should be the one. |
|
Sorry about that. The links on the actual WinLibs site are wrong. |
No dramas. I went straight to github, although it's about seven pages in under the releases... |
|
The external libs compile pretty well under GCC 10. There is a test failure for netcdf I have yet to look into closely but I think (hope) that this is testing utilities we don't distribute anyway. The others are all known (libgdbm, libsodium and libssh2). I think it makes sense to upload these to a github release on the extlibs repo, flagged as a pre-release. That will allow other people to get started with the builds. I'll also push the 5.36 scripts to a branch on this repo. WRT the perl builds, we are hitting issues with IO.xs (see below). These are already fixed in the perl core but after 5.36 was released. @sisyphus - Are there other such issues lurking? Meanwhile I'll try adding the patch from 19663. |
|
After patching with Perl/perl5#19663 and also https://rt.cpan.org/Transaction/Display.html?id=2258362 the core build completes. Now I need to rerun with core tests on and see how far it gets. |
|
Two more notes: The patch.exe that comes with strawberry perl 5.32 does not work when applying the patches (5.32 is being used to bootstrap the process and is currently at the front of the path). The winlibs version does, however, work. I have yet to add the two defines flagged in #56 (comment) +#define HAS_BUILTIN_EXPECT /* we have gcc-8.3 */
+#define HAS_BUILTIN_CHOOSE_EXPR /* we have gcc-8.3 */ |
|
Current files are in the wip_536 branch: https://github.com/StrawberryPerl/Perl-Dist-Strawberry/tree/wip_536 |
(Sorry, my "memory" reckoned that the requisite perl patches had been applied to the perl source much earlier than they actually were.) Beyond that, I can't think of any issues until you get to using gcc-12 compiler - where you might want to apply Perl/perl5@51634b4 to win32/GNUmakefile Be aware that these recent compilers from winlibs set _WIN32_WINNT to 0x0a00 which implies that the system is Windows 10. Cheers, |
|
Thanks Rob. I might open a separate issue to flag the GCC 12 changes for future reference. Incidentally, the core tests passed except for these two. Maybe another define is needed? (Edit - that's a general question, not for @sisyphus specifically). Edit 2: And this one, although it is to be expected since Socket.xs was patched. And perhaps more importantly there is |
Yes, some expert intervention might be needed here. Cheers, |
|
|
|
Hmmm ... I've now built perl-5.36.0 on Windows 11 using winlibs 64-bit gcc-10.3.0. All other files are as per original perl-5.36.0 source. I installed that perl anyway - just in case that might prove useful somewhere down the track, and ran There's apparently some issue with gcc-10.3.0 that's not afflicting 11.3.0. Cheers, |
|
@sisyphus - Just to be sure, did you use the winlibs versions ? If so, which specific links? It could also be a difference in the build env. FWIW, my builds are running in a cmd shell on a mingw bash shell on a powershell in a windows 10 docker image on a windows 10 box. |
Yes: The winlibs website currently lists a later version (release 3) of 11.3.0, containing llvm-14.0.3.
Yes - but on my Windows 11 machine, in exactly the same environment, 10.3.0 is failing to handle the same perl source that 11.3.0 handles just fine. Incidentally, last night on Windows 7, I built perl-5.36.0 straight out of the box (no patches) using that same gcc-10.3.0. I'll just mention that, as regards the upcoming 5.38.0 release: Actually, I do build with one (unnecessary) patch that puts the mingw-runtime version of the toolset that built perl into the currently empty $Config{gnulibc_version}. Cheers, |
|
Thanks @sisyphus @genio - any thoughts on the GCC 11? The version without LLVM is at https://github.com/brechtsanders/winlibs_mingw/releases/download/11.2.0-14.0.0-9.0.0-msvcrt-r7/winlibs-x86_64-posix-seh-gcc-11.2.0-mingw-w64msvcrt-9.0.0-r7.zip |
|
When I started the task of getting the build environment together using the Dockerfile, gcc11 was the current version available on WinLibs. The latest available was my only selection criteria (also, I hoped the UCRT version would work well); there wasn't any other reasoning behind that selection. If gcc 10, 11, or 12 works, then I'm all for whichever one makes our lives easier. |
|
Thanks. Will keep building with GCC 10 for now, largely because I have all the extlibs done (with known caveats). I updated the config.gc values as below (just showing additions) so the libs from cpan go to the right vendor dir, and now we see additional fails in the main build.
Do the +archlib='~INST_TOP~\lib'
+archlibexp='~INST_TOP~\lib'
+bin='~INST_TOP~\bin'
+binexp='~INST_TOP~\bin'
+d_builtin_choose_expr='define'
+d_builtin_expect='define'
+d_mkstemp='define'
+d_ndbm='define'
+d_vendorarch='define'
+d_vendorbin='define'
+d_vendorlib='define'
+d_vendorscript='define'
+dlext='xs.dll'
+i_db='define'
+i_dbm='define'
+i_gdbm='define'
+i_ndbm='define'
+installarchlib='~INST_TOP~\lib'
+installbin='~INST_TOP~\bin'
+installhtmldir=''
+installhtmlhelpdir=''
+installman1dir=''
+installman3dir=''
+installprefix='~INST_TOP~'
+installprefixexp='~INST_TOP~'
+installprivlib='~INST_TOP~\lib'
+installscript='~INST_TOP~\bin'
+installsitearch='~INST_TOP~\site\lib'
+installsitebin='~INST_TOP~\site\bin'
+installsitelib='~INST_TOP~\site\lib'
+installsitescript='~INST_TOP~\site\bin'
+installvendorarch='~INST_TOP~\vendor\lib'
+installvendorbin='~INST_TOP~\bin'
+installvendorlib='~INST_TOP~\vendor\lib'
+installvendorscript='~INST_TOP~\bin'
+man1dir=''
+man1direxp=''
+man3dir=''
+man3direxp=''
+myuname='Win32 strawberry-perl 5.32.0.1 #1 Sun Aug 2 17:07:44 2020 i386'
+osvers='6.1'
+perlpath='~INST_TOP~\bin\perl.exe'
+privlib='~INST_TOP~\lib'
+privlibexp='~INST_TOP~\lib'
+scriptdir='~INST_TOP~\bin'
+scriptdirexp='~INST_TOP~\bin'
+sitearch='~INST_TOP~\site\lib'
+sitearchexp='~INST_TOP~\site\lib'
+sitebin='~INST_TOP~\site\bin'
+sitebinexp='~INST_TOP~\site\bin'
+sitelib='~INST_TOP~\site\lib'
+sitelibexp='~INST_TOP~\site\lib'
+siteprefix='~INST_TOP~\site'
+siteprefixexp='~INST_TOP~\site'
+sitescript='~INST_TOP~\site\bin'
+sitescriptexp='~INST_TOP~\site\bin'
+usevendorprefix='define'
+usrinc='C:\strawberry\c\include'
+vendorarch='~INST_TOP~\vendor\lib'
+vendorarchexp='~INST_TOP~\vendor\lib'
+vendorbin='~INST_TOP~\bin'
+vendorbinexp='~INST_TOP~\bin'
+vendorlib='~INST_TOP~\vendor\lib'
+vendorlibexp='~INST_TOP~\vendor\lib'
+vendorprefix='~INST_TOP~\vendor'
+vendorprefixexp='~INST_TOP~\vendor'
+vendorscript='~INST_TOP~\bin'
+vendorscriptexp='~INST_TOP~\bin' |
|
And just a data point. The MSYS2 project seems to have a working perl 5.32 using UCRT. See for example this patch. They have a few other patches, some of which might be useful for Strawberry Perl. One of their patches is similar to what @sisyphus noted in an earlier comment and which I think is now also in blead? (If so is it a candidate for 5.36.1 or does it not fit the backport policy?) (Edited for grammar) |
Steve Hay is currently preparing for the 5.36.1 release. It feels to me that such a backport should be allowable as a "portability" fix, but I'm not sure that I understand the rules all that well. AIUI, you'd be wanting the patches to dist/IO/poll.h as well as the ones given in @Biswa96's patch at https://github.com/msys2/MINGW-packages/blob/master/mingw-w64-perl/007-use-winsock-socket-functions.patch. Applying @Biswa96's patches are (I think) as simple as syncing the current cpan Socket source into the perl-5.36.1 source. Cheers, PS |
|
Thanks Rob. The build is currently applying both those patches, assuming they are the same as in Perl/perl5#19663 and Perl/perl5#18911 So, even if the changes don't make it in to 5.36.1, Strawberry Perl will include them. Edit: ...although Perl/perl5#18911 is still open so incomplete? If it is then it is unlikely to be up for backport. |
I believe it was still open due to oversight ;-) Cheers, |
|
Just returning to the original aim of this issue, I have added code on the wip_536 branch to use hashes of values to update the The approach is a bit clunky as it hijacks the target name with a keyword so the destination is the hash of changes, so the source and destination are reversed in such cases. An alternate approach would be to also support an array of arrays so then the hash can be first. f8f3b97 (The branch will need a bit of cleanup before any PR) Edit - Also, while the implementation is clunky, this approach is much simpler and more future proof than working with patch files. |
|
The implementation works. Any modifications can be done under their own issues. |
The 5.32 and earlier builds override several of the configuration files under the perl/win32 dir before building perl. The files live under the share dir, for example https://github.com/StrawberryPerl/Perl-Dist-Strawberry/tree/master/share/perl-5.32
Some of these are to change build args such as NDBM and GDBM (although see StrawberryPerl/build-extlibs#13 for current build status), others are changing the installation dirs, dll extensions etc.
However, others change pointer sizes and the like.
There have been changes in the perl core since the 5.32 files were generated, so how much do the strawberry perl versions of these files need to be updated?
Diffs comparing the perl 5.36 versions with the strawberry perl 5.32 versions, excluding whitespace, are attached.
Many of the differences are typo fixes and some are just file reordering (HAS_MSG in Config.gc), but there are cases where new arguments in perl are not in the strawberry perl versions (e.g. HASATTRIBUTE_ALWAYS_INLINE in Config.gc).
Some of the missing parameters might not matter as they will be left undefined by default anyway. Right?
Others maybe should be defined? If so which?
config.gc.patch
config_H.gc.patch
FindExt.pm.patch
The text was updated successfully, but these errors were encountered: