Add glibc binary #70

cosmicexplorer · 2018-06-08T23:21:49Z

See pantsbuild/pants#5943. We are providing the glibc binary (55 MB tar.gz file), which we build from source. We provide version 2.27, the newest version. The script works on my machine and the archive produced works with the GLibc BinaryTool from pantsbuild/pants#5943 on my machine as well (the unskipped tests pass).

stuhood

This does not appear to build out of the box on our centos6 image, due to a requirement on bison 2.7 (centos6 has bison 2.4). That image is using devtoolset-7-gcc, but it doesn't appear to include bison.

cosmicexplorer · 2018-06-11T17:38:14Z

I should have checked that before making this pr! I will see if it's reasonable to build or download bison 2.7 as part of the build process for the glibc package.

- also add some TODOs about a possible utils.v2.bash

stuhood · 2018-06-11T20:41:59Z

Let me know when this is reviewable again.

cosmicexplorer · 2018-06-11T21:33:17Z

@stuhood after fd2f10c, this should be reviewable -- it builds on my exact-same-docker-image. Once bison was built, we also needed to install the m4 package, and downgrade the glibc version to 2.23. I will update this in pantsbuild/pants#5943.

cosmicexplorer · 2018-06-11T21:43:25Z

That's wrong! There's a later build failure. Sorry, will be 100% sure next time.

stuhood · 2018-06-11T21:57:45Z

Can you confirm that the downgraded glibc actually needs the newer bison? It looks sufficiently old to maybe work in the centos image.

EDIT: I confirmed that it still needs bison 2.7. But the INSTALL note indicates:

If you wish to regenerate the 'yacc' parser code in the 'intl'
subdirectory you will need

   * GNU 'bison' 2.7 or later

...so perhaps it's possible to skip that step in the configure script.

cosmicexplorer · 2018-06-11T22:12:09Z

perhaps it's possible to skip that step in the configure script.

I believe it is, I had seen that in the release notes but did not think more of it. Will implement that now.

cosmicexplorer · 2018-06-12T21:13:22Z

@stuhood I have not been able to get this to build in a finite amount of time in my centos6 image. Setting -j2 causes errors that seem to be due to parallelism, so I've set it to build in serial, which obviously doesn't help. I would appreciate it if you could try running the script here, but just let me know if it fails and I will investigate -- it seems like it might be a faster path to getting this done than waiting another day for my laptop to get around to it.

stuhood · 2018-06-13T18:33:43Z

I have this (post 10d1827) running in the background.

stuhood · 2018-06-13T18:51:17Z

It failed with:

<snip>
make[2]: Leaving directory `/pantsbuild-binaries/build-support/bin/glibc/linux/x86_64/2.23/glibc-2.23-linux/glibc-2.23/libio'
make  subdir=dlfcn -C dlfcn ..=../ subdir_lib
make[2]: Entering directory `/pantsbuild-binaries/build-support/bin/glibc/linux/x86_64/2.23/glibc-2.23-linux/glibc-2.23/dlfcn'
.././scripts/mkinstalldirs /pantsbuild-binaries/build-support/bin/glibc/linux/x86_64/2.23/glibc-2.23-linux/glibc-build/dlfcn
mkdir -p -- /pantsbuild-binaries/build-support/bin/glibc/linux/x86_64/2.23/glibc-2.23-linux/glibc-build/dlfcn
.././scripts/mkinstalldirs /pantsbuild-binaries/build-support/bin/glibc/linux/x86_64/2.23/glibc-2.23-linux/glibc-build/dlfcn
.././scripts/mkinstalldirs /pantsbuild-binaries/build-support/bin/glibc/linux/x86_64/2.23/glibc-2.23-linux/glibc-build/dlfcn
echo '' > /pantsbuild-binaries/build-support/bin/glibc/linux/x86_64/2.23/glibc-2.23-linux/glibc-build/dlfcn/stamp.osT
echo '' > /pantsbuild-binaries/build-support/bin/glibc/linux/x86_64/2.23/glibc-2.23-linux/glibc-build/dlfcn/stamp.oST
mv -f /pantsbuild-binaries/build-support/bin/glibc/linux/x86_64/2.23/glibc-2.23-linux/glibc-build/dlfcn/stamp.osT /pantsbuild-binaries/build-support/bin/glibc/linux/x86_64/2.23/glibc-2.23-linux/glibc-build/dlfcn/stamp.os
gcc sdlopen.c -c -std=gnu11 -fgnu89-inline  -O2 -Wall -Wundef -Wwrite-strings -fmerge-all-constants -frounding-math -g -Wstrict-prototypes -Wold-style-definition     -ftls-model=initial-exec      -I../include -I/pantsbuild-binaries/build-support/bin/glibc/linux/x86_64/2.23/glibc-2.23-linux/glibc-build/dlfcn  -I/pantsbuild-binaries/build-support/bin/glibc/linux/x86_64/2.23/glibc-2.23-linux/glibc-build  -I../sysdeps/unix/sysv/linux/x86_64/64  -I../sysdeps/unix/sysv/linux/x86_64  -I../sysdeps/unix/sysv/linux/x86  -I../sysdeps/unix/sysv/linux/wordsize-64  -I../sysdeps/x86_64/nptl  -I../sysdeps/unix/sysv/linux/include -I../sysdeps/unix/sysv/linux  -I../sysdeps/nptl  -I../sysdeps/pthread  -I../sysdeps/gnu  -I../sysdeps/unix/inet  -I../sysdeps/unix/sysv  -I../sysdeps/unix/x86_64  -I../sysdeps/unix  -I../sysdeps/posix  -I../sysdeps/x86_64/64  -I../sysdeps/x86_64/fpu/multiarch  -I../sysdeps/x86_64/fpu  -I../sysdeps/x86/fpu/include -I../sysdeps/x86/fpu  -I../sysdeps/x86_64/multiarch  -I../sysdeps/x86_64  -I../sysdeps/x86  -I../sysdeps/ieee754/ldbl-96  -I../sysdeps/ieee754/dbl-64/wordsize-64  -I../sysdeps/ieee754/dbl-64  -I../sysdeps/ieee754/flt-32  -I../sysdeps/wordsize-64  -I../sysdeps/ieee754  -I../sysdeps/generic  -I.. -I../libio -I.   -D_LIBC_REENTRANT -include /pantsbuild-binaries/build-support/bin/glibc/linux/x86_64/2.23/glibc-2.23-linux/glibc-build/libc-modules.h -DMODULE_NAME=libc -include ../include/libc-symbols.h       -o /pantsbuild-binaries/build-support/bin/glibc/linux/x86_64/2.23/glibc-2.23-linux/glibc-build/dlfcn/sdlopen.o -MD -MP -MF /pantsbuild-binaries/build-support/bin/glibc/linux/x86_64/2.23/glibc-2.23-linux/glibc-build/dlfcn/sdlopen.o.dt -MT /pantsbuild-binaries/build-support/bin/glibc/linux/x86_64/2.23/glibc-2.23-linux/glibc-build/dlfcn/sdlopen.o
mv -f /pantsbuild-binaries/build-support/bin/glibc/linux/x86_64/2.23/glibc-2.23-linux/glibc-build/dlfcn/stamp.oST /pantsbuild-binaries/build-support/bin/glibc/linux/x86_64/2.23/glibc-2.23-linux/glibc-build/dlfcn/stamp.oS
mv: cannot stat `/pantsbuild-binaries/build-support/bin/glibc/linux/x86_64/2.23/glibc-2.23-linux/glibc-build/dlfcn/stamp.oST': No such file or directory
make[2]: *** [/pantsbuild-binaries/build-support/bin/glibc/linux/x86_64/2.23/glibc-2.23-linux/glibc-build/dlfcn/stamp.oS] Error 1
make[2]: *** Waiting for unfinished jobs....
make[2]: Leaving directory `/pantsbuild-binaries/build-support/bin/glibc/linux/x86_64/2.23/glibc-2.23-linux/glibc-2.23/dlfcn'
make[1]: *** [dlfcn/subdir_lib] Error 2
make[1]: Leaving directory `/pantsbuild-binaries/build-support/bin/glibc/linux/x86_64/2.23/glibc-2.23-linux/glibc-2.23'
make: *** [all] Error 2
cp: cannot stat `': No such file or directory

cosmicexplorer · 2018-06-13T19:06:55Z

Ok, that was the exact error that the serial make had fixed. I didn't see it repro when I tested it, so I tried adding parallelism again, but I guess that's the thing about concurrency errors. I have reverted that commit and it should work -- if you run into a new issue I will investigate. Thanks for your patience.

stuhood · 2018-06-13T19:07:27Z

...hm. That previous failure was likely because my machine is out of disk space.

cosmicexplorer · 2018-06-13T19:09:16Z

Oh. That might explain why I saw it running concurrently and not single threaded (which I never have yet ran to completion and may have eventually hit the same out of space error?)

…

On Wed, Jun 13, 2018 at 12:07 Stu Hood ***@***.***> wrote: ...hm. That previous failure was likely because my machine is out of disk space. — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <#70 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/ABPqT-gP_U6oAGgYnWj_C-j3605hKGUaks5t8WLwgaJpZM4UhERl> .

This reverts commit 10d1827.

cosmicexplorer · 2018-06-14T03:32:47Z

See discussion in pantsbuild/pants#5943 -- we shouldn't be providing glibc, and we will instead be searching for it on the host, or erroring out.

cosmicexplorer mentioned this pull request Jun 8, 2018

Introduce libc subsystem to find crti.o on linux hosts and unskip the native backend subsystem tests pantsbuild/pants#5943

Merged

cosmicexplorer added 2 commits June 8, 2018 16:22

add glibc binary for linux at version 2.27, built from source

1f89b27

fix typo

8359bb9

cosmicexplorer requested review from stuhood and benjyw June 10, 2018 00:26

benjyw approved these changes Jun 10, 2018

View reviewed changes

stuhood requested changes Jun 11, 2018

View reviewed changes

cosmicexplorer added 2 commits June 11, 2018 13:04

add build-bison.sh, and use our bison to build glibc

0186782

- also add some TODOs about a possible utils.v2.bash

add m4 to build in our centos6 image

24f5256

fix glibc version for compatibility with centos6 kernel

fd2f10c

cosmicexplorer added a commit to cosmicexplorer/pants that referenced this pull request Jun 11, 2018

update default version of glibc as per pantsbuild/binaries#70

bce6634

remove bison script and make glibc build

f61948e

add back parallel make

10d1827

cosmicexplorer added a commit to cosmicexplorer/pants that referenced this pull request Jun 13, 2018

update default version of glibc as per pantsbuild/binaries#70

14b8a7d

cosmicexplorer added 2 commits June 13, 2018 12:10

Revert "add back parallel make"

d642f36

This reverts commit 10d1827.

add back parallel make, and move comments to configure invoke

e8bdce7

cosmicexplorer closed this Jun 14, 2018

cosmicexplorer added a commit to cosmicexplorer/pants that referenced this pull request Jun 23, 2018

update default version of glibc as per pantsbuild/binaries#70

4de4efe

cosmicexplorer added a commit to cosmicexplorer/pants that referenced this pull request Jun 27, 2018

update default version of glibc as per pantsbuild/binaries#70

b6ac0be

cosmicexplorer added a commit to cosmicexplorer/pants that referenced this pull request Jun 29, 2018

update default version of glibc as per pantsbuild/binaries#70

673455c

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add glibc binary #70

Add glibc binary #70

cosmicexplorer commented Jun 8, 2018 •

edited

Loading

stuhood left a comment

cosmicexplorer commented Jun 11, 2018 •

edited

Loading

stuhood commented Jun 11, 2018

cosmicexplorer commented Jun 11, 2018

cosmicexplorer commented Jun 11, 2018

stuhood commented Jun 11, 2018 •

edited

Loading

cosmicexplorer commented Jun 11, 2018

cosmicexplorer commented Jun 12, 2018

stuhood commented Jun 13, 2018

stuhood commented Jun 13, 2018

cosmicexplorer commented Jun 13, 2018

stuhood commented Jun 13, 2018

cosmicexplorer commented Jun 13, 2018 via email

cosmicexplorer commented Jun 14, 2018

Add glibc binary #70

Add glibc binary #70

Conversation

cosmicexplorer commented Jun 8, 2018 • edited Loading

stuhood left a comment

Choose a reason for hiding this comment

cosmicexplorer commented Jun 11, 2018 • edited Loading

stuhood commented Jun 11, 2018

cosmicexplorer commented Jun 11, 2018

cosmicexplorer commented Jun 11, 2018

stuhood commented Jun 11, 2018 • edited Loading

cosmicexplorer commented Jun 11, 2018

cosmicexplorer commented Jun 12, 2018

stuhood commented Jun 13, 2018

stuhood commented Jun 13, 2018

cosmicexplorer commented Jun 13, 2018

stuhood commented Jun 13, 2018

cosmicexplorer commented Jun 13, 2018 via email

cosmicexplorer commented Jun 14, 2018

cosmicexplorer commented Jun 8, 2018 •

edited

Loading

cosmicexplorer commented Jun 11, 2018 •

edited

Loading

stuhood commented Jun 11, 2018 •

edited

Loading