can't use gcc-ar as Bazel's ar

[benjaminp]

gcc provides a wrapper, gcc-ar, over binutils ar that passes the proper --plugin option so that ar can understand LTO objects. gcc-ar does some dodgy munging of the command line, which causes it to fail if all arguments are passed via param file, which is precisely how Bazel likes to pass ar arguments. Would it be possible to work around this in Bazel by putting the ar mode argument on the command line rather than the param file?

[laszlocsomor]

paging passanger @mhlopko at Gate #3760 to answer

[hlopko]

The problem is to decide what are arguments and what are objects. Not for ar, but for msvc, emscripten, and for all exotic wrapper scripts people have. We currently hardcode some logic for linking, and we need to get rid of it as it is confusing, and people actually started relying on the fact that some flags will be moved to the params file and then params file is placed elsewhere as the original flag. We definitely don't want to put more logic for ar.

I see 2 ways forward:

• implement some way of expressing if given flag (defined in the CROSSTOOL) can be moved to params file
• use wrapper scripts

For the short term, I'd suggest using wrapper scripts for your build, in which you'd read the content of params file, and shuffle it as you require.

[benjaminp]

A fairly general solution would be to be add a allowed_in_param_file boolean flag to theFlagGroup message, which would let the crosstool control param file usage precisely. Besides my request, I imagine this flag could be used to move hard-coded logic in LinkCommandLine.extractArgumentsForDynamicLinkParamFile into the crosstool.

(I'm not actually sure what the reason for the special-casing inextractArgumentsForDynamicLinkParamFile is. Can't everything go in the param file?)

[hlopko]

Yes, what you propose is what I had in mind for the first point. It should work (I'm not aware of any reason why it shouldn't). This would solve the same problem extractArgumentsForDynamicLinkParamFile solves. It's possible that this method is older than the crosstool.

That said, for this to work I would have to migrate all internal crosstools to action_configs/features. I'll have to do it at some point, but before that I want to do some more radical experiments with crosstool this quarter. So I won't have time to work on this feature request before EO2017.

But again, you'd need to have a custom crosstool, and with that you can use workaround with wrapper scripts, so you shouldn't be blocked.

[benjaminp]

Yep, I agree this is low priority. Thanks for weighing in, though.

[4ZM]

I have the same problem trying to link with lto. gcc-ar doesn't call ar. But I don't understand the suggested workaround. What would the custom wrapper script for ar in the custom crosstool look like?

[benjaminp]

Your custom wrapper script would read the param file and exec gcc-ar with the arguments directly.

[4ZM]

Still don't understand how this should work. The argument to the wrapper script I create doesn't receive a parameter file as far as I can tell. How can I access the parameter file?

[AndersSundman]

ping on the above question

[benjaminp]

Such questions are probably better asked on a designated help forum like Stack overflow.

[hlopko]

Update: Our current plan is to remove special casing of C++ command lines completely, and instead make it all or nothing, like the rest of other actions are doing. That doesn't change anything for the current issue re gcc-ar, but I wanted to share.

Re above questions, I'll need a repro, which platform are you running on, what compiler version, do you have your own CROSSTOOL? I agree that this issue is not the right place to discuss this. Feel free to open a separate issue with as much info as you have, and clean problem statement. Thanks!

[mpictor]

This should reproduce for you @mhlopko .
Platform amd64, gcc version 7.3.0, bazel 0.17.2. My CROSSTOOL is in the below repo at tools/amd64_compiler/CROSSTOOL

git clone https://github.com:mpictor/bazel-amd64-crosstool

git checkout lto-gcc-ar-bug

bazel build --crosstool_top=//tools/amd64_compiler:toolchain --cpu=k8 --verbose_failures //:lib

---

My results:
$ bazel info release

release 0.17.2

$ bazel build --crosstool_top=//tools/amd64_compiler:toolchain --cpu=k8 --verbose_failures //:lib

(21:38:32) INFO: Current date is 2018-12-02
(21:38:32) INFO: Analysed target //:lib (1 packages loaded).
(21:38:32) INFO: Found 1 target...
(21:38:32) ERROR: /home/mark/build/amd64-crosstool/BUILD:13:1: Linking of rule '//:lib' failed (Exit 1): x86_64-unknown-linux-gnu-gcc-ar failed: error executing command 
  (cd /home/mark/.cache/bazel/_bazel_mark/72eaf95e2fa16b7b926bc447a0b9dafd/execroot/bazel_amd64_crosstool && \
  exec env - \
    PATH='/home/mark/.yarn/bin:/home/mark/.config/yarn/global/node_modules/.bin:~/bin:/usr/x86_64-pc-linux-gnu/gcc-bin/7.3.0:/usr/lib/llvm/6/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/opt/bin:/opt/nvidia-cg-toolkit/bin:/usr/games/bin:/home/mark/go/bin' \
    PWD=/proc/self/cwd \
  tools/amd64_compiler/x86_64-unknown-linux-gnu/x86_64-unknown-linux-gnu-gcc-ar @bazel-out/k8-fastbuild/bin/liblib.a-2.params)

Use --sandbox_debug to see verbose messages from the sandbox
/home/mark/.cache/bazel/_bazel_mark/72eaf95e2fa16b7b926bc447a0b9dafd/external/x86_64_unknown_linux_gnu_gcc_730/bin/../lib/gcc/x86_64-unknown-linux-gnu/7.3.0/../../../../x86_64-unknown-linux-gnu/bin/ar: invalid option -- '@'
Usage: /home/mark/.cache/bazel/_bazel_mark/72eaf95e2fa16b7b926bc447a0b9dafd/external/x86_64_unknown_linux_gnu_gcc_730/bin/../lib/gcc/x86_64-unknown-linux-gnu/7.3.0/../../../../x86_64-unknown-linux-gnu/bin/ar [emulation options] [-]{dmpqrstx}[abcDfilMNoPsSTuvV] [--plugin <name>] [member-name] [count] archive-file file...
       /home/mark/.cache/bazel/_bazel_mark/72eaf95e2fa16b7b926bc447a0b9dafd/external/x86_64_unknown_linux_gnu_gcc_730/bin/../lib/gcc/x86_64-unknown-linux-gnu/7.3.0/../../../../x86_64-unknown-linux-gnu/bin/ar -M [<mri-script]
 commands:
  d            - delete file(s) from the archive
  m[ab]        - move file(s) in the archive
  p            - print file(s) found in the archive
  q[f]         - quick append file(s) to the archive
  r[ab][f][u]  - replace existing or insert new file(s) into the archive
  s            - act as ranlib
  t            - display contents of archive
  x[o]         - extract file(s) from the archive
 command specific modifiers:
  [a]          - put file(s) after [member-name]
  [b]          - put file(s) before [member-name] (same as [i])
  [D]          - use zero for timestamps and uids/gids
  [U]          - use actual timestamps and uids/gids (default)
  [N]          - use instance [count] of name
  [f]          - truncate inserted file names
  [P]          - use full path names when matching
  [o]          - preserve original dates
  [u]          - only replace files that are newer than current archive contents
 generic modifiers:
  [c]          - do not warn if the library had to be created
  [s]          - create an archive index (cf. ranlib)
  [S]          - do not build a symbol table
  [T]          - make a thin archive
  [v]          - be verbose
  [V]          - display the version number
  @<file>      - read options from <file>
  --target=BFDNAME - specify the target object format as BFDNAME
 optional:
  --plugin <p> - load the specified plugin
 emulation options: 
  No emulation specific options
/home/mark/.cache/bazel/_bazel_mark/72eaf95e2fa16b7b926bc447a0b9dafd/external/x86_64_unknown_linux_gnu_gcc_730/bin/../lib/gcc/x86_64-unknown-linux-gnu/7.3.0/../../../../x86_64-unknown-linux-gnu/bin/ar: supported targets: elf64-x86-64 elf32-i386 elf32-iamcu elf32-x86-64 a.out-i386-linux pei-i386 pei-x86-64 elf64-l1om elf64-k1om elf64-little elf64-big elf32-little elf32-big plugin srec symbolsrec verilog tekhex binary ihex
Target //:lib failed to build
(21:38:32) INFO: Elapsed time: 0.545s, Critical Path: 0.30s
(21:38:32) INFO: 0 processes.
(21:38:32) FAILED: Build did NOT complete successfully

[hlopko]

I think I don't understand the problem? The command line is:

tools/amd64_compiler/x86_64-unknown-linux-gnu/x86_64-unknown-linux-gnu-gcc-ar @bazel-out/k8-fastbuild/bin/liblib.a-2.params

And gcc-ar compains that @ is not recognized option. What you need to do is to replace gcc-ar with a custom script, that will read the params file coming after @, and pass those flags to gcc-ar inline. I'll now close this issue, since there's no plan to fix this in Bazel in any other way than with a wrapper script.

[mpictor]

ah, ok. thanks!

[yesudeep]

Wrapper script that I'm using:

#!/usr/bin/env python3
# -*- coding: utf-8 -*-

import subprocess
import sys


def read_params_file(filename):
    params = []
    with open(filename) as f:
        for line in f.readlines():
            line = line.strip()
            if line:
                params.append(line)
    return params


def parse_args(argv):
    new_argv = []
    for arg in argv:
        if arg.startswith("@"):
            new_argv += read_params_file(arg[1:])
        else:
            new_argv.append(arg)
    return new_argv


def main(argv):
    new_args = parse_args(argv)
    subprocess.run(
        ["/usr/bin/gcc-ar-original"] + new_args,
    )


if __name__ == "__main__":
    main(sys.argv[1:])