Build fails when using Bison 3.2 #11

jasonboggess · 2018-11-02T02:50:14Z

Attempted to install via pip on macos 10.14 using python 3.7 and pip3 version 18.0

Got the following error:

    /private/var/folders/_9/fz23mhs14q135p8kh6f4dg980000gn/T/pip-install-g5ph25wz/yaramod/build/src/yaramod/yy/yy_parser.hpp:738:102: error: too many arguments provided to function-like macro invocation
          basic_symbol (typename Base::kind_type t, YY_RVREF (std::pair<nonstd::optional<std::uint64_t>, nonstd::optional<std::uint64_t>>) v, YY_RVREF (location_type) l);

The text was updated successfully, but these errors were encountered:

s3rvac · 2018-11-02T08:56:59Z

Hi. Could you please try building yaramod and its Python bindings manually by following these two steps?

First, ensure that you have Flex >= 2.6 and Bison >= 3.0 and set proper paths (ensure that they exist on your system):

# Apple ships old Flex & Bison, so Homebrew versions should be used.
export CMAKE_INCLUDE_PATH="/usr/local/opt/flex/include"
export CMAKE_LIBRARY_PATH="/usr/local/opt/flex/lib;/usr/local/opt/bison/lib"
export PATH="/usr/local/opt/flex/bin:/usr/local/opt/bison/bin:$PATH"

Clone the repo and run the build:

git clone https://github.com/avast-tl/yaramod.git
cd yaramod
mkdir build
cd build
cmake -DYARAMOD_PYTHON=ON ..
make

If the build fails, please post the output here. If the build passes, could you please try running pip install [..] again but after setting the above variables?

BTW, which of the following two commands did you run?

pip install git+https://github.com/avast-tl/yaramod.git
pip install yaramod

jasonboggess · 2018-11-02T12:30:38Z

Hi! Thanks for the super fast reply. I did the above steps (cloned the repo and tried to build manually) and still got the same errors.

Used homebrew to install flex and bison.

Here are what versions I'm using and what my variables ended up to be:

$ flex --version
flex 2.6.4
$ bison --version
bison (GNU Bison) 3.2
Written by Robert Corbett and Richard Stallman.

Copyright (C) 2018 Free Software Foundation, Inc.
This is free software; see the source for copying conditions.  There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.
$ echo $CMAKE_INCLUDE_PATH
/usr/local/opt/flex/include
$ echo $CMAKE_LIBRARY_PATH
/usr/local/opt/flex/lib;/usr/local/opt/bison/lib
$ cmake --version
cmake version 3.12.3

CMake suite maintained and supported by Kitware (kitware.com/cmake).
$ make --version
GNU Make 3.81
Copyright (C) 2006  Free Software Foundation, Inc.
This is free software; see the source for copying conditions.
There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A
PARTICULAR PURPOSE.

This program built for i386-apple-darwin11.3.0
$ gcc --version
Configured with: --prefix=/Applications/Xcode.app/Contents/Developer/usr --with-gxx-include-dir=/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.14.sdk/usr/include/c++/4.2.1
Apple LLVM version 10.0.0 (clang-1000.11.45.5)
Target: x86_64-apple-darwin18.0.0
Thread model: posix
InstalledDir: /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin

And when running make:

$ make
[  3%] Building CXX object src/CMakeFiles/yaramod.dir/builder/yara_file_builder.cpp.o
In file included from yaramod/src/builder/yara_file_builder.cpp:8:
In file included from yaramod/include/yaramod/parser/parser_driver.h:13:
In file included from yaramod/include/yaramod/parser/lexer.h:13:
yaramod/build/src/yaramod/yy/yy_parser.hpp:738:102: error: too many arguments provided to function-like macro invocation
      basic_symbol (typename Base::kind_type t, YY_RVREF (std::pair<nonstd::optional<std::uint64_t>, nonstd::optional<std::uint64_t>>) v, YY_RVREF (location_type) l);
                                                                                                     ^
yaramod/build/src/yaramod/yy/yy_parser.hpp:80:10: note: macro 'YY_RVREF' defined here
# define YY_RVREF(Type)    Type&&
         ^
yaramod/build/src/yaramod/yy/yy_parser.hpp:738:49: error: unknown type name 'YY_RVREF'
      basic_symbol (typename Base::kind_type t, YY_RVREF (std::pair<nonstd::optional<std::uint64_t>, nonstd::optional<std::uint64_t>>) v, YY_RVREF (location_type) l);
                                                ^
yaramod/build/src/yaramod/yy/yy_parser.hpp:1209:76: error: too many arguments provided to function-like macro invocation
    make_REGEXP_RANGE (YY_COPY (std::pair<nonstd::optional<std::uint64_t>, nonstd::optional<std::uint64_t>>) v, YY_COPY (location_type) l);
                                                                           ^
yaramod/build/src/yaramod/yy/yy_parser.hpp:81:10: note: macro 'YY_COPY' defined here
# define YY_COPY(Type)     Type
         ^
yaramod/build/src/yaramod/yy/yy_parser.hpp:1209:24: error: unknown type name 'YY_COPY'
    make_REGEXP_RANGE (YY_COPY (std::pair<nonstd::optional<std::uint64_t>, nonstd::optional<std::uint64_t>>) v, YY_COPY (location_type) l);
                       ^
yaramod/build/src/yaramod/yy/yy_parser.hpp:1632:128: error: too many arguments provided to function-like macro invocation
   Parser ::basic_symbol<Base>::basic_symbol (typename Base::kind_type t, YY_RVREF (std::pair<nonstd::optional<std::uint64_t>, nonstd::optional<std::uint64_t>>) v, YY_RVREF (location_type) l)
                                                                                                                               ^
yaramod/build/src/yaramod/yy/yy_parser.hpp:80:10: note: macro 'YY_RVREF' defined here
# define YY_RVREF(Type)    Type&&
         ^
yaramod/build/src/yaramod/yy/yy_parser.hpp:1632:75: error: unknown type name 'YY_RVREF'
   Parser ::basic_symbol<Base>::basic_symbol (typename Base::kind_type t, YY_RVREF (std::pair<nonstd::optional<std::uint64_t>, nonstd::optional<std::uint64_t>>) v, YY_RVREF (location_type) l)
                                                                          ^
yaramod/build/src/yaramod/yy/yy_parser.hpp:2717:84: error: too many arguments provided to function-like macro invocation
   Parser ::make_REGEXP_RANGE (YY_COPY (std::pair<nonstd::optional<std::uint64_t>, nonstd::optional<std::uint64_t>>) v, YY_COPY (location_type) l)
                                                                                   ^
yaramod/build/src/yaramod/yy/yy_parser.hpp:81:10: note: macro 'YY_COPY' defined here
# define YY_COPY(Type)     Type
         ^
yaramod/build/src/yaramod/yy/yy_parser.hpp:2717:32: error: unknown type name 'YY_COPY'
   Parser ::make_REGEXP_RANGE (YY_COPY (std::pair<nonstd::optional<std::uint64_t>, nonstd::optional<std::uint64_t>>) v, YY_COPY (location_type) l)
                               ^
yaramod/build/src/yaramod/yy/yy_parser.hpp:1484:7: error: no matching constructor for initialization of 'yaramod::yy::Parser::by_type'
    : Base (YY_MOVE (other))
      ^     ~~~~~~~~~~~~~~~
yaramod/build/src/yaramod/yy/yy_parser.hpp:2103:12: note: in instantiation of member function 'yaramod::yy::Parser::basic_symbol<yaramod::yy::Parser::by_type>::basic_symbol' requested here
    return symbol_type (token::END, YY_MOVE (l));
           ^
yaramod/build/src/yaramod/yy/yy_parser.hpp:2043:22: note: candidate constructor not viable: no known conversion from 'typename remove_reference<basic_symbol<by_type> &>::type' (aka 'yaramod::yy::Parser::basic_symbol<yaramod::yy::Parser::by_type>') to 'const yaramod::yy::Parser::by_type' for 1st argument
   Parser ::by_type::by_type (const by_type& other)
                     ^
yaramod/build/src/yaramod/yy/yy_parser.hpp:2048:22: note: candidate constructor not viable: no known conversion from 'typename remove_reference<basic_symbol<by_type> &>::type' (aka 'yaramod::yy::Parser::basic_symbol<yaramod::yy::Parser::by_type>') to 'yaramod::yy::Parser::token_type' (aka 'yaramod::yy::Parser::token::yytokentype') for 1st argument
   Parser ::by_type::by_type (token_type t)
                     ^
yaramod/build/src/yaramod/yy/yy_parser.hpp:2038:22: note: candidate constructor not viable: requires 0 arguments, but 1 was provided
   Parser ::by_type::by_type ()
                     ^
yaramod/build/src/yaramod/yy/yy_parser.hpp:1488:13: error: cannot initialize object parameter of type 'const yaramod::yy::Parser::by_type' with an expression of type 'yaramod::yy::Parser::basic_symbol<yaramod::yy::Parser::by_type>'
    switch (other.type_get ())
            ^~~~~
10 errors generated.
make[2]: *** [src/CMakeFiles/yaramod.dir/builder/yara_file_builder.cpp.o] Error 1
make[1]: *** [src/CMakeFiles/yaramod.dir/all] Error 2
make: *** [all] Error 2

FYI I ran:

pip3 install yaramod

metthal · 2018-11-02T12:49:48Z

I am not sure if it'll help because the errors are from bison generated parser, but I see some nonstd::optional in the output so I tried to update optional-lite to the newest version in new-optional-lite branch. Can you please try to compile this branch? Thanks.

jasonboggess · 2018-11-02T22:54:46Z

@metthal I deleted the build directory, and checked out your branch, but same errors.

metthal · 2018-11-03T11:25:20Z

It may have something to do with this. From what I see, you use bison 3.2 which was rolled out just few days ago so this might be some bug in the new release which showed itself only with our parser definition. If this is an option for you, could you please try to revert bison to 3.1 or 3.0.x? I'll try to upgrade to bison 3.2 on my machine as soon as I get to it. If it's really problem in the new release, we might need to file a bug there.

s3rvac · 2018-11-03T11:49:22Z

@metthal Good catch. The issue is really caused by Bison 3.2. I was able to reproduce it even on Ubuntu inside Docker via the following Dockerfile:

FROM ubuntu:bionic

RUN apt-get -y update && \
    apt-get install -y build-essential cmake git flex wget

RUN wget https://ftp.gnu.org/gnu/bison/bison-3.2.tar.gz && \
    tar xf bison-3.2.tar.gz && \
    cd bison-3.2 && \
    ./configure && \
    make -j$(nproc) && \
    make install

RUN git clone https://github.com/avast-tl/yaramod && \
    mkdir yaramod/build && \
    cd yaramod/build && \
    cmake .. && \
    make -j$(nproc)

When you run

$ docker build -t yaramod .

the build fails with

In file included from /yaramod/include/yaramod/parser/lexer.h:13:0,
                 from /yaramod/include/yaramod/parser/parser_driver.h:13,
                 from /yaramod/src/builder/yara_file_builder.cpp:8:
/yaramod/build/src/yaramod/yy/yy_parser.hpp:738:134: error: macro "YY_RVREF" passed 2 arguments, but takes just 1
       basic_symbol (typename Base::kind_type t, YY_RVREF (std::pair<nonstd::optional<std::uint64_t>, nonstd::optional<std::uint64_t>>) v, YY_RVREF (location_type) l);
                                                                                                                                      ^
[..]

jasonboggess · 2018-11-03T14:16:26Z

I manually installed 3.1.0 and 3.0.5 of bison and I still see the same issue. I made sure that bison --version was correct and that my env variables were set:

export CMAKE_INCLUDE_PATH="/usr/local/opt/flex/include"
export CMAKE_LIBRARY_PATH="/usr/local/opt/flex/lib;/usr/local/opt/bison/lib"
export PATH="/usr/local/opt/flex/bin:/usr/local/opt/bison/bin:$PATH"

s3rvac · 2018-11-03T14:28:59Z

This is weird. With 3.1.0 and 3.0.5, the build succeeds for me. It only fails with 3.2.0.

s3rvac · 2018-11-03T14:43:13Z

@jasonboggess Based on the output you have provided in the (now deleted) comment, it seems to me that you are running cmake/make from a build directory in which you have already run cmake/make (maybe with Bison 3.2?). Could you please try running cmake .. and make from a completely clean build directory? If you indeed have Bison 3.1, cmake .. should print the following line

-- Found BISON: /usr/local/opt/bison/bin (found version "3.1")

and the build should succeed.

jasonboggess · 2018-11-04T16:15:10Z

Yes, I think that was it. Thanks!

s3rvac · 2018-11-04T17:50:23Z

Great 👍 We will leave the issue open until we fix the build with Bison 3.2.

metthal · 2018-11-08T10:44:10Z

I've resolved the problem with bison 3.2. The should now be able to compile yaramod while having bison 3.2.

slw07g · 2019-02-16T21:51:14Z

I'm having the issue with bison 3.3 (the version homebrew installed)

metthal · 2019-02-16T22:12:03Z

What kind of issues? I see that some of the directives we use are now deprecated but it should work just fine despite the warnings. I'll fix the warnings, thanks for pointing them out, but you should still be able to compile and use it.

slw07g · 2019-02-16T22:28:19Z

$> cmake .
...
-- Found FLEX: /usr/local/homebrew/bin/flex (found version "2.6.4") 
-- Found BISON: /usr/local/homebrew/bin/bison (found version "3.3.2") 
...
$> make
...
$> cat external/src/yaramod-project-stamp/yaramod-project-build-err.log 
parser/yy/parser.y:7.1-5: invalid directive: `%code'
parser/yy/parser.y:7.7-14: syntax error, unexpected identifier
make[5]: *** [src/yaramod/yy/yy_parser.cpp] Error 1
make[4]: *** [src/CMakeFiles/yaramod.dir/all] Error 2
make[3]: *** [all] Error 2
~

metthal · 2019-02-16T22:43:39Z

This error reminds me of reports we had when pre-3 version of bison was used instead. Could you please paste here contents of external/src/yaramod-project-stamp/yaramod-project-configure-out.log?

slw07g · 2019-02-16T22:52:27Z

-- Configuring done
-- Generating done
-- Build files have been written to: /Users/<username>/retdec/external/src/yaramod-project-build

metthal · 2019-02-16T22:58:07Z

Is that everything? Because my looks like this

-- The CXX compiler identification is GNU 8.2.1
-- Check for working CXX compiler: /usr/bin/c++ 
-- Check for working CXX compiler: /usr/bin/c++ -- works
-- Detecting CXX compiler ABI info
-- Detecting CXX compiler ABI info - done
-- Detecting CXX compile features
-- Detecting CXX compile features - done
-- Found FLEX: /usr/bin/flex (found version "2.6.1").
-- Found BISON: /usr/bin/bison (found version "3.0.4").
-- Configuring done
-- Generating done
-- Build files have been written to: /home/<username>/projects/retdec/build/external/src/yaramod-project-build

Could you please try to remove external/src/yaramod-project* completely and run cmake and then make one more time?

slw07g · 2019-02-17T04:41:14Z

Yes, that was everything in the file.

I was able to build yaramod-project succesfully after following your suggestion. (Perhaps that should be a part of make clean?)

The commands I ran:

rm -rf external/src/yaramod-project-*
cmake .
make

s3rvac added O-macos P-build C-python-bindings labels Nov 2, 2018

s3rvac changed the title ~~error: too many arguments provided to function-like macro invocation~~ Build fails when using Bison 3.2 Nov 3, 2018

s3rvac added bug and removed C-python-bindings O-macos labels Nov 3, 2018

jasonboggess mentioned this issue Nov 4, 2018

yaracpp build fails with "configure: error: cannot run C compiled programs" avast/retdec#425

Closed

metthal closed this as completed in a35e710 Nov 8, 2018

orivej mentioned this issue Nov 16, 2018

bison: 3.1 -> 3.2.1 NixOS/nixpkgs#49454

Merged

9 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Build fails when using Bison 3.2 #11

Build fails when using Bison 3.2 #11

jasonboggess commented Nov 2, 2018

s3rvac commented Nov 2, 2018

jasonboggess commented Nov 2, 2018

metthal commented Nov 2, 2018

jasonboggess commented Nov 2, 2018

metthal commented Nov 3, 2018

s3rvac commented Nov 3, 2018 •

edited

Loading

jasonboggess commented Nov 3, 2018

s3rvac commented Nov 3, 2018

s3rvac commented Nov 3, 2018 •

edited

Loading

jasonboggess commented Nov 4, 2018

s3rvac commented Nov 4, 2018

metthal commented Nov 8, 2018

slw07g commented Feb 16, 2019

metthal commented Feb 16, 2019

slw07g commented Feb 16, 2019 •

edited

Loading

metthal commented Feb 16, 2019

slw07g commented Feb 16, 2019

metthal commented Feb 16, 2019

slw07g commented Feb 17, 2019 •

edited

Loading

Build fails when using Bison 3.2 #11

Build fails when using Bison 3.2 #11

Comments

jasonboggess commented Nov 2, 2018

s3rvac commented Nov 2, 2018

jasonboggess commented Nov 2, 2018

metthal commented Nov 2, 2018

jasonboggess commented Nov 2, 2018

metthal commented Nov 3, 2018

s3rvac commented Nov 3, 2018 • edited Loading

jasonboggess commented Nov 3, 2018

s3rvac commented Nov 3, 2018

s3rvac commented Nov 3, 2018 • edited Loading

jasonboggess commented Nov 4, 2018

s3rvac commented Nov 4, 2018

metthal commented Nov 8, 2018

slw07g commented Feb 16, 2019

metthal commented Feb 16, 2019

slw07g commented Feb 16, 2019 • edited Loading

metthal commented Feb 16, 2019

slw07g commented Feb 16, 2019

metthal commented Feb 16, 2019

slw07g commented Feb 17, 2019 • edited Loading

s3rvac commented Nov 3, 2018 •

edited

Loading

s3rvac commented Nov 3, 2018 •

edited

Loading

slw07g commented Feb 16, 2019 •

edited

Loading

slw07g commented Feb 17, 2019 •

edited

Loading