fail to parse the #define macro

[WoolenWang]

Please turn debug info on and reproduce your issue.
The debug info in the log file could help to solve or locate the issue.

Description

Please provide a succinct description of your issue.

Steps to reproduce the problem

Please provide the steps required to reproduce the problem

1. Step A
   parse with include header,setting the include dir ok
2. Step B
   parse the c source code

Expected behavior

parse ok with the sslr ast tree node

Actual behavior

get exception::

`
com.sonar.sslr.api.RecognitionException: Parse error at line 1 column 8:

--> list_inline inlineEOF#define AD_R_OK 0EOF

at org.sonar.sslr.internal.vm.Machine.parse(Machine.java:73)
at com.sonar.sslr.impl.Parser.parse(Parser.java:84)
at com.sonar.sslr.impl.Parser.parse(Parser.java:78)
at org.sonar.cxx.preprocessor.CxxPreprocessor.process(CxxPreprocessor.java:419)
at com.sonar.sslr.impl.Lexer.preprocess(Lexer.java:153)
at com.sonar.sslr.impl.Lexer.preprocess(Lexer.java:143)
at com.sonar.sslr.impl.Lexer.lex(Lexer.java:132)
at com.sonar.sslr.impl.Lexer.lex(Lexer.java:110)
at org.sonar.cxx.preprocessor.CxxPreprocessor.handleIncludeLine(CxxPreprocessor.java:698)
at org.sonar.cxx.preprocessor.CxxPreprocessor.process(CxxPreprocessor.java:449)
at com.sonar.sslr.impl.Lexer.preprocess(Lexer.java:153)
at com.sonar.sslr.impl.Lexer.preprocess(Lexer.java:143)
at com.sonar.sslr.impl.Lexer.lex(Lexer.java:132)
at com.sonar.sslr.impl.Lexer.lex(Lexer.java:110)
at org.sonar.cxx.preprocessor.CxxPreprocessor.handleIncludeLine(CxxPreprocessor.java:698)
at org.sonar.cxx.preprocessor.CxxPreprocessor.process(CxxPreprocessor.java:449)
at com.sonar.sslr.impl.Lexer.preprocess(Lexer.java:153)
at com.sonar.sslr.impl.Lexer.preprocess(Lexer.java:143)
at com.sonar.sslr.impl.Lexer.lex(Lexer.java:132)
at com.sonar.sslr.impl.Lexer.lex(Lexer.java:110)
at com.sonar.sslr.impl.Parser.parse(Parser.java:74)

`

Known workarounds

the parse error header is ::

#ifdef __cplusplus
extern "C" {
#endif

    typedef char                S8;
    typedef unsigned char       U8;
    typedef short               S16;
    typedef unsigned short      U16;
    typedef int                 S32;
    typedef unsigned int        U32;
    typedef long long           S64;
    typedef unsigned long long  U64;
    typedef long                T_P;
    typedef int                 BOOL;
#ifndef TRUE
#define TRUE       1
#endif
#ifndef FALSE
#define FALSE      0
#endif
#define AD_R_FAIL -1
#define AD_R_OK    0

#ifdef __cplusplus
} /* extern "C" */
#endif

LOG file

10:53:17.313 [Ruby-0-Thread-3: E:/git/code_helper/src/code_logger/code_inject/inject_runner.rb:88] WARN org.sonar.cxx.preprocessor.CxxPreprocessor - Cannot parse '#define list_inline inline', ignoring... 10:53:17.313 [Ruby-0-Thread-5: E:/git/code_helper/src/code_logger/code_inject/inject_runner.rb:88] WARN org.sonar.cxx.preprocessor.CxxPreprocessor - Cannot parse '#define AD_R_OK 0', ignoring... 10:53:17.313 [Ruby-0-Thread-3: E:/git/code_helper/src/code_logger/code_inject/inject_runner.rb:88] DEBUG org.sonar.cxx.preprocessor.CxxPreprocessor - Parser exception: '{}'

Related information

• cxx plugin version?
  1.1.0
• SonarQube version?

[guwirth]

@WoolenWang thanks for the description. I have some problems to understand reproduce the problem? Could you give some more information how to reproduce the issue. In best case a reduced source code sample.

[WoolenWang]

@guwirth
I just use the cxx-sslr tool-kit as a jar plugin to java(jruby) and i direct call it's api to parse the code.

the src code use to parse is https://github.com/lua/lua/releases/tag/v5-2-3

get error when i use jruby call api to parse in file lstring.c .(but using the toolkit is ok to parse lstring.c)

the code i write in ruby with jruby as bellow::

c_cpp_sslr_parser.zip

the key code is ::

context = SquidAstVisitorContextImpl.new(SourceProject.new(""))
          config = CxxConfiguration.new(Charset.forName(@encoding))
          config.setErrorRecoveryEnabled(false)
          if is_need_include
            debug "添加include目录:#{self.include_dirs.uniq}"
            config.setIncludeDirectories(self.include_dirs.uniq)
          end
          cpp_language = CppLanguage.new(MapSettings.new.asConfig)
          parser = CxxParser.create(context, config, cpp_language)
          # parser = CxxParser.create(context, config) 新的使用加了一个language,这个language可以直接送一个任意的配置进去就可以了
          @line_offset = LineOffsets.new(@file_str)
          @ast_node = parser.parse(@file_str)

this code i just learn from your code usage:

https://github.com/SonarOpenCommunity/sonar-cxx/blob/master/sslr-cxx-toolkit/src/main/java/org/sonar/cxx/toolkit/CxxConfigurationModel.java#L132

when i call CxxConfiguration.setErrorRecoveryEnabled(true) with getting this error ::

DEBUG org.sonar.cxx.preprocessor.CxxPreprocessor - Parser exception: '{}'
com.sonar.sslr.api.RecognitionException: Parse error at line 1 column 7:

  -->   luai_checknum#if (!definedL(HARDMEM,EST,HARDMEM,EST,c)	{  }??EOF)EOF

	at org.sonar.sslr.internal.vm.Machine.parse(Machine.java:73)
	at com.sonar.sslr.impl.Parser.parse(Parser.java:84)
	at com.sonar.sslr.impl.Parser.parse(Parser.java:78)
	at org.sonar.cxx.preprocessor.CxxPreprocessor.process(CxxPreprocessor.java:419)
	at com.sonar.sslr.impl.Lexer.preprocess(Lexer.java:153)
	at com.sonar.sslr.impl.Lexer.preprocess(Lexer.java:143)
	at com.sonar.sslr.impl.Lexer.lex(Lexer.java:132)
	at com.sonar.sslr.impl.Lexer.lex(Lexer.java:110)
	at org.sonar.cxx.preprocessor.CxxPreprocessor.handleIncludeLine(CxxPreprocessor.java:698)
	at org.sonar.cxx.preprocessor.CxxPreprocessor.process(CxxPreprocessor.java:449)
	at com.sonar.sslr.impl.Lexer.preprocess(Lexer.java:153)
	at com.sonar.sslr.impl.Lexer.preprocess(Lexer.java:143)
	at com.sonar.sslr.impl.Lexer.lex(Lexer.java:132)
	at com.sonar.sslr.impl.Lexer.lex(Lexer.java:110)
	at org.sonar.cxx.preprocessor.CxxPreprocessor.handleIncludeLine(CxxPreprocessor.java:698)
	at org.sonar.cxx.preprocessor.CxxPreprocessor.process(CxxPreprocessor.java:449)
	at com.sonar.sslr.impl.Lexer.preprocess(Lexer.java:153)
	at com.sonar.sslr.impl.Lexer.preprocess(Lexer.java:143)
	at com.sonar.sslr.impl.Lexer.lex(Lexer.java:132)
	at com.sonar.sslr.impl.Lexer.lex(Lexer.java:110)
	at com.sonar.sslr.impl.Parser.parse(Parser.java:74)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)

if i call CxxConfiguration.setErrorRecoveryEnabled(false) with getting this error ::

Parse error at line 51 column 24:

   46: &&
   47:          (a->tsv.tt == LUA_TSHRSTR ? eqshrstr(a, b) : luaS_eqlngstr(a, b));
   48: }
   49: 
   50: 
  -->  unsigned int luaS_hash (const char *str, size_t l, unsigned int seed) {
   52:   unsigned int h = seed ^ cast(unsigned int, l);
   53:   size_t l1;
Parse error at line 51 column 24:

   46: &&
   47:          (a->tsv.tt == LUA_TSHRSTR ? eqshrstr(a, b) : luaS_eqlngstr(a, b));
   48: }
   49: 
   50: 
  -->  unsigned int luaS_hash (const char *str, size_t l, unsigned int seed) {
   52:   unsigned int h = seed ^ cast(unsigned int, l);
   53:   size_t l1;

org.sonar.sslr.internal.vm.Machine.parse(org/sonar/sslr/internal/vm/Machine.java:73)
	com.sonar.sslr.impl.Parser.parse(com/sonar/sslr/impl/Parser.java:84)
	com.sonar.sslr.impl.Parser.parse(com/sonar/sslr/impl/Parser.java:78)
	java.lang.reflect.Method.invoke(java/lang/reflect/Method.java:498)
	org.jruby.javasupport.JavaMethod.invokeDirectWithExceptionHandling(org/jruby/javasupport/JavaMethod.java:453)
	org.jruby.javasupport.JavaMethod.invokeDirect(org/jruby/javasupport/JavaMethod.java:314)
	RUBY.parse_file(E:/git/code_helper/src/file_parser/c_cpp_parser/c_cpp_sslr_parser.rb:73)

is that the way i use this cxx-sslr is not right??

really thanks for you to reply this.

[WoolenWang]

Oh, i found that this error was related to multi thread
when i use only one thread there is not such problem, is that any way to use this with multi thread?

[ivangalkin]

@WoolenWang you are correct: at the moment our plugin is not thread-safe. Unfortunately our preprocessor (which is responsible for the evaluation of #define directives) also has this limitation. We plan to address this issue in the next versions. Therefore your issue is (more or less) a duplication of #1493

@guwirth I already suggested to delete some shared states in preprocessor (see #1638). However the presence of CxxParser:cxxpp and CxxFileVisitor will still prevent us from multi-threading. We should try to utilize com.sonar.sslr.api.Preprocessor::init() instead. This is possible, but we have to prevent the call of init() if preprocessor is recursively executed for include directives.

[guwirth]

Improved with 1.2.2

[WoolenWang]

Look forward to this 1.2.2 version, the thread safe analysis is very cool , waiting for that