Allow configureable WhitespaceBehavior in getTokensForParser

[wilzbach]

Needed for dlang-community/D-Scanner#448 and and dlang-community/D-Scanner#450

I am not sure whether this is the best solution as it requires allocation by the user as const(Token)[] is

	tokens = getTokensForParser(code, config, &cache);
	auto tokensC = tokens; // <-- this can be used by the Dscanner checks
	if (config.whitespaceBehavior == WhitespaceBehavior.include)
		tokensC = tokens.filter!(t => t.type != WhitespaceBehavior.include).array;
...
	dparse.parser.parseModule(tokensC, fileName, p, report, ...)
...

The parser currently doesn't check for whitespace tokens and imho it would unnecessarily complicate its logic to add checks.

[wilzbach]

(rebased to see how CodeCov turns out to work)

[codecov-io]

Codecov Report

Merging #149 into master will increase coverage by 4.32%.
The diff coverage is 40%.

@@            Coverage Diff             @@
##           master     #149      +/-   ##
==========================================
+ Coverage   81.47%   85.79%   +4.32%     
==========================================
  Files           7        7              
  Lines        4528     7041    +2513     
==========================================
+ Hits         3689     6041    +2352     
- Misses        839     1000     +161

Impacted Files	Coverage Δ
src/dparse/lexer.d	84.31% <40%> (+4.29%)	⬆️
src/dparse/ast.d	14.03% <0%> (-1.11%)	⬇️
test/tester.d	100% <0%> (ø)	⬆️
src/dparse/rollback_allocator.d	89.79% <0%> (+0.21%)	⬆️
src/std/experimental/lexer.d	86.66% <0%> (+2.94%)	⬆️
src/dparse/parser.d	90.48% <0%> (+3.01%)	⬆️

---

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 5b009b7...90c37c1. Read the comment docs.

[Hackerpilot]

The getTokensForParser function was given that name deliberately. The parser doesn't handle whitespace tokens so there should not be any whitespace in the value returned from this function.

[Hackerpilot]

The (unchanged) doc comment specifically states that this won't happen.

[wilzbach]

The getTokensForParser function was given that name deliberately. The parser doesn't handle whitespace tokens so there should not be any whitespace in the value returned from this function.

Ok, but

1. What's the purpose of config.whitespaceBehavior then?
2. How do you propose to allow getting the whitespace tokens from libdparse then?(as e.g. needed for r [WIP] Fix #328 - Add check for trailing whitespace D-Scanner#448 or Add check for if constraint indendation D-Scanner#450)

[Hackerpilot]

1. That config option is used inside of the lexer's popFront function.
2. It may be a good idea to attach the whitespace tokens to the non-whitespace tokens the way we do with comments.

[WebFreak001]

there is a new PR (#387) which would replace this PR

[WebFreak001]

closing because no longer needed (superseded by #387, merged)