Add support for arrow functions

[jrfnl]

This adds support for the PHP 7.4 arrow functions to all relevant utility methods, including back-porting support down to PHPCS 2.6.0 (in as far as possible without the T_FN token existing).

These changes are in line with the support for arrow functions as added in PHPCS itself in PHPCS 3.5.3 and 3.5.4.

Also see: squizlabs/PHP_CodeSniffer#2523

To aid both the utility methods in PHPCSUtils as well as sniffs in external standards, two new methods are being introduced:

• FunctionDeclarations::isArrowFunction()
• FunctionDeclarations::getArrowFunctionOpenClose()

More about both can be found in the below commit summary.

Fixes #3

Commit summary

Tokens\Collections: add T_ARRAY to $returnTypeTokens

PHPCS does not adjust the return type token for arrow functions to T_STRING until PHPCS 3.5.3/4.
Depening on the PHPCS version and the specific code, an array return type token for an arrow function may be tokenized as T_ARRAY_HINT or T_ARRAY.

Utils\FunctionDeclarations: new arrow function helper utilities

This introduces two new utilities to allow for detecting and analyzing PHP 7.4 arrow functions.

The arrow function tokens - the PHP native T_FN and the PHPCS native T_FN_ARROW - where backfilled/introduced in PHPCS 3.5.3/4, including assigning parentheses owner/opener/closer and scope owner/opener/closer tokens to the relevant related tokens.

The utility functions in this commit back-fills this functionality for use with older PHPCS versions.

Note: this does mean that sniffs which target functions, will need to add both the T_FN, as well as the T_STRING token to their register() method and need to pass any T_STRING tokens onto these utility functions before processing the token.

Functions:

• isArrowFunction() to detect whether an arbitrary (T_FN/T_STRING) token is in actual fact an arrow function keyword.
• getArrowFunctionOpenClose() to retrieve the token pointers to the open parenthesis, close parenthesis, scope opener and scope closer for arrow functions.

Generally speaking, a sniff should only need to call one of these functions depending on what information is needed.

Includes dedicated unit tests.

The tests are loosely based on the Tokenizer/BackfillFnTokenTest file in PHPCS itself.

Note: in a limited set of very specific circumstances, the backfill in PHPCS 3.5.3 is broken.
While the isArrowFunction() will still give correct results in that case, the getArrowFunctionOpenClose() function will not.
As PHPCS 3.5.3 is explicitly not supported by PHPCSUtils due to the broken backfill of PHP 7.4 numeric literals, no fixes will be added to work-around the broken backfill for arrow functions in PHPCS 3.5.3.

Related ticket in PHPCS itself:

• PHP 7.4: arrow functions squizlabs/PHP_CodeSniffer#2523

Related commits in PHPCS itself:

• squizlabs/PHP_CodeSniffer@8fedf8c
• squizlabs/PHP_CodeSniffer@51afb54
• squizlabs/PHP_CodeSniffer@ae3ffc7
• squizlabs/PHP_CodeSniffer@0b498ad
• squizlabs/PHP_CodeSniffer@30b5457
• squizlabs/PHP_CodeSniffer@0e4fe74

BCFile/Operators::isReference(): allow for arrow functions returning by reference

... for backwards compatibility to the BCFile::isReference() and the sister-method Operators::isReference() as per upstream commit squizlabs/PHP_CodeSniffer@96e69bb which was included in PHPCS 3.5.3.

BCFile/Operators::isReference(): fix backward compatibility with PHPCS < 3.5.3

BCFile/FunctionDeclarations::get[Method]Parameters(): add support for arrow functions

Add support for arrow functions to the BCFile::getMethodParameters() and the sister-method FunctionDeclarations::getParameters() as per upstream commit squizlabs/PHP_CodeSniffer@b74e813 which was included in PHPCS 3.5.3.

BCFile/FunctionDeclarations::get[Method]Parameters(): fix backward compatibility with PHPCS < 3.5.3

• Use the FunctionDeclarations::isArrowFunction() utility to determine whether a token is an arrow function, rather than relying on the existence of a token which wasn't backfilled until PHPCS 3.5.3.
• Stabilize the unit test for when the T_FN token is not yet backfilled.
• Add a unit test covering all supported parameter types.
• Add a unit test covering arrow functions returning by reference.
• Add a unit test for handling of live coding/parse errors.

BCFile/FunctionDeclarations::get[Method]Properties(): add support for arrow functions

Add support for arrow functions to the BCFile::getMethodProperties() and the sister-method FunctionDeclarations::getProperties() as per upstream commit squizlabs/PHP_CodeSniffer@c8fca56 which was included in PHPCS 3.5.3.

BCFile/FunctionDeclarations::get[Method]Properties(): fix backward compatibility with PHPCS < 3.5.3

• Use the FunctionDeclarations::getArrowFunctionOpenClose() utility to determine whether a token is an arrow function and what the relevant opener/closer tokens are, rather than relying on the existence of a token which wasn't backfilled until PHPCS 3.5.3/4.
• Stabilize the unit test for when the T_FN token is not yet backfilled.
• Add a unit test covering arrow functions returning by reference.
• Add a unit test for upstream issue PSR2.Methods.FunctionCallSignature false positive when arrow function has array return type squizlabs/PHP_CodeSniffer#2773.
  This particular test will be skipped on PHPCS 3.5.3 as it won't pass due to the incomplete backfill in PHPCS 3.5.3.
  As PHPCS 3.5.3 is not supported by PHPCSUtils, this will not be fixed.
• Add a unit test for handling of live coding/parse errors.
• Add toggle for the double arrow token to use depending on the PHPCS version as per upstream commit squizlabs/PHP_CodeSniffer@bbd6f63.

FunctionDeclaration::getProperties(): document bug fix - 'has_body' = false for unfinished arrow functions

The fn for a potential arrow function will be tokenized as T_FN, even when there are no parenthesis and there is no function body, like during live coding or in case of a parse error.

In that case, IMO, the has_body index key should be set to false.

This is in line with the behaviour of the FunctionDeclaration::getProperties() function for other function constructs (and in contrast to the behaviour of the BCFile::getMethodProperties() function).

Also see: 1983715

BCFile::getDeclarationName(): allow functions to be called "fn"

... for backwards compatibility.

As per upstream commit squizlabs/PHP_CodeSniffer@37dda44 which was included in PHPCS 3.5.3.

Includes new unit test.

BCFile::getDeclarationName(): fix backward compatibility with PHPCS < 3.5.3

BCFile::findEndOfStatement(): add support for arrow functions

• Add support for arrow functions to the BCFile::findEndOfStatement() method as per upstream commit squizlabs/PHP_CodeSniffer@bf642b2 which was included in PHPCS 3.5.3.
• Improve support for arrow functions in the BCFile::findEndOfStatement() method as per upstream commit squizlabs/PHP_CodeSniffer@1be4196 which was included in PHPCS 3.5.4.
• Add additional unit test as per upstream commit squizlabs/PHP_CodeSniffer@30b5457 which was included in PHPCS 3.5.4.

BCFile::findEndOfStatement(): fix backward compatibility with PHPCS < 3.5.3/3.5.4

BCFile::findStartOfStatement(): add support for arrow functions

• Add support for arrow functions to the BCFile::findStartOfStatement() method as per upstream commit squizlabs/PHP_CodeSniffer@fbf67ef which will be included in PHPCS 3.5.5.

Also see: squizlabs/PHP_CodeSniffer#2523 and squizlabs/PHP_CodeSniffer#2849

Includes adding an initial unit test file for the BCFile::findStartOfStatement() method which was, so far, untested.
The initial unit test only (intentionally) covers the current change.

Arrays::getDoubleArrowPtr(): handle arrow functions

In PHPCS 3.5.3+, the double arrow for arrow functions is tokenized as T_FN_ARROW and will not confuse this utility method anymore.

However, for PHPCS < 3.5.3 in combination with PHP 7.4, the fn keyword will be tokenized as T_FN, however, the double arrow will still tokenize as T_DOUBLE_ARROW.
And for PHPCS < 3.5.3 in combination with PHP 7.4, the fn keyword will be tokenized as T_STRING and the double arrow will tokenize as T_DOUBLE_ARROW.

Both of these would cause incorrect results for this function.

As far as I can see, arrow functions are not usable in an array key, so basically, we know that there will be no (array) arrow as soon as we encounter a T_FN token of a T_STRING token which is an arrow function.

Includes unit tests.

Related: squizlabs/PHP_CodeSniffer#2703

PassedParameters: add unit test with arrow function

Parentheses::getOwner()/isOwnerIn(): add support for arrow functions

Allow for the PHPCSUtils\Utils\Parentheses::getOwner() and PHPCSUtils\Utils\Parentheses::isOwnerIn() methods to recognize arrow functions as parentheses owners in PHP < 7.4 and PHPCS < 3.5.3/4.

Includes unit tests.