Short array list syntax not correctly tokenized if short array is the first content in a file

[jrfnl]

The outer square brackets in the below example are tokenized as T_OPEN_SQUARE_BRACKET and T_CLOSE_SQUARE_BRACKET instead of as T_OPEN_SHORT_ARRAY and T_CLOSE_SHORT_ARRAY.

<?php

[$a, [$b]] = $array;

is tokenized as follows - take note of token 2 and 9:

        *** START PHP TOKENIZING ***
        Process token [0]: T_OPEN_TAG => <?php\n
        Process token [1]: T_WHITESPACE => \n
        Process token  2 : T_OPEN_SQUARE_BRACKET => [
        Process token [3]: T_VARIABLE => $a
        Process token  4 : T_COMMA => ,
        Process token [5]: T_WHITESPACE =>
        Process token  6 : T_OPEN_SQUARE_BRACKET => [
        Process token [7]: T_VARIABLE => $b
        Process token  8 : T_CLOSE_SQUARE_BRACKET => ]
        Process token  9 : T_CLOSE_SQUARE_BRACKET => ]
        Process token [10]: T_WHITESPACE =>
        Process token  11 : T_EQUAL => =
        Process token [12]: T_WHITESPACE =>
        Process token [13]: T_VARIABLE => $array
        Process token  14 : T_SEMICOLON => ;
        Process token [15]: T_WHITESPACE => \n
        *** END PHP TOKENIZING ***
        *** START TOKEN MAP ***
        => Found square bracket opener at 2
                => Found square bracket opener at 6
                => Found square bracket closer at 8 for 6
        => Found square bracket closer at 9 for 2
        *** END TOKEN MAP ***
        *** START SCOPE MAP ***
        *** END SCOPE MAP ***
        *** START LEVEL MAP ***
        Process token 0 on line 1 [col:1;len:5;lvl:0;]: T_OPEN_TAG => <?php\n
        Process token 1 on line 2 [col:1;len:0;lvl:0;]: T_WHITESPACE => \n
        Process token 2 on line 3 [col:1;len:1;lvl:0;]: T_OPEN_SQUARE_BRACKET => [
        Process token 3 on line 3 [col:2;len:2;lvl:0;]: T_VARIABLE => $a
        Process token 4 on line 3 [col:4;len:1;lvl:0;]: T_COMMA => ,
        Process token 5 on line 3 [col:5;len:1;lvl:0;]: T_WHITESPACE =>
        Process token 6 on line 3 [col:6;len:1;lvl:0;]: T_OPEN_SQUARE_BRACKET => [
        Process token 7 on line 3 [col:7;len:2;lvl:0;]: T_VARIABLE => $b
        Process token 8 on line 3 [col:9;len:1;lvl:0;]: T_CLOSE_SQUARE_BRACKET => ]
        Process token 9 on line 3 [col:10;len:1;lvl:0;]: T_CLOSE_SQUARE_BRACKET => ]
        Process token 10 on line 3 [col:11;len:1;lvl:0;]: T_WHITESPACE =>
        Process token 11 on line 3 [col:12;len:1;lvl:0;]: T_EQUAL => =
        Process token 12 on line 3 [col:13;len:1;lvl:0;]: T_WHITESPACE =>
        Process token 13 on line 3 [col:14;len:6;lvl:0;]: T_VARIABLE => $array
        Process token 14 on line 3 [col:20;len:1;lvl:0;]: T_SEMICOLON => ;
        Process token 15 on line 3 [col:21;len:0;lvl:0;]: T_WHITESPACE => \n
        *** END LEVEL MAP ***
        *** START ADDITIONAL PHP PROCESSING ***
        * token 6 on line 3 changed from T_OPEN_SQUARE_BRACKET to T_OPEN_SHORT_ARRAY
        * token 8 on line 3 changed from T_CLOSE_SQUARE_BRACKET to T_CLOSE_SHORT_ARRAY
        *** END ADDITIONAL PHP PROCESSING ***

[gsherwood]

This happened because the short array was the first content in the file and the loop wasn't going all the way back to the first token. It would have worked if there was even a space before the open tag, so this error looks pretty localised.

[jrfnl]

@gsherwood Thanks for the quick fix. I actually extrapolated the example from a larger file. I'll see if I can figure out the determining factor when in a larger context to create a small reproducible example, or maybe this has already fixed that too.

I'll let you know how I fare.

[gsherwood]

I'll let you know how I fare.

Thanks. I'll close this report because it's now really specific, so please create another one if you find that code again.