PHP 7.4: backfill T_BAD_CHARACTER

[jrfnl]

Tokenizer:
token_get_all() will now emit a T_BAD_CHARACTER token for unexpected
characters instead of leaving behind holes in the token stream.

PHP 7.4 (re-)introduces the T_BAD_CHARACTER token which previously already existed in PHP, but was removed in PHP 7.0.

IMO, this token should be back-filled for the PHP versions missing it.

Refs:

• https://github.com/php/php-src/blob/3775d47eee38f3b34f800a0b23f840ec7a94e4c7/UPGRADING#L152-L154
• php/php-src@9ad094e
• php/php-src@e9f99e3

[gsherwood]

Thanks for letting me know about this. I've made the token available to all versions, although it's not going to appear in older versions so there wasn't anything else I could do with it.

[jrfnl]

Thanks. FYI: As far as I know it was a token which could appear in the token stream pre-PHP 7.0 as well.