GH1422: Add support for Unicode globbing

[tom-dudley]

I've added Unicode support to the Globber by rewriting the GlobTokenizer. This fixes #1422

• Tokens are lazily loaded into a queue
• GlobTokenizer now specifies 'non-identifier characters' rather than 'identifier characters'
• Globbing is 15% faster
• Unicode supported

[dnfclas]

@tom-dudley,
Thanks for your contribution.
To ensure that the project team has proper rights to use your work, please complete the Contribution License Agreement at https://cla2.dotnetfoundation.org.

It will cover your contributions to all .NET Foundation-managed open source projects.
Thanks,
.NET Foundation Pull Request Bot

[dnfclas]

@tom-dudley, thanks for signing the contribution license agreement. We will now validate the agreement and then the pull request.

Thanks, .NET Foundation Pull Request Bot

[tom-dudley]

If the OS X bitrise error is genuine (I read elsewhere you might be having issues with bitrise), could you make the output publicly viewable or let me know what the failure was? Thanks!

[patriksvensson]

@tom-dudley I took a look at it and it doesn't seem to be related to your changes.

[devlead]

@tom-dudley @patriksvensson reran the Bitrise Mac Build and it completed successfuly

[SIkebe]

@tom-dudley Thanks for taking this issue up!

[devlead]

LGTM 👍

[devlead]

@tom-dudley your changes have been merged, thanks for your contribution 👍