[stdlib] Add more utf-8 validation unit tests

[gabrieldemarmiesse]

Part of my work on utf-8 validation. The new algorithm is more complexe, thus to improve our trust in it, I added new unit tests.

[JoeLoser]

Nice, thanks for improving the tests! Do you or other people in the community you've been working with have a more sketched out broader/longer-term design for incorporating UTF-8 strings in the stdlib more generally speaking (e.g. should it be String type, a distinct type, how do we think about different ways of indexing (by byte, code point, etc.)?

[JoeLoser]

Suggested change

	def test_combinaison_good_utf8_sequences():
	def test_combination_good_utf8_sequences():

[JoeLoser]

I'll fix this and the other similar typo internally when I import it.

[gabrieldemarmiesse]

oupsy, my french is leaking

[JoeLoser]

!sync

[gabrieldemarmiesse]

About utf-8 and Strings longer term, there was quite a bit of discussion on the discord about this. (internal encoding + null terminator) and while there wasn't a consensus, there is a subset of ideas that everyone agreed on. I propose we do the non-controversial work first and then we'll be able to discuss the more controversial changes.

Non-controversial changes

Everyone agreed about the necessity of having utf-8 strings in one shape or another with a null terminator for compatibility with the OS. I propose we go in this direction first and make our String type an utf-8 string with a null terminator always present.

Controversial changes

Now for the rest of the ideas that we can discuss later, there was:

• Adding a struct parameter to String Saying if the null terminator was present or not, and not present by default.
• Adding a struct parameter to String indicating the internal encoding, notably to have an internal encoding compatible with python, which does not have utf-8 string. This parameter can also indicate that a string is ASCII only, type promotion can be applied when mixing strings with different internal representations.
• Add a table for faster indexing in utf-8 strings.

Those three changes were controversial, and complexe. Especially with Python devs talking about changing the internal representation of str to use UTF-8, see faster-cpython/ideas#684 for their faster-cpython initiative.

So yeah the immediate goal would be to integrate utf-8 to our String struct. We'll see later for the rest.

[modularbot]

✅🟣 This contribution has been merged 🟣✅

Your pull request has been merged to the internal upstream Mojo sources. It will be reflected here in the Mojo repository on the nightly branch during the next Mojo nightly release, typically within the next 24-48 hours.

We use Copybara to merge external contributions, click here to learn more.

[modularbot]

Landed in b40ec35! Thank you for your contribution 🎉