[stdlib] Move StringRef find() implementation to Span

[martinvuyk]

Move StringRef find() implementation to Span

[JoeLoser]

Question It seems a bit weird from an API design perspective to have this "find" function in span, can you help me understand why we'd want this? It feels more coupled to string algorithms, and Span isn't just for a view over string data.

[martinvuyk]

It can be used if you want to find occurrences of a specific scalar value sequence. My main thought was that we'll lose the ability to do that once StringSlice.find() is fixed to use unicode codepoints and it introduces quite some overhead for algorithms that are faster using raw bytes. We could also make this private WDYT?

Also, once PR's #3577 DType.get_dtype() gets merged, List.index() can also delegate to this

[martinvuyk]

@JoeLoser gentle ping and also to add something, since it seems like we are moving toward unifying Python's bytes into Span[UInt8]in PR #3636, request #3634 and many other places. Python's bytes.find is actually a method, and many other things that we put in StringSlice and StringRef are actually operations from bytes.

[msaelices]

I think this is a great change for overall performance when manipulating slices. It opens a lot of possibilities.

[martinvuyk]

@JoeLoser CI is failing because of some bugs I'll solve later.

I refactored normalize_index and used it here. I also refactored memrmem and memrchr to be vectorized like their counterparts. These are algorithms that apply not only to string data, you can use them to find anything. I had tried to make a similar algorithm for Array in my repo but failed to make it as efficient as these, kudos to whoever implemented them. Can we please have this? They will be used a lot. I've also found that .replace() and .count() will use this function besides .split(), .splitlines(), List.index(), and others.

[martinvuyk]

@JoeLoser would it be possible to land this before the next stable release? I'd like to use this to be able to bypass string's .find() for algorithms which work on raw bytes. And using that bypass, I'd like to make the breaking change to switch over to full unicode:

• String.__len__() should return unicode length and not byte length
• String.find() should work on unicode offsets
• String.__getitem__() should work on unicode offsets

I would like to make the change as soon as possible because it will break a lot of projects, and many more are coming because the community is growing.