Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[stdlib] Add _count_utf8_continuation_bytes() #3529

Closed
wants to merge 51 commits into from
Closed

[stdlib] Add _count_utf8_continuation_bytes() #3529

wants to merge 51 commits into from

Conversation

martinvuyk
Copy link
Contributor

Add _count_utf8_continuation_bytes()

@martinvuyk martinvuyk requested a review from a team as a code owner September 23, 2024 01:20
@soraros
Copy link
Contributor

soraros commented Sep 23, 2024

Can we have it take a byte Span?

@martinvuyk martinvuyk marked this pull request as draft September 23, 2024 02:45
@martinvuyk martinvuyk marked this pull request as ready for review September 24, 2024 13:40
@martinvuyk
add count_utf8_continuation_bytes
e04b3d6 
Signed-off-by: martinvuyk <martin.vuyklop@gmail.com>
@martinvuyk
fix detail
dc98262 
Signed-off-by: martinvuyk <martin.vuyklop@gmail.com>
@martinvuyk
add tests
9c8a753 
Signed-off-by: martinvuyk <martin.vuyklop@gmail.com>
@martinvuyk
change to span as suggested by @soraros
fc7700e 
Signed-off-by: martinvuyk <martin.vuyklop@gmail.com>
@martinvuyk
fix detail
3994181 
Signed-off-by: martinvuyk <martin.vuyklop@gmail.com>
@martinvuyk
fix stuff
e9e4042 
Signed-off-by: martinvuyk <martin.vuyklop@gmail.com>
@martinvuyk
fix implementation
2184125 
Signed-off-by: martinvuyk <martin.vuyklop@gmail.com>
@martinvuyk
add todo
600086d 
Signed-off-by: martinvuyk <martin.vuyklop@gmail.com>
@martinvuyk
mojo format
648e68f 
Signed-off-by: martinvuyk <martin.vuyklop@gmail.com>
@martinvuyk
fix details
f4616f6 
Signed-off-by: martinvuyk <martin.vuyklop@gmail.com>
@martinvuyk
retry github CI
c819775 
Signed-off-by: martinvuyk <martin.vuyklop@gmail.com>
@martinvuyk
retry github CI
81b51ae 
Signed-off-by: martinvuyk <martin.vuyklop@gmail.com>
@martinvuyk
retry github CI
f6ae98e 
Signed-off-by: martinvuyk <martin.vuyklop@gmail.com>
@martinvuyk
delete garbage from another PR
d049d32 
Signed-off-by: martinvuyk <martin.vuyklop@gmail.com>
@martinvuyk
mojo format
a3ddffb 
Signed-off-by: martinvuyk <martin.vuyklop@gmail.com>
@martinvuyk martinvuyk force-pushed the add-count-utf8-continuation-bytes branch from 640916a to a3ddffb Compare September 24, 2024 13:52
@martinvuyk
delete garbage from another PR
5e715d1 
Signed-off-by: martinvuyk <martin.vuyklop@gmail.com>
@martinvuyk
delete garbage from another PR
6c0d6a9 
Signed-off-by: martinvuyk <martin.vuyklop@gmail.com>
@martinvuyk
delete garbage from another PR
0e35afa 
Signed-off-by: martinvuyk <martin.vuyklop@gmail.com>
This was referenced Sep 24, 2024
Open
Open
martinvuyk and others added 5 commits September 25, 2024 16:13
@martinvuyk
Merge branch 'nightly' into add-count-utf8-continuation-bytes
2656292
@martinvuyk
simplify implementation
3844561 
Signed-off-by: martinvuyk <martin.vuyklop@gmail.com>
@martinvuyk
retry github CI
366b247 
Signed-off-by: martinvuyk <martin.vuyklop@gmail.com>
@martinvuyk
retry github CI
47d6fd1 
Signed-off-by: martinvuyk <martin.vuyklop@gmail.com>
@martinvuyk
simplify tests
381afcf 
Signed-off-by: martinvuyk <martin.vuyklop@gmail.com>
@martinvuyk
fix detail
5e443b8 
Signed-off-by: martinvuyk <martin.vuyklop@gmail.com>
@martinvuyk
fix detail
1a23e9a 
Signed-off-by: martinvuyk <martin.vuyklop@gmail.com>
@martinvuyk
fix detail
43bffb3 
Signed-off-by: martinvuyk <martin.vuyklop@gmail.com>
martinvuyk
martinvuyk commented Sep 26, 2024
View reviewed changes
stdlib/test/collections/test_string.mojo
Comment on lines -1338 to +1351
var utf8_sequence_lengths = List(5, 12, 9, 5, 7, 6, 5, 5, 2, 3, 12)
var items_amount_characters = List(5, 12, 9, 5, 7, 6, 5, 5, 2, 3, 12)
for item_idx in range(len(items)):
var item = items[item_idx]
var utf8_sequence_len = 0
var ptr = item.unsafe_ptr()
var amnt_characters = 0
var byte_idx = 0
for v in item:
var byte_len = v.byte_length()
assert_equal(item[byte_idx : byte_idx + byte_len], v)
for i in range(byte_len):
assert_equal(ptr[byte_idx + i], v.unsafe_ptr()[i])
byte_idx += byte_len
utf8_sequence_len += 1
assert_equal(utf8_sequence_len, utf8_sequence_lengths[item_idx])
amnt_characters += 1

assert_equal(amnt_characters, items_amount_characters[item_idx])
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is unrelated to the main topic of this PR, I just realized this is another place where indexing was assuming byte offset so I fixed it and also renamed some variables with unclear names.

@JoeLoser JoeLoser self-assigned this Oct 13, 2024
martinvuyk and others added 8 commits October 13, 2024 19:47
@martinvuyk
Merge remote-tracking branch 'upstream/nightly' into add-count-utf8-c…
3dfa872 
…ontinuation-bytes
@martinvuyk
fix detail
a6c1047 
Signed-off-by: martinvuyk <martin.vuyklop@gmail.com>
@martinvuyk
fix detail
ada559a 
Signed-off-by: martinvuyk <martin.vuyklop@gmail.com>
@martinvuyk
fix detail
585c6d1 
Signed-off-by: martinvuyk <martin.vuyklop@gmail.com>
@martinvuyk
fix detail
88b2bbc 
Signed-off-by: martinvuyk <martin.vuyklop@gmail.com>
@martinvuyk
fix detail
f9f1c69 
Signed-off-by: martinvuyk <martin.vuyklop@gmail.com>
@martinvuyk
fix detail
f634923 
Signed-off-by: martinvuyk <martin.vuyklop@gmail.com>
@martinvuyk
Merge branch 'nightly' into add-count-utf8-continuation-bytes
99de02d
@JoeLoser
Copy link
Collaborator

!sync

@modularbot modularbot added the imported-internally Signals that a given pull request has been imported internally. label Oct 14, 2024
@modularbot
Copy link
Collaborator

✅🟣 This contribution has been merged 🟣✅

Your pull request has been merged to the internal upstream Mojo sources. It will be reflected here in the Mojo repository on the nightly branch during the next Mojo nightly release, typically within the next 24-48 hours.

We use Copybara to merge external contributions, click here to learn more.

@modularbot modularbot added the merged-internally Indicates that this pull request has been merged internally label Oct 14, 2024
@modularbot
Copy link
Collaborator

Landed in 495fa9f! Thank you for your contribution 🎉

@modularbot modularbot added the merged-externally Merged externally in public mojo repo label Oct 16, 2024
modularbot added a commit that referenced this pull request Oct 16, 2024
@modularbot @martinvuyk
[External] [stdlib] Add _count_utf8_continuation_bytes() (#49049)
495fa9f 
[External] [stdlib] Add `_count_utf8_continuation_bytes()`

Add `_count_utf8_continuation_bytes()`

ORIGINAL_AUTHOR=martinvuyk
<110240700+martinvuyk@users.noreply.github.com>
PUBLIC_PR_LINK=#3529

Co-authored-by: martinvuyk <110240700+martinvuyk@users.noreply.github.com>
Closes #3529
MODULAR_ORIG_COMMIT_REV_ID: 994f648ac650ccd29096946d29b290e855bce057
@modularbot modularbot closed this Oct 16, 2024
@martinvuyk martinvuyk deleted the add-count-utf8-continuation-bytes branch October 16, 2024 01:16
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@soraros soraros soraros left review comments

Assignees

@JoeLoser JoeLoser

Labels
imported-internally Signals that a given pull request has been imported internally. merged-externally Merged externally in public mojo repo merged-internally Indicates that this pull request has been merged internally
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
4 participants
@martinvuyk @soraros @JoeLoser @modularbot