Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[C++] To support the utf8 utf16 and using simd to accelerate the optimization #1964

Open
pandalee99 opened this issue Dec 2, 2024 · 1 comment · May be fixed by #1990
Open

[C++] To support the utf8 utf16 and using simd to accelerate the optimization #1964

pandalee99 opened this issue Dec 2, 2024 · 1 comment · May be fixed by #1990
Assignees
Labels
c++ enhancement New feature or request

Comments

@pandalee99
Copy link
Contributor

Feature Request

Earlier, I implemented utf16 to utf8 and used simd to speed up the process.

std::string utf16ToUtf8(const std::u16string &utf16, bool is_little_endian)

Now, you also need to implement utf8 to utf16.
Because now in java, using utf8, 16, latin1 is becoming very common, now we need to implement it in py as well.
But for speed reasons, we might consider using a direct call to a c++ module.

For simd acceleration, it is necessary to adapt to x86 architecture, arm and risc-v.
The solution is that you can use the avx2 instruction set under x86 and so on...

Is your feature request related to a problem? Please describe

Module lack utf8 utf16.

Describe the solution you'd like

Implement utf8 to utf16, and use simd acceleration.

Describe alternatives you've considered

No response

Additional context

No response

@pandalee99
Copy link
Contributor Author

This requirement will be completed this week

@pandalee99 pandalee99 linked a pull request Dec 25, 2024 that will close this issue
2 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
c++ enhancement New feature or request
Projects
None yet
Development

Successfully merging a pull request may close this issue.

1 participant