Fix : emoji length cause overlapping

[keithCuniah]

Description

Because Emoji not always have a size of one, this can cause overlapping in prediction and annotation.

Please include a summary of the changes and the related issue. Please also include relevant motivation and context. List any dependencies that are required for this change.

Closes #2353

Type of change
Changes in Back and Front need to be done

Front

• repair test recordHasEmoji in RecordTokenClassification
• performance => move computed to static variable in RecordTokenClassification
• ensure that we don't have an offset depending of the emoji length

Back

• ensure that we don't have an offset depending of the emoji length

(Please delete options that are not relevant. Remember to title the PR according to the type of change)

• Bug fix (non-breaking change which fixes an issue)
• New feature (non-breaking change which adds functionality)
• Breaking change (fix or feature that would cause existing functionality to not work as expected)
• Refactor (change restructuring the codebase without changing functionality)
• Improvement (change adding some improvement to an existing functionality)
• Documentation update

How Has This Been Tested

(Please describe the tests that you ran to verify your changes. And ideally, reference tests)

• Token classification

Checklist

• I have merged the original branch into my forked branch
• I added relevant documentation
• follows the style guidelines of this project
• I did a self-review of my code
• I made corresponding changes to the documentation
• My changes generate no new warnings
• I have added tests that prove my fix is effective or that my feature works
• I have added relevant notes to the CHANGELOG.md file (See https://keepachangelog.com/)

[cceyda]

I'm looking forward to the resolution of this bug, but I don't think this PR is exactly the solution. As the issue is not limited to emojis

[keithCuniah]

Hi @cceyda, sorry for the delay of answer. You are right, the solution will be update. We will tackle this fix as soon as possible :)

[keithCuniah]

Hello @cceyda, sorry again for the delay of answer.
The problem in the front is how we calculate the token length with the library stringz in RecordTextClassification component : with this library, we consider always a character to have a length equal to one (normal character or emoji). So to implement the solution you have suggested with the spread operator [...emoji_or_character], we need to calculate the prediction at the character level and not at the token level as it is right now.
The solution is quite complex to tackle right now, we will implement it in another issue.