Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Splitting tokens using zones sometimes does not work #5082

Closed
reckart opened this issue Oct 2, 2024 · 0 comments
Closed

Splitting tokens using zones sometimes does not work #5082

reckart opened this issue Oct 2, 2024 · 0 comments
Assignees
@reckart
Labels
🐛Bug Something isn't working
Milestone
34.1

Comments

@reckart
Copy link
Member

reckart commented Oct 2, 2024

Describe the bug
When tokenizing a text using token splitting zones, the resulting tokenization may be incorrect if the sentences do not neatly partition the text.

Expected behavior
Token zones should be properly applied within sentences, irrespective of where they start and end (non-overlapping)-

Please complete the following information:

  • Version and build ID: 34.0
@reckart reckart added the 🐛Bug Something isn't working label Oct 2, 2024
@reckart reckart added this to the 34.1 milestone Oct 2, 2024
@reckart reckart self-assigned this Oct 2, 2024
@reckart reckart added this to Kanban Oct 2, 2024
@github-project-automation github-project-automation bot moved this to 🔖 To do in Kanban Oct 2, 2024
reckart added a commit that referenced this issue Oct 2, 2024
@reckart
#5082 - Splitting tokens using zones sometimes does not work
7498df7 
- Fixed issue and added tests
reckart added a commit that referenced this issue Oct 4, 2024
@reckart
Merge pull request #5083 from inception-project/bugfix/5082-Splitting…
96c038e 
…-tokens-using-zones-sometimes-does-not-work

#5082 - Splitting tokens using zones sometimes does not work
@reckart reckart closed this as completed Oct 4, 2024
@github-project-automation github-project-automation bot moved this from 🔖 To do to 🍹 Done in Kanban Oct 4, 2024
reckart added a commit that referenced this issue Oct 6, 2024
@reckart
Merge branch 'release/34.x'
15d04b3 
* release/34.x:
  #5082 - Splitting tokens using zones sometimes does not work
reckart added a commit that referenced this issue Oct 6, 2024
@reckart
Merge branch 'main' into refactoring/3415-Move-all-configuration-opti…
c4e301b 
…ons-to-Spring-configuration-beans

* main:
  #5082 - Splitting tokens using zones sometimes does not work
  #5078 - Upgrade dependencies - rhino 1.7.14 -> 1.7.15 - ant 1.10.14 -> 1.10.15 - junit 5.11.0 -> 5.11.1 - junit-platform 1.11.0 -> 1.11.1 - mockito 5.12.0 -> 5.14.1 - testcontainers 1.20.1 -> 1.20.2 - spring 6.1.12 -> 6.1.13 - spring-boot 3.3.2 -> 3.3.4 - spring-security 6.3.1 -> 6.3.3 - swagger 2.2.22 -> 2.2.24 - log4j 2.23.1 -> 2.24.1 - tomcat 10.1.28 -> 10.1.30 - jetty 12.0.12 -> 12.0.14 - postgres-driver 42.7.3 -> 43.7.4 - wicket 10.1.0 -> 10.2.0 - wicketstuff 10.1.1 -> 10.2.0 - wicket-bootstrap 7.0.5 -> 7.0.8 - wicket-jquery-selectors 4.0.5 -> 4.0.6 - wicket-webjars 4.0.4 -> 4.0.5 - opensearch 2.16.0 -> 2.17.0 - jackson 2.17.2 -> 2.18.0 - snakeyaml 2.2 -> 2.3 - okio 3.9.0 -> 3.9.1 - byte-buddy 1.14.19 -> 1.15.3 - commons-csv 1.11.0 -> 1.12.0 - commons-lang3 3.16.0 -> 3.17.0 - commons-compress 1.27.0 -> 1.27.1 - commons-io 2.16.1 -> 2.17.0 - jinjava 2.7.2 -> 2.7.3 - nimbus-jose-jwt 9.40 -> 9.41.2 - json-schema-validator 1.5.1 -> 1.5.2 - snappy 1.1.10.6 -> 1.1.10.7 - jquery 1.13.3 -> 1.14.0
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@reckart reckart

Labels
🐛Bug Something isn't working
Projects
Kanban
Status: 🍹 Done
Milestone
34.1
Development

No branches or pull requests
1 participant
@reckart