Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Return token_strings in the response for Tokenize #30

Merged
merged 1 commit into from
Aug 26, 2022
Merged

Conversation

mkozakov
Copy link
Contributor

The API response of tokenize now includes a token_strings field. This PR adds it to the SDK.

Example:
cohere.Tokenize({"hello world"})

&TokenizeResponse{
	Tokens:       []int64{33555, 1114},
	TokenStrings: []string{"hello", " world"},
}

@mkozakov mkozakov requested a review from lfayoux August 25, 2022 14:11
go.mod Show resolved Hide resolved
Copy link
Contributor

@lfayoux lfayoux left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

lgtm!

@mkozakov mkozakov merged commit 0ac5999 into main Aug 26, 2022
@mkozakov mkozakov deleted the feature/tokenize branch August 26, 2022 17:43
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants