Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Optionally ignore utf-8 decoding error for scripted C++ tokenizers. (… #2134

Merged
merged 2 commits into from
Mar 29, 2023
Merged

Optionally ignore utf-8 decoding error for scripted C++ tokenizers. (… #2134

merged 2 commits into from
Mar 29, 2023

Commits on Mar 29, 2023

  1. Optionally ignore utf-8 decoding error for scripted C++ tokenizers. (p… 

    …ytorch#2128)

Summary:
Pull Request resolved: pytorch#2128

Binding and test to make sure we can use 'ignore' option for utf-8 decoding added to pytorch in D43970697( pytorch/pytorch#97282).

Reviewed By: Nayef211

Differential Revision: D44315169

fbshipit-source-id: d42fcacafd429cf586c631faf826abc172b173d3
    @shuminghu @Nayef211
    shuminghu authored and Nayef211 committed Mar 29, 2023
    Configuration menu
    Copy the full SHA
    3b687ae View commit details
    Browse the repository at this point in the history

  2. Linter fixes

    @Nayef211
    Nayef211 committed Mar 29, 2023
    Configuration menu
    Copy the full SHA
    ff1aa87 View commit details
    Browse the repository at this point in the history