[retry] Use pytorch-labs/tokenizers and remove tokenizer/ (#1401)

[larryliu0820]

Summary: Retry of #1401

Test Plan: Re-run the repro command in #1413:

python3 torchchat.py generate llama3.2-1b-base --prompt "write me a story about a boy and his bear"

Reviewers:

Subscribers:

Tasks:

Tags:

[pytorch-bot]

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/pytorch/torchchat/1443

• 📄 Preview Python docs built from this PR

Note: Links to docs will display an error until the docs builds have been completed.

❌ 1 New Failure, 1 Pending, 1 Unrelated Failure

As of commit dee39b9 with merge base 019f76f ():

NEW FAILURE - The following job has failed:

• Run the aoti runner with CUDA using stories / test-runner-aot-cuda / linux-job (gh)
  RuntimeError: Command docker exec -t e6136f50b7cea17bff58df27946a2105e3cd90fa5ca7f884c5cb5fe543223e78 /exec failed with exit code 1

BROKEN TRUNK - The following job failed but were present on the merge base:

👉 Rebase onto the `viable/strict` branch to avoid these failures

• pull / runner-aoti (16-core-ubuntu) (gh) (trunk failure)
  Process completed with exit code 1.

This comment was automatically generated by Dr. CI and updates every 15 minutes.