[tokenizer] fixes trust_remote_code issues for DJL converter

[bryanktliu]

Description

Brief description of what this PR is about

• If this change is a backward incompatible change, why must this change be made?
• Interesting edge cases to note here

[codecov-commenter]

⚠️ Please install the to ensure uploads and comments are reliably processed by Codecov.

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 72.41%. Comparing base (c0d3a37) to head (2923077).

❗ Your organization needs to install the Codecov GitHub app to enable full functionality.

Additional details and impacted files

@@             Coverage Diff              @@
##             master    #3569      +/-   ##
============================================
- Coverage     72.43%   72.41%   -0.03%     
+ Complexity     7421     7419       -2     
============================================
  Files           674      674              
  Lines         33087    33087              
  Branches       3526     3526              
============================================
- Hits          23966    23959       -7     
- Misses         7490     7497       +7     
  Partials       1631     1631              

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.