PerTok Tokenizer

[JLenzy]

This is Lemonaide's new Tokenizer, called PerTok (performance tokenizer)

---

📚 Documentation preview 📚: https://miditok--191.org.readthedocs.build/en/191/

[Natooz]

I think from here we can take two paths:

1. implement the basic micro-timing (without overlapping beat_res) in the microtiming-draft branch and make it work, then pull these changes to this branch and make the few adaptations, that shouldn't requite to change much things;
2. Put the microtiming-draft and MT feature aside from now and just focus on PerTok, which would however need to fetch it's parameters solely from config.additional_params (for now) and undo the PerTok-only changes made to the config and MusicTokenizer classes.

In both cases we'll have to make the test pass.

For 1., I can take care of it (I don't know when, I'm quite busy for now, but hopefully this shouldn't take long, most of it is already implemented).
WDYT?

[JLenzy]

implement the basic micro-timing (without overlapping beat_res) in the microtiming-draft branch and make it work, then pull these changes to this branch and make the few adaptations, that shouldn't requite to change much things;
Put the microtiming-draft and MT feature aside from now and just focus on PerTok, which would however need to fetch it's parameters solely from config.additional_params (for now) and undo the PerTok-only changes made to the config and MusicTokenizer classes.

At this stage I would prefer to focus on the second option, as my experience has been that the duration/microtiming issue quickly becomes a rabbit-hole when it is implemented through the base classes. This would help get this update ready sooner and I do have to be conscious of time constraints here.

One other question I have: Would it be okay for me to add something like "use_full_resolution" to the TokenizerConfig, in addition to "use_microtiming"?
The reason being that PerTok needs to have full-resolution (not downsampled) files, regardless of whether it is using MicroShift tokens or not. This is how it is able to implement the overlapping Timeshift/Duration values. Right now, if I set 'use_microtiming=False' then it breaks the entire tokenizer, because the original MidiTok downsampling logic comes into play.

[Natooz]

Noted, let’s focus on the second one then!

For ‘use_full_resolution’, as long as it can be implemented only in PerTok that’s possible

[Natooz]

Just added a few lint suggestions for the main files

[Natooz]

Lint fixes in suggestions

[codecov]

Codecov Report

Attention: Patch coverage is 79.09091% with 69 lines in your changes missing coverage. Please review.

Project coverage is 90.15%. Comparing base (9d9781c) to head (07b3d1d).
Report is 15 commits behind head on main.

Files with missing lines	Patch %	Lines
miditok/tokenizations/pertok.py	77.82%	55 Missing ⚠️
tests/utils_tests.py	76.66%	14 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main     #191      +/-   ##
==========================================
- Coverage   90.65%   90.15%   -0.50%     
==========================================
  Files          40       41       +1     
  Lines        6011     6575     +564     
==========================================
+ Hits         5449     5928     +479     
- Misses        562      647      +85     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.