Restrict Text Recognization to Digit Only

[jidalii]

Hi, I am trying to do the text recognition of a table with digits only. Sometimes, the digits are recognized as English characters. I wonder whether there is a way to restrict the scope of the text recognition to digits only. Thanks!

[felixdittrich92]

Hi @jidalii 👋,

Unfortunately, we don’t have built-in logic for blacklisting or whitelisting characters yet.
There’s already a related ticket: #988.

I’ve tried several approaches, including:

• Logits masking: Hard avoidance of blacklisted characters by setting their indices to -inf on the logits before applying softmax.
• Logits scaling: Applying a penalty by multiplying the logits of blacklisted characters before softmax.
• Filtering after softmax: Replacing blacklisted characters with the next most probable character.
• Refined iteration: Computing softmax -> masking blacklisted character indicies -> recomputing the last feature representation (from the head weight) and passing the masked features through the final linear layer.

However, I wasn’t really satisfied with the results. In some cases, the model started predicting seemingly random characters to fill the restricted positions (e.g., when "a" was blacklisted, instead of choosing the closer "á," it predicted "R").

Maybe @frgfm has some ideas I haven’t tested yet ? 🤗

In general, I think the simplest approach would be to introduce a RecognitionLogitsProcessor, allowing users to manipulate the logit vector before softmax. However, this would require some understanding of the process, so I’m not sure how many users would take advantage of it.

[frgfm]

Hey there 👋

I think the logits masking would be the best approach here. We could argue it does extra computation for nothing but for decoding it's much easier than changing the number of possible output characters.

In some regards, we could think about a Logitsprocessor in the same spirit than structured generation for LLMs. But I think that would require heavy refactoring 😅

[felixdittrich92]

Logitsprocessor should not take to much effort we have the base class :

doctr/doctr/models/recognition/core.py

Line 39 in 225210c

class RecognitionPostProcessor(NestedObject):

and all our recognition post processor's inerhit from it - where we pass the logits and compute softmax :)

Mh about computation I totally agree, but if I remember correct with masking before softmax it replaced it with "random" chars because the distribution is to close

[felixdittrich92]

However, you have to "know what you are doing there" with a LogitsProcessor... I think not many people would use that or it needs extensive explanation and/or help

something like ocr_predictor(.. whitelist=VOCABS["french"] + VOCABS["german"]) for example seems to be more user friendly ..but would require a more robust solution / logic on our end 😅

[jidalii]

Thank you so much for your suggestions! I would definitely try that approach of having an extra layer of LogitsProcessor.

[felixdittrich92]

It's on the roadmap to find a hopefully more user friendly solution 👍
CC @SiddhantBahuguna