Infinite recursion for tesseract --list-langs with conda-forge binary

[jonashaag]

Current Behavior

tesseract --list-langs goes into infinite loop on macOS if TESSDATA_PREFIX is empty.

macOS Instruments shows infinite recursion in addAvailableLanguages, and a LOT of stat64 calls (multiple 10k per second).

Expected Behavior

Should not go into infinite recursion

Suggested Fix

No response

tesseract -v

tesseract 5.3.4
leptonica-1.83.1
libgif 5.2.1 : libjpeg 8d (libjpeg-turbo 3.0.0) : libpng 1.6.43 : libtiff 4.6.0 : zlib 1.2.13 : libwebp 1.3.2 : libopenjp2 2.5.2
Found NEON

Operating System

macOS 14 Sonoma

Other Operating System

No response

uname -a

Darwin ... 23.4.0 Darwin Kernel Version 23.4.0: Fri Mar 15 00:12:41 PDT 2024; root:xnu-10063.101.17~1/RELEASE_ARM64_T8103 arm64

Compiler

From conda-forge

CPU

No response

Virtualization / Containers

None

Other Information

No response

[zdenop]

Can you please exactly describe how to reproduce the problem?
How did you install tesseract?

I do not have macOS, but on Windows, I got this result:

> set TESSDATA_PREFIX=""
> tesseract --list-langs
List of available languages in """/" (0):

>set TESSDATA_PREFIX=
>tesseract --list-langs
List of available languages in "./" (1):
Downloads/dotslayer

On the Linux I got this result:

$ export TESSDATA_PREFIX=""
$ tesseract --list-langs
List of available languages in "/usr/share/tesseract-ocr/5/tessdata/" (2):
eng
osd

[jonashaag]

On macOS, your Linux commands reproduce the infinite loop

[zdenop]

@jonashaag : how did you installed tesseract?

@stweil : Can you have a look at this?

[stweil]

I don't get an infinite loop on macOS (installed with Homebrew). An empty TESSDATA_PREFIX is handled like an unset one:

% tesseract --list-langs|head -3
List of available languages in "/opt/homebrew/share/tessdata/" (682):
Fraktur
Greek
% TESSDATA_PREFIX= tesseract --list-langs|head -3
List of available languages in "/opt/homebrew/share/tessdata/" (682):
Fraktur
Greek

[stweil]

@jonashaag, addAvailableLanguages starts searching for Tesseract model files at a well defined start location and is called recursively only if it finds a directory. Unless you have a cyclic directory structure, there won't be an infinite recursion. macOS and Linux use the same code.

Can you see the arguments of the stat64 calls? Where does the search start? Maybe there are simply a lot of files and directories below your start location. If there is really an endless recursion, it will have a first directory entry which will occur again and again.

[jonashaag]

Aha, interesting, the Homebrew version works fine. The conda-forge version that I've been using doesn't.

Output from macOS's fs_usage:

11:50:07.100355  open              F=3        (R_____N____X)  xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx/.pixi/envs/default/share/tessdata                                                                                            0.000020   tesseract.1362835
11:50:07.100364  fstatfs64         F=3                                                                                                                                                                                        0.000002   tesseract.1362835
11:50:07.100439  getdirentries64   F=3    B=0x14d0                                                                                                                                                                            0.000074   tesseract.1362835
11:50:07.100443  stat64                                 xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx/.pixi/envs/default/share/tessdata                                                                                                  0.000003   tesseract.1362835
11:50:07.100450  open              F=4        (R_____N____X)  xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx/.pixi/envs/default/share/tessdata                                                                                            0.000007   tesseract.1362835
11:50:07.100451  fstatfs64         F=4                                                                                                                                                                                        0.000001   tesseract.1362835
11:50:07.100501  getdirentries64   F=4    B=0x14d0                                                                                                                                                                            0.000050   tesseract.1362835
11:50:07.100503  stat64                                 xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx/.pixi/envs/default/share/tessdata                                                                                                  0.000002   tesseract.1362835
11:50:07.100509  open              F=5        (R_____N____X)  xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx/.pixi/envs/default/share/tessdata                                                                                            0.000006   tesseract.1362835
11:50:07.100510  fstatfs64         F=5                                                                                                                                                                                        0.000001   tesseract.1362835
11:50:07.100559  getdirentries64   F=5    B=0x14d0                                                                                                                                                                            0.000049   tesseract.1362835
11:50:07.100561  stat64                                 xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx/.pixi/envs/default/share/tessdata                                                                                                  0.000002   tesseract.1362835
[... repeated indefinitely ...]

[jonashaag]

There are no symlinks in .pixi/envs/default/share/tessdata or any of its children.

$ file --no-dereference .pixi/envs/default/share/tessdata/** | grep sym
[empty]

[stweil]

Conda links:

• https://anaconda.org/conda-forge/tesseract

I am still searching for a description of the build process which was used for the Conda Tesseract package.

[jonashaag]

It is here:

• https://github.com/conda-forge/tesseract-feedstock/blob/main/recipe/build.sh
• https://github.com/conda-forge/tesseract-feedstock/blob/main/recipe/meta.yaml

[stweil]

Thanks. I could build it locally, and the result works fine. It looks like the error is in libtesseract.5.dylib which is not surprising because that library contains the function addAvailableLanguages.

In my tests with the conda-forge binary I get some more strange results:

• tesseract --list-langs works if the tessdata directory is empty.
• It also terminates if there is a single entry (file or directy) below tessdata, but fails to report if that file is a model file.
• It runs endless if there are two or more entries below tessdata.

I cannot explain that strange results with the Tesseract code, and debugging without debug symbols is rather time consuming. Therefore I suggest to report the issue to conda-forge.

[jonashaag]

I think it is related to Conda's prefix replacement. The prefix ends up being empty or something like that. I will build a package with debug symbols/prints and check what's going on.

[stweil]

That's a good hint which helped me find the root of the problem.

Patching TESSDATA_PREFIX in the compiled library does currently not work for Tesseract because that string is assigned to std::string datadir in CCUtil::main_setup. This datadir gets a different string value after the patch, but its length is not patched and still has the value of the unpatched string. Therefore datadir won't get the required terminating '/' character.

This is not restricted to installations on macOS. Installations with Conda on Linux will have the same problem.

I think there is a simple fix which I will try later.

[stweil]

Pull request #4239 should fix this issue.

[jonashaag]

Fantastic! Thanks a million!

[stweil]

@jonashaag, I am sorry, but my patch does not work. I'll try a different fix tomorrow.

[stweil]

Pull request #4240 has a different fix which I now tested successfully on Linux and on macOS.