Support CTC Beam Search Decoder (KenLM Lexicon)

[carolineechen]

Add support for a CTC Beam Search Decoder with KenLM language model support and lexicon constraint

• bindings from flashlight / wav2letter project
• decoder API similar to fairseq's KenLMDecoder
• factory function for constructing the decoder (while core decoder API is not finalized)

[mthrok]

Thanks for the PR.

A) Can you import this diff and see if there is any unexpected memory issue?

Then, Could you split the PR (maybe leaving this PR as-is and make new ones) into

1. The code ported from FL with only minor cosmetic changes
2. The custom code you added for torchaudio (C++ and Python)
3. Build update. (CMake and stuff)

RE 1. we can land it with simple review without actual build process as long as A) does not yield any issue.
Then 2 and 3 become easier to handle.

[facebook-github-bot]

@carolineechen has imported this pull request. If you are a Facebook employee, you can view this diff on Phabricator.

[nateanl]

Looks like the CI test failures are from Python 3.6. Rebasing the PR should resolve it.

[nateanl]

This file can be put somewhere more explicit IMO. For example, CONTRIBUTING.md or in prototype directory.

[nateanl]

The factory function is not imported in prototype/__init__.py. Could you add kenlm_lexicon_decoder to it?

[mthrok]

prototype/__init__.py should not import anything (with external library dependencies).
I missed it when I was reviewing emformer/rnn-t, but they are pure-python implementation, and they won't cause segmentation fault, so they are exceptionally okay.

If ctcdecoder is imported in prototype/__init__.py, then KenLM becomes mandatory dependency for using anything in prototype module. It will force user who do not care about this decoder but want to use other prototype module to install KenLM, which is inconvenient.

[nateanl]

I see. If users want to try prototype feature, are they expected to add it to __init__.py manually?

[mthrok]

Nope. If we put everything in a separate submodule under prototype, and as long as prototype.__init__.py does not import this submodule, users should be able to perform import normally (from torchaudio.prototype.ctc_decoder import foo), while we mitigate the risk of broken dependency.

torchaudio
└── prototype
    ├── __init__.py  # does not import ctc_decoder module
    └── ctc_decoder
        ├── __init__.py  # Initialize extension module here
        └── ctc_decoder.py

[nateanl]

Got it. Sounds good to me.

[mthrok]

Suggested change

	#include <kenlm/lm/model.hh>
	#include "lm/model.hh"

Although this is not wrong, but the upstream KenLM defines the header structure this way, so we need to use the local include here. It causes an error like this https://fburl.com/sandcastle/jivo6e9y

[nateanl]

Actually I had a similar issue when installing torchaudio with ctc decoder. The kenlm repository has to be inside the audio directory in order to be built. You can try by installing KenLM outside of audio directory and build torchaudio.

Do you have any idea how to mitigate this issue?

[mthrok]

I have some ideas for build process, but I need to experiment it as it has to be compatible with local build/CI build/fbcode build.

[carolineechen]

would "kenlm/lm/model.hh" work? the current suggestion "lm/model.hh" causes my local build to fail

[facebook-github-bot]

@carolineechen has imported this pull request. If you are a Meta employee, you can view this diff on Phabricator.