Remove old CTC decoder (Fixes #1675)

[reuben]

• Remove old decoder code and build system integration
• Remove references in code and tests
• Update documentation

[reuben]

@lissyx I'm having trouble getting the ctcdecode package to build on ARM64, getting relocation errors in libm.a when doing the final linking of the Python extension. I see that in native_client/BUILD we link against libstdc++.a, is that a requirement on our ARM platforms? The strange thing is, it's building fine on ARMv7...

[lissyx]

@lissyx I'm having trouble getting the ctcdecode package to build on ARM64, getting relocation errors in libm.a when doing the final linking of the Python extension. I see that in native_client/BUILD we link against libstdc++.a, is that a requirement on our ARM platforms? The strange thing is, it's building fine on ARMv7...

They should share the same linking strategy, but maybe you lack some -PIE or something like that?

[lissyx]

I don't think it's a good idea, completion time of linux-amd64-cpu-opt is much higher, it's going to delay everything behind, we should re-use linux-amd64-ctc-opt to build the new python wheel instead

[reuben]

The problem is that it's not a new Python wheel, it's several Python wheels, one for each architecture/Python version, so I tried to leverage the existing infrastructure for CPU builds.

[lissyx]

Sure, but it's still better IMHO to do that outside, we can easily hack around do_deepspeech_python_build

[reuben]

Initially the compiler error mentioned -fPIC, but compiling with that flag made no difference, it still fails with relocation issues in libm.a.

[reuben]

I think the problem is that definitions.mk doesn't cover all that's needed to get a build working on ARM, as it's missing some of the stuff that was added in the tensorflow CROSSTOOL file like pointing to a different linker.

[lissyx]

I think the problem is that definitions.mk doesn't cover all that's needed to get a build working on ARM, as it's missing some of the stuff that was added in the tensorflow CROSSTOOL file like pointing to a different linker.

It's more simply just unable to find the proper libm because of absolute paths in ld scripts. Quick hack that works for me:

diff --git a/native_client/ctcdecode/Makefile b/native_client/ctcdecode/Makefile
index 14cc591..39c5e63 100644
--- a/native_client/ctcdecode/Makefile
+++ b/native_client/ctcdecode/Makefile
@@ -4,6 +4,8 @@ include ../definitions.mk

 NUM_PROCESSES ?= 1

+LDFLAGS_NEEDED += $(RASPBIAN)/lib/aarch64-linux-gnu/libm-2.24.so
+
 all: bindings

 clean:
@@ -16,4 +18,4 @@ bindings:
        AS=$(AS) CC=$(CC) CXX=$(CXX) LD=$(LD) CFLAGS="$(CFLAGS) $(CXXFLAGS)" LDFLAGS="$(LDFLAGS_NEEDED)" $(PYTHON_PATH) $(NUMPY_INCLUDE) python ./setup.py build_ext --num_processes $(NUM_PROCESSES) $(PYTHON_PLATFORM_NAME) $(SETUP_FLAGS)
        find temp_build -type f -name "*.o" -delete
        AS=$(AS) CC=$(CC) CXX=$(CXX) LD=$(LD) CFLAGS="$(CFLAGS) $(CXXFLAGS)" LDFLAGS="$(LDFLAGS_NEEDED)" $(PYTHON_PATH) $(NUMPY_INCLUDE) python ./setup.py bdist_wheel --num_processes $(NUM_PROCESSES) $(PYTHON_PLATFORM_NAME) $(SETUP_FLAGS)
-       rm -rf temp_build
\ No newline at end of file
+       rm -rf temp_build
diff --git a/native_client/ctcdecode/setup.py b/native_client/ctcdecode/setup.py
index e6b3855..ec31f9b 100644
--- a/native_client/ctcdecode/setup.py
+++ b/native_client/ctcdecode/setup.py
@@ -89,10 +89,6 @@ LIBS = []
 ARGS = ['-O3', '-DNDEBUG', '-DKENLM_MAX_ORDER=6', '-std=c++11',
         '-Wno-unused-local-typedef', '-Wno-sign-compare']

-if os.environ['TARGET'] == 'rpi3-armv8':
-    ARGS.append('-static-libstdc++')
-
-
 decoder_module = Extension(
     name='ds_ctcdecoder._swigwrapper',
     sources=['swigwrapper.i'] + FILES + glob.glob('*.cpp'),

[reuben]

We have liftoff! Thanks @lissyx!

[reuben]

In 38b5447 I've updated the docs and added some convenience code in util/taskcluster.py to get the URL of the ds_ctcdecoder package for easy installation.

[reuben]

@tilmankamp I ended up removing the COORD global entirely as it's only used in the train function now. Now globals related code is in util/config.py and the coordinator remains in util/coordinator.py.

[kdavis-mozilla]

The majority of the PR is fine.

However there are a few places that some substitutions, e.g. cluster to C.cluster, seem to have been missed and a few other small nits that should be changed, e.g. from X import * to from X import a, b, c.

[reuben]

Everything should be addressed now.

[kdavis-mozilla]

LGTM

[JRMeyer]

@reuben, does this PR mean we don't have to generate the old trie model (i.e. from v0.3.0) for training and new trie model for testing?

[reuben]

Yes.

…

-- reuben

On 15 Nov 2018, at 03:17, Josh Meyer ***@***.***> wrote: @reuben, does this PR mean we don't have to generate the old trie model (i.e. from v0.3.0) for training and new trie model for testing? — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub, or mute the thread.

[JRMeyer]

@reuben I'm loading in an older checkpoint (v0.3.0), and I get the following error:

KeyError: 'CTCBeamSearchDecoderWithLM'

After some searching, I think this is related to a missing libctc_decoder_with_kenlm.so file (c.f. #1306, #950)

The following is the script I'm running to load the checkpoint:

import tensorflow as tf
from tensorflow.python.tools import freeze_graph
from ds_ctcdecoder import ctc_beam_search_decoder, Scorer

sess = tf.Session()
sess.run(tf.global_variables_initializer())
 
tf.contrib.rnn

tf.load_op_library('../tmp/native_client/libdeepspeech.so')

new_saver = tf.train.import_meta_graph('../v030/ckpt/model.v0.2.0.meta')
new_saver.restore(sess, '../v030/ckpt/model.v0.2.0')

Is this error a by-product of this PR? Are the older checkpoints on the release page not forwards compatible after this PR?

-josh

[reuben]

This works fine for me:

python -u DeepSpeech.py --checkpoint_dir v0.3_checkpoint --epoch -1 --train_files data/ldc93s1/ldc93s1.csv --dev_files data/ldc93s1/ldc93s1.csv --test_files data/ldc93s1/ldc93s1.csv
Preprocessing ['data/ldc93s1/ldc93s1.csv']
Preprocessing done
Preprocessing ['data/ldc93s1/ldc93s1.csv']
Preprocessing done
W Parameter --validation_step needs to be >0 for early stopping to work
I STARTING Optimization
I Training epoch 250473...
I Training of Epoch 250473 - loss: 50.891705
I FINISHED Optimization - training time: 0:00:30
100% (1 of 1) |##########################################################################################################################################################| Elapsed Time: 0:00:00 Time:  0:00:00
Preprocessing ['data/ldc93s1/ldc93s1.csv']
Preprocessing done
Computing acoustic model predictions...
100% (1 of 1) |##########################################################################################################################################################| Elapsed Time: 0:00:01 Time:  0:00:01
Decoding predictions...
100% (1 of 1) |##########################################################################################################################################################| Elapsed Time: 0:00:00 Time:  0:00:00
Test - WER: 0.363636, CER: 9.000000, loss: 17.909977
--------------------------------------------------------------------------------
WER: 0.363636, CER: 9.000000, loss: 17.909977
 - src: "she had your dark suit in greasy wash water all year"
 - res: "she had a dark suit and grease with water all year"
--------------------------------------------------------------------------------

[reuben]

(On latest master)

[JRMeyer]

Strange, but good news.

[lock]

This thread has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.