Cordova plugin (5.x) for the MMIR framework that uses the Nuance SpeechKit (Dragon Mobile SDK) for speech recognition and synthesis
See https://dragonmobile.nuancemobiledeveloper.com
This Cordova plugin is specifically targeted to be used with the MMIR framework:
On adding the plugin, 2 MMIR "modules" (for recognition and synthesis) will be copied
into the platform's resource folders <www assets>/mmirf/env/media/nuance*.js
It is recommended to create a local copy of the plugin/repository and place your resources and credentials for Dragon Mobile SDK there (i.e. not the installation directly from the GIT repository).
See the section below for installing from a local copy of the repository and the section about the Nuance libraries and credentials.
For additional information you can also visit Nuance's site on Phonegap/Cordova development: https://nuancedev.github.io/samples/#phonegap
execute the following command in Cordova project's root directory:
cordova plugin add https://github.com/mmig/mmir-plugin-speech-nuance.git
(1) check out the repository into a local directory (or download its ZIP file and decompress it)
(2) add the libaries, resources, and your credentials for the Nuance Dragon Mobile SDK (see section below)
(3) add the plugin to the Cordova project:
use command:
cordova plugin add <file path to plugin directory> --nofetch
If plugin source code (from this repository) is located in directory:
D:\DevProjects\Eclipse_workplace\mmir-plugin-nuancespeech
execute the following command in Cordova project's root directory:
cordova plugin add D:\DevProjects\Eclipse_workplace\mmir-plugin-nuancespeech --nofetch
You need to add the native libararies and resources from your the Dragon Mobile SDK into
the plugin's /res directories.
/res/android/
./libs/
./res/
...
/res/ios/frameworks/SpeechKit.framework/
./Headers/
./Resources/
./SpeechKit
...
During development you can use the Credentials helper class for supplying the credentials via Cordova's config.xml.
NOTE that for production you should use appropriate mechanisms/implementations for securing the credentials
within the generated platform-specific app files. Your should not put your credentials in the config.xml
when publishing you app, since this file might be readable/extractable as plain-text in the generated app file,
depending on the target platform (e.g. for Android you could use ProGuard for obfuscation or use the
Android NDK and put your credentials in a native library in order to increase the barrier
for reverse-engineering).
Edit the Cordova config.xml in the project's root directory and add your credentials (the example below
only contains "mock" data and will not work):
<?xml version='1.0' encoding='utf-8'?>
<widget id="...
<preference name="nuanceAppKey" value="{
(byte)0x... <copy & paste your app key here, in Android format>
}"/>
<preference name="nuanceAppId" value="NMDPTRIAL_...<copy & paste your app ID here>"/>
<preference name="nuanceServerUrl" value="<copy & paste the server URL for the SpeechKit service>"/>
<preference name="nuanceServerPort" value="<copy & paste the port number>"/>
<preference name="nuanceServerSsl" value="<use true, if you are using SSL>"/>
<!-- OPTIONAL (can be omitted)
if SSL is used, this settings can be used for strengthening the encryption
-->
<preference name="nuanceCertSummary" value="<copy & paste the cert summary>"/>
<!-- OPTIONAL (can be omitted)
if SSL is used, this settings can be used for strengthening the encryption
-->
<!-- preference name="nuanceCertData" value="<copy & paste the cert data>"/-->
...
</widget>the decompressed resources for the Android SDK should contain a directory /libs
with the following contents (that need to be copied to the plugin's /res/android directory):
/libs/*
/libs/nmdp_speech_kit.jar
/libs/arm64-v8a/libnmsp_sk_speex.so
/libs/armeabi/libnmsp_sk_speex.so
/libs/armeabi-v7a/libnmsp_sk_speex.so
/libs/x86/libnmsp_sk_speex.so
the decompressed resources for the iOS SDK should contain a directory /libs
with the following contents (that need to be copied to the plugin's /res/ios directory):
/SpeechKit.framework/Versions/<current version>/*
/Headers/*
/Resources/*
/SpeechKit
(if you encounter problems, you should try to avoid using the symbolic links in /SpeechKit.framework,
and instead use the actual files in /SpeechKit.framework/Versions/<current version>)
the MMIR modules the give access to the speech recognition / synthesis will be copied from the plugin directory
/www/nuanceAudioInput.js
/www/nuanceTextToSpeech.js
into into the platform folders of the www-resource files to:
/www/mmirf/env/media/*
TIP: if you are using _Eclipse_ you can add _links_ to these files in your project, so that they appear in your app's `/www` directory at `/www/mmirf/env/media/*` Either use _Eclipse_'s `New File` dialog or edit `/.project` by adding the following somewhere within the `` tag: ```xml ... www/mmirf/env/media/nuanceAudioInput.js 1 $%7BPROJECT_LOC%7D/plugins/dfki-mmir-plugin-speech-nuance/www/nuanceAudioInput.js www/mmirf/env/media/nuanceTextToSpeech.js 1 $%7BPROJECT_LOC%7D/plugins/dfki-mmir-plugin-speech-nuance/www/nuanceTextToSpeech.js ... ``` ## MMIR CONFIGURATION
for configuring the MMIR app to use this plugin/module for its speech input/output do the following:
edit the configuration file in
/www/config/configuration.json
modify or add (if it does not exist already) the configuration entries for the MediaManager plugins, i.e. edit the JSON file to:
{
...
"mediaManager": {
"plugins": {
"browser": ["html5AudioOutput.js",
"webkitAudioInput.js",
"maryTextToSpeech.js"
],
"cordova": ["cordovaAudioOutput.js",
"nuanceAudioInput.js",
"nuanceTextToSpeech.js"
]
}
}
...
}change the "cordova" array entries to "nuanceAudioInput.js" (for ASR) and
"nuanceTextToSpeech.js" (for TTS) in order to use the Nuance ASR- and TTS-
engine, when the application is run as Cordova app.
NOTE: "building" is not necessary for using the plugin, but it may provide helpful feedback during plugin development.
This project requires Cordova 5.x for building the Java source.
You can checkout the CordovaLib project from a repository and then reference the checked-out project from this project:
(1) checkout the Cordova5Lib project into the same Eclipse workspace:
t.b.d.: XXXX/Cordova5Lib
(2) (in Eclipse) open the project Properties for this project, goto "Java Build Path", open tab "Projects" and add the CordovaLib project (you may also need to clean / rebuild the project).
t.b.d.
NOTE 1: for recognition interface (audio input) see the wiki's general Speech Input API and Speech Output API
NOTE 2: the functions of this module are exported to the framework's mmir.media module)