Speech Synthesizer behavior change in V4

[alokraj68]

Version

4.6 vis CDN

Describe the bug

In V3, we had an option to stop the speech from code and after that, the speech synthesizes won't work until the user again uses speech recognizer.

In V4, this behavior has changed. The bot speaks out every message even after stoping it from code.

To Reproduce

Steps to reproduce the behavior:

1. Go to any voice-enabled samples.
2. Click on mic
3. Speak something to the bot and wait for the bot to speak back.
4. Execute the code segment
   speechServicesPonyfill.speechSynthesis.queue.stop(); speechServicesPonyfill.speechSynthesis.cancel();
   The speech will stop now.
5. Now type something to the bot. The reply will be again spoken by the bot.

Expected behavior

The bot should not speak once this code segment is executed. This was the behavior in V3.

Additional context

This is an upgrade issue and the context is that if we minimize the bot and we trigger any message from the user to the bot via piggybacking, the reply will be spoken by the bot even when it is minimized.

[Bug]

[alokraj68]

@corinagum, Can you please follow these steps and help me out resolve this?

[compulim]

1. The API of speechServicesPonyfill.speechSynthesis conforms to Web Speech API spec. Since the speechSynthesis.queue is not specified in the spec, calling queue.stop is not a supported feature and intended to change from time to time
2. In Web Chat, instead of using SpeechSynthesis to control the queue (e.g. stop/suspend/resume synthesis), we build our own queue for controlling how utterance are being synthesized
   • This is because the spec lack of a way to cancel/dequeue a single utterance, which Web Chat requires
   • As you may explored into this area already, the SpeechSynthesis object only support cancelling all utterances, but not a single one
3. If you would want to control the queue, today, you can create a custom store and dispatch few actions to it
   • Dispatching WEB_CHAT/STOP_SPEAKING will stop Web Chat from synthesizing the current and upcoming activities
     • This feature may change when we implement Design: Switching input mode #2211 in our R8 release, which is currently planned for Spring 2020
     • In R8, both the dev and the user will have better control on how/when to synthesize an activity

[corinagum]

Closing this issue for organizational purposes. For those wanting to add anything, please take your discussion to #2211. Thank you!

[alokraj68]

@compulim,
In V3 we had a method call speechOptions.speechSynthesizer.stopSpeaking(); Which will stop the utterance speaking and speechOptions.speechSynthesizer._requestQueue = []; which will stop the synthesizes from speaking out any more utterance from the bot. When the user presses the mic button again, the synthesis will again be turned on.

I am looking for the same equivalent since this behavior is not there now with the V4 speech synthesis.

Also, Cancelling all utterance does not disable the speech until the user press the mic button.

[compulim]

@alokraj68, for stop speaking, can you implement point 3 as in this comment?

[alokraj68]

@compulim, @corinagum, I did implement the same. But once we send a Proactive message, It is also spoken by the bot which we do not want. Also, we tried to run it via piggybacking from webchat like this:

var StartActivity = { from: { id: "id", name: "name", role: "user", }, name: 'startConversation', type: 'message', value: 'Welcome', }; directLine.postActivity(StartActivity).subscribe(function (id) { stopSpeech(); // dispatches the code to stop speech here. });

The stop speech does not execute as expected since the stop speech is sent before the speech file from server is received by cognitive services.

The only crude method we found now is as follows

if (speechServicesPonyfill && speechServicesPonyfill.speechSynthesis) { if (speechServicesPonyfill.speechSynthesis.speaking) { speechServicesPonyfill.speechSynthesis.cancel(); store.dispatch({ type: 'WEB_CHAT/STOP_SPEAKING' }); if (speechServicesPonyfill.speechSynthesis.speaking) { setTimeout(function () { stopSpeech(); }, 500); // we dont want to run it in a setTimeout. We need an alternative for this. } } }

Please suggest an alternative.

[compulim]

@alokraj68 are you trying to stop all speech synthesis?

If yes, this sample will help, https://github.com/microsoft/BotFramework-WebChat/tree/master/samples/03.speech/d.cognitive-speech-services-speech-recognition-only.

This sample build a partial adapter that enable speech recognition but disable speech synthesis, which will aid in your case if you are stopping all speech synthesis.