Directline speech stops working if user clicks on the Mic during recognition

[collegewap]

Screenshots

Version

4.9.2

Describe the bug

Steps to reproduce

1. Go to https://compulim.github.io/webchat-loader/
2. Select "MockBot (Direct Line Speech)"
3. Click on "Open Web Chat in a new window"
4. Speak something
5. Before the bot responds, click on Mic again
6. The screen would be hung showing "Starting.."

The issue is replicable in some other scenarios as well. I am unable to mention the exact steps. If you play around the mic button, it gets hung.

Expected behavior

Webchat should not get hung.

[Bug]

[carlosscastro]

Thanks for reporting. We'll investigate and prioritize during our planning and update as necessary. Thanks!

[carlosscastro]

@compulim if possible please add to the R11 investigate backlog.

[collegewap]

Happens in mock bot also.

Steps to replicate:

1. Open 3a
2. Say hello
3. After bot speaks, click on Mic and don't say anything
4. Now I get the second message
5. After the bot finishes speaking, click on Mic.
6. It will get hung on "Starting..."

This happens more often in my custom bot

[peterinnesmsft]

@compulim is there any update on this one? Outside of SLA, so if we could prioritize a response, that would be great.

[peterinnesmsft]

@collegewap we will investigate and handle this issue as part of our next upcoming release cycle.

[compulim]

Sorry for late response. I am starting to investigate and will post my investigation notes here.

[compulim]

Minimal repro

• Connect using DLSpeech to any bots
  • For example, /01.getting-started/d.es5-direct-line-speech/index.html
• After connected, click on microphone button and remain silent
  • This should trigger InitialSilenceTimeout event on the service side
• The bot will say "Unknown command: ''. Type help to learn more."
• Click on microphone button again
• Expected: the microphone can be turned on again
• Actual: it just say "Starting..." and hung there

Investigation

1

It seems Speech SDK has an issue after InitialSilenceTimeout event.

In the second click, we called DialogServiceConnector.listenOnceAsync. But the function return immediately because the check isTurnComplete is false. It seems Speech SDK did not stop completely after InitialSilenceTimeout.

2

After bumped to microsoft-cognitiveservices-speech-sdk@1.13.1 (in PR #3432), I am still seeing the problem of InitialSilenceTimeout not able to reset the isTurnComplete flag to true.

Will investigate more on this code path.

3

This is related to microsoft/cognitive-services-speech-sdk-js#211.

We need to wait for the Cognitive Services Speech team to fix the issue in their SDK > 1.13.1.

Next steps

• Bumping Speech SDK to latest version and see if it still repro.
  • Bumped to 1.13.1 in PR [Post 4.10.1] Bump to microsoft-cognitiveservices-speech-sdk@1.13.1 #3432
• Convert to external bug.
• Wait for Cognitive Services team to fix it.
• Bump our dependency and verify the fix.

[cwhitten]

Moving from R11 with the understanding that we won't have enough lead time to integrate a drop of the speech sdk

[edwgj]

Hi, another bug is reproduced here:

1. Go to https://compulim.github.io/webchat-loader/
2. Select "MockBot (Direct Line Speech)"
3. Click on "Open Web Chat in a new window"
4. Type help
5. The bot sends cards with lots of options, click on mic
6. While the bot is listening, click on any card option
7. Web chat UI disappear, the screen gets blank

[corinagum]

This is related to #3575. Keeping this issue open to ensure we verify fixed behavior post-bump.

I removed the blocked label since the new sdk version is available now.

[compulim]

This seems fixed in Speech SDK 1.15.0.

However, while operating in DLSpeech mode, Speech SDK does not support abort. That means, all speech sending to the bot cannot be stopped even the microphone button is clicked (it's in disabled state anyway).