Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

SpeechTestDataCI will attempt to test currently training models #67

Open
KatieProchilo opened this issue Jun 11, 2020 · 0 comments
Open
Labels
P1 Good to fix.

Comments

@KatieProchilo
Copy link
Contributor

When the SpeechTestDataCI workflow runs after a test data update, it grabs the latest Custom Speech model to test. However, the list of Custom Speech models includes models that are Succeeded and also models that are Processing.

This results in undesirable behavior when the SpeechTestDataCI workflow gets a model that is Processing. The test will still run, but it will wait the full 40+ min. for that model to finish training before doing so.

This also means that the whole point of the SpeechTestDataCI workflow - the fair comparison - is not happening when testing and training data are updated at the same time. The SpeechTestDataCI workflow will test the new model instead of the benchmark or baseline model.

Solution

The fix for this should be pretty easy. In the output of the azurespeechcli for the speech model list command, we already filter based on the model kind. We can also now filter to only keep models that are Succeeded which will solve this problem.

@KatieProchilo KatieProchilo added P1 Good to fix. and removed P1 Good to fix. labels Jun 11, 2020
@cmayomsft cmayomsft added the P1 Good to fix. label Jun 12, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
P1 Good to fix.
Projects
None yet
Development

No branches or pull requests

2 participants