Prioritizing Natural Language Test Cases Based on Highly-Used Game Features

This repository contains the source code of the experiments that we performed for our multi-objective optimization approach for prioritization of manual test cases.

---

Our approach was applied and evaluated with the data of the Prodigy Math game from our industry partner Prodigy Education. Our approach consists of two main steps: (1) automatic identification of the game feature(s) that are covered in manual test cases described in natural language and (2) prioritization of test cases based on the game features that are highly-used by players.

(1) Zero-shot classification: we used three techniques/models with strong zero-shot classification capabilities and different combinations (ensembles) of those techniques/models to automatically identify the covered game features from the textual description of test cases:

• BartLargeMNLI - facebook/bart-large-mnli
• CrossEncoderNLI - cross-encoder/nli-distilroberta-base
• LatentEmb - latent-embeddings

(2) Multi-objective optimization to prioritize the execution of test cases: we performed two main experiments with the NSGA-II genetic algorithm to find optimal test case execution orderings:

• Without feature usage: our objectives functions are number of covered game features (maximize) and test execution time (minimize)
• With feature usage: our objectives functions are number of covered highly-used game features (maximize) and test execution time (minimize)

---

Structure of directories

The following directories contains the source code of all the approaches that were part of our experiments.

• zero-shot-classification-experiments: contains the notebook and scripts with the source code of our experiments with different zero-shot techniques for text classifications and their combinations.

• multi-objective-optimization-experiments: contains the notebooks and scripts with the source code of our experiments with different optimization approaches.

---

Dependencies

The following dependencies are required to run the notebooks and scripts on your local machine:

• Python 3.9

• Transformers 4.21.2

  pip install transformers

• Torch 1.10.0+cu113

  pip3 install -f torch torchvision

• SentenceTransformers 2.2.2

  pip install sentence-transformers

• Scikit-learn 1.1.2

  pip install scikit-learn

• Scipy 1.9.1

  pip install scipy

• cliffs_delta 1.0.0

  pip install cliffs-delta

• NLTK 3.4.1

  pip install nltk

• fasttext 0.9.2

  pip install fasttext

• Gensim 4.2.0

  pip install gensim

• Numpy 1.22.4

  pip install numpy

• Pandas 1.4.3

  pip install pandas

• Matplotlib 3.5.3

  pip install matplotlib

• Plotly 5.10.0

  pip install plotly

• Seaborn 0.12.0

  pip install seaborn

• MLFlow 1.29.0

  pip install mlflow

Any questions about the work can be sent to the first author by email (markosviggiato [at] gmail.com)