Exception: Statement is not part of it's test case

[Jacob-yen]

Hi~
I encountered a problem when I attempted to generate test cases for a toy program named test_script.py shown as below:

import numpy as np


def add1(a:np.ndarray,b:np.ndarray)->np.ndarray:
    if a.shape == b.shape:
        return a+b
    else:
        return np.array([])
if __name__ == "__main__":
    print(add1(np.array([1,2,3]),np.array([5,6,7,9])))

I placed test_pynguin.py at /data02/projects/test_pynguin and ran the command:

cd /data02/projects/test_pynguin
pynguin --project-path . --output-path /data02/projects/output --module-name test_script

Then I received an exception:

╭─────────────────────────────── Traceback (most recent call last) ────────────────────────────────╮
│                                                                                                  │
│ /data02/anaconda3/envs/py38/bin/pynguin:8 in <module>                                    │
│                                                                                                  │
│   5 from pynguin.cli import main                                                                 │
│   6 if __name__ == '__main__':                                                                   │
│   7 │   sys.argv[0] = re.sub(r'(-script\.pyw|\.exe)?$', '', sys.argv[0])                         │
│ ❱ 8 │   sys.exit(main())                                                                         │
│   9                                                                                              │
│ /data02/anaconda3/envs/py38/lib/python3.8/site-packages/pynguin/cli.py:172 in main       │
│                                                                                                  │
│   169 │                                                                                          │
│   170 │   set_configuration(parsed.config)                                                       │
│   171 │   with console.status("Running Pynguin..."):                                             │
│ ❱ 172 │   │   return run_pynguin().value                                                         │
│   173                                                                                            │
│   174                                                                                            │
│   175 if __name__ == "__main__":                                                                 │
│                                                                                                  │
│ /data02/anaconda3/envs/py38/lib/python3.8/site-packages/pynguin/generator.py:98 in       │
│ run_pynguin                                                                                      │
│                                                                                                  │
│    95 │   """                                                                                    │
│    96 │   try:                                                                                   │
│    97 │   │   _LOGGER.info("Start Pynguin Test Generation…")                                     │
│ ❱  98 │   │   return _run()                                                                      │
│    99 │   finally:                                                                               │
│   100 │   │   _LOGGER.info("Stop Pynguin Test Generation…")                                      │
│   101                                                                                            │
│                                                                                                  │
│ /data02/anaconda3/envs/py38/lib/python3.8/site-packages/pynguin/generator.py:266 in _run │
│                                                                                                  │
│   263 │   │   executor, test_cluster                                                             │
│   264 │   )                                                                                      │
│   265 │   _LOGGER.info("Start generating test cases")                                            │
│ ❱ 266 │   generation_result = algorithm.generate_tests()                                         │
│   267 │   if algorithm.stopping_condition.is_fulfilled():                                        │
│   268 │   │   _LOGGER.info("Used up all resources (%s).", algorithm.stopping_condition)          │
│   269 │   else:                                                                                  │
│                                                                                                  │
│ /data02/anaconda3/envs/py38/lib/python3.8/site-packages/pynguin/generation/algorithms/dy │
│ namosastrategy.py:70 in generate_tests                                                           │
│                                                                                                  │
│    67 │   │   │   self.create_test_suite(self._archive.solutions)                                │
│    68 │   │   )                                                                                  │
│    69 │   │   while self.resources_left() and len(self._archive.uncovered_goals) > 0:            │
│ ❱  70 │   │   │   self.evolve()                                                                  │
│    71 │   │   │   self.after_search_iteration(self.create_test_suite(self._archive.solutions))   │
│    72 │   │                                                                                      │
│    73 │   │   self.after_search_finish()                                                         │
│                                                                                                  │
│ /data02/anaconda3/envs/py38/lib/python3.8/site-packages/pynguin/generation/algorithms/dy │
│ namosastrategy.py:84 in evolve                                                                   │
│                                                                                                  │
│    81 │   │   """Runs one evolution step."""                                                     │
│    82 │   │   offspring_population: List[                                                        │
│    83 │   │   │   tcc.TestCaseChromosome                                                         │
│ ❱  84 │   │   ] = self._breed_next_generation()                                                  │
│    85 │   │                                                                                      │
│    86 │   │   # Create union of parents and offspring                                            │
│    87 │   │   union: List[tcc.TestCaseChromosome] = []                                           │
│                                                                                                  │
│ /data02/anaconda3/envs/py38/lib/python3.8/site-packages/pynguin/generation/algorithms/ab │
│ stractmosastrategy.py:53 in _breed_next_generation                                               │
│                                                                                                  │
│    50 │   │   │   │   │   continue                                                               │
│    51 │   │   │                                                                                  │
│    52 │   │   │   # Apply mutation on offspring_1                                                │
│ ❱  53 │   │   │   self._mutate(offspring_1)                                                      │
│    54 │   │   │   if offspring_1.has_changed() and offspring_1.size() > 0:                       │
│    55 │   │   │   │   offspring_population.append(offspring_1)                                   │
│    56                                                                                            │
│                                                                                                  │
│ /data02/anaconda3/envs/py38/lib/python3.8/site-packages/pynguin/generation/algorithms/ab │
│ stractmosastrategy.py:83 in _mutate                                                              │
│                                                                                                  │
│    80 │                                                                                          │
│    81 │   @staticmethod                                                                          │
│    82 │   def _mutate(offspring: tcc.TestCaseChromosome) -> None:                                │
│ ❱  83 │   │   offspring.mutate()                                                                 │
│    84 │   │   if not offspring.has_changed():                                                    │
│    85 │   │   │   # if offspring is not changed, we try to mutate it once again                  │
│    86 │   │   │   offspring.mutate()                                                             │
│                                                                                                  │
│ /data02/anaconda3/envs/py38/lib/python3.8/site-packages/pynguin/ga/testcasechromosome.py │
│ :129 in mutate                                                                                   │
│                                                                                                  │
│   126 │   │   │   randomness.next_float()                                                        │
│   127 │   │   │   <= config.configuration.search_algorithm.test_change_probability               │
│   128 │   │   ):                                                                                 │
│ ❱ 129 │   │   │   if self._mutation_change():                                                    │
│   130 │   │   │   │   changed = True                                                             │
│   131 │   │                                                                                      │
│   132 │   │   if (                                                                               │
│                                                                                                  │
│ /data02/anaconda3/envs/py38/lib/python3.8/site-packages/pynguin/ga/testcasechromosome.py │
│ :182 in _mutation_change                                                                         │
│                                                                                                  │
│   179 │   │   │   │   │   continue                                                               │
│   180 │   │   │   │   old_distance = statement.ret_val.distance                                  │
│   181 │   │   │   │   ret_val = statement.ret_val                                                │
│ ❱ 182 │   │   │   │   if statement.mutate():                                                     │
│   183 │   │   │   │   │   changed = True                                                         │
│   184 │   │   │   │   else:                                                                      │
│   185 │   │   │   │   │   assert self._test_factory, "Mutation requires a test factory."         │
│                                                                                                  │
│ /data02/anaconda3/envs/py38/lib/python3.8/site-packages/pynguin/testcase/statement.py:10 │
│ 16 in mutate                                                                                     │
│                                                                                                  │
│   1013 │   │   if mutable_param_count > 0:                                                       │
│   1014 │   │   │   p_per_param = 1.0 / mutable_param_count                                       │
│   1015 │   │   │   changed |= self._mutate_special_parameters(p_per_param)                       │
│ ❱ 1016 │   │   │   changed |= self._mutate_parameters(p_per_param)                               │
│   1017 │   │   return changed                                                                    │
│   1018 │                                                                                         │
│   1019 │   def _mutable_argument_count(self) -> int:                                             │
│                                                                                                  │
│ /data02/anaconda3/envs/py38/lib/python3.8/site-packages/pynguin/testcase/statement.py:10 │
│ 52 in _mutate_parameters                                                                         │
│                                                                                                  │
│   1049 │   │   changed = False                                                                   │
│   1050 │   │   for param_name in self._generic_callable.inferred_signature.parameters:           │
│   1051 │   │   │   if randomness.next_float() < p_per_param:                                     │
│ ❱ 1052 │   │   │   │   changed |= self._mutate_parameter(                                        │
│   1053 │   │   │   │   │   param_name, self._generic_callable.inferred_signature                 │
│   1054 │   │   │   │   )                                                                         │
│   1055                                                                                           │
│                                                                                                  │
│ /data02/anaconda3/envs/py38/lib/python3.8/site-packages/pynguin/testcase/statement.py:11 │
│ 16 in _mutate_parameter                                                                          │
│                                                                                                  │
│   1113 │   │   │   │   │   },                                                                    │
│   1114 │   │   │   │   ),                                                                        │
│   1115 │   │   │   )                                                                             │
│ ❱ 1116 │   │   │   copy.mutate()                                                                 │
│   1117 │   │   │   possible_replacements.append(copy.ret_val)                                    │
│   1118 │   │                                                                                     │
│   1119 │   │   # TODO(fk) Use param_type instead of to_mutate.variable_type,                     │
│                                                                                                  │
│ /data02/anaconda3/envs/py38/lib/python3.8/site-packages/pynguin/testcase/statement.py:10 │
│ 16 in mutate                                                                                     │
│                                                                                                  │
│   1013 │   │   if mutable_param_count > 0:                                                       │
│   1014 │   │   │   p_per_param = 1.0 / mutable_param_count                                       │
│   1015 │   │   │   changed |= self._mutate_special_parameters(p_per_param)                       │
│ ❱ 1016 │   │   │   changed |= self._mutate_parameters(p_per_param)                               │
│   1017 │   │   return changed                                                                    │
│   1018 │                                                                                         │
│   1019 │   def _mutable_argument_count(self) -> int:                                             │
│                                                                                                  │
│ /data02/anaconda3/envs/py38/lib/python3.8/site-packages/pynguin/testcase/statement.py:10 │
│ 52 in _mutate_parameters                                                                         │
│                                                                                                  │
│   1049 │   │   changed = False                                                                   │
│   1050 │   │   for param_name in self._generic_callable.inferred_signature.parameters:           │
│   1051 │   │   │   if randomness.next_float() < p_per_param:                                     │
│ ❱ 1052 │   │   │   │   changed |= self._mutate_parameter(                                        │
│   1053 │   │   │   │   │   param_name, self._generic_callable.inferred_signature                 │
│   1054 │   │   │   │   )                                                                         │
│   1055                                                                                           │
│                                                                                                  │
│ /data02/anaconda3/envs/py38/lib/python3.8/site-packages/pynguin/testcase/statement.py:10 │
│ 72 in _mutate_parameter                                                                          │
│                                                                                                  │
│   1069 │   │   current = self._args.get(param_name, None)                                        │
│   1070 │   │   param_type = inf_sig.parameters[param_name]                                       │
│   1071 │   │   possible_replacements = self.test_case.get_objects(                               │
│ ❱ 1072 │   │   │   param_type, self.get_position()                                               │
│   1073 │   │   )                                                                                 │
│   1074 │   │                                                                                     │
│   1075 │   │   # Param has to be optional, otherwise it would be set.                            │
│                                                                                                  │
│ /data02/anaconda3/envs/py38/lib/python3.8/site-packages/pynguin/testcase/statement.py:17 │
│ 6 in get_position                                                                                │
│                                                                                                  │
│    173 │   │   for idx, stmt in enumerate(self._test_case.statements):                           │
│    174 │   │   │   if stmt == self:                                                              │
│    175 │   │   │   │   return idx                                                                │
│ ❱  176 │   │   raise Exception("Statement is not part of it's test case")                        │
│    177 │                                                                                         │
│    178 │   def add_assertion(self, assertion: ass.Assertion) -> None:                            │
│    179 │   │   """Add the given assertion to this statement.                                     │
╰──────────────────────────────────────────────────────────────────────────────────────────────────╯

Looking forward to your reply and thanks in advance!

[stephanlukasczyk]

Thank you for your interest in Pynguin and the bug report.

I've just published Pynguin v0.15.0, which should fix the bug (and also prevents another issue by switching back to our simple assertion-generation strategy).

[Jacob-yen]

@stephanlukasczyk
Hi~ Thanks for your prompt reply and for fixing the bug. I met an awkward problem with the newly released Pynguin: my remote machine just shut down (losing connection with my remote machine) after I executed the command mentioned before. Could you successfully run the toy example I provided?

Hope to get your reply soon. Thanks in advance.
BTW, I really appreciate your working on developing and sharing Pynguin. I love Pynguin so much. :)

[Wooza]

Hi,

I hope that this shutdown is not related to Pynguin's execution, though it might be possible, if any of the tested code or it's transitive dependencies interact with the system. In such case, Pynguin might execute something like os.system("shutdown now"), which is why we highly recommend to run Pynguin within some form of isolation from the host system.

I was able to run your example using the following command: pynguin --project-path ../demo/ --module-name test_script --output-path /tmp/pynguin-results/ -v --budget 10, i.e., with a budget of only 10 seconds, which resulted in 50% coverage and the two files:

# Automatically generated by Pynguin.
import numpy as module_0
import test_script as module_1


def test_case_0():
    tuple_0 = None
    list_0 = [tuple_0]
    ndarray_0 = module_0.ndarray(*list_0)
    assert ndarray_0 is not None
    ndarray_1 = module_1.add1(ndarray_0, ndarray_0)
    assert ndarray_1 is not None

and

# Automatically generated by Pynguin.
import test_script as module_0


def test_case_0():
    try:
        ndarray_0 = None
        ndarray_1 = module_0.add1(ndarray_0, ndarray_0)
    except BaseException:
        pass

The produced test cases are far from ideal, because Pynguin struggles to generate meaningful ndarrays. See this explanation by Stephan, as to why this is the case.

[Jacob-yen]

Hi, @Wooza
I appreciate your prompt reply and my machine works well with pynguin now. ：）