Added scripts to get metadata (runner arguments) associated with the …

…python_testing scripts for automating the process (project-chip#32752)

* Create unit test to test metadata parser

* Create test script metadata parser

* Create test environment file

* Added Copyright Code

* Added Copyright code

* Added copyright code

* Added run arguments comment header

* Created BUILD.gn for metadata parser

* Added reference to the metadata parser BUILD.gn

* Added TC_RR_1_1_Test.py for testing purpose

* Reverted back the testing changes done to the original file

* Updated TC_RR_1_1_Test.py with comments as it was

* Update test_metadata.py to correct the location for TC_RR* test script

* Updated BUILD.gn with the correct naming tests

* Rename env.yaml to test_env.yaml as it is for testing only

* Updated test_metadata.py with the corrected test env file

* Updated and rename TC_RR_1_1_Test.py to simple_run_args.py

Just keeping the runner arguments required for testing purpose and removing all the unwanted test script code for this current testing use case

* Updated test_metadata.py to the new name reference of the testing file

* Renamed test method to test_run_metadata_parser

* Removed the unwanted print method

* Removed handling for int/bool/hex args and trace args

* Remove try-catch block

* Replaced == to is for null comparsion

* Removed argument validation method

* Added the dataclass to be optional

* Updated the test script to be a .txt file and changed the designed to be all run arguments in one line

* Rename simple_run_args.py to simple_run_args.txt

* Updated BUILD.gn

* Rename scripts/tests/py/test_env.yaml to env_example.yaml

* Rename env_example.yaml to scripts/tests/py/env_example.yaml

* Updated path of the environment file

* Change the test script file type

* Updated metadata_parser reference

* Rename env_example.yaml to env_test.yaml

* Update environment file name

* Created __init__.py file

* Added reference for __init__.py in the source

* Created pyproject.toml file

* Created setup.py file

* Created setup.cfg file

* Changed the path of the folder where the scripts are located

* Added print statement

* Updated print statement

* Added inputs to the BUILD.gn file

* Updated paths for file references

* Added test script path to the inputs

* Added sources for the file not find error fix

* Updated metadata.py changes as suggested in the PR comments review

* Updated test script

* Updated license comment

* Removed from pathlib import Path

* Added Print Statement

* Removed the print debug line

* Updated it to use relative path instead of absolute path

* Changed copyright year and removed unwanted sources

* Updated copyright year to be 2024

* Updated copyright year to be 2024

* Updated copyright year to be 2024

* Updated copyright year to be 2024

* Updated copyright year to be 2024

* Updated copyright year to be 2024

* Update comment

Co-authored-by: Andrei Litvin <andy314@gmail.com>

* Updated the class name to MetadataReader

* Updated class name to MetadataReader

* Removed optional on factoryreset

* Renamed env_yaml_file_path and solved inline inside constructor

* Update metadata.py

* Update metadata.py

* Delete scripts/tests/py/simple_run_args.txt

* Update BUILD.gn

* Updated test_metadata.py

* Updated metadata.py

* Create simple_run_args.txt

* Update BUILD.gn

* Delete scripts/tests/py/simple_run_args.txt

* Update BUILD.gn

* Updated test_metadata.py

* Restyled by whitespace

* Restyled by gn

* Restyled by prettier-yaml

* Restyled by autopep8

* Restyled by isort

* Fixed the spell error

* Fixed comments from the PR

* Removed extra print line

* Restyled by autopep8

* Test via comment space

* Test via removed comment space

* Test via comment space

* Restyled by autopep8

* Fixed mypy erros and test script line readability fixes

* Restyled by autopep8

* Removed unawanted variables

* Removing unused Union

* Restyled by autopep8

---------

Co-authored-by: Andrei Litvin <andy314@gmail.com>
Co-authored-by: Restyled.io <commits@restyled.io>

BUILD.gn

@@ -233,6 +233,7 @@ if (current_toolchain != "${dir_pw_toolchain}/default:default") {
  "//scripts/build:build_examples.tests",
  "//scripts/py_matter_idl:matter_idl.tests",
  "//scripts/py_matter_yamltests:matter_yamltests.tests",
+ "//scripts/tests/py:metadata_parser.tests",
  "//src:tests_run",
  ]
 

scripts/tests/py/BUILD.gn

@@ -0,0 +1,36 @@
+# Copyright (c) 2024 Project CHIP Authors
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+import("//build_overrides/build.gni")
+import("//build_overrides/chip.gni")
+
+import("//build_overrides/pigweed.gni")
+import("$dir_pw_build/python.gni")
+
+pw_python_package("metadata_parser") {
+ setup = [
+ "setup.py",
+ "setup.cfg",
+ "pyproject.toml",
+ ]
+
+ inputs = [ "env_test.yaml" ]
+
+ sources = [
+ "__init__.py",
+ "metadata.py",
+ ]
+
+ tests = [ "test_metadata.py" ]
+}

scripts/tests/py/env_test.yaml

@@ -0,0 +1,32 @@
+# Copyright (c) 2024 Project CHIP Authors
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+# test environment argument definitions for metadata parser
+discriminator: 1234
+KVS: kvs1
+trace-to-appjson: json:out/trace_data/app-{SCRIPT_BASE_NAME}.json
+enable-key: 000102030405060708090a0b0c0d0e0f
+
+storage-path: admin_storage.json
+commissioning-method: on-network
+passcode: 20202021
+endpoint: 1
+trace-to-testjson: json:out/trace_data/test-{SCRIPT_BASE_NAME}.json
+trace-to-testperfetto: perfetto:out/trace_data/test-{SCRIPT_BASE_NAME}.perfetto
+manual-code: 10054912339
+PICS: src/app/tests/suites/certification/ci-pics-values
+
+app:
+ all-clusters: out/linux-x64-all-clusters-ipv6only-no-ble-no-wifi-tsan-clang-test/chip-all-clusters-app
+ lock: out/linux-x64-lock-ipv6only-no-ble-no-wifi-tsan-clang-test/chip-lock-app

scripts/tests/py/metadata.py

@@ -0,0 +1,217 @@
+#!/usr/bin/python3
+# Copyright (c) 2024 Project CHIP Authors
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+import re
+from dataclasses import dataclass
+from typing import Dict, List, Optional
+
+import yaml
+
+
+@dataclass
+class Metadata:
+ py_script_path: Optional[str] = None
+ run: Optional[str] = None
+ app: Optional[str] = None
+ discriminator: Optional[str] = None
+ passcode: Optional[str] = None
+
+ def copy_from_dict(self, attr_dict: Dict[str, str]) -> None:
+ """
+ Sets the value of the attributes from a dictionary.
+
+ Attributes:
+
+ attr_dict:
+ Dictionary that stores attributes value that should
+ be transferred to this class.
+ """
+
+ if "app" in attr_dict:
+ self.app = attr_dict["app"]
+
+ if "run" in attr_dict:
+ self.run = attr_dict["run"]
+
+ if "discriminator" in attr_dict:
+ self.discriminator = attr_dict["discriminator"]
+
+ if "passcode" in attr_dict:
+ self.passcode = attr_dict["passcode"]
+
+ if "py_script_path" in attr_dict:
+ self.py_script_path = attr_dict["py_script_path"]
+
+ # TODO - set other attributes as well
+
+
+class MetadataReader:
+ """
+ A class to parse run arguments from the test scripts and 
+ resolve them to environment specific values.
+ """
+
+ def __init__(self, env_yaml_file_path: str):
+ """
+ Reads the YAML file and Constructs the environment object
+
+ Parameters:
+
+ env_yaml_file_path:
+ Path to the environment file that contains the YAML configuration.
+ """
+ with open(env_yaml_file_path) as stream:
+ self.env = yaml.safe_load(stream)
+
+ def __resolve_env_vals__(self, metadata_dict: Dict[str, str]) -> None:
+ """
+ Resolves the argument defined in the test script to environment values.
+ For example, if a test script defines "all_clusters" as the value for app
+ name, we will check the environment configuration to see what raw value is
+ assocaited with the "all_cluster" variable and set the value for "app" option
+ to this raw value.
+
+ Parameter:
+
+ metadata_dict:
+ Dictionary where each key represent a particular argument and its value represent
+ the value for that argument defined in the test script.
+ """
+
+ for run_arg, run_arg_val in metadata_dict.items():
+
+ if not type(run_arg_val) == str or run_arg == "run":
+ metadata_dict[run_arg] = run_arg_val
+ continue
+
+ if run_arg_val is None:
+ continue
+
+ sub_args = run_arg_val.split('/')
+
+ if len(sub_args) not in [1, 2]:
+ err = """The argument is not in the correct format. 
+ The argument must follow the format of arg1 or arg1/arg2. 
+ For example, arg1 represents the argument type and optionally arg2 
+ represents a specific variable defined in the environment file whose
+ value should be used as the argument value. If arg2 is not specified,
+ we will just use the first value associated with arg1 in the environment file."""
+ raise Exception(err)
+
+ if len(sub_args) == 1:
+ run_arg_val = self.env.get(sub_args[0])
+
+ elif len(sub_args) == 2:
+ run_arg_val = self.env.get(sub_args[0]).get(sub_args[1])
+
+ # if a argument has been specified in the comment header
+ # but can't be found in the env file, consider it to be
+ # boolean value.
+ if run_arg_val is None:
+ run_arg_val = True
+
+ metadata_dict[run_arg] = run_arg_val
+
+ def __read_args__(self, run_args_lines: List[str]) -> Dict[str, str]:
+ """
+ Parses a list of lines and extracts argument
+ values from it.
+
+ Parameters:
+
+ run_args_lines:
+ Line in test script header that contains run argument definition.
+ Each line will contain a list of run arguments separated by a space.
+ Line below is one example of what the run argument line will look like:
+ "app/all-clusters discriminator KVS storage-path"
+
+ In this case the line defines that app, discriminator, KVS, and storage-path
+ are the arguments that should be used with this run.
+
+ An argument can be defined multiple times in the same line or in different lines.
+ The last definition will override any previous definition. For example,
+ "KVS/kvs1 KVS/kvs2 KVS/kvs3" line will lead to KVS value of kvs3.
+ """
+ metadata_dict = {}
+
+ for run_line in run_args_lines:
+ for run_arg_word in run_line.strip().split():
+ '''
+ We expect the run arg to be defined in one of the 
+ following two formats:
+ 1. run_arg
+ 2. run_arg/run_arg_val
+
+ Examples: "discriminator" and "app/all_clusters"
+
+ '''
+ run_arg = run_arg_word.split('/', 1)[0]
+ metadata_dict[run_arg] = run_arg_word
+
+ return metadata_dict
+
+ def parse_script(self, py_script_path: str) -> List[Metadata]:
+ """
+ Parses a script and returns a list of metadata object where
+ each element of that list representing run arguments associated
+ with a particular run.
+
+ Parameter:
+
+ py_script_path:
+ path to the python test script
+
+ Return:
+
+ List[Metadata]
+ List of Metadata object where each Metadata element represents
+ the run arguments associated with a particular run defined in
+ the script file.
+ """
+
+ runs_def_ptrn = re.compile(r'^\s*#\s*test-runner-runs:\s*(.*)$')
+ args_def_ptrn = re.compile(r'^\s*#\s*test-runner-run/([a-zA-Z0-9_]+):\s*(.*)$')
+
+ runs_arg_lines: Dict[str, List[str]] = {}
+ runs_metadata = []
+
+ with open(py_script_path, 'r', encoding='utf8') as py_script:
+ for line in py_script.readlines():
+
+ runs_match = runs_def_ptrn.match(line.strip())
+ args_match = args_def_ptrn.match(line.strip())
+
+ if runs_match:
+ for run in runs_match.group(1).strip().split():
+ runs_arg_lines[run] = []
+
+ elif args_match:
+ runs_arg_lines[args_match.group(1)].append(args_match.group(2))
+
+ for run, lines in runs_arg_lines.items():
+ metadata_dict = self.__read_args__(lines)
+ self.__resolve_env_vals__(metadata_dict)
+
+ # store the run value and script location in the
+ # metadata object
+ metadata_dict['py_script_path'] = py_script_path
+ metadata_dict['run'] = run
+
+ metadata = Metadata()
+
+ metadata.copy_from_dict(metadata_dict)
+ runs_metadata.append(metadata)
+
+ return runs_metadata

scripts/tests/py/pyproject.toml

@@ -0,0 +1,17 @@
+# Copyright (c) 2024 Project CHIP Authors
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+[build-system]
+requires = ['setuptools', 'wheel']
+build-backend = 'setuptools.build_meta'

scripts/tests/py/setup.cfg

@@ -0,0 +1,19 @@
+# Copyright (c) 2024 Project CHIP Authors
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+[metadata]
+name = metadata_parser
+version = 0.0.1
+author = Project CHIP Authors
+description = Scripts to get metadata (runner arguments) associated with the python_testing scripts

scripts/tests/py/setup.py

@@ -0,0 +1,20 @@
+# Copyright (c) 2024 Project CHIP Authors
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+
+"""The metadata_parser package."""
+
+import setuptools # type: ignore
+
+setuptools.setup() # Package definition in setup.cfg