Jetstream: Adding a workload to execute the #1102

scojac01 · 2020-06-04T11:37:22Z

jetstream suite of browser based tests.

marcbonnici · 2020-06-04T12:01:48Z

wa/workloads/jetstream/__init__.py

+    tests = ['3d-cube-SP', '3d-raytrace-SP', 'acorn-wtb', 'ai-astar', 'Air', 'async-fs', 'Babylon', 'babylon-wtb',
+             'base64-SP', 'Basic', 'bomb-workers', 'Box2D', 'cdjs', 'chai-wtb', 'coffeescript-wtb', 'crypto',
+             'crypto-aes-SP', 'crypto-md5-SP', 'crypto-sha1-SP', 'date-format-tofte-SP', 'date-format-xparb-SP', 'delta-blue',
+             'earley-boyer', 'espree-wtb', 'first-inspector-code-load', 'FlightPlanner', 'float-mm.c', 'gaussian-blur', 
+             'gbemu', 'hash-map', 'jshint-wtb', 'json-parse-inspector', 
+             'json-stringify-inspector', 'lebab-wtb', 'mandreel', 'ML', 'multi-inspector-code-load', 'n-body-SP',
+             'navier-stokes', 'octane-code-load', 'octane-zlib', 'OfflineAssembler', 'pdfjs', 'prepack-wtb', 'raytrace',
+             'regex-dna-SP', 'regexp', 'richards', 'segmentation',
+             'splay', 'stanford-crypto-aes', 'stanford-crypto-pbkdf2', 'stanford-crypto-sha256', 'string-unpack-code-SP',
+             'tagcloud-SP', 'typescript', 'uglify-js-wtb', 'UniPoker']


Can these be organized a bit more uniformly?

marcbonnici · 2020-06-04T12:02:30Z

wa/workloads/jetstream/__init__.py

+                    context.add_metric(test_name, result, 'score', lower_is_better=False)
+                else:
+                    raise WorkloadError('score {} cannot be found'.format(regex))
+


pylint: W293 blank line contains whitespace

marcbonnici · 2020-06-04T12:06:48Z

wa/workloads/jetstream/__init__.py

+        self.target.execute("uiautomator dump")
+        self.target.pull('/sdcard/window_dump.xml', os.path.join(context.output_directory, 'screenDump.xml'))


Can an explicit path using target.working_directory be passed to these commands?

marcbonnici · 2020-06-04T12:08:17Z

wa/workloads/jetstream/__init__.py

+        self.target.execute("uiautomator dump")
+        self.target.pull('/sdcard/window_dump.xml', os.path.join(context.output_directory, 'screenDump.xml'))


Also out of curiosity have you found this is a more reliable method of extracting the scores rather than performing this from the java side?

We tried originally with the standard way of IDing and logging in the java side but it was constantly missing elements when we looped through it. This method is the most reliable for this workload.

marcbonnici · 2020-06-04T14:06:55Z

wa/workloads/jetstream/__init__.py

+from wa.framework.exception import ValidationError, WorkloadError
+from wa.utils.types import list_of_strs
+from wa.utils.misc import unique


Except for WorkloadError are these unused imports?

marcbonnici · 2020-06-04T14:09:51Z

wa/workloads/jetstream/uiauto/app/src/main/java/com/arm/wa/uiauto/jetstream/UiAutomation.java

+        pressEnter();
+    }
+
+    public void clearTabs() throws Exception {


This seems like it becoming a common function, s it worth adding it as a clearChromeTabs function to BaseUiAutomation?

marcbonnici · 2020-06-04T15:02:25Z

wa/workloads/jetstream/__init__.py

+
+    def __init__(self, target, **kwargs):
+        super(Jetstream, self).__init__(target, **kwargs)
+        self.gui.timeout = 1500


On my device this timeout was reached. (took ~33 mins to complete). Can this be raised to be greater then the timeout from the java side?

marcbonnici

And can the commit message be updated?

marcbonnici

Can the commit message title be fixed to fit on a single line?

Pylint fixes: https://travis-ci.org/github/ARM-software/workload-automation/builds/695014614

marcbonnici · 2020-06-10T10:15:21Z

wa/workloads/jetstream/uiauto/app/src/main/java/com/arm/wa/uiauto/jetstream/UiAutomation.java

+    /*public void clearTabs() throws Exception {
+        UiObject tabselector =
+            mDevice.findObject(new UiSelector().resourceId("com.android.chrome:id/tab_switcher_button")
+                .className("android.widget.ImageButton"));
+        if (!tabselector.exists()){
+            return;
+        }
+        tabselector.click();
+        UiObject menu =
+            mDevice.findObject(new UiSelector().resourceId("com.android.chrome:id/menu_button")
+                .className("android.widget.ImageButton"));
+        menu.click();
+        UiObject closetabs =
+            mDevice.findObject(new UiSelector().textContains("Close all tabs"));
+        if (closetabs.exists()){
+            closetabs.click();
+        }
+    }*/
+}


Remove commented code.

marcbonnici · 2020-06-10T12:03:07Z

wa/workloads/jetstream/__init__.py

+        self.target.execute('uiautomator dump')
+        self.target.pull(os.path.join('/sdcard/window_dump.xml'), os.path.join(context.output_directory, 'screenDump.xml'))
+


Can an explicit path using target.working_directory be passed to these commands to prevent relying on defaults?

Jetstream is a browser based benchmark. This workload executes the full suite of Jetstream tests and also includes the addition of the clearChromeTabs function to the base ui automation class.

robF8 · 2020-07-08T13:26:17Z

Updated commit message to be one liner
Removed commented code
Added explicit path for xml dump
Fixed pylint errors
Fixed pep8 errors

marcbonnici · 2020-07-27T16:40:01Z

wa/workloads/jetstream/uiauto/app/src/main/java/com/arm/wa/uiauto/jetstream/UiAutomation.java

+        UiObject scores =
+            mDevice.findObject(new UiSelector().resourceId("result-summary"));
+        scores.waitForExists(2100000);
+    }


Maybe allow for the runtime of the benchmark to be configurable as the timeout is still hit on my device.
Also the results screen should be checked for here and raise the appropriate error otherwise the error about a missing score is misleading when the workload just hasn't finished yet.

marcbonnici · 2020-07-27T16:40:48Z

wa/workloads/jetstream/__init__.py

+            total_score_match = total_score_regex.search(xml_str)
+            if total_score_match:
+                total_score = float(total_score_match.group(1))
+                context.add_metric('jetstream', total_score, 'score', lower_is_better=False)
+            else:
+                raise WorkloadError('Total score for jetstream could not be found')
+            for regex in self.regex_tests:
+                match = regex.search(xml_str)
+                if match:
+                    result = float(match.group(1))
+                    test_name = match.group(2)
+                    context.add_metric(test_name, result, 'score', lower_is_better=False)
+                else:
+                    raise WorkloadError('score {} cannot be found'.format(regex))


This does not need to be within the with statement.

marcbonnici · 2020-07-27T16:51:06Z

wa/workloads/jetstream/__init__.py

+        for test in self.tests:
+            formatted_string = 'text="([\d.]+)" resource-id="results-cell-({})-score"'.format(test)
+            self.regex_tests.append(re.compile(formatted_string))
+        # Add regex for tests with annoyingly different resource id's
+        self.regex_tests.append(re.compile(r'text="([\d.]+)" resource-id="wasm-score-id(gcc-loops-wasm)"'))
+        self.regex_tests.append(re.compile(r'text="([\d.]+)" resource-id="wasm-score-id(HashSet-wasm)"'))
+        self.regex_tests.append(re.compile(r'text="([\d.]+)" resource-id="wasm-score-id(quicksort-wasm)"'))
+        self.regex_tests.append(re.compile(r'text="([\d.]+)" resource-id="wasm-score-id(richards-wasm)"'))
+        self.regex_tests.append(re.compile(r'text="([\d.]+)" resource-id="wasm-score-id(tsf-wasm)"'))
+        self.regex_tests.append(re.compile(r'text="([\d.]+)" resource-id="(wsl)-score-score"'))


Is there a particular reason for all the tests to be specified explicitly? The workload doesn't seem to be configurable which tests are ran and looking at the code there appears to be 3 unique patterns for matching results so would it not be easier to match on the "generic" patterns?

marcbonnici requested changes Jun 4, 2020

View reviewed changes

marcbonnici reviewed Jun 4, 2020

View reviewed changes

scojac01 force-pushed the robf-jetstream branch from 1ed5263 to f29f6d5 Compare June 5, 2020 12:58

marcbonnici reviewed Jun 10, 2020

View reviewed changes

robF8 force-pushed the robf-jetstream branch from f29f6d5 to a8a6bf6 Compare July 8, 2020 13:06

Add jetstream workload

4b1c446

Jetstream is a browser based benchmark. This workload executes the full suite of Jetstream tests and also includes the addition of the clearChromeTabs function to the base ui automation class.

robF8 force-pushed the robf-jetstream branch from a8a6bf6 to 4b1c446 Compare July 8, 2020 13:23

marcbonnici reviewed Jul 27, 2020

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Jetstream: Adding a workload to execute the #1102

Jetstream: Adding a workload to execute the #1102

scojac01 commented Jun 4, 2020

marcbonnici Jun 4, 2020

marcbonnici Jun 4, 2020

marcbonnici Jun 4, 2020

marcbonnici Jun 4, 2020

scojac01 Jun 5, 2020

marcbonnici Jun 4, 2020

marcbonnici Jun 4, 2020

marcbonnici Jun 4, 2020

marcbonnici left a comment

marcbonnici left a comment

marcbonnici Jun 10, 2020

marcbonnici Jun 10, 2020

robF8 commented Jul 8, 2020

marcbonnici Jul 27, 2020

marcbonnici Jul 27, 2020

marcbonnici Jul 27, 2020

		self.target.execute("uiautomator dump")
		self.target.pull('/sdcard/window_dump.xml', os.path.join(context.output_directory, 'screenDump.xml'))

		self.target.execute('uiautomator dump')
		self.target.pull(os.path.join('/sdcard/window_dump.xml'), os.path.join(context.output_directory, 'screenDump.xml'))

Jetstream: Adding a workload to execute the #1102

Are you sure you want to change the base?

Jetstream: Adding a workload to execute the #1102

Conversation

scojac01 commented Jun 4, 2020

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

marcbonnici left a comment

Choose a reason for hiding this comment

marcbonnici left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

robF8 commented Jul 8, 2020

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment