Issues with some workloads

[vickariofillis]

**
I have some issues with using WA with a Qualcomm development board.

1. I tried running Geekbench 5 but I got errors about no internet connection even though the development is connected to the internet via WiFi.
2. After installing Geekbench 6 as well, there is an error about multiple matches (5 & 6). Is it possible to specify the version with a command line argument? If not, do I have to specify it in the yaml config file (and if yes, where is it located)?
3. I tried copying some of the YAML file examples from the User Guide but I keep getting the same error.

ERROR Error in "configs/test.yaml":
ERROR "config" entry specified inside config/global section; If this is defined in a config file, move the entry content into the top level

Thank you.

[marcbonnici]

1. The internet connectivity check is performed by attempting to ping google's DNS servers from the device and detecting if this operation is successful or not. [1] Could you confirm whether this operation succeeds from your device?
2. This is specified in the agenda (yaml config file) as WA has detected both version on your device and is unsure which you want to run. You can specify this via the version parameter so you can set this to either 5 or 6 and WA should run the correct version (The full list of allowed versions can be found here [2]).
3. If you're having trouble generating the the agenda you can use the built in create command to generate one for reference. E.g.
   wa create agenda geekbench -o my_geekbench_agenda.yaml
   This can then be customized so an agenda to run version 6 should look something like:

config:
    augmentations: []
    iterations: 1
workloads:
-   name: geekbench
    params:
        loops: 1
        prefer_host_package: true
        strict: false
        timeout: 3600
        uninstall: false
        variant: null
        version: 6
        ...

Also would you be able to point me to the documentation this causing this confusion and I can see if this can be clarified?

[1] https://github.com/ARM-software/devlib/blob/be988bb42b766574bb9bd9191fa3532a65091716/devlib/target.py#LL1499C1-L1499C1
[2] https://github.com/ARM-software/devlib/blob/be988bb42b766574bb9bd9191fa3532a65091716/devlib/target.py#LL1499C1-L1499C1

[vickariofillis]

1. Pinging works fine.

adb shell ping -c1 www.google.com
PING www.google.com (142.251.41.68) 56(84) bytes of data.
64 bytes from yyz10s20-in-f4.1e100.net (142.251.41.68): icmp_seq=1 ttl=255 time=4.86 ms

--- www.google.com ping statistics --- 1 packets transmitted, 1 received, 0% packet loss, time 0ms rtt min/avg/max/mdev = 4.863/4.863/4.863/0.000 ms

2. The 2nd link is the same as the 1st. Do you mean this file?

3. Creating the agenda works fine. However, when I try to run it I get the same error.

ERROR Error in "configs/geekbench6.yaml":
ERROR "config" entry specified inside config/global section; If this is defined in a config file, move the entry content into the top level

I have made a folder called configs (containing the geekbench6.yaml created config file) inside the workload-automation directory. Not sure if that's the issue and the config file should be somewhere else.

Also would you be able to point me to the documentation this causing this confusion and I can see if this can be clarified?

I copied the simple example that's here in page 33 (Section 2.1. User Information - Create an Agenda).

[marcbonnici]

1. Interesting, do you have the run.log to see if that indicates what is failing? Worst case we can define a custom target to override the connectivity check on your device [3]
2. Apologies I had meant to link to the workload reference [4]
3. Ok how are invoking your wa command? Using the above agenda I would expect something like wa run configs/geekbench6.yaml -d my_output_directory should work for you?

created config file) inside the workload-automation directory. Not sure if that's the issue and the config file should be somewhere else

That shouldn't impact the configs, however you may run into issues with conflicts between the wa sub directory and wa command itself so just something to be aware of.

[3] https://workload-automation.readthedocs.io/en/latest/developer_information.html?highlight=custom%20target#adding-a-custom-target
[4] https://workload-automation.readthedocs.io/en/latest/plugins/workloads.html#id15

[vickariofillis]

1. Here's the run.log contents from running YouTube which also requires an internet connection.

2. I have created a new agenda for geekbench where I've specified the version, but I'm running into the config issue. I'll try it after I figure out that.

3. I'm trying to run it with the following command.

wa run geekbench -f -c configs/geekbench6.yaml

[marcbonnici]

1. Thanks for the logs, this error message is generated when WA detects "100% packet loss" when attempting to ping the servers [5] so I am unsure why your target would be reporting this.
   In which case depending on your set up, you could define a custom target that overrides this check and always reports True.
   Modifying the example provided in [3] something like the following should do the job

# Import the Target that our custom implementation will be based on
from wa import add_description_for_target

# Import the Target that our custom implementation will be based on
from devlib import AndroidTarget

class DevBoardTarget(AndroidTarget):
    # Provide the name that will be used to identify your custom target
    name = 'android_dev_internet_workaround'

    def is_network_connected(self):
        return True

description = '''An custom target to override the internet conectivity check'''

# Call the helper function with our newly created function and its description.
add_description_for_target(DevBoardTarget, description)

If you save this in your ~/.workload_automation/plugins config directory as something like ~/.workload_automation/plugins/android_dev_internet_workaround.py, WA will automatically pickup your new device and will enable you to use the newly defined android_dev_internet_workaround device in your agendas.

Modifying the example above this would look something like

config:
    augmentations: []
    iterations: 1
    device: android_dev_internet_workaround
workloads:
-   name: geekbench
    params:
        loops: 1
        prefer_host_package: true
        strict: false
        timeout: 3600
        uninstall: false
        variant: null
        version: 6
        ...

3. Right ok, so the syntax of the wa run command is
   wa run <workload or agenda> [<arguments>]

So you would either use wa run geekbench or wa run configs/geekbench6.yaml.

The former command is a short cut which internally generates an agenda using default values and that is why you're receiving this error as the two agendas cannot be merged together.
Any config provided with -c is designed to be additional configuration that is merged on top of the original config, and therefore doesn't require the same toplevel config entry that is present in the examples. This mechanism is outlined in more detail here [6].

[5] https://github.com/ARM-software/devlib/blob/master/devlib/target.py#L1512
[6] https://workload-automation.readthedocs.io/en/latest/user_information.html#configuration-merging

[vickariofillis]

Thanks for the help Marc. It now runs fine. I have one final question.

For Geekbench, I'd like to run the Compute benchmarks besides CPU. Correct me if I'm wrong, but in order to do that I have to modify the UiAutomation.java file *1. Currently, there's only a runCpuBenchmarks function.

*1: Location is ~/Documents/workload-automation/wa/workloads/geekbench/uiauto/app/src/main/java/com/arm/wa/uiauto/geekbench.

Is that the way to go? I'd like to do that with other benchmarks as well (e.g., gfxbench). Any pointers/things to look for in the process?

[marcbonnici]

Ok great and yes you are correct the workload would need to be modified to support the additional test cases. Likely both on the java side to trigger the tests and the python side to allow the desired test case to be selected.
We have some basic introduction / explanation on this process in the docs at [1] if that is of use, otherwise let us know if you have any further questions.

[1] https://workload-automation.readthedocs.io/en/latest/developer_information.html#adding-a-apkuiautomator-workload