Workload Automation v2 in LAVA (WAv2)
*************************************
This directory contains test definitnions and supporting files to run Workload Automation in LAVA. It needs to be noted that some setup in LAVA is required prior to running WAv2 as an automated test.
Job Construction
================
Coordination of the workload automation *host* and the Versatile Express TC2 *target* is done through a multi-node job definition (wav2multi.json). The actions in the JSON file are assigned to specific roles (host & target). The roles are then invoked using lava ACTIONs and in these jobs, the roles execute respective lava-test-shell tests defined in YAML (wa2host.yaml) and wa2target.yaml). The two devices coordinate their execution using synchronization API functions within the lava-test-shell tests. In this test there are 3 api calls used.
* **lava-wait** (waits for data from **lava-send**)
* **lava-sync** (waits for all devices)
Parameter description for wa2host.yaml, wa2target.yaml and wa2host_postprocessing.yaml
* wa2host.yaml
* **AGENDA** - name of the agenda file to run specific workload on TC2. Agenda files are stored in this repository in templates directory.
* **JOB_NAME** - name that identifies job or set of jobs that aggregate results in a single database
* **MODE** - mp (HMP mode booting to A7 cluster), iks (In Kernel Switcher), a7only (only A7 cluster online), a15only (only A15 cluster online)
* **SIGNAL_PREFIX** - identifies the host-target pair of lava-test-shells
* wa2target.yaml
* **SIGNAL_PREFIX** - identifies the host-target pair of lava-test-shells
* wa2host_postprocessing.yaml
* **JOB_NAME** - name that identifies job or set of jobs that aggregate results in a single database. Postprocessing YAML uses this database
* **DELETE_DB** - if set to true, database is deleted after postprocessing is done. Default: false
More information on multi-node can be found in the LAVA documentation:
* Documentation: https://validation.linaro.org/static/docs/writing-multinode.html
* API: https://validation.linaro.org/static/docs/multinodeapi.html?highlight=multinode
Data postprocessing
===================
To simplify the post processing of data from multiple lava actions and possibly multiple lava test runs, the test uses an external sqlite database (hosted on the Xen KVM) and the WAv2 options to store the results until they are ready to be processed (after the entire WAv2 run). Database schema and contents are created by WAv2. Database is located in /root/db_results/ directory. Database name is defined in wa2host.yaml as a parameter JOB_NAME. The parameter has to have the same value for all the jobs running on the same build in order to store the results in the same database.
Database data postprocessing is done using separate YAML definition - wa2host_postprocessing.yaml. It takes one parameter that defines the name of the database to use. The parameter name is the same as for testing jobs: JOB_NAME
Overview of LAVA Architecture & Security
========================================
Due to licensing, WAv2-in-LAVA uses capabilities that restrict access to a specific user or group in LAVA.
Job submission security
-----------------------
Only authorized users are allowed to submit jobs onto the protected device. For the WAv2 testing, the protected device in LAVA is an SSH connection onto a Xen server in the lab that contains the workload automation secure repository, see diagram below.
Results security
----------------
The test jobs we execute in LAVA submit results into private bundle streams with restricted access controls.
Security block diagram
----------------------
::
wa2multi.json
+---------------------------------------------------------+
| wa2host.yaml wa2target.yaml |
| +-----------------------+ +----------------------+ |
| |host: dummy-ssh | |target: vexpress-tc2 | |
| | Xen server | |deploy: android (lsk) | |
| | +WAv2 Repository | IP | | |
| | |<-----| | |
| |restricted to NDA list | | | |
| +----|------------------+ +----------------------+ |
+------|--------------------------------------------------+
v
+---------------------+
| Xen Server |
| + WAv2 Repository |
+---------------------+
Xen Server Configuration
========================
The Xen server is a Linux server with the licensed WAv2 repository installed. LAVA connects to the Xen server over an SSH connection using a dummy-ssh device-type in LAVA[1].
The Xen server is a *persistent* device under control in LAVA, which means that data is persistent over multiple test runs and that only 1 LAVA test can run at a time to prevent accidental access collisions. In an environment that has multiple WAv2 setups, you would need to scale up your WAv2 *host* architecture.
Filesystem & software Requirements:
* /root/templates - contents
WAv2-in-LAVA uses adb over ethernet. WAv2 supports adb over ethernet since version 2.1.4. In order to point WAv2 to correct device config.py file needs to have correct contents. Since config.py is a part of WAv2 and is restricted by the license, it is not available as a part of this git repository. config.py template has to be located in /root/templates/ directory. The template has to have entry for adb_name as $ipaddr. The template is copied to the job directory when LAVA job is started on host and $ipaddr is replaced with correct value of targer IP address.
Install WAv2 on dummy-ssh
=========================
Get Workload Automation sources from below link. (check for access permission to below repo before you clone)::
git clone ssh://user-name@linaro-private.git.linaro.org/srv/landing-teams.git.linaro.org/arm-wa2/wa2
this clone downloads a file called "wlauto-2.1.4dev-external.sh"::
$ sudo sh wlauto-2.1.4dev-external.sh
Accept agreement and follow the instruction from RELEASE_README::
$ pip install wlauto-2.1.4dev.tar.gz
Quick-view:
-----------
- Install SDK build-tools under /usr/
Android expects aapt to be installed under /usr/build-tools/<Android-version>/aapt
- Install latest adb
Workload Automation connects target device via adb over ethernet. So latest adb is required.
Install pre-requirements
------------------------
- apt-get install python-pip
- apt-get install python-dev
- apt-get install python-pexpect (possibly don't need this)
- apt-get install python-serial
- apt-get install python-imaging
- apt-get install libc6:i386
- apt-get install zlib1g:i386
- apt-get install libstdc++6:i386
- Install bbench, Audio and third-party benchmarks
Workload automation expects packages to be available on dummy-ssh server. please refer "APK Files" section
::
$ ls .workload_automation/dependencies/bbench/
bbench.js exit.html forms.js index.html index_noinput.html mich_engin.png results.html sites
$ ls .workload_automation/dependencies/Canon_in_D_Piano.mp3
.workload_automation/dependencies/Canon_in_D_Piano.mp3
$ find /usr/local/lib/python2.7/dist-packages/wlauto/workloads/* | grep apk
/usr/local/lib/python2.7/dist-packages/wlauto/workloads/andebench/andebench-release.apk
/usr/local/lib/python2.7/dist-packages/wlauto/workloads/antutu/antutu_benchmark_4.0.3.apk
/usr/local/lib/python2.7/dist-packages/wlauto/workloads/benchmarkpi/gr.androiddev.BenchmarkPi-1.apk
/usr/local/lib/python2.7/dist-packages/wlauto/workloads/caffeinemark/com.flexycore.caffeinemark_9.apk
/usr/local/lib/python2.7/dist-packages/wlauto/workloads/cfbench/eu.chainfire.cfbench.apk
/usr/local/lib/python2.7/dist-packages/wlauto/workloads/geekbench/com.primatelabs.geekbench3.apk
/usr/local/lib/python2.7/dist-packages/wlauto/workloads/linpack/com.greenecomputing.linpackpro-1.apk
/usr/local/lib/python2.7/dist-packages/wlauto/workloads/quadrant/com.aurorasoftworks.quadrant.ui.professional-1.apk
/usr/local/lib/python2.7/dist-packages/wlauto/workloads/smartbench/com.smartbench.twelve-1.apk
/usr/local/lib/python2.7/dist-packages/wlauto/workloads/sqlite/com.redlicense.benchmark.sqlite-1.apk
/usr/local/lib/python2.7/dist-packages/wlauto/workloads/vellamo/com.quicinc.vellamo.apk
Run WAv2 on dummy-ssh
=====================
Running workload automation will be taken care by run-workload.sh files
**Note:** config.py is having reboot policy as never. Because rebooting of target will be taken care by LAVA.
**Note:** Default config.py is available under ~/.workload_automation/config.py
Debug mode::
$ run_workloads some-agenda-file.yaml -v -d output_dir -c config.py
Please refer example agenda::
/usr/local/lib/python2.7/dist-packages/wlauto/agenda_example.yaml
* other? TODO
WAv2 uses the LAVA dummy-ssh device-type, an SSH connection to an external device that can be controlled by LAVA.
For more information, see the documentation at:
* dummy-ssh: https://validation.linaro.org/static/docs/dummy-deploy.html#configuration-dummy-ssh
