aboutsummaryrefslogtreecommitdiff
diff options
context:
space:
mode:
authorNaresh Kamboju <naresh.kamboju@linaro.org>2015-08-11 11:48:31 +0530
committerNaresh Kamboju <naresh.kamboju@linaro.org>2015-08-11 11:48:31 +0530
commit6891e20194027250d0a15554dfded8e3f13ec403 (patch)
tree00308d5db27cffcd85e35126b1e8f39190406905
parentf236f32abf8419090574da4c8aa6939acf3fe973 (diff)
KVM: Adding KVM device as host to run wa tool
Following features are added * KVM device replacing dummy-ssh devices * Dynamically fetch and install wa tool on kvm device * KVM connects Android device over IP and run wa tool Change-Id: If292ec59d83b2f0d5d41160a8485a3b6014889be Signed-off-by: Naresh Kamboju <naresh.kamboju@linaro.org>
-rw-r--r--README152
-rw-r--r--install-wa2.sh35
-rwxr-xr-xrun-workload.sh2
-rw-r--r--templates/config.py26
-rw-r--r--wa2host.yaml42
-rw-r--r--wa2target.yaml1
6 files changed, 119 insertions, 139 deletions
diff --git a/README b/README
index 213fc82..a320bb4 100644
--- a/README
+++ b/README
@@ -1,10 +1,10 @@
Workload Automation v2 in LAVA (WAv2)
*************************************
-This directory contains test definitnions and supporting files to run Workload Automation in LAVA. It needs to be noted that some setup in LAVA is required prior to running WAv2 as an automated test.
+This directory contains test definitnions and supporting files to run Workload Automation in LAVA.
Job Construction
================
-Coordination of the workload automation *host* and the Versatile Express TC2 *target* is done through a multi-node job definition (wav2multi.json). The actions in the JSON file are assigned to specific roles (host & target). The roles are then invoked using lava ACTIONs and in these jobs, the roles execute respective lava-test-shell tests defined in YAML (wa2host.yaml) and wa2target.yaml). The two devices coordinate their execution using synchronization API functions within the lava-test-shell tests. In this test there are 3 api calls used.
+Coordination of the workload automation *host* and the Versatile Express TC2 or juno *target* is done through a multi-node job definition (wav2multi.json). The actions in the JSON file are assigned to specific roles (host & target). The roles are then invoked using lava ACTIONs and in these jobs, the roles execute respective lava-test-shell tests defined in YAML (wa2host.yaml) and wa2target.yaml). The two devices coordinate their execution using synchronization API functions within the lava-test-shell tests. In this test there are 3 api calls used.
* **lava-wait** (waits for data from **lava-send**)
* **lava-sync** (waits for all devices)
@@ -38,23 +38,20 @@ More information on multi-node can be found in the LAVA documentation:
Data postprocessing
===================
-To simplify the post processing of data from multiple lava actions and possibly multiple lava test runs, the test uses an external sqlite database (hosted on the Xen KVM) and the WAv2 options to store the results until they are ready to be processed (after the entire WAv2 run). Database schema and contents are created by WAv2. Database is located in /root/db_results/ directory. Database name is defined in wa2host.yaml as a parameter JOB_NAME. The parameter has to have the same value for all the jobs running on the same build in order to store the results in the same database.
+To simplify the post processing of data from multiple lava actions and possibly multiple lava test runs, the test uses an external sqlite database (hosted on the KVM) and the WAv2 options to store the results until they are ready to be processed (after the entire WAv2 run). Database schema and contents are created by WAv2. Database is located in /root/db_results/ directory. Database name is defined in wa2host.yaml as a parameter JOB_NAME. The parameter has to have the same value for all the jobs running on the same build in order to store the results in the same database.
-Database data postprocessing is done using separate YAML definition - wa2host_postprocessing.yaml. It takes one parameter that defines the name of the database to use. The parameter name is the same as for testing jobs: JOB_NAME
+Database data postprocessing is done using separate YAML definition - wa2host_postprocessing.yaml. It takes one parameter that defines the name of the database to use. The parameter name is the same as for testing jobs: JOB_NAME
-Overview of LAVA Architecture & Security
+Overview of LAVA Architecture
========================================
-Due to licensing, WAv2-in-LAVA uses capabilities that restrict access to a specific user or group in LAVA.
+WAv2-in-LAVA uses multinode setup to run workload automation testing.
+In multinode setup kvm device act as host and vexpress-tc2 or juno are target devices.
-Job submission security
------------------------
-Only authorized users are allowed to submit jobs onto the protected device. For the WAv2 testing, the protected device in LAVA is an SSH connection onto a Xen server in the lab that contains the workload automation secure repository, see diagram below.
-
-Results security
+Results
----------------
-The test jobs we execute in LAVA submit results into private bundle streams with restricted access controls.
+The test jobs we execute in LAVA submit results into private bundle streams with access controls.
-Security block diagram
+Block diagram
----------------------
::
@@ -62,119 +59,18 @@ Security block diagram
+---------------------------------------------------------+
| wa2host.yaml wa2target.yaml |
| +-----------------------+ +----------------------+ |
- | |host: dummy-ssh | |target: vexpress-tc2 | |
- | | Xen server | |deploy: android (lsk) | |
- | | +WAv2 Repository | IP | | |
- | | |<-----| | |
- | |restricted to NDA list | | | |
- | +----|------------------+ +----------------------+ |
- +------|--------------------------------------------------+
- v
- +---------------------+
- | Xen Server |
- | + WAv2 Repository |
- +---------------------+
-
-Xen Server Configuration
-========================
-The Xen server is a Linux server with the licensed WAv2 repository installed. LAVA connects to the Xen server over an SSH connection using a dummy-ssh device-type in LAVA[1].
-
-The Xen server is a *persistent* device under control in LAVA, which means that data is persistent over multiple test runs and that only 1 LAVA test can run at a time to prevent accidental access collisions. In an environment that has multiple WAv2 setups, you would need to scale up your WAv2 *host* architecture.
-
-Filesystem & software Requirements:
-
-* /root/templates - contents
-
- WAv2-in-LAVA uses adb over ethernet. WAv2 supports adb over ethernet since version 2.1.4. In order to point WAv2 to correct device config.py file needs to have correct contents. Since config.py is a part of WAv2 and is restricted by the license, it is not available as a part of this git repository. config.py template has to be located in /root/templates/ directory. The template has to have entry for adb_name as $ipaddr. The template is copied to the job directory when LAVA job is started on host and $ipaddr is replaced with correct value of targer IP address.
-
-Install WAv2 on dummy-ssh
-=========================
-Get Workload Automation sources from below link. (check for access permission to below repo before you clone)::
-
- git clone ssh://user-name@linaro-private.git.linaro.org/srv/landing-teams.git.linaro.org/arm-wa2/wa2
-
-this clone downloads a file called "wlauto-2.1.4dev-external.sh"::
-
- $ sudo sh wlauto-2.1.4dev-external.sh
-
-Accept agreement and follow the instruction from RELEASE_README::
-
- $ pip install wlauto-2.1.4dev.tar.gz
-
-Quick-view:
------------
-- Install SDK build-tools under /usr/
-
- Android expects aapt to be installed under /usr/build-tools/<Android-version>/aapt
-
-- Install latest adb
-
- Workload Automation connects target device via adb over ethernet. So latest adb is required.
-
-Install pre-requirements
-------------------------
-
-- apt-get install python-pip
-
-- apt-get install python-dev
-
-- apt-get install python-pexpect (possibly don't need this)
-
-- apt-get install python-serial
-
-- apt-get install python-imaging
-
-- apt-get install libc6:i386
-
-- apt-get install zlib1g:i386
-
-- apt-get install libstdc++6:i386
-
-- Install bbench, Audio and third-party benchmarks
-
- Workload automation expects packages to be available on dummy-ssh server. please refer "APK Files" section
-
-::
-
- $ ls .workload_automation/dependencies/bbench/
- bbench.js exit.html forms.js index.html index_noinput.html mich_engin.png results.html sites
-
- $ ls .workload_automation/dependencies/Canon_in_D_Piano.mp3
- .workload_automation/dependencies/Canon_in_D_Piano.mp3
-
- $ find /usr/local/lib/python2.7/dist-packages/wlauto/workloads/* | grep apk
- /usr/local/lib/python2.7/dist-packages/wlauto/workloads/andebench/andebench-release.apk
- /usr/local/lib/python2.7/dist-packages/wlauto/workloads/antutu/antutu_benchmark_4.0.3.apk
- /usr/local/lib/python2.7/dist-packages/wlauto/workloads/benchmarkpi/gr.androiddev.BenchmarkPi-1.apk
- /usr/local/lib/python2.7/dist-packages/wlauto/workloads/caffeinemark/com.flexycore.caffeinemark_9.apk
- /usr/local/lib/python2.7/dist-packages/wlauto/workloads/cfbench/eu.chainfire.cfbench.apk
- /usr/local/lib/python2.7/dist-packages/wlauto/workloads/geekbench/com.primatelabs.geekbench3.apk
- /usr/local/lib/python2.7/dist-packages/wlauto/workloads/linpack/com.greenecomputing.linpackpro-1.apk
- /usr/local/lib/python2.7/dist-packages/wlauto/workloads/quadrant/com.aurorasoftworks.quadrant.ui.professional-1.apk
- /usr/local/lib/python2.7/dist-packages/wlauto/workloads/smartbench/com.smartbench.twelve-1.apk
- /usr/local/lib/python2.7/dist-packages/wlauto/workloads/sqlite/com.redlicense.benchmark.sqlite-1.apk
- /usr/local/lib/python2.7/dist-packages/wlauto/workloads/vellamo/com.quicinc.vellamo.apk
-
-Run WAv2 on dummy-ssh
-=====================
-Running workload automation will be taken care by run-workload.sh files
-
-**Note:** config.py is having reboot policy as never. Because rebooting of target will be taken care by LAVA.
-
-**Note:** Default config.py is available under ~/.workload_automation/config.py
-
-Debug mode::
-
- $ run_workloads some-agenda-file.yaml -v -d output_dir -c config.py
-
-Please refer example agenda::
-
- /usr/local/lib/python2.7/dist-packages/wlauto/agenda_example.yaml
-
-* other? TODO
-
-WAv2 uses the LAVA dummy-ssh device-type, an SSH connection to an external device that can be controlled by LAVA.
-For more information, see the documentation at:
-
-* dummy-ssh: https://validation.linaro.org/static/docs/dummy-deploy.html#configuration-dummy-ssh
+ | |host: kvm | |target: vexpress-tc2 | |
+ | | | |target: juno | |
+ | |deploy: kvm image | |deploy: android | |
+ | |install: | IP | | |
+ | | + WAv2 Repository |<-----| | |
+ | | + WAv2 dependencies | | | |
+ | | + SDK build tools | adb | | |
+ | |run: |----->| | |
+ | | + WAv2 run tests | | | |
+ | +-----------------------+ +----------------------+ |
+ +---------------------------------------------------------+
+KVM Configuration
+========================
+Please refer wa2host.yaml and install-wa2.sh for KVM configuration
diff --git a/install-wa2.sh b/install-wa2.sh
new file mode 100644
index 0000000..52fd7f5
--- /dev/null
+++ b/install-wa2.sh
@@ -0,0 +1,35 @@
+# install SDK build-tools
+echo "install SDK build-tools"
+cd /usr/
+pwd
+# TODO: build-tools.tar.gz only accessible from Linaro LAVA lab
+wget http://testdata.validation.linaro.org/apks/workload-automation/build-tools.tar.gz
+tar -xvf build-tools.tar.gz
+cd -
+
+# clone workload-automation git repo
+echo "clone workload-automation git repo from github"
+git clone https://github.com/ARM-software/workload-automation.git
+cd workload-automation
+echo "python setup.py sdist"
+python setup.py sdist
+echo "sudo pip install dist/wlauto-*.tar.gz"
+sudo pip install dist/wlauto-*.tar.gz
+
+# install workloads dependencies bbench and APKs.
+echo "install workloads bbench and APKs"
+cd /root
+id
+cd; pwd
+# TODO: workload_automation_home.tar.gz only accessible from Linaro LAVA lab
+wget http://testdata.validation.linaro.org/apks/workload-automation/workload_automation_home.tar.gz
+tar -xvf workload_automation_home.tar.gz
+
+# create folder for database results
+mkdir -p /root/db_results/
+
+# check wa install path
+echo $PATH
+which wa
+
+exit 0
diff --git a/run-workload.sh b/run-workload.sh
index 852eeb6..9cc099e 100755
--- a/run-workload.sh
+++ b/run-workload.sh
@@ -47,6 +47,6 @@ adb -s $IPADDR:5555 shell "cat /sys/devices/system/cpu/possible"
echo -n "IKS enabled: "
adb -s $IPADDR:5555 shell "cat /sys/kernel/bL_switcher/active"
echo -e "**********************\n"
-
+echo "wa run $1 -v -f -d $2 -c config.py"
wa run $1 -v -f -d $2 -c config.py
exit 0
diff --git a/templates/config.py b/templates/config.py
new file mode 100644
index 0000000..43f18df
--- /dev/null
+++ b/templates/config.py
@@ -0,0 +1,26 @@
+reboot_policy = 'never'
+instrumentation = [
+ 'execution_time',
+ #'daq',
+ #'interrupts',
+ #'cpufreq',
+ 'hwmon',
+ #'perf',
+ #'streamline',
+]
+
+result_processors = [
+ 'standard',
+ 'csv',
+ 'json',
+ 'sqlite'
+]
+
+logging = {
+ 'file format': '%(asctime)s %(levelname)-8s %(name)s: %(message)s',
+ 'verbose format': '%(asctime)s %(levelname)-8s %(name)s: %(message)s',
+}
+
+device_config = dict(
+ adb_name="$ipaddr"
+)
diff --git a/wa2host.yaml b/wa2host.yaml
index 62cc493..b474e40 100644
--- a/wa2host.yaml
+++ b/wa2host.yaml
@@ -18,15 +18,37 @@ params:
DEVICE: ""
SIGNAL_PREFIX: "workload"
+install:
+ steps:
+ - './install-wa2.sh'
+ deps:
+ - git
+ - wget
+ - zip
+ - xz-utils
+ - python
+ - python-yaml
+ - python-lxml
+ - python-setuptools
+ - python-numpy
+ - python-colorama
+ - python-pip
+ - sqlite3
+ - libstdc++6:i386
+ - libgcc1:i386
+ - zlib1g:i386
+ - libncurses5:i386
+ - android-tools-adb
+ - android-tools-fastboot
+
run:
steps:
- - lava-test-case step1-$SIGNAL_PREFIX-copy-templates --shell cp /root/templates/config.py templates/
- - lava-test-case step2-$SIGNAL_PREFIX-prepare-agenda --shell ./prepare-agenda.py --agenda $AGENDA --job-name $JOB_NAME
- - lava-test-case step3-$SIGNAL_PREFIX-wait-for-ip --shell lava-wait $SIGNAL_PREFIX-send-ip
- - lava-test-case step4-$SIGNAL_PREFIX-prepare-config --shell ./prepare-config.py --prefix $SIGNAL_PREFIX
- - lava-test-case step5-$SIGNAL_PREFIX-prepare-device --shell ./prepare-device.sh $MODE $DEVICE
- - lava-test-case step6-$SIGNAL_PREFIX-run-workload --shell ./run-workload.sh $AGENDA $JOB_NAME
- - if [ -f ./$JOB_NAME/results.csv ]; then lava-test-case-attach step7-$SIGNAL_PREFIX-run-workload ./$JOB_NAME/results.csv; fi
- - if [ -f ./$JOB_NAME/run.log ]; then lava-test-case-attach step8-$SIGNAL_PREFIX-run-workload ./$JOB_NAME/run.log; fi
- - lava-test-case step9-$SIGNAL_PREFIX-tear-down --shell ./tear-down.sh $JOB_NAME
- - lava-test-case step10-$SIGNAL_PREFIX-multinode-sync --shell lava-sync $SIGNAL_PREFIX-finished
+ - lava-test-case step1-$SIGNAL_PREFIX-prepare-agenda --shell ./prepare-agenda.py --agenda $AGENDA --job-name $JOB_NAME
+ - lava-test-case step2-$SIGNAL_PREFIX-wait-for-ip --shell lava-wait $SIGNAL_PREFIX-send-ip
+ - lava-test-case step3-$SIGNAL_PREFIX-prepare-config --shell ./prepare-config.py --prefix $SIGNAL_PREFIX
+ - lava-test-case step4-$SIGNAL_PREFIX-prepare-device --shell ./prepare-device.sh $MODE $DEVICE
+ - lava-test-case step5-$SIGNAL_PREFIX-run-workload --shell ./run-workload.sh $AGENDA $JOB_NAME
+ - if [ -f ./$JOB_NAME/results.csv ]; then lava-test-case-attach step6-$SIGNAL_PREFIX-run-workload ./$JOB_NAME/results.csv; fi
+ - if [ -f ./$JOB_NAME/run.log ]; then lava-test-case-attach step7-$SIGNAL_PREFIX-run-workload ./$JOB_NAME/run.log; fi
+ - lava-test-case step8-$SIGNAL_PREFIX-tear-down --shell ./tear-down.sh $JOB_NAME
+ - lava-test-case step9-$SIGNAL_PREFIX-multinode-sync --shell lava-sync $SIGNAL_PREFIX-finished
diff --git a/wa2target.yaml b/wa2target.yaml
index 6c05ad5..4a63fef 100644
--- a/wa2target.yaml
+++ b/wa2target.yaml
@@ -17,6 +17,7 @@ params:
run:
steps:
+ - 'mount -o rw,remount /'
- lava-test-case step1-$SIGNAL_PREFIX-cat-build-info --shell cat /system/build.prop
- lava-test-case step2-$SIGNAL_PREFIX-get-adb --shell getprop service.adb.tcp.port
- lava-test-case step3-$SIGNAL_PREFIX-set-adb --shell setprop service.adb.tcp.port 5555