v2: initial commit
- Added utils, bin, and lib folders and related tools
- Added Linux smoke test case as an example
- Added test plan example that used to test test-runner
Change-Id: I73693108db219cb67afc99525d80cab93f1eb9ea
Signed-off-by: Chase Qi <chase.qi@linaro.org>
diff --git a/automated/COPYING b/automated/COPYING
new file mode 100644
index 0000000..6d45519
--- /dev/null
+++ b/automated/COPYING
@@ -0,0 +1,340 @@
+ GNU GENERAL PUBLIC LICENSE
+ Version 2, June 1991
+
+ Copyright (C) 1989, 1991 Free Software Foundation, Inc.
+ 51 Franklin St, Fifth Floor, Boston, MA 02110-1301 USA
+ Everyone is permitted to copy and distribute verbatim copies
+ of this license document, but changing it is not allowed.
+
+ Preamble
+
+ The licenses for most software are designed to take away your
+freedom to share and change it. By contrast, the GNU General Public
+License is intended to guarantee your freedom to share and change free
+software--to make sure the software is free for all its users. This
+General Public License applies to most of the Free Software
+Foundation's software and to any other program whose authors commit to
+using it. (Some other Free Software Foundation software is covered by
+the GNU Library General Public License instead.) You can apply it to
+your programs, too.
+
+ When we speak of free software, we are referring to freedom, not
+price. Our General Public Licenses are designed to make sure that you
+have the freedom to distribute copies of free software (and charge for
+this service if you wish), that you receive source code or can get it
+if you want it, that you can change the software or use pieces of it
+in new free programs; and that you know you can do these things.
+
+ To protect your rights, we need to make restrictions that forbid
+anyone to deny you these rights or to ask you to surrender the rights.
+These restrictions translate to certain responsibilities for you if you
+distribute copies of the software, or if you modify it.
+
+ For example, if you distribute copies of such a program, whether
+gratis or for a fee, you must give the recipients all the rights that
+you have. You must make sure that they, too, receive or can get the
+source code. And you must show them these terms so they know their
+rights.
+
+ We protect your rights with two steps: (1) copyright the software, and
+(2) offer you this license which gives you legal permission to copy,
+distribute and/or modify the software.
+
+ Also, for each author's protection and ours, we want to make certain
+that everyone understands that there is no warranty for this free
+software. If the software is modified by someone else and passed on, we
+want its recipients to know that what they have is not the original, so
+that any problems introduced by others will not reflect on the original
+authors' reputations.
+
+ Finally, any free program is threatened constantly by software
+patents. We wish to avoid the danger that redistributors of a free
+program will individually obtain patent licenses, in effect making the
+program proprietary. To prevent this, we have made it clear that any
+patent must be licensed for everyone's free use or not licensed at all.
+
+ The precise terms and conditions for copying, distribution and
+modification follow.
+
+ GNU GENERAL PUBLIC LICENSE
+ TERMS AND CONDITIONS FOR COPYING, DISTRIBUTION AND MODIFICATION
+
+ 0. This License applies to any program or other work which contains
+a notice placed by the copyright holder saying it may be distributed
+under the terms of this General Public License. The "Program", below,
+refers to any such program or work, and a "work based on the Program"
+means either the Program or any derivative work under copyright law:
+that is to say, a work containing the Program or a portion of it,
+either verbatim or with modifications and/or translated into another
+language. (Hereinafter, translation is included without limitation in
+the term "modification".) Each licensee is addressed as "you".
+
+Activities other than copying, distribution and modification are not
+covered by this License; they are outside its scope. The act of
+running the Program is not restricted, and the output from the Program
+is covered only if its contents constitute a work based on the
+Program (independent of having been made by running the Program).
+Whether that is true depends on what the Program does.
+
+ 1. You may copy and distribute verbatim copies of the Program's
+source code as you receive it, in any medium, provided that you
+conspicuously and appropriately publish on each copy an appropriate
+copyright notice and disclaimer of warranty; keep intact all the
+notices that refer to this License and to the absence of any warranty;
+and give any other recipients of the Program a copy of this License
+along with the Program.
+
+You may charge a fee for the physical act of transferring a copy, and
+you may at your option offer warranty protection in exchange for a fee.
+
+ 2. You may modify your copy or copies of the Program or any portion
+of it, thus forming a work based on the Program, and copy and
+distribute such modifications or work under the terms of Section 1
+above, provided that you also meet all of these conditions:
+
+ a) You must cause the modified files to carry prominent notices
+ stating that you changed the files and the date of any change.
+
+ b) You must cause any work that you distribute or publish, that in
+ whole or in part contains or is derived from the Program or any
+ part thereof, to be licensed as a whole at no charge to all third
+ parties under the terms of this License.
+
+ c) If the modified program normally reads commands interactively
+ when run, you must cause it, when started running for such
+ interactive use in the most ordinary way, to print or display an
+ announcement including an appropriate copyright notice and a
+ notice that there is no warranty (or else, saying that you provide
+ a warranty) and that users may redistribute the program under
+ these conditions, and telling the user how to view a copy of this
+ License. (Exception: if the Program itself is interactive but
+ does not normally print such an announcement, your work based on
+ the Program is not required to print an announcement.)
+
+These requirements apply to the modified work as a whole. If
+identifiable sections of that work are not derived from the Program,
+and can be reasonably considered independent and separate works in
+themselves, then this License, and its terms, do not apply to those
+sections when you distribute them as separate works. But when you
+distribute the same sections as part of a whole which is a work based
+on the Program, the distribution of the whole must be on the terms of
+this License, whose permissions for other licensees extend to the
+entire whole, and thus to each and every part regardless of who wrote it.
+
+Thus, it is not the intent of this section to claim rights or contest
+your rights to work written entirely by you; rather, the intent is to
+exercise the right to control the distribution of derivative or
+collective works based on the Program.
+
+In addition, mere aggregation of another work not based on the Program
+with the Program (or with a work based on the Program) on a volume of
+a storage or distribution medium does not bring the other work under
+the scope of this License.
+
+ 3. You may copy and distribute the Program (or a work based on it,
+under Section 2) in object code or executable form under the terms of
+Sections 1 and 2 above provided that you also do one of the following:
+
+ a) Accompany it with the complete corresponding machine-readable
+ source code, which must be distributed under the terms of Sections
+ 1 and 2 above on a medium customarily used for software interchange; or,
+
+ b) Accompany it with a written offer, valid for at least three
+ years, to give any third party, for a charge no more than your
+ cost of physically performing source distribution, a complete
+ machine-readable copy of the corresponding source code, to be
+ distributed under the terms of Sections 1 and 2 above on a medium
+ customarily used for software interchange; or,
+
+ c) Accompany it with the information you received as to the offer
+ to distribute corresponding source code. (This alternative is
+ allowed only for noncommercial distribution and only if you
+ received the program in object code or executable form with such
+ an offer, in accord with Subsection b above.)
+
+The source code for a work means the preferred form of the work for
+making modifications to it. For an executable work, complete source
+code means all the source code for all modules it contains, plus any
+associated interface definition files, plus the scripts used to
+control compilation and installation of the executable. However, as a
+special exception, the source code distributed need not include
+anything that is normally distributed (in either source or binary
+form) with the major components (compiler, kernel, and so on) of the
+operating system on which the executable runs, unless that component
+itself accompanies the executable.
+
+If distribution of executable or object code is made by offering
+access to copy from a designated place, then offering equivalent
+access to copy the source code from the same place counts as
+distribution of the source code, even though third parties are not
+compelled to copy the source along with the object code.
+
+ 4. You may not copy, modify, sublicense, or distribute the Program
+except as expressly provided under this License. Any attempt
+otherwise to copy, modify, sublicense or distribute the Program is
+void, and will automatically terminate your rights under this License.
+However, parties who have received copies, or rights, from you under
+this License will not have their licenses terminated so long as such
+parties remain in full compliance.
+
+ 5. You are not required to accept this License, since you have not
+signed it. However, nothing else grants you permission to modify or
+distribute the Program or its derivative works. These actions are
+prohibited by law if you do not accept this License. Therefore, by
+modifying or distributing the Program (or any work based on the
+Program), you indicate your acceptance of this License to do so, and
+all its terms and conditions for copying, distributing or modifying
+the Program or works based on it.
+
+ 6. Each time you redistribute the Program (or any work based on the
+Program), the recipient automatically receives a license from the
+original licensor to copy, distribute or modify the Program subject to
+these terms and conditions. You may not impose any further
+restrictions on the recipients' exercise of the rights granted herein.
+You are not responsible for enforcing compliance by third parties to
+this License.
+
+ 7. If, as a consequence of a court judgment or allegation of patent
+infringement or for any other reason (not limited to patent issues),
+conditions are imposed on you (whether by court order, agreement or
+otherwise) that contradict the conditions of this License, they do not
+excuse you from the conditions of this License. If you cannot
+distribute so as to satisfy simultaneously your obligations under this
+License and any other pertinent obligations, then as a consequence you
+may not distribute the Program at all. For example, if a patent
+license would not permit royalty-free redistribution of the Program by
+all those who receive copies directly or indirectly through you, then
+the only way you could satisfy both it and this License would be to
+refrain entirely from distribution of the Program.
+
+If any portion of this section is held invalid or unenforceable under
+any particular circumstance, the balance of the section is intended to
+apply and the section as a whole is intended to apply in other
+circumstances.
+
+It is not the purpose of this section to induce you to infringe any
+patents or other property right claims or to contest validity of any
+such claims; this section has the sole purpose of protecting the
+integrity of the free software distribution system, which is
+implemented by public license practices. Many people have made
+generous contributions to the wide range of software distributed
+through that system in reliance on consistent application of that
+system; it is up to the author/donor to decide if he or she is willing
+to distribute software through any other system and a licensee cannot
+impose that choice.
+
+This section is intended to make thoroughly clear what is believed to
+be a consequence of the rest of this License.
+
+ 8. If the distribution and/or use of the Program is restricted in
+certain countries either by patents or by copyrighted interfaces, the
+original copyright holder who places the Program under this License
+may add an explicit geographical distribution limitation excluding
+those countries, so that distribution is permitted only in or among
+countries not thus excluded. In such case, this License incorporates
+the limitation as if written in the body of this License.
+
+ 9. The Free Software Foundation may publish revised and/or new versions
+of the General Public License from time to time. Such new versions will
+be similar in spirit to the present version, but may differ in detail to
+address new problems or concerns.
+
+Each version is given a distinguishing version number. If the Program
+specifies a version number of this License which applies to it and "any
+later version", you have the option of following the terms and conditions
+either of that version or of any later version published by the Free
+Software Foundation. If the Program does not specify a version number of
+this License, you may choose any version ever published by the Free Software
+Foundation.
+
+ 10. If you wish to incorporate parts of the Program into other free
+programs whose distribution conditions are different, write to the author
+to ask for permission. For software which is copyrighted by the Free
+Software Foundation, write to the Free Software Foundation; we sometimes
+make exceptions for this. Our decision will be guided by the two goals
+of preserving the free status of all derivatives of our free software and
+of promoting the sharing and reuse of software generally.
+
+ NO WARRANTY
+
+ 11. BECAUSE THE PROGRAM IS LICENSED FREE OF CHARGE, THERE IS NO WARRANTY
+FOR THE PROGRAM, TO THE EXTENT PERMITTED BY APPLICABLE LAW. EXCEPT WHEN
+OTHERWISE STATED IN WRITING THE COPYRIGHT HOLDERS AND/OR OTHER PARTIES
+PROVIDE THE PROGRAM "AS IS" WITHOUT WARRANTY OF ANY KIND, EITHER EXPRESSED
+OR IMPLIED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF
+MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE. THE ENTIRE RISK AS
+TO THE QUALITY AND PERFORMANCE OF THE PROGRAM IS WITH YOU. SHOULD THE
+PROGRAM PROVE DEFECTIVE, YOU ASSUME THE COST OF ALL NECESSARY SERVICING,
+REPAIR OR CORRECTION.
+
+ 12. IN NO EVENT UNLESS REQUIRED BY APPLICABLE LAW OR AGREED TO IN WRITING
+WILL ANY COPYRIGHT HOLDER, OR ANY OTHER PARTY WHO MAY MODIFY AND/OR
+REDISTRIBUTE THE PROGRAM AS PERMITTED ABOVE, BE LIABLE TO YOU FOR DAMAGES,
+INCLUDING ANY GENERAL, SPECIAL, INCIDENTAL OR CONSEQUENTIAL DAMAGES ARISING
+OUT OF THE USE OR INABILITY TO USE THE PROGRAM (INCLUDING BUT NOT LIMITED
+TO LOSS OF DATA OR DATA BEING RENDERED INACCURATE OR LOSSES SUSTAINED BY
+YOU OR THIRD PARTIES OR A FAILURE OF THE PROGRAM TO OPERATE WITH ANY OTHER
+PROGRAMS), EVEN IF SUCH HOLDER OR OTHER PARTY HAS BEEN ADVISED OF THE
+POSSIBILITY OF SUCH DAMAGES.
+
+ END OF TERMS AND CONDITIONS
+
+ How to Apply These Terms to Your New Programs
+
+ If you develop a new program, and you want it to be of the greatest
+possible use to the public, the best way to achieve this is to make it
+free software which everyone can redistribute and change under these terms.
+
+ To do so, attach the following notices to the program. It is safest
+to attach them to the start of each source file to most effectively
+convey the exclusion of warranty; and each file should have at least
+the "copyright" line and a pointer to where the full notice is found.
+
+ <one line to give the program's name and a brief idea of what it does.>
+ Copyright (C) <year> <name of author>
+
+ This program is free software; you can redistribute it and/or modify
+ it under the terms of the GNU General Public License as published by
+ the Free Software Foundation; either version 2 of the License, or
+ (at your option) any later version.
+
+ This program is distributed in the hope that it will be useful,
+ but WITHOUT ANY WARRANTY; without even the implied warranty of
+ MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+ GNU General Public License for more details.
+
+ You should have received a copy of the GNU General Public License
+ along with this program; if not, write to the Free Software
+ Foundation, Inc., 51 Franklin St, Fifth Floor, Boston, MA 02110-1301 USA
+
+
+Also add information on how to contact you by electronic and paper mail.
+
+If the program is interactive, make it output a short notice like this
+when it starts in an interactive mode:
+
+ Gnomovision version 69, Copyright (C) year name of author
+ Gnomovision comes with ABSOLUTELY NO WARRANTY; for details type `show w'.
+ This is free software, and you are welcome to redistribute it
+ under certain conditions; type `show c' for details.
+
+The hypothetical commands `show w' and `show c' should show the appropriate
+parts of the General Public License. Of course, the commands you use may
+be called something other than `show w' and `show c'; they could even be
+mouse-clicks or menu items--whatever suits your program.
+
+You should also get your employer (if you work as a programmer) or your
+school, if any, to sign a "copyright disclaimer" for the program, if
+necessary. Here is a sample; alter the names:
+
+ Yoyodyne, Inc., hereby disclaims all copyright interest in the program
+ `Gnomovision' (which makes passes at compilers) written by James Hacker.
+
+ <signature of Ty Coon>, 1 April 1989
+ Ty Coon, President of Vice
+
+This General Public License does not permit incorporating your program into
+proprietary programs. If your program is a subroutine library, you may
+consider it more useful to permit linking proprietary applications with the
+library. If this is what you want to do, use the GNU Library General
+Public License instead of this License.
diff --git a/automated/README b/automated/README
new file mode 100644
index 0000000..26636f8
--- /dev/null
+++ b/automated/README
@@ -0,0 +1,56 @@
+Getting started
+===============
+ git clone https://git.linaro.org/qa/test-definitions.git
+ cd ./test-definitions
+ . ./automated/bin/setenv.sh
+ test-runner -h
+
+Running test
+============
+Both test script and corresponding test definition files provided in the same
+folder that named with the test case name.
+
+Test script can be run independently, you may need to specify parameters
+manually as needed.
+
+Test definition file in YAML format created for each test to
+ - Define default test parameters
+ - Run test with local test-runner, which supports to run a set of tests
+ defined in agenda file, customized parameters are supported as well.
+ - Run test in LAVA(Linaro Automation and Validation Architecture)
+
+Running test script
+------------------
+linux:
+ cd ./automated/linux/smoke/
+ ./smoke.sh
+ # skip package installation.
+ ./smoke.sh -s true
+
+android:
+ cd ./automated/android/dd-wr-speed/
+ ./dd-wr-speed.sh
+ # Specify SN when more than one device connected.
+ ./dd-wr-speed.sh -s "serial_no"
+ # Specify other params.
+ ./dd-wr-speed.sh -i "10" -p "/dev/block/mmcblk1p1"
+
+Using test-runner
+----------------
+single test run:
+ test-runner -d ./automated/linux/smoke/smoke.yaml
+ # skip package install
+ test-runner -d ./automated/linux/smoke/smoke.yaml -s
+
+running test plan:
+ # Run a set of tests defined in agenda file.
+ test-runner -a ./plans/linux-example.yaml
+
+Collecting result
+=================
+Please find test results in the following locations.
+ - ${OUTPUT} and ${RESULT_FILE} for each test are defined in the test case.
+ - test-runner parse stdout.log and save result to ${test_path}
+ - test-runner combines results of executed tests and save them in
+ ${OUTPUT}/results.csv,
+ - ${OUTPUT} path of test-runner can be modified with '-o'
diff --git a/automated/bin/arm64/busybox b/automated/bin/arm64/busybox
new file mode 100755
index 0000000..6d09a07
--- /dev/null
+++ b/automated/bin/arm64/busybox
Binary files differ
diff --git a/automated/bin/armeabi/busybox b/automated/bin/armeabi/busybox
new file mode 100755
index 0000000..cebf6cf
--- /dev/null
+++ b/automated/bin/armeabi/busybox
Binary files differ
diff --git a/automated/bin/setenv.sh b/automated/bin/setenv.sh
new file mode 100755
index 0000000..c0a3f79
--- /dev/null
+++ b/automated/bin/setenv.sh
@@ -0,0 +1,13 @@
+#!/bin/sh
+
+export REPO_PATH="$(pwd)"
+echo "REPO_PATH: ${REPO_PATH}"
+
+if ! [ -d "${REPO_PATH}/automated/bin" ]; then
+ echo "ERROR: Please execute the below command from test-definitions DIR"
+ echo " . ./automated/bin/setenv.sh"
+ exit 1
+fi
+
+export PATH="${REPO_PATH}/automated/bin":"${PATH}"
+echo "BIN_PATH: ${PATH}"
diff --git a/automated/bin/test-runner b/automated/bin/test-runner
new file mode 120000
index 0000000..0e7f8bc
--- /dev/null
+++ b/automated/bin/test-runner
@@ -0,0 +1 @@
+../utils/test-runner.py
\ No newline at end of file
diff --git a/automated/lib/android-test-lib b/automated/lib/android-test-lib
new file mode 100755
index 0000000..034b153
--- /dev/null
+++ b/automated/lib/android-test-lib
@@ -0,0 +1,58 @@
+#!/bin/sh
+
+initialize_adb() {
+ if [ -z "${SN}" ]; then
+ local number="$(adb devices | grep -wc 'device')"
+ if [ "${number}" -gt 1 ]; then
+ warn_msg "Device not specified; define SN or use '-s'"
+ error_msg "More than one device or emulator found"
+ elif [ "${number}" -eq 1 ]; then
+ SN="$(adb devices | grep -w 'device' | awk '{print $1}')"
+ export SN
+ else
+ error_msg "Device NOT found"
+ fi
+ fi
+
+ adb -s "${SN}" shell ls / > /dev/null 2>&1
+ if [ $? -eq 0 ]; then
+ info_msg "Connected to device ${SN} successfully"
+ else
+ error_msg "Unable to connect to device ${SN}"
+ fi
+}
+
+detect_abi() {
+ # "| tr -d '\r'" is needed here, refer to the below issue.
+ # https://code.google.com/p/android/issues/detail?id=2482
+ abi="$(adb -s "${SN}" shell uname -m | tr -d '\r')"
+ case $abi in
+ armv7|armv7l|armv7el|armv7lh) abi="armeabi" ;;
+ arm64|armv8|arm64-v8a|aarch64) abi="arm64" ;;
+ *) error_msg "Unknown architecture" ;;
+ esac
+ info_msg "ABI: ${abi}"
+}
+
+install() {
+ local file_path="$1"
+ local file_name="$(basename "${file_path}")"
+
+ if adb -s "${SN}" shell mount | grep system | grep -q ro; then
+ # Remounts the /system partition on the device read-write
+ info_msg "/system partition is read-only, remounting it read-write..."
+ adb -s "${SN}" remount
+ fi
+
+ info_msg "Installing ${file_name}"
+ adb -s "${SN}" push "${file_path}" "/system/bin/"
+ adb -s "${SN}" shell chmod 755 "/system/bin/${file_name}"
+}
+
+pull_output() {
+ local device_output="$1"
+ local host_output="$2"
+
+ info_msg "Pulling output from devcie ${SN}"
+ adb -s "${SN}" pull "${device_output}" "${host_output}"
+}
diff --git a/automated/lib/sh-test-lib b/automated/lib/sh-test-lib
new file mode 100755
index 0000000..15ef4e0
--- /dev/null
+++ b/automated/lib/sh-test-lib
@@ -0,0 +1,121 @@
+#!/bin/sh
+
+LANG=C
+export LANG
+
+error_msg() {
+ local msg="$1"
+ test -z "${msg}" && msg="Unknown error"
+ printf "ERROR: %s\n" "${msg}" >&2
+ exit 1
+}
+
+warn_msg() {
+ local msg="$1"
+ test -z "${msg}" && msg="Unknown error"
+ printf "WARNING: %s\n" "${msg}" >&2
+}
+
+info_msg() {
+ local msg="$1"
+ test -z "${msg}" && msg="Unknown info"
+ printf "INFO: %s\n" "${msg}" >&1
+}
+
+check_return_fail() {
+ if [ $? -ne 0 ]; then
+ fail_test "$1"
+ return 0
+ else
+ return 1
+ fi
+}
+
+fail_test() {
+ local reason="$1"
+ echo "${test}: fail - ${reason}"
+}
+
+pass_test() {
+ echo "${test}: pass"
+}
+
+check_root() {
+ if [ "$(id -ru)" -eq 0 ]; then
+ return 0
+ else
+ return 1
+ fi
+}
+
+check_return() {
+ local exit_code="$?"
+ local test="$1"
+
+ test -z "${test}" && warn_msg "Test name is empty"
+
+ if [ "${exit_code}" -ne 0 ]; then
+ echo "${test} fail" | tee -a "${RESULT_FILE}"
+ return "${exit_code}"
+ else
+ echo "${test} pass" | tee -a "${RESULT_FILE}"
+ return 0
+ fi
+}
+
+add_metric() {
+ local test="$1"
+ local measurement="$2"
+ local units="$3"
+
+ test -z "${test}" && warn_msg "Test name is empty"
+ test -z "${measurement}" && warn_msg "Test measurement is empty"
+ test -z "${units}" && warn_msg "Test units is empty"
+
+ echo "${test} pass ${measurement} ${units}" | tee -a "${RESULT_FILE}"
+}
+
+dist_name() {
+ if [ -x /usr/bin/lsb_release ]; then
+ dist="$(lsb_release -si)"
+ elif [ -f /etc/lsb-release ]; then
+ . /etc/lsb-release
+ dist="${DISTRIB_ID}"
+ elif [ -f /etc/debian_version ]; then
+ dist="Debian"
+ elif [ -f /etc/fedora-release ]; then
+ dist="Fedora"
+ elif [ -f /etc/centos-release ]; then
+ dist="CentOS"
+ else
+ dist="Unknown"
+ warn_msg "Unsupported distro: cannot determine distribution name"
+ fi
+}
+
+install_deps() {
+ local pkgs="$1"
+ local skip_install="$2"
+
+ if [ "${skip_install}" = "True" ] || [ "${skip_install}" = "true" ]; then
+ info_msg "install_deps skipped"
+ else
+ info_msg "Installing ${pkgs}"
+ dist_name
+ case "${dist}" in
+ Debian|Ubuntu)
+ apt-get update
+ apt-get install -y -q ${pkgs}
+ ;;
+ CentOS)
+ yum -e 0 -y install ${pkgs}
+ ;;
+ Fedora)
+ dnf -e 0 -y install ${pkgs}
+ ;;
+ Unknown)
+ warn_msg "Unsupported distro: package install skipped"
+ ;;
+ esac
+ fi
+}
diff --git a/automated/linux/smoke/smoke.sh b/automated/linux/smoke/smoke.sh
new file mode 100755
index 0000000..42d4c90
--- /dev/null
+++ b/automated/linux/smoke/smoke.sh
@@ -0,0 +1,51 @@
+#!/bin/sh
+
+. ../../lib/sh-test-lib
+OUTPUT="$(pwd)/output"
+mkdir -p "${OUTPUT}"
+RESULT_FILE="${OUTPUT}/result.txt"
+export RESULT_FILE
+
+usage() {
+ echo "Usage: $0 [-s <true|false>]" 1>&2
+ exit 1
+}
+
+while getopts "s:" o; do
+ case "$o" in
+ s) SKIP_INSTALL="${OPTARG}" ;;
+ *) usage ;;
+ esac
+done
+
+install() {
+ dist_name
+ case "${dist}" in
+ Debian|Ubuntu) pkgs="lsb-release" ;;
+ Fedora|CentOS) pkgs="redhat-lsb-core" ;;
+ esac
+
+ install_deps "${pkgs}" "${SKIP_INSTALL}"
+}
+
+run() {
+ local test="$1"
+ test_case_id="$(echo "${test}" | awk '{print $1}')"
+ echo
+ info_msg "Running ${test_case_id} test..."
+ eval "${test}"
+ check_return "${test_case_id}"
+}
+
+# Test run.
+[ -f "${OUTPUT}" ] && \
+mv "${OUTPUT}" "${OUTPUT}_$(date +%Y%m%d%H%M%S)"
+
+install
+run "pwd"
+run "lsb_release -a"
+run "uname -a"
+run "ip a"
+run "lscpu"
+run "vmstat"
+run "lsblk"
diff --git a/automated/linux/smoke/smoke.yaml b/automated/linux/smoke/smoke.yaml
new file mode 100644
index 0000000..f20e6db
--- /dev/null
+++ b/automated/linux/smoke/smoke.yaml
@@ -0,0 +1,27 @@
+metadata:
+ format: Lava-Test Test Definition 1.0
+ name: smoke-tests-basic
+ description: "Run basic smoke tests."
+ maintainer:
+ - milosz.wasilewski@linaro.org
+ - chase.qi@linaro.org
+ os:
+ - debian
+ - fedora
+ scope:
+ - functional
+ devices:
+ - panda
+ - panda-es
+ - juno
+ - hi6220-hikey
+ - apq8016-sbc
+
+params:
+ SKIP_INSTALL: "False"
+
+run:
+ steps:
+ - cd ./automated/linux/smoke/
+ - ./smoke.sh -s "${SKIP_INSTALL}"
+ - ../../utils/send-to-lava.sh ./output/result.txt
diff --git a/automated/utils/send-to-lava.sh b/automated/utils/send-to-lava.sh
new file mode 100755
index 0000000..55cbc6d
--- /dev/null
+++ b/automated/utils/send-to-lava.sh
@@ -0,0 +1,31 @@
+#!/bin/sh
+
+RESULT_FILE="$1"
+
+if [ -f "${RESULT_FILE}" ]; then
+ while read line; do
+ if echo "${line}" | egrep -q ".* +(pass|fail|skip)$"; then
+ test="$(echo "${line}" | awk '{print $1}')"
+ result="$(echo "${line}" | awk '{print $2}')"
+
+ if command -v lava-test-case; then
+ lava-test-case "${test}" --result "${result}"
+ else
+ echo "<TEST_CASE_ID=${test} RESULT=${result}>"
+ fi
+ elif echo "${line}" | egrep -q ".* +(pass|fail|skip) +[0-9.E-]+ [A-Za-z./]+$"; then
+ test="$(echo "${line}" | awk '{print $1}')"
+ result="$(echo "${line}" | awk '{print $2}')"
+ measurement="$(echo "${line}" | awk '{print $3}')"
+ units="$(echo "${line}" | awk '{print $4}')"
+
+ if command -v lava-test-case; then
+ lava-test-case "${test}" --result "${result}" --measurement "${measurement}" --units "${units}"
+ else
+ echo "<TEST_CASE_ID=${test} RESULT=${result} UNITS=${units} MEASUREMENT=${measurement}>"
+ fi
+ fi
+ done < "${RESULT_FILE}"
+else
+ echo "WARNING: result file is missing!"
+fi
diff --git a/automated/utils/test-runner.py b/automated/utils/test-runner.py
new file mode 100755
index 0000000..5e58add
--- /dev/null
+++ b/automated/utils/test-runner.py
@@ -0,0 +1,354 @@
+#!/usr/bin/env python
+import argparse
+import csv
+import json
+import logging
+import os
+import pexpect
+import re
+import shutil
+import sys
+import time
+import yaml
+from uuid import uuid4
+
+
+class TestPlan(object):
+ """
+ Analysis args specified, then generate test plan.
+ """
+
+ def __init__(self, args):
+ self.output = args.output
+ self.test_def = args.test_def
+ self.test_plan = args.test_plan
+ self.timeout = args.timeout
+ self.skip_install = args.skip_install
+ self.logger = logging.getLogger('RUNNER.TestPlan')
+
+ def test_list(self):
+ if self.test_def:
+ if not os.path.exists(self.test_def):
+ self.logger.error(' %s NOT found, exiting...' % self.test_def)
+ sys.exit(1)
+
+ test_list = [{'path': self.test_def}]
+ test_list[0]['uuid'] = str(uuid4())
+ test_list[0]['timeout'] = self.timeout
+ test_list[0]['skip_install'] = self.skip_install
+ elif self.test_plan:
+ if not os.path.exists(self.test_plan):
+ self.logger.error(' %s NOT found, exiting...' % self.test_plan)
+ sys.exit(1)
+
+ with open(self.test_plan, 'r') as f:
+ test_plan = yaml.safe_load(f)
+ try:
+ test_list = test_plan['requirements']['tests']['automated']
+ for test in test_list:
+ test['uuid'] = str(uuid4())
+ except KeyError as e:
+ self.logger.error("%s is missing from test plan" % str(e))
+ sys.exit(1)
+ else:
+ self.logger.error('Plese specify a test or test plan.')
+ sys.exit(1)
+
+ return test_list
+
+
+class TestSetup(object):
+ """
+ Create directories required, then copy files needed to these directories.
+ """
+
+ def __init__(self, test, args):
+ self.output = os.path.realpath(args.output)
+ self.test_name = os.path.splitext(test['path'].split('/')[-1])[0]
+ self.uuid = test['uuid']
+ self.test_uuid = self.test_name + '_' + self.uuid
+ self.test_path = os.path.join(self.output, self.test_uuid)
+ self.logger = logging.getLogger('RUNNER.TestSetup')
+
+ def validate_env(self):
+ # Inspect if environment set properly.
+ try:
+ self.repo_path = os.environ['REPO_PATH']
+ except KeyError:
+ self.logger.error('KeyError: REPO_PATH')
+ self.logger.error("Please run '. ./bin/setenv.sh' to setup test environment")
+ sys.exit(1)
+
+ def create_dir(self):
+ if not os.path.exists(self.output):
+ os.makedirs(self.output)
+ self.logger.info('Output directory created: %s' % self.output)
+
+ def copy_test_repo(self):
+ self.validate_env()
+ shutil.rmtree(self.test_path, ignore_errors=True)
+ shutil.copytree(self.repo_path, self.test_path, symlinks=True)
+ self.logger.info('Test repo copied to: %s' % self.test_path)
+
+ def create_uuid_file(self):
+ with open('%s/uuid' % self.test_path, 'w') as f:
+ f.write(self.uuid)
+
+
+class TestDefinition(object):
+ """
+ Convert test definition to testdef.yaml, testdef_metadata and run.sh.
+ """
+
+ def __init__(self, test, args):
+ self.output = os.path.realpath(args.output)
+ self.test_def = test['path']
+ self.test_name = os.path.splitext(self.test_def.split('/')[-1])[0]
+ self.test_uuid = self.test_name + '_' + test['uuid']
+ self.test_path = os.path.join(self.output, self.test_uuid)
+ self.logger = logging.getLogger('RUNNER.TestDef')
+ self.skip_install = args.skip_install
+ if 'skip_install' in test:
+ self.skip_install = test['skip_install']
+ self.custom_params = None
+ if 'parameters' in test:
+ self.custom_params = test['parameters']
+ if 'params' in test:
+ self.custom_params = test['params']
+ with open(self.test_def, 'r') as f:
+ self.testdef = yaml.safe_load(f)
+
+ def definition(self):
+ with open('%s/testdef.yaml' % self.test_path, 'w') as f:
+ f.write(yaml.dump(self.testdef, encoding='utf-8', allow_unicode=True))
+
+ def metadata(self):
+ with open('%s/testdef_metadata' % self.test_path, 'w') as f:
+ f.write(yaml.dump(self.testdef['metadata'], encoding='utf-8', allow_unicode=True))
+
+ def run(self):
+ with open('%s/run.sh' % self.test_path, 'a') as f:
+ f.write('#!/bin/sh\n')
+
+ self.parameters = self.handle_parameters()
+ if self.parameters:
+ for line in self.parameters:
+ f.write(line)
+
+ f.write('set -e\n')
+ f.write('export TESTRUN_ID=%s\n' % self.testdef['metadata']['name'])
+ f.write('cd %s\n' % self.test_path)
+ f.write('UUID=`cat uuid`\n')
+ f.write('echo "<STARTRUN $TESTRUN_ID $UUID>"\n')
+ steps = self.testdef['run'].get('steps', [])
+ if steps:
+ for cmd in steps:
+ if '--cmd' in cmd or '--shell' in cmd:
+ cmd = re.sub(r'\$(\d+)\b', r'\\$\1', cmd)
+ f.write('%s\n' % cmd)
+ f.write('echo "<ENDRUN $TESTRUN_ID $UUID>"\n')
+
+ os.chmod('%s/run.sh' % self.test_path, 0755)
+
+ def handle_parameters(self):
+ ret_val = ['###default parameters from test definition###\n']
+
+ if 'params' in self.testdef:
+ for def_param_name, def_param_value in list(self.testdef['params'].items()):
+ # ?'yaml_line'
+ if def_param_name is 'yaml_line':
+ continue
+ ret_val.append('%s=\'%s\'\n' % (def_param_name, def_param_value))
+ elif 'parameters' in self.testdef:
+ for def_param_name, def_param_value in list(self.testdef['parameters'].items()):
+ if def_param_name is 'yaml_line':
+ continue
+ ret_val.append('%s=\'%s\'\n' % (def_param_name, def_param_value))
+ else:
+ return None
+
+ ret_val.append('######\n')
+
+ ret_val.append('###custom parameters from test plan###\n')
+ if self.custom_params:
+ for param_name, param_value in list(self.custom_params.items()):
+ if param_name is 'yaml_line':
+ continue
+ ret_val.append('%s=\'%s\'\n' % (param_name, param_value))
+
+ if self.skip_install:
+ ret_val.append('SKIP_INSTALL="True"\n')
+ ret_val.append('######\n')
+
+ return ret_val
+
+
+class TestRun(object):
+ def __init__(self, test, args):
+ self.output = os.path.realpath(args.output)
+ self.test_name = os.path.splitext(test['path'].split('/')[-1])[0]
+ self.test_uuid = self.test_name + '_' + test['uuid']
+ self.test_path = os.path.join(self.output, self.test_uuid)
+ self.test_timeout = args.timeout
+ if 'timeout' in test:
+ self.test_timeout = test['timeout']
+ self.logger = logging.getLogger('RUNNER.TestRun')
+ self.logger.info('Executing %s/run.sh' % self.test_path)
+ shell_cmd = '%s/run.sh 2>&1 | tee %s/stdout.log' % (self.test_path, self.test_path)
+ self.child = pexpect.spawn('/bin/sh', ['-c', shell_cmd])
+
+ def check_output(self):
+ if self.test_timeout:
+ self.logger.info('Test timeout: %s' % self.test_timeout)
+ test_end = time.time() + self.test_timeout
+
+ while self.child.isalive():
+ if self.test_timeout and time.time() > test_end:
+ self.logger.warning('%s test timed out, killing test process...' % self.test_uuid)
+ self.child.terminate(force=True)
+ break
+ try:
+ self.child.expect('\r\n')
+ print(self.child.before)
+ except pexpect.TIMEOUT:
+ continue
+ except pexpect.EOF:
+ self.logger.info('%s test finished.\n' % self.test_uuid)
+ break
+
+
+class ResultParser(object):
+ def __init__(self, test, args):
+ self.output = os.path.realpath(args.output)
+ self.test_name = os.path.splitext(test['path'].split('/')[-1])[0]
+ self.test_uuid = self.test_name + '_' + test['uuid']
+ self.result_path = os.path.join(self.output, self.test_uuid)
+ self.metrics = []
+ self.results = {}
+ self.results['test'] = self.test_name
+ self.results['id'] = self.test_uuid
+ self.logger = logging.getLogger('RUNNER.ResultParser')
+
+ def run(self):
+ self.parse_stdout()
+ self.dict_to_json()
+ self.dict_to_csv()
+ self.logger.info('Result files saved to: %s' % self.result_path)
+ print('--- Printing result.csv ---')
+ with open('%s/result.csv' % self.result_path) as f:
+ print(f.read())
+
+ def parse_stdout(self):
+ with open('%s/stdout.log' % self.result_path, 'r') as f:
+ for line in f:
+ if re.match(r'\<(|LAVA_SIGNAL_TESTCASE )TEST_CASE_ID=.*', line):
+ line = line.strip('\n').strip('<>').split(' ')
+ data = {'test_case_id': '',
+ 'result': '',
+ 'measurement': '',
+ 'units': ''}
+
+ for string in line:
+ parts = string.split('=')
+ if len(parts) == 2:
+ key, value = parts
+ key = key.lower()
+ data[key] = value
+
+ self.metrics.append(data.copy())
+
+ self.results['metrics'] = self.metrics
+
+ def dict_to_json(self):
+ with open('%s/result.json' % self.result_path, 'w') as f:
+ json.dump(self.results, f, indent=4)
+
+ def dict_to_csv(self):
+ with open('%s/result.csv' % self.result_path, 'w') as f:
+ fieldnames = ['test_case_id', 'result', 'measurement', 'units']
+ writer = csv.DictWriter(f, fieldnames=fieldnames)
+
+ writer.writeheader()
+ for metric in self.results['metrics']:
+ writer.writerow(metric)
+
+ with open('%s/result.csv' % self.output, 'a') as f:
+ writer = csv.DictWriter(f, fieldnames=fieldnames)
+
+ for metric in self.results['metrics']:
+ writer.writerow(metric)
+
+
+def get_args():
+ parser = argparse.ArgumentParser()
+ parser.add_argument('-o', '--output', default='/root/output', dest='output',
+ help='''
+ specify a directory to store test and result files.
+ Default: /root/output
+ ''')
+ parser.add_argument('-p', '--test_plan', default=None, dest='test_plan',
+ help='''
+ specify an test plan file which has tests and related
+ params listed in yaml format.
+ ''')
+ parser.add_argument('-d', '--test_def', default=None, dest='test_def',
+ help='''
+ base on test definition repo location, specify relative
+ path to the test definition to run.
+ Format example: "ubuntu/smoke-tests-basic.yaml"
+ ''')
+ parser.add_argument('-t', '--timeout', type=int, default=None,
+ dest='timeout', help='Specify test timeout')
+ parser.add_argument('-s', '--skip_install', dest='skip_install',
+ default=False, action='store_true',
+ help='skip install section defined in test definition.')
+ args = parser.parse_args()
+ return args
+
+
+def main():
+ # Setup logger.
+ logger = logging.getLogger('RUNNER')
+ logger.setLevel(logging.DEBUG)
+ ch = logging.StreamHandler()
+ ch.setLevel(logging.DEBUG)
+ formatter = logging.Formatter('%(asctime)s - %(name)s: %(levelname)s: %(message)s')
+ ch.setFormatter(formatter)
+ logger.addHandler(ch)
+
+ if os.geteuid() != 0:
+ logger.error("Sorry, you need to run this as root")
+ sys.exit(1)
+
+ # Generate test plan.
+ args = get_args()
+ test_plan = TestPlan(args)
+ test_list = test_plan.test_list()
+ logger.info('Tests to run:')
+ for test in test_list:
+ print(test)
+
+ # Run tests.
+ for test in test_list:
+ # Create directories and copy files needed.
+ setup = TestSetup(test, args)
+ setup.create_dir()
+ setup.copy_test_repo()
+ setup.create_uuid_file()
+
+ # Convert test definition.
+ test_def = TestDefinition(test, args)
+ test_def.definition()
+ test_def.metadata()
+ test_def.run()
+
+ # Run test.
+ test_run = TestRun(test, args)
+ test_run.check_output()
+
+ # Parse test output, save results in json and csv format.
+ result_parser = ResultParser(test, args)
+ result_parser.run()
+
+if __name__ == "__main__":
+ main()