[test-suite] Introduce a new CMake+LIT based harness

This introduces a replacement for the Makefile based build system. This replacement build system uses CMake for configuring and compiling the tests, and LIT for running them. LIT will report both compile and execution times as metrics, available by using the '-o' option to output JSON or '-v' (without '-s').

The advantages of this setup are:
  * Use of the same technologies used in LLVM itself means there's more expertise available to debug, fix and extend, and fewer disparate command line flags to remember.
  * A common-off-the-shelf solution means there's much less custom code to maintain, only glue.
  * More readable output by default, more debuggable by far.
  * The ability to run just one test, quickly and easily.
  * The ability to perform test repeats without cleaning and recompiling first.
  * Because the command line is simple, LNT is not required to run a test.

The disadvantages are:
  * Not all of the features in the Makefile system are supported or are going to be. For example, llvm-gcc support, compiling with opt & llc. Clang is stable and expected now - we don't need these workarounds.

This commit introduces the CMake and LIT glue and a python script that can parse all the Makefiles in SingleSource & MultiSource and can generate equivalent CMakeLists.txt. The intention is that this is run once and the resulting files checked in then never used again.

With this script run, the entire test-suite builds and runs. Example command lines:

  $ mkdir build; cd build
  $ cmake -DCMAKE_C_COMPILER=/path/to/my/clang .. # CMake finds clang++ and introspects the target OS/arch/endianness
  $ make -j12
  $ llvm-lit -sv .

Ninja also works, but the way the ninja makefiles are generated means there is no way to build all tests in an arbitrary subtree (as LNT's --only-test does). With ninja you can build all tests or a single test.

All feedback welcome!

git-svn-id: https://llvm.org/svn/llvm-project/test-suite/trunk@251431 91177308-0d34-0410-b5e6-96231b3b80d8

1 files changed, 46 insertions, 0 deletions

diff --git a/lit.cfg b/lit.cfg
new file mode 100644
index 00000000..11809d97
--- /dev/null
+++ b/lit.cfg
@@ -0,0 +1,46 @@
+import lit.formats
+import lit
+import os, glob, re
+
+def getUserTimeFromTimeOutput(f):
+    with open(f) as fd:
+        l = [l for l in fd.readlines()
+             if l.startswith('user')]
+    assert len(l) == 1
+        
+    m = re.match(r'user\s+([0-9.]+)', l[0])
+    return float(m.group(1))
+
+class TestSuiteTest(lit.formats.ShTest):
+    def __init__(self):
+        lit.formats.ShTest.__init__(self, False)
+        
+    def execute(self, test, litConfig):
+        result = lit.formats.ShTest.execute(self, test, litConfig)
+        basepath = os.path.dirname(test.getFilePath())
+        
+        if not result.code.isFailure:
+            # Collect the timing information.
+            timeglob = os.path.join(basepath, 'Output', '*.time')
+            times = glob.glob(timeglob)
+            assert len(times) == 1
+            time = getUserTimeFromTimeOutput(times[0])
+
+            result.addMetric('exec_time', lit.Test.toMetricValue(time))
+
+        # For completeness, attempt to find compile time information too.
+        compile_time = 0.0
+        for path, subdirs, files in os.walk(basepath):
+            for file in files:
+                if file.endswith('.o.time'):
+                    compile_time += getUserTimeFromTimeOutput(os.path.join(path, file))
+        result.addMetric('compile_time', lit.Test.toMetricValue(compile_time))
+        
+        return result
+
+config.name = 'test-suite'
+
+config.test_format = TestSuiteTest()
+config.suffixes = ['.test']
+config.test_source_root = os.path.dirname(__file__)
+config.excludes = ['ABI-Testsuite']