Add subcommand for running complex scenarios

The primary purpose of this subcommand is to help us reproduce
buildbots, but I think in the long run it will be flexible enough to
accommodate other goals as well (e.g. cross-compilation).

There are some very important bits that are clearly missing at the
moment, but we can work on them incrementally. One of those is that at
the moment we take the path to an LLVM worktree as input - in the future
we should be able to take an SVN revision and setup the LLVM sources as
part of the script. We also want to be able to configure which
subprojects to use for each stage, what flags to use for each stage and
for the test-suite etc (yes, there will be lots of parameters, but that
will become less of a concern when we introduce support for config
files).

In the future, we'll probably want to change the interface. At the
moment, we pass a lot of things separately that maybe don't need all
this flexibility. In this version of the patch, we would pass something
like:
"--source-dir <sourceDir> --stage1-build-dir <buildDir1> \
 --stage2-build-dir <buildDir2> --test-suite <testSuiteDir> \
 --sandbox <sandboxDir> --lnt <lntDir>"
The script then knows that we want 2 stages because we have a
"--stage2-build-dir", and also that we want to run the test-suite
because we have a "--test-suite".

Given the nature of this subcommand and how in the long run we will
probably want it to setup everything on its own, we'd be better off with
a single "--root-dir <rootDir>" argument, and hardcode the path to the
sources as "<rootDir>/llvm", the path to the build directories as
"<rootDir>/stageN" etc. We could also probably assume that the repos are
in "<rootDir>/repos/test-suite" and respectively "<rootDir>/repos/lnt".
We could then have a much simpler interface along the lines of:
"--root-dir <rootDir> --enable-stage2 --enable-test-suite"

For the other subcommands, we prefer to be very specific about
everything, because they're meant to be very flexible and usable from
different contexts (with or without wrapper scripts to help make things
less tedious). This new subcommand is pretty different though, since it
combines more than one functionality. We may even consider making it a
separate script instead of a subcommand of llvm.py, but for the time
being it's easier to prototype in this form.

Also, we may want to change the name :)

Change-Id: Ia6bf1a2f034d5b84572879215fe48f58addbfe2c

2 files changed, 283 insertions, 0 deletions

diff --git a/scripts/llvm.py b/scripts/llvm.py
index 7c3fb85..b8362ec 100644
--- a/scripts/llvm.py
+++ b/scripts/llvm.py
@@ -163,6 +163,71 @@ def run_the_test_suite(args):
     except RuntimeError as exc:
         die("Failed to run the test-suite because:\n{}".format(str(exc)))
 
+
+def build_and_test(args):
+    """
+    Build a set of LLVM subprojects, in one or two stages, with or without a
+    test-suite run.
+    """
+
+    proj = Proj()
+
+    dryRun = args.dry
+    llvmWorktreeRoot = args.sources
+
+    stage1BuildDir = args.stage1
+    stage2BuildDir = args.stage2
+    testSuiteDir = args.test_suite
+    sandboxDir = args.sandbox
+    lntDir = args.lnt
+
+    if dryRun:
+        consumer = CommandPrinter()
+    else:
+        consumer = CommandRunner()
+
+    try:
+        sourceConfig = LLVMSourceConfig(proj, llvmWorktreeRoot, args.dry)
+
+        if not dryRun and not os.path.exists(stage1BuildDir):
+            os.makedirs(stage1BuildDir)
+
+        buildConfig1 = LLVMBuildConfig(sourceConfig, stage1BuildDir, consumer)
+        buildConfig1.cmake([], "Ninja")
+        buildConfig1.build()
+        testedBuildDir = stage1BuildDir
+
+        if stage2BuildDir is not None:
+            if not dryRun and not os.path.exists(stage2BuildDir):
+                os.makedirs(stage2BuildDir)
+
+            buildConfig2 = LLVMBuildConfig(sourceConfig, stage2BuildDir,
+                                           consumer)
+
+            # TODO: Make sure clang is actually built in this config (preferably
+            # before reaching this point)
+            buildConfig2.cmake(
+                [
+                    "-DCMAKE_C_COMPILER={}/bin/clang".format(stage1BuildDir),
+                    "-DCMAKE_CXX_COMPILER={}/bin/clang++".format(stage1BuildDir)],
+                "Ninja")
+            buildConfig2.build()
+            testedBuildDir = stage2BuildDir
+
+        if testSuiteDir is not None:
+            setup_test_suite(consumer, sandboxDir, lntDir)
+
+            # TODO: Make sure clang is actually built in this config (preferably
+            # before reaching this point)
+            lit = os.path.join(testedBuildDir, "bin", "llvm-lit")
+            clang = os.path.join(testedBuildDir, "bin", "clang")
+            run_test_suite(consumer, sandboxDir, testSuiteDir, lit,
+                           ["--cc={}".format(clang)])
+
+    except RuntimeError as exc:
+        die("Failed because:\n{}".format(str(exc)))
+
+
 ##########################################################################
 # Command line parsing                                                   #
 ##########################################################################
@@ -359,6 +424,44 @@ runTestSuite.add_argument(
     help="Print the commands instead of executing them.")
 runTestSuite.set_defaults(run_command=run_the_test_suite)
 
+buildAndTest = subcommands.add_parser(
+    'build-and-test',  # TODO: This really needs a better name...
+    help="Run complex build scenarios with one or two stages of clang and "
+         "optionally a test-suite run. This should be flexible enough to allow "
+         "us to reproduce any buildbot configuration, but it can obviously be "
+         "used for other purposes as well.")
+buildAndTest.set_defaults(run_command=build_and_test)
+buildAndTest.add_argument(
+    '--source-dir',
+    dest='sources',
+    required=True,
+    help="Path to the directory containing the LLVM worktree that we're going "
+         "to build from.")
+buildAndTest.add_argument(
+    '--stage1-build-dir',
+    dest='stage1',
+    required=True,
+    help="Path to the build directory for stage 1.")
+buildAndTest.add_argument(
+    '--stage2-build-dir',
+    dest='stage2',
+    help="Path to the build directory for stage 2.")
+buildAndTest.add_argument(
+    "--test-suite",
+    help="Path to the test-suite repo.")
+buildAndTest.add_argument(
+    "--sandbox",
+    help="Path to the sandbox where the test-suite should be setup.")
+buildAndTest.add_argument(
+    "--lnt",
+    help="Path to the LNT repo.")
+buildAndTest.add_argument(
+    '-n', '--dry-run',
+    dest='dry',
+    action='store_true',
+    default=False,
+    help="Print the commands instead of executing them.")
+
 args = options.parse_args()
 if args.subcommand == "projects" and args.add and not args.repos:
     projs.error(
diff --git a/tests/cli/testbuildandtest.py b/tests/cli/testbuildandtest.py
new file mode 100644
index 0000000..948fe3a
--- /dev/null
+++ b/tests/cli/testbuildandtest.py
@@ -0,0 +1,180 @@
+"""Command line interface tests for llvm.py build-and-test.
+
+Note that although this uses the unittest framework, it does *not* contain unit
+tests.
+
+"""
+import os
+
+from llvmtestcase import LLVMTestCase, require_command_arg, debug
+
+
+class Testllvmbuildandtest(LLVMTestCase):
+
+    @classmethod
+    def llvm_build_and_test(cls, *args, **kwargs):
+        return cls.command_with_defaults("build-and-test", *args, **kwargs)
+
+    def test_default_stage1(self):
+        """
+        Test that we dump the correct commands for a single stage build of LLVM.
+        """
+        sourceDir = "path-to-sources"
+        buildDir = "path-to-stage1"
+
+        output = self.run_with_output(
+            self.llvm_build_and_test(
+                "--dry-run",
+                "--source-dir", sourceDir,
+                "--stage1-build-dir", buildDir))
+
+        commands = output.splitlines()
+
+        self.assertRegex(commands[0],
+                         "{build}\$ cmake -G Ninja .* {sources}".format(
+                             build=buildDir, sources=sourceDir))
+
+        self.assertRegex(commands[1],
+                         "{build}\$ ninja".format(build=buildDir))
+
+    def test_stage1_and_testsuite(self):
+        """
+        Test that we dump the correct commands for a single stage build of LLVM
+        and a run of the test-suite with the resulting compiler.
+        """
+        sourceDir = "path-to-sources"
+        buildDir = "path-to-stage1"
+        testSuiteDir = "path-to-test-suite"
+        sandboxDir = "path-to-sandbox"
+        lntDir = "path-to-lnt"
+
+        output = self.run_with_output(
+            self.llvm_build_and_test(
+                "--dry-run",
+                "--source-dir", sourceDir,
+                "--stage1-build-dir", buildDir,
+                "--test-suite", testSuiteDir,
+                "--sandbox", sandboxDir,
+                "--lnt", lntDir))
+
+        commands = output.splitlines()
+
+        self.assertRegex(commands[0],
+                         "{build}\$ cmake -G Ninja .* {sources}".format(
+                             build=buildDir, sources=sourceDir))
+
+        self.assertRegex(commands[1],
+                         "{build}\$ ninja".format(build=buildDir))
+
+        self.assertRegex(
+            commands[2], ".*\$ virtualenv {sandbox}".format(sandbox=sandboxDir))
+
+        self.assertRegex(
+            commands[3],
+            ".*\$ {sandbox}/bin/python {lnt}/setup.py develop".format(
+                sandbox=sandboxDir,
+                lnt=lntDir))
+
+        self.assertRegex(
+            commands[4],
+            ".*\$ {sandbox}/bin/python {sandbox}/bin/lnt runtest test-suite "
+            "--sandbox={sandbox} --test-suite={testsuite} "
+            "--use-lit={build}/bin/llvm-lit --cc={build}/bin/clang".format(
+                sandbox=sandboxDir, testsuite=testSuiteDir, build=buildDir))
+
+    def test_default_stage2(self):
+        """
+        Test that we dump the correct commands for a 2-stage build of LLVM.
+        """
+        sourceDir = "path-to-sources"
+        buildDir1 = "path-to-stage1"
+        buildDir2 = "path-to-stage2"
+
+        output = self.run_with_output(
+            self.llvm_build_and_test(
+                "--dry-run",
+                "--source-dir", sourceDir,
+                "--stage1-build-dir", buildDir1,
+                "--stage2-build-dir", buildDir2))
+
+        commands = output.splitlines()
+
+        self.assertRegex(
+            commands[0],
+            "{stage1}\$ cmake -G Ninja .* {sources}".format(
+                stage1=buildDir1, sources=sourceDir))
+
+        self.assertRegex(
+            commands[1],
+            "{stage1}\$ ninja".format(stage1=buildDir1))
+
+        self.assertRegex(
+            commands[2], "{stage2}\$ cmake -G Ninja .* "
+            "-DCMAKE_C_COMPILER={stage1}/bin/clang "
+            "-DCMAKE_CXX_COMPILER={stage1}/bin/clang\+\+ {sources}".format(
+                stage1=buildDir1, stage2=buildDir2, sources=sourceDir))
+
+        self.assertRegex(
+            commands[3],
+            "{stage2}\$ ninja".format(stage2=buildDir2))
+
+    def test_stage2_and_testsuite(self):
+        """
+        Test that we dump the correct commands for a 2-stage build of LLVM and a
+        run of the test-suite with the resulting compiler.
+        """
+        sourceDir = "path-to-sources"
+        buildDir1 = "path-to-stage1"
+        buildDir2 = "path-to-stage2"
+        testSuiteDir = "path-to-test-suite"
+        sandboxDir = "path-to-sandbox"
+        lntDir = "path-to-lnt"
+
+        output = self.run_with_output(
+            self.llvm_build_and_test(
+                "--dry-run",
+                "--source-dir", sourceDir,
+                "--stage1-build-dir", buildDir1,
+                "--stage2-build-dir", buildDir2,
+                "--test-suite", testSuiteDir,
+                "--sandbox", sandboxDir,
+                "--lnt", lntDir))
+
+        commands = output.splitlines()
+
+        self.assertRegex(
+            commands[0],
+            "{stage1}\$ cmake -G Ninja .* {sources}".format(
+                stage1=buildDir1,
+                sources=sourceDir))
+
+        self.assertRegex(
+            commands[1],
+            "{stage1}\$ ninja".format(stage1=buildDir1))
+
+        self.assertRegex(
+            commands[2], "{stage2}\$ cmake -G Ninja .* "
+            "-DCMAKE_C_COMPILER={stage1}/bin/clang "
+            "-DCMAKE_CXX_COMPILER={stage1}/bin/clang\+\+ {sources}".format(
+                stage1=buildDir1, stage2=buildDir2, sources=sourceDir))
+
+        self.assertRegex(
+            commands[3],
+            "{stage2}\$ ninja".format(stage2=buildDir2))
+
+        self.assertRegex(
+            commands[4],
+            ".*\$ virtualenv {sandbox}".format(sandbox=sandboxDir))
+
+        self.assertRegex(
+            commands[5],
+            ".*\$ {sandbox}/bin/python {lnt}/setup.py develop".format(
+                sandbox=sandboxDir,
+                lnt=lntDir))
+
+        self.assertRegex(
+            commands[6],
+            ".*\$ {sandbox}/bin/python {sandbox}/bin/lnt runtest test-suite "
+            "--sandbox={sandbox} --test-suite={testsuite} "
+            "--use-lit={build}/bin/llvm-lit --cc={build}/bin/clang".format(
+                sandbox=sandboxDir, testsuite=testSuiteDir, build=buildDir2))