[llvmprojs] Rewrite llvm-projs in python

This is the first step in moving all our scripts to python. For now we keep the
current content of the repo, but we'll start adding a new directory hierarchy
for the python stuff:

  * modules: should contain most of the python code, organized as a package that
             can be imported by the scripts
  * scripts: the scripts themselves, which should be parsing the command line
             and calling stuff from 'modules' to do the actual work; can be
             broken down into the same categories we had before (helpers, bisect
             etc), or we could just have one big pile
  * tests: should contain unittests (for the stuff in modules) and command line
           interface tests (for the scripts)

The code is heavily using functionality from the tcwg-release-tools repo (which
probably needs to be renamed / reorganized), so you should have that in your
PYTHONPATH when trying to run any of the scripts. To run the tests, just invoke
check.sh.

One of the important changes is that we'll be using python's argparse module to
parse command line flags, which means we'll have to stick to a more traditional
interface for the scripts. In particular, we can't have short options like "+c"
anymore. This isn't much of a problem, because we will keep the bash scripts as
they are and just make them invoke a tool written in python (scripts/llvm.py) to
do the work. The tool will have subcommands for any functionality that we want,
for instance the new interface for adding/removing subprojects is:

llvm.py projects [-a subproject subproject ... subproject]
                 [-r subproject ... subproject]

The -a and -r options (followed by any number of subprojects) can be used to
selectively enable/disable things. You have to pass the full name of the
subproject (e.g. llvmprojs.py -a clang lld -r libcxx) for it to work. This is
invoked by the llvm-projs bash script, which has the old, more convenient
interface.

For now the bash scripts will live in the same directories as they used to, but
after the transition is complete we will want to move them to the scripts
directory.

Note that we're also eliding any dashes in the names of the scripts, in keeping
with Python best practices for module names (i.e. using valid Python identifiers
as names).

Change-Id: I9ec08632dbb17464673240d6f6881a90f45d5371

2 files changed, 224 insertions, 0 deletions

diff --git a/modules/__init__.py b/modules/__init__.py
new file mode 100644
index 0000000..e69de29
--- /dev/null
+++ b/modules/__init__.py
diff --git a/modules/llvm.py b/modules/llvm.py
new file mode 100644
index 0000000..8922c83
--- /dev/null
+++ b/modules/llvm.py
@@ -0,0 +1,224 @@
+import os
+
+from linaropy.git.worktree import Worktree
+
+
+class LLVMSubproject(object):
+    """
+    Class that keeps track of everything related to an LLVM subproject (clang,
+    lld, compiler-rt etc): repo URL, location preferred by CMake,
+    CMake variable for adding or removing it from the build etc.
+    """
+
+    def __init__(self, cmake_path, cmake_var):
+        """Create an LLVMSubproject with the provided info.
+
+        Parameters
+        ----------
+        cmake_path
+            Path relative to the LLVM source root where this subproject should
+            live so that CMake can automatically pick it up during the build.
+        cmake_var
+            The name of the CMake variable that can be used to enable/disable
+            building this subproject.
+        """
+        self.cmake_path = cmake_path
+        self.cmake_var = cmake_var
+
+    def get_cmake_path(self, llvm_source_directory):
+        """
+        Get the path where this subproject should live in the given LLVM tree so
+        that CMake can pick it up by default.
+        """
+        return os.path.join(llvm_source_directory, self.cmake_path)
+
+    @classmethod
+    def get_all_subprojects(cls):
+        """
+        Return a dictionary of all the LLVM subprojects.
+        At the moment, llvm itself is not part of the subprojects, because it
+        always needs to be there and everything is relative to it.
+        """
+        return {
+            "clang":
+                cls(os.path.join("tools", "clang"), "LLVM_TOOL_CLANG_BUILD"),
+            "compiler-rt":
+                cls(os.path.join("projects", "compiler-rt"),
+                    "LLVM_TOOL_COMPILER_RT_BUILD"),
+            "libcxx":
+                cls(os.path.join("projects", "libcxx"),
+                    "LLVM_TOOL_LIBCXX_BUILD"),
+            "libcxxabi":
+                cls(os.path.join("projects", "libcxxabi"),
+                    "LLVM_TOOL_LIBCXXABI_BUILD"),
+            "libunwind":
+                cls(os.path.join("projects", "libunwind"),
+                    "LLVM_TOOL_LIBUNWIND_BUILD"),
+            "lld":
+                cls(os.path.join("tools", "lld"), "LLVM_TOOL_LLD_BUILD"),
+            "lldb":
+                cls(os.path.join("tools", "lldb"), "LLVM_TOOL_LLDB_BUILD"),
+            "test-suite":
+                cls(os.path.join("projects", "test-suite"), None),
+        }
+
+
+class LLVMSourceConfig(object):
+    """Class for managing an LLVM source tree.
+
+    It keeps track of which subprojects are enabled in a given tree and provides
+    functionality for adding / removing them.
+    """
+
+    def __init__(self, proj, sourcePath,
+                 subprojs=LLVMSubproject.get_all_subprojects()):
+        """ Create a source configuration.
+
+        Parameters
+        ----------
+        proj : Proj
+            Temporary project directory (used mostly for logging).
+        sourcePath
+            Must point to a valid LLVM source tree.
+        subprojs : dictionary
+            Dictionary containing a number of LLVMSubproject objects.
+            By default, this contains all the LLVM subprojects as returned by
+            LLVMSubproject.get_all_subprojects(), but any subset would work just
+            as well.
+            The keys will be used to identify the subprojects in any of this
+            class's methods. It is an error to invoke them with a subproj that
+            does not exist in this dictionary.
+        """
+        sourcePath = str(sourcePath)
+        if not os.path.isdir(sourcePath):
+            raise EnvironmentError("Invalid path to LLVM source tree")
+
+        self.proj = proj
+        self.llvmSourceTree = Worktree(proj, sourcePath)
+        self.subprojs = subprojs
+
+    def get_enabled_subprojects(self):
+        """Get a list of the subprojects enabled in this configuration."""
+        enabled = []
+        for subproj in self.subprojs:
+            if self.__is_enabled(subproj):
+                enabled.append(subproj)
+        return enabled
+
+    def update(self, add={}, remove=[]):
+        """Update the configuration by adding/removing subprojects.
+
+        Parameters
+        ----------
+        add : dictionary
+            A map of (subproject name, repo) to add to the config. The order in
+            which the subprojects are added is unspecified. Subprojects that are
+            already enabled in the config are ignored.
+        remove: list
+            A list of subproject names to remove from the config. The order in
+            which the subprojects are removed is unspecified. Duplicates and
+            subprojects that aren't enabled in the config are ignored. A
+            subproject may be removed even if it is in an invalid state.
+
+        For both add and remove, the subproject name must exist in the
+        dictionary that the config was initialized with and the repo must be a
+        valid GitRepo object.
+
+        TODO: If adding/removing a project fails, this function should try to
+        recover.
+        """
+
+        # Make the inputs friendly
+        if add is None:
+            add = {}
+
+        if remove is None:
+            remove = []
+
+        for subproj in add.keys():
+            if subproj in remove:
+                raise ValueError("Can't add and remove {} at the same time"
+                                 .format(subproj))
+
+        for (subproj, repo) in add.items():
+            self.__add_subproject(subproj, repo)
+
+        for subproj in remove:
+            self.__remove_subproject(subproj)
+
+    def __get_subproj_object(self, subprojName):
+        """Get the LLVMSubproject object corresponding to subprojName."""
+        if not subprojName in self.subprojs.keys():
+            raise ValueError("Unknown llvm subproject {0}".format(subprojName))
+        return self.subprojs[subprojName]
+
+    def __get_subproj_cmake_path(self, subprojName):
+        """Get the full path to subprojName in this source tree."""
+        subproj = self.__get_subproj_object(subprojName)
+        return subproj.get_cmake_path(self.llvmSourceTree.repodir)
+
+    def __is_enabled(self, subprojName):
+        """
+        Check if subproj is enabled in this configuration. A subproj is
+        considered to be enabled if it is a worktree on the correct branch.
+        """
+        try:
+            # If this succeeds, the subproject has already been added.
+            existing = Worktree(self.proj,
+                                self.__get_subproj_cmake_path(subprojName))
+        except EnvironmentError:
+            # If it's not a worktree (for whatever reason), it's not enabled.
+            return False
+
+        # If it is a worktree, but on the wrong branch, it is not enabled.
+        return existing.getbranch() == self.llvmSourceTree.getbranch()
+
+    # TODO: add_subproject, remove_subproject and is_enabled should live in
+    # another object (AddRemoveStrategy?) that would know what we want to add
+    # (worktrees, links, whatever)
+    def __add_subproject(self, subprojName, subprojRepo):
+        """Add a given subproject to this configuration.
+
+        This will make sure the subproject's sources are visible in the proper
+        place in the LLVM source tree that the configuration was created with.
+
+        We currently achieve this by creating a worktree for the subproject, but
+        if more flexibility is needed we can add a config option. The branch
+        that the worktree will be created with is the same branch that the
+        existing LLVM source tree is on, and it will be tracking a branch
+        corresponding to the one that the LLVM branch was forked from.
+        """
+        path = self.__get_subproj_cmake_path(subprojName)
+
+        if self.__is_enabled(subprojName):
+            # Subproject has already been added, nothing to do.
+            return
+
+        if os.path.exists(path):
+            raise EnvironmentError(
+                "{} already exists but is not a valid subproject directory."
+                    .format(path) +
+                "Please make sure it is a worktree on the same branch as LLVM.")
+
+        # Create a new worktree
+        branch = self.llvmSourceTree.getbranch()
+        if subprojRepo.branchexists(branch):
+            Worktree.create(self.proj, subprojRepo, path, branch)
+        else:
+            trackedBranch = "master"  # TODO: track proper branch
+            Worktree.create(
+                self.proj,
+                subprojRepo,
+                path,
+                branch,
+                trackedBranch)
+
+    def __remove_subproject(self, subprojName):
+        """Remove a given subproject from this build configuration."""
+        path = self.__get_subproj_cmake_path(subprojName)
+
+        if not os.path.isdir(path):
+            return
+
+        worktree = Worktree(self.proj, path)
+        worktree.clean(False)