Use git worktree in the LLVM helper scripts

This commit adds a new script, llvm-env, which sets up the environment for
working with LLVM. It looks for an environment variable LLVM_ROOT and tries to
create the following hierarchy:

$LLVM_ROOT
  `- repos
  |    `- llvm
  |    `- clang
  |    `- compiler-rt
  |     [...]
  `- <branch1>
  |    `- llvm
  |    `- build
  |    `- debug
  `- <branch2>
  |    `- llvm
  |    `- build
  |    `- debug
  [...]

The $LLVM_ROOT/repos directory contains all the repositories, as checked out by
llvm-prepare, and will always track master. For other branches,
llvm-env <branch_name> will create a new directory, $LLVM_ROOT/<branch_name>,
and will add an llvm worktree directory there. If -d is passed, it will also
create a debug directory there, otherwise it will create a build directory.
Notice that these 2 can live in parallel, and we can switch between them at any
time by invoking llvm-env. It will set LLVM_SRC and LLVM_BLD accordingly, and
also modify the path to point to the binaries in LLVM_BLD.

The other scripts will now work with the LLVM_SRC and LLVM_BLD set by llvm-env
in the current shell. Because llvm-env controls whether or not we're doing a
debug build, llvm-build will no longer take a -d flag (it will instead look
after a LLVM_DEBUG environment variable, also set by llvm-env). There are
changes in llvm-projs, too, because now it no longer creates links - instead it
creates worktree directories in the corresponding $LLVM_ROOT/<branch>/llvm.
Other scripts have also been updated accordingly.

To make things easier, here are some of the changes that I had to make that are
not particularly important for the review (pretty mechanical stuff):
* Moved function has() from llvm-branch to llvm-common, so I could reuse it
* Because of this, I had to rename the has() function in llvm-projs to
has_link(), which is actually a better name for it anyway
* Disable the checks for LLVM_SRC and LLVM_BLD in llvm-common

Change-Id: I9e02f6d8e0c803e79838845013b81331dffba99c

17 files changed, 609 insertions, 237 deletions

diff --git a/helpers/.llvm-env-add b/helpers/.llvm-env-add
new file mode 100755
index 0000000..c886aab
--- /dev/null
+++ b/helpers/.llvm-env-add
@@ -0,0 +1,23 @@
+#!/usr/bin/env bash
+
+. llvm-common
+
+if [ "$1" = "" -o "$2" = "" -o "$3" = "" ]; then
+  echo "Usage: $0 <branch> <llvm-repo-dir> <llvm-worktree-dir>"
+  exit 1
+fi
+
+branch=$1
+llvm_repo_dir=$2
+llvm_worktree_dir=$3
+
+if [ ! -d $llvm_repo_dir ]; then
+  echo "Couldn't find llvm repo in $llvm_repo_dir"
+  exit 1
+fi
+
+if [ ! -d $llvm_worktree_dir ]; then
+  # First time we're checking out this branch
+  safe_run add_worktree $llvm_repo_dir $llvm_worktree_dir $branch
+fi
+
diff --git a/helpers/.llvm-env-remove b/helpers/.llvm-env-remove
new file mode 100755
index 0000000..fba6deb
--- /dev/null
+++ b/helpers/.llvm-env-remove
@@ -0,0 +1,44 @@
+#!/usr/bin/env bash
+
+# This script removes a worktree environment for LLVM and all other projects:
+# * it removes all the worktrees and corresponding  build directories
+# * it runs git worktree prune in all the repos
+# * it DOES NOT remove the corresponding branch
+
+. llvm-common
+
+if [ "$1" = "" -o "$2" = "" ]; then
+  echo "Usage: $0 <llvm-repo-dir> <llvm-worktree-dir>"
+  exit 1
+fi
+
+llvm_repo_dir=$1
+llvm_worktree_dir=$2
+
+if [ ! -d $llvm_repo_dir ]; then
+  echo "Couldn't find llvm repo in $llvm_repo_dir"
+  exit 1
+fi
+
+if [ ! -d $llvm_worktree_dir ]; then
+  echo "Couldn't find llvm worktree in $llvm_worktree_dir"
+  exit 1
+fi
+
+# Clean up the worktrees (if we remove the LLVM worktree dir, all the other
+# worktrees are removed too, so we only need to prune the repos)
+safe_run remove_worktree $llvm_repo_dir $llvm_worktree_dir
+safe_run remove_worktree $llvm_repo_dir/../clang
+safe_run remove_worktree $llvm_repo_dir/../clang-tools-extra
+safe_run remove_worktree $llvm_repo_dir/../compiler-rt
+safe_run remove_worktree $llvm_repo_dir/../lld
+safe_run remove_worktree $llvm_repo_dir/../libcxx
+safe_run remove_worktree $llvm_repo_dir/../libcxxabi
+safe_run remove_worktree $llvm_repo_dir/../libunwind
+safe_run remove_worktree $llvm_repo_dir/../lldb
+safe_run remove_worktree $llvm_repo_dir/../test-suite
+
+# Clean up build directories too
+env_root_dir=`readlink -m $llvm_worktree_dir/../`
+echo "Removing everything in $env_root_dir"
+safe_run rm -rf $env_root_dir
diff --git a/helpers/README.txt b/helpers/README.txt
index 1875f7a..9577192 100644
--- a/helpers/README.txt
+++ b/helpers/README.txt
@@ -34,28 +34,39 @@ This type of repo is meant for dev boards, benchmarks and stress tests.
 Environment Variables
 ---------------------
 
-These scripts rely on three environment variables:
-
-LLVM_SRC   : The path to the LLVM source directory. All other repos will be
-             checked out relative to it in ($LLVM_SRC/../<project>) and linked
-             to the right place inside $LLVM_SRC. It is recommended that you
-             set it to a place where all source dirs will be, and only those,
-             for example ~/devel/llvm/src/llvm.
-LLVM_BLD   : The path to the (out-of-tree) directory for the release+asserts
-             build. It is recommended that you set it two levels down from the
-             source dirs, so you can have multiple build directories (debug,
-             no-asserts, lld, etc), example ~/devel/llvm/build/llvm.
-LLVM_GITRW : Git-Read-Write mode, which also means using Git-Svn or not.
-             Values can be 'yes' or 'no'.
+In order to use these scripts, you will have to set the following environment
+variables:
+
+LLVM_ROOT    : A path where the scripts can checkout the LLVM & related
+               repositories and create build directories for them
+LLVM_GITRW   : Git-Read-Write mode, which also means using Git-Svn or not.
+               Values can be 'yes' or 'no'.
+LLVM_GITUSER : User for 'origin'.
+LLVM_SVNUSER : User for the upstream SVN repo (only if LLVM_GITRW = 'yes').
+
+Based on these variables, the llvm-env script sets up the environment needed by
+the other helper scripts. You should usually source llvm-env before running any
+of the helper scripts.
 
 Dependencies
 ------------
 
 In order to fully use these scripts, you're going to need to install a few
 dependencies. Since people use different Linux distributions, the scripts
-don't even try to install that automatically. The dependencies are:
-
-Scripts/Build: libjson-perl cmake ninja-build perl python2.7 ccache
-LLVM: libxml2-dev zlib1g-dev libtinfo-dev python-sphinx binutils-gold
-Test-Suite: python-virtualenv bison groff gawk
+don't even try to install that automatically.
+
+The scripts themselves require Git-SVN and a version of Git that can handle
+worktrees (e.g. git 2.5.0, may work with earlier versions too).
+
+The requirements for LLVM and the related subprojects are listed in the upstream
+documentation:
+llvm:        http://llvm.org/docs/GettingStarted.html#requirements
+clang:       http://clang.llvm.org/get_started.html
+compiler-rt: http://compiler-rt.llvm.org/
+lld:         http://lld.llvm.org/getting_started.html
+lldb:        http://lldb.llvm.org/build.html
+libcxx:      http://libcxx.llvm.org/docs/BuildingLibcxx.html
+libcxxabi:   http://libcxxabi.llvm.org/
+test-suite:  http://llvm.org/docs/TestingGuide.html#requirements
+lnt:         http://llvm.org/docs/lnt/intro.html?highlight=requirements
 
diff --git a/helpers/git-pull b/helpers/git-pull
index 3dbd3e1..6db6029 100755
--- a/helpers/git-pull
+++ b/helpers/git-pull
@@ -4,7 +4,7 @@
 # not to keep the prefix. If the branch already exists, try to merge. If not,
 # check out from origin.
 #
-# Syntax: git-pull linaro-local/[branch] -f
+# Syntax: git-pull linaro-local/[branch]
 
 . llvm-common
 
@@ -12,10 +12,6 @@ branch=$1
 if [[ $branch = '' ]]; then
   branch=`get_branch`
 fi
-force=$2
-if [[ $force != '-f' ]]; then
-  force=
-fi
 
 if [[ $branch = 'master' ]]; then
   echo "Can't pull the master branch."
@@ -37,17 +33,9 @@ fi
 
 # If the branch exists already, merge
 if git branch | grep -q $branch; then
-  if [[ $force = '-f' ]]; then
-    echo " + Force check out $branch"
-    safe_run git checkout master
-    safe_run git branch -D $branch
-    safe_run git checkout -b $branch origin/$prefix$branch
-    exit 0
-  fi
-
   echo " + Merging the origin branch..."
   if not git merge --ff-only origin/$prefix$branch; then
-    echo "Error merging the branch. Use -f to force."
+    echo "Error merging the branch."
     exit 1
   fi
 # If not, just check out the branch
diff --git a/helpers/git-rebase b/helpers/git-rebase
new file mode 100755
index 0000000..114afbe
--- /dev/null
+++ b/helpers/git-rebase
@@ -0,0 +1,34 @@
+#!/usr/bin/env bash
+
+# This script rebases the current branch on the current master. It does not
+# pull upstream master beforehand. If you need a fresh master, use git-refresh.
+#
+# Syntax: git-rebase
+
+. llvm-common
+
+branch=`get_branch`
+echo "Rebasing $branch"
+
+if [[ `git diff | head -1` != '' ]]; then
+  echo "You have uncommitted changes in your repo, bailing"
+  echo "Please, stash your changes and run this script again"
+  exit 2
+fi
+
+if [[ `git branch | grep $branch` = '' ]]; then
+  echo "Branch $branch doesn't exist in this repository"
+  exit 3
+fi
+
+if [[ "$branch" = "master" ]]; then
+  echo "Can't rebase master"
+  exit 4
+fi
+
+git rebase master
+if [[ $? != 0 ]]; then
+  echo "Rebase failed, aborting rebase..."
+  safe_run git rebase --abort
+  exit 1
+fi
diff --git a/helpers/git-rebase-all b/helpers/git-rebase-all
deleted file mode 100755
index 6b09246..0000000
--- a/helpers/git-rebase-all
+++ /dev/null
@@ -1,49 +0,0 @@
-#!/usr/bin/env bash
-
-# This script rebases all branches based on the current master. It does not
-# pull upstream master beforehand. If you need a fresh master, use git-refresh.
-# If you want to rebase only one branch, use the first argument for that.
-# To protect broken branches from being merged (and forcing a manual merge),
-# rename it to <name>-disabled and it'll be skipped.
-#
-# Syntax: git-rebase-all [branch (default=all)]
-
-. llvm-common
-
-echo "Rebasing all branches"
-branch=`get_branch`
-
-if [[ `git diff | head -1` != '' ]]; then
-  echo "You have uncommitted changes in your repo, bailing"
-  echo "Please, stash your changes and run this script again"
-  exit 2
-fi
-
-if [[ $1 != '' ]]; then
-  if [[ `git branch | grep $1` = '' ]]; then
-    echo "Branch '$1' doesn't exist in this repository"
-    exit 3
-  fi
-  branch=$1
-fi
-
-branches=`get_branches`
-for br in $branches; do
-  if [[ $br != 'master' && `echo $br | grep "^disabled"` == '' ]]; then
-    safe_run git checkout $br
-    git rebase master
-    if [[ $? != 0 ]]; then
-      echo "Rebase failed, aborting rebase..."
-      safe_run git rebase --abort
-      safe_run git checkout master
-      exit 1
-    fi
-    safe_run git checkout master
-  fi
-done
-
-final_branch=`get_branch`
-if [[ $branch != $final_branch ]]; then
-  echo "Back to original branch"
-  safe_run git checkout $branch
-fi
diff --git a/helpers/git-refresh b/helpers/git-refresh
index d27fea5..100654f 100755
--- a/helpers/git-refresh
+++ b/helpers/git-refresh
@@ -9,6 +9,11 @@
 . llvm-common
 
 # Update from origin to upstream
+master_dir=`git rev-parse --git-common-dir`
+
+pushdq $master_dir/../
+trap popdq EXIT
+
 safe_run git checkout master
 echo " + Fetching origin..."
 safe_run git fetch origin
diff --git a/helpers/git-svn-commit b/helpers/git-svn-commit
index 49cb0d3..f27bb77 100755
--- a/helpers/git-svn-commit
+++ b/helpers/git-svn-commit
@@ -50,11 +50,11 @@ if [[ $? != 0 ]]; then
   else
     echo "Not in base dir, won't try anything. But if you"
     echo "saw the message 'resource out of date', remove"
-    echo "the .git/svn directory and trying again."
+    echo "the .git/svn directory and try again."
     exit 1
   fi
 fi
 
 # Refresh / rebase to avoid future conflicts
 safe_run git-refresh
-safe_run git-rebase-all
+safe_run git-rebase
diff --git a/helpers/llvm-branch b/helpers/llvm-branch
index 98a9504..5343914 100755
--- a/helpers/llvm-branch
+++ b/helpers/llvm-branch
@@ -1,18 +1,24 @@
 #!/usr/bin/env bash
 
 # This script helps you check which branches are checked out on git and make
-# sure they're coherent with the feature you're developing. For instance,
-# if a feature spans across both LLVM and Clang, you could name the branch on
-# each repo the same and use this script to seamlessly move between them.
+# sure all repos linked in the current worktree are on the same branch (either
+# by checking it out, or by creating it if it doesn't exist). The branch must
+# exist in at least one of the repos. If it doesn't you can force its creation
+# with -b.
 
-# Syntax: llvm-branch [-d] [branch]
-# -d: delete chosen branch
-# branch: find on llvm and clang and check it out
-# Without arguments, the script lists all branches and highlight the current one
+# Syntax: llvm-branch [-d|-b] [branch]
+#   -d: delete chosen branch
+#   -b: create branch in all repos
+#   branch: checkout or create branch on all repos linked in the current tree
+# Without arguments, the script lists all branches and highlights the current one
 
 . llvm-common
 
+safe_run verify_env
+
 function print_branches() {
+  # TODO: put a star or something next to branches that exist in other
+  # worktrees? (can be found by iterating through LLVM_ROOT)
   local current=$1
   shift
   local space=0
@@ -29,22 +35,6 @@ function print_branches() {
   done
 }
 
-function has() {
-  if [[ $1 = '' || $2 = '' ]]; then
-    echo no
-    return
-  fi
-  local item=$1
-  shift
-  for each in $*; do
-    if [[ $item = $each ]]; then
-      echo yes
-      return
-    fi
-  done
-  echo no
-}
-
 function switch() {
   SRC=$1
   branch=$2
@@ -53,15 +43,17 @@ function switch() {
     cd $SRC
     pwd
     if [[ $in = 'yes' ]]; then
-      if [[ $DEL = '' ]]; then
+      if [[ $DELETE = '' ]]; then
+        echo " + Checking out $branch in $SRC"
         safe_run git checkout $branch
       else
-        safe_run git checkout master
-        safe_run git branch $DEL $branch
+        echo " + Deleting $branch in $SRC"
+        safe_run git branch $DELETE $branch
       fi
     else
-      if [[ $DEL = '' ]]; then
-        safe_run git checkout master
+      if [[ $DELETE = '' ]]; then
+        echo " + Creating $branch in $SRC"
+        safe_run git checkout -b $branch
       fi
     fi
   fi
@@ -129,9 +121,21 @@ if [ -d $LLDB_SRC ]; then
 fi
 
 # Delete chosen branch
-DEL=''
+DELETE=''
 if [[ $1 = '-d' || $1 = '-D' ]]; then
-  DEL=$1
+  DELETE=$1
+  shift
+fi
+
+# Force creation
+CREATE=''
+if [[ $1 = '-b' ]]; then
+  if [[ $DELETE != '' ]]; then
+    echo "Can't create and delete branch at the same time"
+    exit 1
+  fi
+
+  CREATE=$1
   shift
 fi
 
@@ -186,12 +190,50 @@ fi
 # Search for branch name
 branch=$1
 
-if [[ $DEL = 1 && $branch = 'master' ]]; then
-  echo "Cannot delete the master branch"
-  exit 1
+if [[ $DELETE != '' ]]; then
+  if [[ $branch = 'master' ]]; then
+    echo "Cannot delete the master branch"
+    exit 1
+  fi
+  if [[ $branch = $llvm_branch ]]; then
+    echo "Cannot delete $branch - it is checked out in llvm"
+    exit 2
+  fi
+  if [[ $branch = $clang_branch ]]; then
+    echo "Cannot delete $branch - it is checked out in clang"
+    exit 2
+  fi
+  if [[ $branch = $clang_extra_branch ]]; then
+    echo "Cannot delete $branch - it is checked out in clang tools extra"
+    exit 2
+  fi
+  if [[ $branch = $rt_branch ]]; then
+    echo "Cannot delete $branch - it is checked out in compiler rt"
+    exit 2
+  fi
+  if [[ $branch = $cxx_branch ]]; then
+    echo "Cannot delete $branch - it is checked out in libcxx"
+    exit 2
+  fi
+  if [[ $branch = $cxxabi_branch ]]; then
+    echo "Cannot delete $branch - it is checked out in libcxxabi"
+    exit 2
+  fi
+  if [[ $branch = $unw_branch ]]; then
+    echo "Cannot delete $branch - it is checked out in libunwind"
+    exit 2
+  fi
+  if [[ $branch = $lld_branch ]]; then
+    echo "Cannot delete $branch - it is checked out in lld"
+    exit 2
+  fi
+  if [[ $branch = $lldb_branch ]]; then
+    echo "Cannot delete $branch - it is checked out in lldb"
+    exit 2
+  fi
 fi
 
-# Check which projects the branch is
+# Check which projects have the branch
 in_llvm=`has $branch $llvm_branches`
 in_clang=`has $branch $clang_branches`
 in_clang_extra=`has $branch $clang_extra_branches`
@@ -201,11 +243,14 @@ in_cxxabi=`has $branch $cxxabi_branches`
 in_unw=`has $branch $unw_branches`
 in_lld=`has $branch $lld_branches`
 in_lldb=`has $branch $lldb_branches`
-if [[ $in_clang = 'no' && $in_clang_extra && $in_llvm = 'no' && \
-      $in_rt = 'no' && $in_cxx = 'no' && $in_cxxabi = 'no' && \
-      $in_unw = 'no' && $in_lld = 'no' && $in_lldb = 'no' ]]; then
-  echo "Branch $branch doesn't exist on any repository"
-  exit 1
+if [[ $CREATE = '' ]]; then
+  if [[ $in_clang = 'no' && $in_clang_extra && $in_llvm = 'no' && \
+        $in_rt = 'no' && $in_cxx = 'no' && $in_cxxabi = 'no' && \
+        $in_unw = 'no' && $in_lld = 'no' && $in_lldb = 'no' ]]; then
+    echo "Branch $branch doesn't exist on any repository"
+    echo "Force its creation with -b"
+    exit 1
+  fi
 fi
 
 # DO IT
diff --git a/helpers/llvm-build b/helpers/llvm-build
index 938caa5..b674fa9 100755
--- a/helpers/llvm-build
+++ b/helpers/llvm-build
@@ -8,6 +8,8 @@
 
 . llvm-common
 
+safe_run verify_env
+
 ## CMD line options and defaults
 CPUs=`grep -c proc /proc/cpuinfo`
 build_dir=$LLVM_BLD
@@ -16,13 +18,12 @@ build_type=Release
 shared=
 targets=
 prog=`basename $0`
-syntax="Syntax: $prog [-u(pdate)] [-c(check-all)] [-i(nstall)] [-d(debug)]"
+syntax="Syntax: $prog [-u(pdate)] [-c(check-all)] [-i(nstall)]"
 update=false
 check=false
 master=false
-debug=false
 inst=false
-while getopts "ucimd" opt; do
+while getopts "ucim" opt; do
   case $opt in
     u)
       update=true
@@ -36,10 +37,6 @@ while getopts "ucimd" opt; do
     m)
       master=true
       ;;
-    d)
-      debug=true
-      build_dir=$LLVM_BLD/../debug
-      ;;
     *)
       echo $syntax
       exit 1
@@ -68,7 +65,7 @@ if which ninja 2>&1 > /dev/null; then
 fi
 
 ## Debug mode, make it lighter
-if $debug; then
+if [ "$LLVM_DEBUG" = true ]; then
   build_type=Debug
   shared=-DBUILD_SHARED_LIBS=True
   targets=-DLLVM_TARGETS_TO_BUILD="ARM;X86;AArch64"
diff --git a/helpers/llvm-common b/helpers/llvm-common
index 47b4937..33a9129 100755
--- a/helpers/llvm-common
+++ b/helpers/llvm-common
@@ -2,16 +2,17 @@
 
 # The common script is only meant to be included from other LLVM helper scripts
 
-if [[ $LLVM_SRC = '' ]]; then
-  echo "Please, define \$LLVM_SRC"
-  exit -1
-fi
-if [[ $LLVM_BLD = '' ]]; then
-  echo "Please, define \$LLVM_BLD"
+# Verify the compulsory environment variables - these must be set for all the
+# helper scripts.
+if [[ $LLVM_ROOT = '' ]]; then
+  echo "Please, define \$LLVM_ROOT to point to the root"
+  echo "path where the worktree setup should be performed"
   exit -1
 fi
+
 if [[ $LLVM_GITRW = '' ]]; then
-  echo "Please, define \$LLVM_GITRW"
+  echo "Please, define \$LLVM_GITRW to yes if you"
+  echo "want to set up Git-SVN and no otherwise"
   exit -1
 fi
 if [[ $LLVM_GITUSER = '' ]]; then
@@ -24,6 +25,36 @@ if [[ $LLVM_GITRW = 'yes' ]] && [[ $LLVM_SVNUSER = '' ]]; then
   exit 1
 fi
 
+# Verify the environment variables that should be set by llvm-env
+verify_env() {
+  if [[ $LLVM_SRC = '' ]]; then
+    echo "Please, define \$LLVM_SRC to point to the current LLVM"
+    echo "worktree directory, or run llvm-env to set it for you"
+    exit -1
+  fi
+  if [[ $LLVM_BLD = '' ]]; then
+    echo "Please, define \$LLVM_BLD to point to the current LLVM"
+    echo "build directory, or run llvm-env to set it for you"
+    exit -1
+  fi
+}
+
+has() {
+  if [[ $1 = '' || $2 = '' ]]; then
+    echo no
+    return
+  fi
+  local item=$1
+  shift
+  for each in $*; do
+    if [[ $item = $each ]]; then
+      echo yes
+      return
+    fi
+  done
+  echo no
+}
+
 get_branch() {
   branch=`git rev-parse --abbrev-ref HEAD`
   if [[ $? != 0 ]]; then
@@ -44,6 +75,13 @@ get_branches() {
   echo $branches
 }
 
+has_branch() {
+  branch=$1
+  branches=`get_branches`
+  result=`has $branch $branches`
+  echo $result
+}
+
 safe_run() {
   "$@"
   if [[ $? != 0 ]]; then
@@ -59,3 +97,45 @@ is_git() {
 is_git_svn() {
   test -f `git rev-parse --show-toplevel`/.git/svn/.metadata
 }
+
+# Quiet pushd & popd
+pushdq() {
+  pushd "$@" > /dev/null
+}
+
+popdq() {
+  popd "$@" > /dev/null
+}
+
+add_worktree() {
+  repo_dir=$1
+  worktree_dir=$2
+  branch=$3
+
+  pushdq $repo_dir
+
+  if [ `get_branch` != 'master' ]; then
+    echo "$repo_dir isn't on master, bailing out"
+    exit 1
+  fi
+
+  if [ `has_branch $branch` = "yes" ]; then
+    safe_run git worktree add $worktree_dir $branch
+  else
+    safe_run git worktree add -b $branch $worktree_dir
+  fi
+  popdq
+}
+
+remove_worktree() {
+  repo_dir=$1
+
+  if [ "$2" != "" ]; then
+    worktree_dir=$2
+    safe_run rm -rf $worktree_dir
+  fi
+
+  pushdq $repo_dir
+  safe_run git worktree prune
+  popdq
+}
diff --git a/helpers/llvm-env b/helpers/llvm-env
new file mode 100755
index 0000000..d90c767
--- /dev/null
+++ b/helpers/llvm-env
@@ -0,0 +1,130 @@
+#!/usr/bin/env bash
+
+# This script manages the environment needed by the helper scripts
+# It needs to have one environment variable set in order to do its job:
+# LLVM_ROOT - the directory under which the llvm sources and build directories
+# will live
+
+# This script is meant to be sourced, so we don't want things to exit on error,
+# because that would kill the whole shell. Instead, we trap and return from the
+# script (which leaves the shell running, so people can see what went wrong).
+clean_exit="trap '' ERR; return"
+
+trap "$clean_exit" ERR
+
+. llvm-common
+
+# Check that we have actually been sourced
+if [ -z "$PS1" ]; then
+  echo "This script must be sourced. You might want to create an alias llvm-env=\". llvm-env\""
+  exit 1
+fi
+
+if [ "$1" = "" ]; then
+  if [ -z "$LLVM_SRC" ]; then
+    echo "You haven't set up a worktree"
+    eval $clean_exit
+  fi
+
+  worktree=`basename $(readlink -m $LLVM_SRC/../)`
+  echo $worktree
+  eval $clean_exit
+elif [ "$1" = "-h" ]; then
+  echo "Usage: $0 <branch> [-cleanup] [-d]"
+  echo "  <branch> : the name of the branch you want to set the env for;"
+  echo "             it will be created if it does not already exist."
+  echo "  -cleanup : will clean up the environment corresponding to <branch>;"
+  echo "             this includes the worktrees for all the subprojects, as"
+  echo "             well as any directories in that environment (especially the"
+  echo "             build dirs!)"
+  echo "             it will not clean up any branches (you should use"
+  echo "             llvm-branch for that)"
+  echo "  -d : build the debug version."
+  eval $clean_exit
+fi
+
+branch=$1
+shift
+
+if [[ $branch = -* ]]; then
+  echo "Invalid branch name $branch"
+  eval $clean_exit
+fi
+
+if [ "$branch" = "master" ]; then
+  echo "Building master directly is not supported."
+  echo "You can instead run 'llvm-env TrackMaster'"
+  echo "to create an environment that tracks master."
+  eval $clean_exit
+fi
+
+clean=false
+if [ "$1" = "-cleanup" ]; then
+  clean=true
+  shift
+fi
+
+debug_build=false
+OPTIND=0
+while getopts "d" opt; do
+  if [ "$opt" = "d" ]; then
+    debug_build=true
+  else
+    eval $clean_exit
+  fi
+  shift
+done
+
+if [ "$1" != "" ]; then
+  echo "Too many args?"
+  eval $clean_exit
+fi
+
+llvm_dir=$LLVM_ROOT/repos/llvm
+llvm_worktree_dir=$LLVM_ROOT/$branch/llvm
+
+llvm_build_dir=$LLVM_ROOT/$branch/build
+if $debug_build; then
+  llvm_build_dir=$LLVM_ROOT/$branch/debug
+fi
+
+if [ "$clean" = true ]; then
+  .llvm-env-remove $llvm_dir $llvm_worktree_dir
+else
+  .llvm-env-add $branch $llvm_dir $llvm_worktree_dir
+fi
+
+# Changes to the environment should be confined to the end of the script, to
+# make sure we don't change half the environment and then fail out
+
+if [ "$clean" = true ]; then
+  # Clean up the environment to make sure the other scripts error out
+  # accordingly instead of trying anything stupid
+  export LLVM_SRC=
+  export LLVM_BLD=
+  export LLVM_DEBUG=
+  if [ ! -z "$LLVM_OLD_PATH" ]; then
+    export PATH=$LLVM_OLD_PATH
+  fi
+  eval $clean_exit
+fi
+
+export LLVM_SRC=$llvm_worktree_dir
+export LLVM_BLD=$llvm_build_dir
+export LLVM_DEBUG=$debug_build # For llvm-build to know
+
+# Make it possible to undo changes to the PATH: we export an LLVM_OLD_PATH, and
+# instead of appending the binary dir to $PATH, we append to $LLVM_OLD_PATH (if
+# it exists)
+# This is intended to support scenarios where you want to switch between a
+# release and debug build of the same branch in the same shell, without growing
+# a huge PATH
+path_to_add_to=$PATH
+if [ ! -z "$LLVM_OLD_PATH" ]; then
+  path_to_add_to=$LLVM_OLD_PATH
+fi
+
+export LLVM_OLD_PATH=$path_to_add_to
+export PATH=$llvm_build_dir/bin:$path_to_add_to
+
+eval $clean_exit
diff --git a/helpers/llvm-prepare b/helpers/llvm-prepare
index 46a1780..d9f8a2a 100755
--- a/helpers/llvm-prepare
+++ b/helpers/llvm-prepare
@@ -34,43 +34,43 @@ else
   svnroot=https://$LLVM_SVNUSER@llvm.org/svn/llvm-project
 fi
 
+llvm_repos=$LLVM_ROOT/repos
+
 echo " + Setting up LLVM"
-prepare $LLVM_SRC $gitroot/llvm $svnroot/llvm/trunk
+prepare $llvm_repos/llvm $gitroot/llvm $svnroot/llvm/trunk
 
 echo " + Setting up Clang"
-prepare $LLVM_SRC/../clang $gitroot/clang $svnroot/cfe/trunk
+prepare $llvm_repos/clang $gitroot/clang $svnroot/cfe/trunk
 
 echo " + Setting up Clang Tools Extra"
-prepare $LLVM_SRC/../clang-tools-extra \
+prepare $llvm_repos/clang-tools-extra \
         $gitroot/clang-tools-extra $svnroot/clang-tools-extra/trunk
 
 echo " + Setting up Compiler-RT"
-prepare $LLVM_SRC/../compiler-rt $gitroot/compiler-rt $svnroot/compiler-rt/trunk
+prepare $llvm_repos/compiler-rt $gitroot/compiler-rt $svnroot/compiler-rt/trunk
 
 echo " + Setting up LLD Linker"
-prepare $LLVM_SRC/../lld $gitroot/lld $svnroot/lld/trunk
+prepare $llvm_repos/lld $gitroot/lld $svnroot/lld/trunk
 
 echo " + Setting up LibC++"
-prepare $LLVM_SRC/../libcxx $gitroot/libcxx $svnroot/libcxx/trunk
+prepare $llvm_repos/libcxx $gitroot/libcxx $svnroot/libcxx/trunk
 
 echo " + Setting up LibC++ABI"
-prepare $LLVM_SRC/../libcxxabi $gitroot/libcxxabi $svnroot/libcxxabi/trunk
+prepare $llvm_repos/libcxxabi $gitroot/libcxxabi $svnroot/libcxxabi/trunk
 
 echo " + Setting up LibUnwind"
-prepare $LLVM_SRC/../libunwind $gitroot/libunwind $svnroot/libunwind/trunk
+prepare $llvm_repos/libunwind $gitroot/libunwind $svnroot/libunwind/trunk
 
 echo " + Setting up LLDB Debugger"
-prepare $LLVM_SRC/../lldb $gitroot/lldb $svnroot/lldb/trunk
+prepare $llvm_repos/lldb $gitroot/lldb $svnroot/lldb/trunk
 
 echo " + Setting up Test Suite"
-prepare $LLVM_SRC/../test-suite $gitroot/test-suite $svnroot/test-suite/trunk
+prepare $llvm_repos/test-suite $gitroot/test-suite $svnroot/test-suite/trunk
 
 echo " + Setting up LNT Test harness"
-prepare $LLVM_SRC/../lnt $gitroot/lnt $svnroot/lnt/trunk
+prepare $llvm_repos/lnt $gitroot/lnt $svnroot/lnt/trunk
 
 echo " + Setting up Zorg Buildbot Config"
-prepare $LLVM_SRC/../zorg $gitroot/zorg $svnroot/zorg/trunk
-
-mkdir -p $LLVM_BLD
+prepare $llvm_repos/zorg $gitroot/zorg $svnroot/zorg/trunk
 
 echo " + Done"
diff --git a/helpers/llvm-projs b/helpers/llvm-projs
index f9cc706..3429b89 100755
--- a/helpers/llvm-projs
+++ b/helpers/llvm-projs
@@ -6,6 +6,8 @@
 
 . llvm-common
 
+safe_run verify_env
+
 prog=`basename $0`
 syntax() {
   echo "Syntax: $prog [clang|lldb|lld|rt|libs|all|none] {+/-}[c|x|r|k|d|l|a|u|t]"
@@ -51,18 +53,18 @@ tests_dir=test-suite
 tests_link=projects/test-suite
 
 # Check if link exists
-has() {
+has_link() {
   link=$1
-  [ -s "$LLVM_SRC/$link" ]
+  [ -d "$LLVM_SRC/$link" ]
 }
 
 # Initialise status
 init() {
   link=$1
-  if has $link; then
-    echo "true";
+  if has_link $link; then
+    echo "ON";
   else
-    echo "false"
+    echo "OFF"
   fi
 }
 
@@ -71,25 +73,42 @@ update() {
   dir=$1
   link=$2
   need=$3
-  if $need; then
-    ln -sf $LLVM_SRC/../$dir $LLVM_SRC/$link
+  if [ "$need" = ON ]; then
+    if ! has_link $link; then
+      pushdq $LLVM_SRC
+      branch=`get_branch`
+      popdq
+
+      safe_run add_worktree $LLVM_ROOT/repos/$dir $LLVM_SRC/$link $branch
+    fi
   else
-    rm -f $LLVM_SRC/$link
+    safe_run remove_worktree $LLVM_ROOT/repos/$dir $LLVM_SRC/$link
+  fi
+}
+
+# Enable/disable projects in the CMake cache
+update_build_dirs() {
+  if [ -d $LLVM_BLD/../build ]; then
+    safe_run cmake $* $LLVM_BLD/../build
+  fi
+
+  if [ -d $LLVM_BLD/../debug ]; then
+    safe_run cmake $* $LLVM_BLD/../debug
   fi
 }
 
-# Lists linked projects and quit
+# Lists linked projects
 list_all() {
   echo "Projects linked:"
-  has $clang_link && echo " + Clang"
-  has $clang_extra_link && echo " + Clang Tools Extra"
-  has $rt_link && echo " + Compiler-RT"
-  has $libcxx_link && echo " + LibC++"
-  has $libcxxabi_link && echo " + LibC++abi"
-  has $libunwind_link && echo " + LibUnwind"
-  has $lld_link && echo " + LLD"
-  has $lldb_link && echo " + LLDB"
-  has $tests_link && echo " + Test-Suite"
+  has_link $clang_link && echo " + Clang"
+  has_link $clang_extra_link && echo " + Clang Tools Extra"
+  has_link $rt_link && echo " + Compiler-RT"
+  has_link $libcxx_link && echo " + LibC++"
+  has_link $libcxxabi_link && echo " + LibC++abi"
+  has_link $libunwind_link && echo " + LibUnwind"
+  has_link $lld_link && echo " + LLD"
+  has_link $lldb_link && echo " + LLDB"
+  has_link $tests_link && echo " + Test-Suite"
   echo
 }
 
@@ -129,43 +148,43 @@ tests=`init $tests_link`
 opt=$1
 case $opt in
   clang)
-    need_all false
-    clang=true
-    clang_extra=true
+    need_all OFF
+    clang=ON
+    clang_extra=ON
     shift
     ;;
   lldb)
-    need_all false
-    clang=true
-    lldb=true
+    need_all OFF
+    clang=ON
+    lldb=ON
     shift
     ;;
   lld)
-    need_all false
-    clang=true
-    lld=true
+    need_all OFF
+    clang=ON
+    lld=ON
     shift
     ;;
   rt)
-    need_all false
-    clang=true
-    rt=true
+    need_all OFF
+    clang=ON
+    rt=ON
     shift
     ;;
   libs)
-    need_all false
-    clang=true
-    libcxx=true
-    libcxxabi=true
-    libunwind=true
+    need_all OFF
+    clang=ON
+    libcxx=ON
+    libcxxabi=ON
+    libunwind=ON
     shift
     ;;
   all)
-    need_all true
+    need_all ON
     shift
     ;;
   none)
-    need_all false
+    need_all OFF
     shift
     ;;
   list)
@@ -183,9 +202,9 @@ while ! test -z $1; do
   opt=$1
 
   sign=${opt:0:1}
-  flag=true
+  flag=ON
   if [ "$sign" = "-" ]; then
-    flag=false
+    flag=OFF
     opt=${opt:1}
   elif [ "$sign" = "+" ]; then
     opt=${opt:1}
@@ -233,7 +252,7 @@ done
 # clang and clang-tools-extra have a special relationship: we can't enable
 # clang-tools-extra without enabling clang, and we also can't disable clang
 # without also disabling clang-tools-extra
-if [ "$clang_extra" = true -a "$clang" = false ]; then
+if [ "$clang_extra" = ON -a "$clang" = OFF ]; then
   echo "Can't have Clang Tools Extra without Clang! Try to add +c or -x"
   exit
 fi
@@ -249,4 +268,12 @@ update $libcxx_dir $libcxx_link $libcxx
 update $rt_dir $rt_link $rt
 update $clang_dir $clang_link $clang
 update $clang_extra_dir $clang_extra_link $clang_extra
+update_build_dirs -DLLVM_TOOL_LLDB_BUILD=$lldb \
+  -DLLVM_TOOL_LLD_BUILD=$lld \
+  -DLLVM_TOOL_LIBUNWIND_BUILD=$libunwind \
+  -DLLVM_TOOL_LIBCXXABI_BUILD=$libcxxabi \
+  -DLLVM_TOOL_LIBCXX_BUILD=$libcxx \
+  -DLLVM_TOOL_COMPILER_RT_BUILD=$rt \
+  -DLLVM_TOOL_CLANG_BUILD=$clang \
+  -DLLVM_TOOL_CLANG_TOOLS_EXTRA_BUILD=$clang_extra
 list_all
diff --git a/helpers/llvm-reset b/helpers/llvm-reset
index 3c88eef..37a1b00 100755
--- a/helpers/llvm-reset
+++ b/helpers/llvm-reset
@@ -15,15 +15,17 @@ reset_git_svn() {
   fi
 }
 
-reset_git_svn LLVM $LLVM_SRC
-reset_git_svn Clang $LLVM_SRC/../clang
-reset_git_svn ClangToolsExtra $LLVM_SRC/../clang-tools-extra
-reset_git_svn Compiler-RT $LLVM_SRC/../compiler-rt
-reset_git_svn Libc++ $LLVM_SRC/../libcxx
-reset_git_svn Libc++abi $LLVM_SRC/../libcxxabi
-reset_git_svn LibUnwind $LLVM_SRC/../libunwind
-reset_git_svn Linker $LLVM_SRC/../lld
-reset_git_svn Debugger $LLVM_SRC/../lldb
-reset_git_svn LNT $LLVM_SRC/../lnt
-reset_git_svn Zorg $LLVM_SRC/../zorg
-reset_git_svn Test-Suite $LLVM_SRC/../test-suite
+llvm_repos=$LLVM_ROOT/repos
+
+reset_git_svn LLVM $llvm_repos/llvm
+reset_git_svn Clang $llvm_repos/clang
+reset_git_svn ClangToolsExtra $llvm_repos/clang-tools-extra
+reset_git_svn Compiler-RT $llvm_repos/compiler-rt
+reset_git_svn Libc++ $llvm_repos/libcxx
+reset_git_svn Libc++abi $llvm_repos/libcxxabi
+reset_git_svn LibUnwind $llvm_repos/libunwind
+reset_git_svn Linker $llvm_repos/lld
+reset_git_svn Debugger $llvm_repos/lldb
+reset_git_svn LNT $llvm_repos/lnt
+reset_git_svn Zorg $llvm_repos/zorg
+reset_git_svn Test-Suite $llvm_repos/test-suite
diff --git a/helpers/llvm-runlit b/helpers/llvm-runlit
index 32143fb..9843174 100755
--- a/helpers/llvm-runlit
+++ b/helpers/llvm-runlit
@@ -2,6 +2,10 @@
 
 # Helper script for running a single LIT test inside the build tree.
 
+. llvm-common
+
+safe_run verify_env
+
 if [[ $1 = '' ]]; then
   echo "Syntax: $0 test-dir"
   exit -1
diff --git a/helpers/llvm-sync b/helpers/llvm-sync
index 0374221..9784006 100755
--- a/helpers/llvm-sync
+++ b/helpers/llvm-sync
@@ -1,21 +1,39 @@
 #!/usr/bin/env bash
 
-# This script checks out all repositories from upstream and pushes master
-# to the origin repo on Linaro's Git server, rebasing all local branches,
-# but *not* pushing them, too.
+# This script pulls all repositories selected with llvm-projs and rebases the
+# current branch on top of the new master.
 
 . llvm-common
 
+safe_run verify_env
+
 function repo_sync () {
-  if [ ! -d $2 ]; then
-    echo " + Not updating $1 (couldn't find source directory)"
+  project=$1
+  repo_dir=$2
+  worktree_dir=$3
+
+  if [ ! -d $repo_dir ]; then
+    echo " + Not updating $project (couldn't find source directory)"
+    return;
+  fi
+
+  if [ ! -d $worktree_dir ]; then
+    echo " + Not updating $project (couldn't find worktree directory)"
     return;
   fi
-  echo " + Updating $1"
-  cd $2
-  branch=`get_branch`
+
+  echo " + Updating $project"
+
+  # To pull master, we must be in the repo dir (we shouldn't checkout master in
+  # the worktree dirs - by default, master lives in the repo dir)
+  cd $repo_dir
   safe_run git-refresh
-  safe_run git-rebase-all $branch
+
+  # To rebase the current branch, we must be in the worktree dir (we shouldn't
+  # checkout the current branch in the repo dir, because it is already checked
+  # out in the worktree dir)
+  cd $worktree_dir
+  safe_run git-rebase
 }
 
 prog=`basename $0`
@@ -29,15 +47,26 @@ linker=false
 debug=false
 tests=false
 web=false
-if [ -d $LLVM_SRC/tools/clang ]; then clang=true; fi
-if [ -d $LLVM_SRC/tools/clang/tools/extra ]; then cextra=true; fi
-if [ -d $LLVM_SRC/projects/compiler-rt ]; then rt=true; fi
-if [ -d $LLVM_SRC/projects/libcxx ]; then libcxx=true; fi
-if [ -d $LLVM_SRC/projects/libcxxabi ]; then libcxxabi=true; fi
-if [ -d $LLVM_SRC/projects/libunwind ]; then libunwind=true; fi
-if [ -d $LLVM_SRC/tools/lld ]; then linker=true; fi
-if [ -d $LLVM_SRC/tools/lldb ]; then debug=true; fi
-if [ -d $LLVM_SRC/projects/test-suite ]; then tests=true; fi
+
+clang_workdir=$LLVM_SRC/tools/clang
+extra_workdir=$LLVM_SRC/tools/clang/tools/extra
+rt_workdir=$LLVM_SRC/projects/compiler-rt
+libcxx_workdir=$LLVM_SRC/projects/libcxx
+libcxxabi_workdir=$LLVM_SRC/projects/libcxxabi
+libunwind_workdir=$LLVM_SRC/projects/libunwind
+lld_workdir=$LLVM_SRC/tools/lld
+lldb_workdir=$LLVM_SRC/tools/lldb
+test_workdir=$LLVM_SRC/projects/test-suite
+
+if [ -d $clang_workdir ]; then clang=true; fi
+if [ -d $extra_workdir ]; then cextra=true; fi
+if [ -d $rt_workdir ]; then rt=true; fi
+if [ -d $libcxx_workdir ]; then libcxx=true; fi
+if [ -d $libcxxabi_workdir ]; then libcxxabi=true; fi
+if [ -d $libunwind_workdir ]; then libunwind=true; fi
+if [ -d $lld_workdir ]; then linker=true; fi
+if [ -d $lldb_workdir ]; then debug=true; fi
+if [ -d $test_workdir ]; then tests=true; fi
 
 while getopts "wa" opt; do
   case $opt in
@@ -65,59 +94,61 @@ while getopts "wa" opt; do
   esac
 done
 
+llvm_repos=$LLVM_ROOT/repos
+
 # Compulsory updates
-repo_sync LLVM $LLVM_SRC
+repo_sync LLVM $llvm_repos/llvm $LLVM_SRC
 
 if $clang; then
-  repo_sync Clang $LLVM_SRC/../clang
+  repo_sync Clang $llvm_repos/clang $clang_workdir
 fi
 
 # Optional updates
 if $cextra; then
-  repo_sync ClangToolsExtra $LLVM_SRC/../clang-tools-extra
+  repo_sync ClangToolsExtra $llvm_repos/clang-tools-extra $extra_workdir
 fi
 
 if $rt; then
-  repo_sync RT $LLVM_SRC/../compiler-rt
+  repo_sync RT $llvm_repos/compiler-rt $rt_workdir
 fi
 
 if $libcxx; then
-  repo_sync LibC++ $LLVM_SRC/../libcxx
+  repo_sync LibC++ $llvm_repos/libcxx $libcxx_workdir
 fi
 
 if $libcxxabi; then
-  repo_sync LibC++ABI $LLVM_SRC/../libcxxabi
+  repo_sync LibC++ABI $llvm_repos/libcxxabi $libcxxabi_workdir
 fi
 
 if $libunwind; then
-  repo_sync LibUnwind $LLVM_SRC/../libunwind
+  repo_sync LibUnwind $llvm_repos/libunwind $libunwind_workdir
 fi
 
 if $linker; then
-  repo_sync Linker $LLVM_SRC/../lld
+  repo_sync Linker $llvm_repos/lld $lld_workdir
 fi
 
 if $debug; then
-  repo_sync Debugger $LLVM_SRC/../lldb
+  repo_sync Debugger $llvm_repos/lldb $lldb_workdir
 fi
 
 if $tests; then
-  repo_sync Test-Suite $LLVM_SRC/../test-suite
-  repo_sync LNT $LLVM_SRC/../lnt
-  repo_sync Zorg $LLVM_SRC/../zorg
+  repo_sync Test-Suite $llvm_repos/test-suite $test_workdir
+  repo_sync LNT $llvm_repos/lnt $llvm_repos/lnt
+  repo_sync Zorg $llvm_repos/zorg $llvm_repos/zorg
 fi
 
 # These are pure SVN repos, not enabled with -a
 # You have to manually force with -w
 if $web; then
-  if [ -d $LLVM_SRC/../www ]; then
+  if [ -d $llvm_repos/www ]; then
     echo " + Updating WWW"
-    cd $LLVM_SRC/../www
+    cd $llvm_repos/www $llvm_repos/www
     svn up
   fi
-  if [ -d $LLVM_SRC/../pubs ]; then
+  if [ -d $llvm_repos/pubs ]; then
     echo " + Updating Pubs"
-    cd $LLVM_SRC/../pubs
+    cd $llvm_repos/pubs $llvm_repos/pubs
     svn up
   fi
 fi