|Age||Commit message (Collapse)||Author|
Both the test-suite and stage 2 need clang, so check early that it is
built as part of stage 1.
I'm not 100% certain on the former, since nowadays the test-suite
contains some bitcode tests which might be able to run without clang,
but it's unlikely that we'll ever be interested in such a scenario.
Use the built-in support from the argparse module to allow our
subcommand to read flags from configuration files.
This is achieved by specifying a prefix (in our case '@') that indicates
to argparse that it should read arguments from the file ('@filepath'),
one per line.
They are then processed as if they had been given on the command line in
the position where the filename is. This makes it possible for flags in
the file to override or be overriden by flags given on the command line,
depending on their position relative to the filename. Flags that
accumulate all the values that they receive cannot be overriden, but
will instead accumulate everything regardless of whether it came from
the command line or the file.
Allow the user to pass flags to LNT when running the test-suite.
Same possibilities as stage 1: subprojects, CMake definitions and build
Add support for custom subprojects, custom CMake definitions and custom
build flags/targets for the first stage of the build. The subprojects
are preserved for stage 2, but the other flags aren't. They will be
We're still hardcoding the CMake generator, but we can fix that later.
This also extracts a couple of helpers for processing the command line
arguments, which we can use both for this subcommand and for the others.
This is necessary so we can add subprojects to our source config. It
assumes that each subproject has a repo directly under the repos-dir.
Since we now have a path to *all* the repos, we can use that to get the
test-suite and LNT repos as well, instead of specifying each of them
manually. This simplifies the interface a bit (we get rid of the
parameters for passing the paths to the test-suite and LNT repo, and
instead add a single parameter for enabling the test-suite).
The primary purpose of this subcommand is to help us reproduce
buildbots, but I think in the long run it will be flexible enough to
accommodate other goals as well (e.g. cross-compilation).
There are some very important bits that are clearly missing at the
moment, but we can work on them incrementally. One of those is that at
the moment we take the path to an LLVM worktree as input - in the future
we should be able to take an SVN revision and setup the LLVM sources as
part of the script. We also want to be able to configure which
subprojects to use for each stage, what flags to use for each stage and
for the test-suite etc (yes, there will be lots of parameters, but that
will become less of a concern when we introduce support for config
In the future, we'll probably want to change the interface. At the
moment, we pass a lot of things separately that maybe don't need all
this flexibility. In this version of the patch, we would pass something
"--source-dir <sourceDir> --stage1-build-dir <buildDir1> \
--stage2-build-dir <buildDir2> --test-suite <testSuiteDir> \
--sandbox <sandboxDir> --lnt <lntDir>"
The script then knows that we want 2 stages because we have a
"--stage2-build-dir", and also that we want to run the test-suite
because we have a "--test-suite".
Given the nature of this subcommand and how in the long run we will
probably want it to setup everything on its own, we'd be better off with
a single "--root-dir <rootDir>" argument, and hardcode the path to the
sources as "<rootDir>/llvm", the path to the build directories as
"<rootDir>/stageN" etc. We could also probably assume that the repos are
in "<rootDir>/repos/test-suite" and respectively "<rootDir>/repos/lnt".
We could then have a much simpler interface along the lines of:
"--root-dir <rootDir> --enable-stage2 --enable-test-suite"
For the other subcommands, we prefer to be very specific about
everything, because they're meant to be very flexible and usable from
different contexts (with or without wrapper scripts to help make things
less tedious). This new subcommand is pretty different though, since it
combines more than one functionality. We may even consider making it a
separate script instead of a subcommand of llvm.py, but for the time
being it's easier to prototype in this form.
Also, we may want to change the name :)
This makes it possible to know what build tool to use in dry run mode
when the build directory has not been configured yet.
Make it possible to use LLVMSourceConfig in dry run mode. This is useful
because otherwise we can't have a dry run on *any* scripts that use an
LLVMSourceConfig, since it might throw errors that wouldn't occur in
This is different from non-dry run in the following ways:
* we do not perform any validation of the source directory (since in dry
run mode it is reasonable to expect that the directory doesn't even
* we do not try to add or remove any subprojects on disk
Instead, we do try to keep track of our updates (which projects are
added and which are removed). We don't however look at what may exist on
disk since that increases the complexity of the code significantly and
it is very likely that scripts will create a source directory from
scratch rather than use one where subprojects have already been added.
We may want to fix that after a bit of refactoring.
Add a subcommand 'setup-test-suite' which sets up a sandbox that can be
used for running the test-suite.
Add a llvm.py subcommand 'run-test-suite' which runs the test-suite in a
given sandbox. This uses the 'test-suite' producer (as opposed to the
old 'nt' one) since this is what the buildbots use these days.
We don't update the helpers in the 'test-suite' directory because they
are obsolete (and using the 'nt' producer) and hopefully nobody has been
using them lately. Instead we add a really simple helper
'llvm-test-suite' which runs the test-suite with the compiler from
$LLVM_BLD. We keep this helper simple for the time being, but in the
future we may want to add more features here (e.g. a benchmarking mode,
core/number of threads detection etc).
We don't currently setup the sandbox. That will be a different commit.
Get rid of some of the duplication from the command line interface tests
by extracting a decorator that checks that the proper exception is
thrown when we run a command without one of its required arguments.
Update comment to describe what happens if one of the parameters is
This makes it possible for us to see large diffs between the actual and
expected values when debugging a test.
Add a llvm.py subcommand for building LLVM in a given build directory.
The build directory must have already been configured with CMake. The
build command that is run will be either a 'ninja' command or a 'make'
command, depending on whether the build directory contains a
'build.ninja' or a 'Makefile'. If it contains neither of those, an error
I initially wanted to put the implementation for this in the
LLVMBuildConfig, but it doesn't really need anything from there. For now
I left it as a free-standing function. It doesn't have any knowledge of
LLVM whatsoever, so an argument could be made for moving it into the
"utils" package rather than the "llvm" package. In the future, we may
want to remove the LLVMBuildConfig class and see if there's a better way
to organize the code for configuring and building.
One disadvantage with using python here is that we lose the nice
progress bars that we could see when running 'ninja check' from the bash
scripts. I'm not sure how to fix that, suggestions welcome.
When launching a command, we now collect its output during execution and
print it as soon as it becomes available. This is necessary in order to
be able to see progress in long-running commands such as CMake or Ninja.
This requires some changes to the way in which we use the Subprocess
module, since we need to use the Popen constructor. This does not throw
an exception if the launched process returns an error - we now need to
manually check the return code and throw an exception ourselves.
Remove the --env argument, which was compulsory for all subcommands, and
replace it with a --source-dir argument, only for those subcommands that
We do this for 2 reasons:
* Not all subcommands care about the source directory (e.g. llvm.py
build won't need it)
* llvm.py should not have any knowledge about environments - that
concept only makes sense for the helper scripts. llvm.py should
instead receive very specific info about where the source and build
Forgot to git add this file when adding support for llvm.py configure.
Move some helpers that aren't directly dealing with LLVM knowledge from
modules/llvm.py to modules/utils.py.
Add a subcommand that runs CMake in a given build directory, with a
custom generator and custom CMake definitions. Also update llvm-build to
use it instead of calling CMake directly, and add calls to it in
llvm-projs as well to make sure we update the build directories whenever
we enable or disable a project.
One known issue with the current code is that the output of the CMake
command is not printed out live, but rather after the command has
finished execution. This is going to be more important for llvm.py build
than for llvm.py configure, so we can fix it in a future commit.
Add a method that runs an action on every subproject known by the source
config. The action will be passed the subproject and a boolean
indicating whether or not it is enabled.
The --repos flag is not used by all the subcommands, so there's no
reason for it to be compulsory. This commit moves it from the generic
llvm.py options to the options for the projects subcommand, where it is
only required when adding subprojects.
* Remove all references to clang-tools-extra in helpers
We do not support clang tools extra in the python scripts, since it's
too complicated and nobody is working on it yet. Therefore, clean up all
references to it in the helpers as well, to avoid any confusion.
* Add a bit more info to llvm.py --help
Get llvm.py push to print the name of the remote branch that it has
pushed to. This is helpful so developers don't need to guess what
namespaces have been prepended to the branch name, and also serves as
quick visual confirmation.
This required a bit of refactoring, since the name of the branch was
computed when needed. We now compute it separately and pass it in to the
function that pushes the branch. We rely on the fact that the branch
name will be the same for all the subprojects involved.
Most of the changes are mechanical, to account for this refactoring
(including new names for some of the existing functions, which make more
sense in this new context).
git-push is uninteresting and nobody uses it. Instead, we could use a
wrapper over llvm.py push, which pushes the current branch for all
Add support for pushing the current branch in all linked subprojects in
a given config. If they are not on the same branch, an error is thrown.
The branches are pushed as linaro-local/$user/$branch, where $branch is
the local name of the branch and $user is the user as seen in
LLVM_GITUSER (we might want to change to a more intelligent way to get
the user in the future).
We achieve this by adding a new method to LLVMSourceConfig:
for_each_enabled, which takes an action and applies it to all the
enabled subprojects. If an exception is thrown by the action, it is
rethrown by the action as a RuntimeError specifying which project the
action was being applied to. Note that this does not behave like a
transaction - the action may have been applied to some of the projects
already, and there's no way to undo it. It's also a bit difficult to
tell which projects the action was applied on (in the future we should
probably log that info).
We also provide an action that simply pushes the current branch to
origin with the updated name. This makes sure the remote branch will
live in an isolated namespace where it won't conflict with other
We also update some of the internals of LLVMSourceConfig to throw
exceptions if some of the subprojects that are linked are not worktrees
on the same branch as LLVM. In the past they were just ignored, but that
doesn't seem like a sane default anymore.
Create an LLVMTestCase class that will contain common helpers for
writing tests for llvm.py subcommands.
Avoid reviews being accidentally submitted on the wrong branch
by setting the default branch in the .gitreview file.
Before this patch, we had an LLVM_ROOT directory which contained the
repos and a series of environments (worktrees + build dicretories).
However, there is no good reason to have such a rigid layout, and in
fact it prevents us from sharing the repos in any meaningful way.
This patch removes LLVM_ROOT and instead introduces two command line
parameters, repos and env, which represent the path to the directory
containing the LLVM repositories and respectively the path to the
environment that we intend to modify. The two paths may be completely
independent. A side effect of this is that we can now also have
different environments in different, unrelated locations (whereas
previously they had to be siblings in LLVM_ROOT).
For the time being, both parameters are compulsory (not only for
llvm-projects, but for all subcommands). There may be situations where
we can get away with only one of them (e.g. `llvm-projs <env>`, which
just lists the projects enabled in the given <env> - it does not need to
know where the repos are in order to do this). There may also be
situations where one of them doesn't make any sense at all (e.g.
`llvm-prepare`, which is not implemented yet in python, only needs the
path to the repos and it would not know what to do with an environment).
Making them optional would require some additional validation logic as
well as more tests, and it feels a bit early on to introduce all that.
It would be nice to have at least one or 2 more subcommands implemented
before we invest time in it.
Add an extra parameter to llvm.py representing the environment that we
want the command to refer to. We then compute the path to the LLVM
source tree based on that and LLVM_ROOT (which is still an environment
variable for the time being).
The bash helper keeps its old interface and uses LLVM_SRC to compute the
environment that it will pass down to llvm.py. We also perform a small
number of drive-by fixes to this helper (e.g. replacing python with
Most of the changes have been made by 2to3. Some manual fiddling was
needed to make sure we're calling python3 instead of just python in the
tests. Two of the helpers in the tests were reworked into proper methods
rather than partials because we had to convert their returned value to
str and this seems like the most straightforward/easy-to-read way.
This is the first step in moving all our scripts to python. For now we keep the
current content of the repo, but we'll start adding a new directory hierarchy
for the python stuff:
* modules: should contain most of the python code, organized as a package that
can be imported by the scripts
* scripts: the scripts themselves, which should be parsing the command line
and calling stuff from 'modules' to do the actual work; can be
broken down into the same categories we had before (helpers, bisect
etc), or we could just have one big pile
* tests: should contain unittests (for the stuff in modules) and command line
interface tests (for the scripts)
The code is heavily using functionality from the tcwg-release-tools repo (which
probably needs to be renamed / reorganized), so you should have that in your
PYTHONPATH when trying to run any of the scripts. To run the tests, just invoke
One of the important changes is that we'll be using python's argparse module to
parse command line flags, which means we'll have to stick to a more traditional
interface for the scripts. In particular, we can't have short options like "+c"
anymore. This isn't much of a problem, because we will keep the bash scripts as
they are and just make them invoke a tool written in python (scripts/llvm.py) to
do the work. The tool will have subcommands for any functionality that we want,
for instance the new interface for adding/removing subprojects is:
llvm.py projects [-a subproject subproject ... subproject]
[-r subproject ... subproject]
The -a and -r options (followed by any number of subprojects) can be used to
selectively enable/disable things. You have to pass the full name of the
subproject (e.g. llvmprojs.py -a clang lld -r libcxx) for it to work. This is
invoked by the llvm-projs bash script, which has the old, more convenient
For now the bash scripts will live in the same directories as they used to, but
after the transition is complete we will want to move them to the scripts
Note that we're also eliding any dashes in the names of the scripts, in keeping
with Python best practices for module names (i.e. using valid Python identifiers
The stress, helper and bisect build scripts needed the LLVM_LINK_JOBS option
to make sure it doesn't run out of memory while linking, especially in SoCs
that have a lot more cores than RAM (ex. HiKey).
This patch calculates the jobs by seeing how much RAM the target has and adding
+1, as link jobs rarely take more than 1GB, but the final ones do use several
hundred. Also making sure we don't exceed the number of cores.
A few other small changes:
* Replace "else if" by "elif"
* Build only minimal targets on ARM (even o full build)
* Fixed a -j$PARALLEL bug introduced by a prvious commit
* Converted a few backticks into $()
Adding a direct link to the buil on the build number, to avoid having
to click on the bot page *then* the link. With the buildmaster *that* slow,
this saves several seconds per lookup, which sometimes can be several dozens
This is achieved by setting the CMake option LLVM_LIT_ARGS. If we don't do this,
lit may decide that it only needs to run on one core.
While we're at it, also update the run.sh that we use for bisects to force ninja
to use the right number of cores while building.
In addition to debug builds, this teaches llvm-env to create self-hosted
bots that will use the "build" directory of the branch to compile a self-hosted
version of the toolchain. It'll use clang and also lld, if available.
* Check for clang, not just build dir
* Use -x instead of -f
* Always export LLVM_SELFHOST
* Quote strings for good measure
* Using LLVM_CMAKE_FLAGS
Allow the user to override the number of CPUs used for building LLVM. The
default remains unchanged, so if you're not going to use the -j flag it won't
The install directory will be <Worktree>/install, right next to the build and
source dirs, and it will be exported in LLVM_INSTALL.
Note that this won't affect existing worktree directories, you should either
nuke the build directories or manually update CMakeCache.txt for them.
Forgot to add a patch set before merging (I don't know how I didn't notice
Errata for Change-Id: I787594ce1ecea0de68ceb0e518194b783da408b3
When running llvm-env without any arguments, it now dumps the current worktrees,
as well as the branch that is currently checked out in them (only for llvm, to
make it easy to spot places where this is different from the environment name).
This is a very hacky implementation, because newer versions of git will come
with git worktree list, which does exactly what we want. We can nuke this when
we're all ready to move to newer gits.
This is necessary in order to allow people outside TCWG to use our scripts
This should allow us to run make/ninja on a given target, e.g.
`llvm-build llc` will build only the llc target
`llvm-build check-llvm-codegen-aarch64` will only run a subset of the tests
This also adds a -h option.
This commit adds a new script, llvm-env, which sets up the environment for
working with LLVM. It looks for an environment variable LLVM_ROOT and tries to
create the following hierarchy:
| `- llvm
| `- clang
| `- compiler-rt
| `- llvm
| `- build
| `- debug
| `- llvm
| `- build
| `- debug
The $LLVM_ROOT/repos directory contains all the repositories, as checked out by
llvm-prepare, and will always track master. For other branches,
llvm-env <branch_name> will create a new directory, $LLVM_ROOT/<branch_name>,
and will add an llvm worktree directory there. If -d is passed, it will also
create a debug directory there, otherwise it will create a build directory.
Notice that these 2 can live in parallel, and we can switch between them at any
time by invoking llvm-env. It will set LLVM_SRC and LLVM_BLD accordingly, and
also modify the path to point to the binaries in LLVM_BLD.
The other scripts will now work with the LLVM_SRC and LLVM_BLD set by llvm-env
in the current shell. Because llvm-env controls whether or not we're doing a
debug build, llvm-build will no longer take a -d flag (it will instead look
after a LLVM_DEBUG environment variable, also set by llvm-env). There are
changes in llvm-projs, too, because now it no longer creates links - instead it
creates worktree directories in the corresponding $LLVM_ROOT/<branch>/llvm.
Other scripts have also been updated accordingly.
To make things easier, here are some of the changes that I had to make that are
not particularly important for the review (pretty mechanical stuff):
* Moved function has() from llvm-branch to llvm-common, so I could reuse it
* Because of this, I had to rename the has() function in llvm-projs to
has_link(), which is actually a better name for it anyway
* Disable the checks for LLVM_SRC and LLVM_BLD in llvm-common
YouCompleteMe helper, updates the YCM config file once YCM-Generator
creates one for you. The steps are:
* Install YCM / Generator
* Inside Vim, on $LLVM_SRC, run :YcmGenerateConfig
* Outside vim, with llvm-env on the same workdir run: llvm-ycm
* Re-start vim, all is good
Why is this needed?
1. YCM relies on .ycm_extra_conf.py to be precise and correct
2. LLVM includes headers and libraries from both source and build dirs
3. YCM-Generator has no idea about the build dir, so it uses /tmp
* You will need to do this for each workdir, as the path has to be absolute
* Don't use CMake's compilation_database.json, or header files won't work
Adding the number of cores to the -j option for test-release.
Also, fix a bunch of shellcheck warnings.