| Age | Commit message (Collapse) | Author |
|---|
|
Both the test-suite and stage 2 need clang, so check early that it is
built as part of stage 1.
I'm not 100% certain on the former, since nowadays the test-suite
contains some bitcode tests which might be able to run without clang,
but it's unlikely that we'll ever be interested in such a scenario.
Change-Id: I6668557d9c2ef8cbad8a309a3c73c690d24533d1
|
|
Use the built-in support from the argparse module to allow our
subcommand to read flags from configuration files.
This is achieved by specifying a prefix (in our case '@') that indicates
to argparse that it should read arguments from the file ('@filepath'),
one per line.
They are then processed as if they had been given on the command line in
the position where the filename is. This makes it possible for flags in
the file to override or be overriden by flags given on the command line,
depending on their position relative to the filename. Flags that
accumulate all the values that they receive cannot be overriden, but
will instead accumulate everything regardless of whether it came from
the command line or the file.
Change-Id: I22a9a92a0f5ba49dd085ef5860c37bd4fc1009e1
|
|
Allow the user to pass flags to LNT when running the test-suite.
Change-Id: Idb66ac25934281613f6392750b38d5c53f99761a
|
|
Same possibilities as stage 1: subprojects, CMake definitions and build
flags.
Change-Id: Ib10be5d2f7687040979c79113e7eeb0af5903df1
|
|
Add support for custom subprojects, custom CMake definitions and custom
build flags/targets for the first stage of the build. The subprojects
are preserved for stage 2, but the other flags aren't. They will be
configured separately.
We're still hardcoding the CMake generator, but we can fix that later.
This also extracts a couple of helpers for processing the command line
arguments, which we can use both for this subcommand and for the others.
Change-Id: Id6f8773b075815eb394eddff48adc9a7123f0da9
|
|
This is necessary so we can add subprojects to our source config. It
assumes that each subproject has a repo directly under the repos-dir.
Since we now have a path to *all* the repos, we can use that to get the
test-suite and LNT repos as well, instead of specifying each of them
manually. This simplifies the interface a bit (we get rid of the
parameters for passing the paths to the test-suite and LNT repo, and
instead add a single parameter for enabling the test-suite).
Change-Id: I818952969aa9335720965af7273fa8273223a64e
|
|
The primary purpose of this subcommand is to help us reproduce
buildbots, but I think in the long run it will be flexible enough to
accommodate other goals as well (e.g. cross-compilation).
There are some very important bits that are clearly missing at the
moment, but we can work on them incrementally. One of those is that at
the moment we take the path to an LLVM worktree as input - in the future
we should be able to take an SVN revision and setup the LLVM sources as
part of the script. We also want to be able to configure which
subprojects to use for each stage, what flags to use for each stage and
for the test-suite etc (yes, there will be lots of parameters, but that
will become less of a concern when we introduce support for config
files).
In the future, we'll probably want to change the interface. At the
moment, we pass a lot of things separately that maybe don't need all
this flexibility. In this version of the patch, we would pass something
like:
"--source-dir <sourceDir> --stage1-build-dir <buildDir1> \
--stage2-build-dir <buildDir2> --test-suite <testSuiteDir> \
--sandbox <sandboxDir> --lnt <lntDir>"
The script then knows that we want 2 stages because we have a
"--stage2-build-dir", and also that we want to run the test-suite
because we have a "--test-suite".
Given the nature of this subcommand and how in the long run we will
probably want it to setup everything on its own, we'd be better off with
a single "--root-dir <rootDir>" argument, and hardcode the path to the
sources as "<rootDir>/llvm", the path to the build directories as
"<rootDir>/stageN" etc. We could also probably assume that the repos are
in "<rootDir>/repos/test-suite" and respectively "<rootDir>/repos/lnt".
We could then have a much simpler interface along the lines of:
"--root-dir <rootDir> --enable-stage2 --enable-test-suite"
For the other subcommands, we prefer to be very specific about
everything, because they're meant to be very flexible and usable from
different contexts (with or without wrapper scripts to help make things
less tedious). This new subcommand is pretty different though, since it
combines more than one functionality. We may even consider making it a
separate script instead of a subcommand of llvm.py, but for the time
being it's easier to prototype in this form.
Also, we may want to change the name :)
Change-Id: Ia6bf1a2f034d5b84572879215fe48f58addbfe2c
|
|
This makes it possible to know what build tool to use in dry run mode
when the build directory has not been configured yet.
Change-Id: Ib620167c62f4c290621ace582d6ad893ad475bc3
|
|
Make it possible to use LLVMSourceConfig in dry run mode. This is useful
because otherwise we can't have a dry run on *any* scripts that use an
LLVMSourceConfig, since it might throw errors that wouldn't occur in
non-dry mode.
This is different from non-dry run in the following ways:
* we do not perform any validation of the source directory (since in dry
run mode it is reasonable to expect that the directory doesn't even
exist yet)
* we do not try to add or remove any subprojects on disk
Instead, we do try to keep track of our updates (which projects are
added and which are removed). We don't however look at what may exist on
disk since that increases the complexity of the code significantly and
it is very likely that scripts will create a source directory from
scratch rather than use one where subprojects have already been added.
We may want to fix that after a bit of refactoring.
Change-Id: Ic12c292b9cdc932aa5de3fa780af5e608dcba01c
|
|
Add a subcommand 'setup-test-suite' which sets up a sandbox that can be
used for running the test-suite.
Change-Id: If6426acd0e4e8ed54d310d1159a545f762affe74
|
|
Add a llvm.py subcommand 'run-test-suite' which runs the test-suite in a
given sandbox. This uses the 'test-suite' producer (as opposed to the
old 'nt' one) since this is what the buildbots use these days.
We don't update the helpers in the 'test-suite' directory because they
are obsolete (and using the 'nt' producer) and hopefully nobody has been
using them lately. Instead we add a really simple helper
'llvm-test-suite' which runs the test-suite with the compiler from
$LLVM_BLD. We keep this helper simple for the time being, but in the
future we may want to add more features here (e.g. a benchmarking mode,
core/number of threads detection etc).
We don't currently setup the sandbox. That will be a different commit.
Change-Id: I2bc661aa483f1ba40cb9bdc01f8d297cd1fb99d0
|
|
Add a llvm.py subcommand for building LLVM in a given build directory.
The build directory must have already been configured with CMake. The
build command that is run will be either a 'ninja' command or a 'make'
command, depending on whether the build directory contains a
'build.ninja' or a 'Makefile'. If it contains neither of those, an error
is generated.
I initially wanted to put the implementation for this in the
LLVMBuildConfig, but it doesn't really need anything from there. For now
I left it as a free-standing function. It doesn't have any knowledge of
LLVM whatsoever, so an argument could be made for moving it into the
"utils" package rather than the "llvm" package. In the future, we may
want to remove the LLVMBuildConfig class and see if there's a better way
to organize the code for configuring and building.
One disadvantage with using python here is that we lose the nice
progress bars that we could see when running 'ninja check' from the bash
scripts. I'm not sure how to fix that, suggestions welcome.
Change-Id: I48cf464a6412238a26eb5bcfb4723946983c86f2
|
|
Remove the --env argument, which was compulsory for all subcommands, and
replace it with a --source-dir argument, only for those subcommands that
need it.
We do this for 2 reasons:
* Not all subcommands care about the source directory (e.g. llvm.py
build won't need it)
* llvm.py should not have any knowledge about environments - that
concept only makes sense for the helper scripts. llvm.py should
instead receive very specific info about where the source and build
directories are.
Change-Id: Iffaeef95559e8923bd883d5c51fed8c306287280
|
|
Move some helpers that aren't directly dealing with LLVM knowledge from
modules/llvm.py to modules/utils.py.
Change-Id: I792286b06a99852facc7bc77b7c39ed67988e18f
|
|
Add a subcommand that runs CMake in a given build directory, with a
custom generator and custom CMake definitions. Also update llvm-build to
use it instead of calling CMake directly, and add calls to it in
llvm-projs as well to make sure we update the build directories whenever
we enable or disable a project.
One known issue with the current code is that the output of the CMake
command is not printed out live, but rather after the command has
finished execution. This is going to be more important for llvm.py build
than for llvm.py configure, so we can fix it in a future commit.
Change-Id: I263b2d47c2083a1778608253bbd437149375c539
|
|
The --repos flag is not used by all the subcommands, so there's no
reason for it to be compulsory. This commit moves it from the generic
llvm.py options to the options for the projects subcommand, where it is
only required when adding subprojects.
Change-Id: I7ad10bda6e1b4efeb594b09e5fda200850d99c07
|
|
* Remove all references to clang-tools-extra in helpers
We do not support clang tools extra in the python scripts, since it's
too complicated and nobody is working on it yet. Therefore, clean up all
references to it in the helpers as well, to avoid any confusion.
* Add a bit more info to llvm.py --help
Change-Id: I5990925cebafce182b081227a4d33aecead7b4ac
|
|
Get llvm.py push to print the name of the remote branch that it has
pushed to. This is helpful so developers don't need to guess what
namespaces have been prepended to the branch name, and also serves as
quick visual confirmation.
This required a bit of refactoring, since the name of the branch was
computed when needed. We now compute it separately and pass it in to the
function that pushes the branch. We rely on the fact that the branch
name will be the same for all the subprojects involved.
Most of the changes are mechanical, to account for this refactoring
(including new names for some of the existing functions, which make more
sense in this new context).
Change-Id: Id7e496bcded60080803e3a850d73329428bc3bac
|
|
Add support for pushing the current branch in all linked subprojects in
a given config. If they are not on the same branch, an error is thrown.
The branches are pushed as linaro-local/$user/$branch, where $branch is
the local name of the branch and $user is the user as seen in
LLVM_GITUSER (we might want to change to a more intelligent way to get
the user in the future).
We achieve this by adding a new method to LLVMSourceConfig:
for_each_enabled, which takes an action and applies it to all the
enabled subprojects. If an exception is thrown by the action, it is
rethrown by the action as a RuntimeError specifying which project the
action was being applied to. Note that this does not behave like a
transaction - the action may have been applied to some of the projects
already, and there's no way to undo it. It's also a bit difficult to
tell which projects the action was applied on (in the future we should
probably log that info).
We also provide an action that simply pushes the current branch to
origin with the updated name. This makes sure the remote branch will
live in an isolated namespace where it won't conflict with other
people's branches.
We also update some of the internals of LLVMSourceConfig to throw
exceptions if some of the subprojects that are linked are not worktrees
on the same branch as LLVM. In the past they were just ignored, but that
doesn't seem like a sane default anymore.
Change-Id: I9c917658d65b5d0e7bad3310efce07a0fe5bce5e
|
|
Before this patch, we had an LLVM_ROOT directory which contained the
repos and a series of environments (worktrees + build dicretories).
However, there is no good reason to have such a rigid layout, and in
fact it prevents us from sharing the repos in any meaningful way.
This patch removes LLVM_ROOT and instead introduces two command line
parameters, repos and env, which represent the path to the directory
containing the LLVM repositories and respectively the path to the
environment that we intend to modify. The two paths may be completely
independent. A side effect of this is that we can now also have
different environments in different, unrelated locations (whereas
previously they had to be siblings in LLVM_ROOT).
For the time being, both parameters are compulsory (not only for
llvm-projects, but for all subcommands). There may be situations where
we can get away with only one of them (e.g. `llvm-projs <env>`, which
just lists the projects enabled in the given <env> - it does not need to
know where the repos are in order to do this). There may also be
situations where one of them doesn't make any sense at all (e.g.
`llvm-prepare`, which is not implemented yet in python, only needs the
path to the repos and it would not know what to do with an environment).
Making them optional would require some additional validation logic as
well as more tests, and it feels a bit early on to introduce all that.
It would be nice to have at least one or 2 more subcommands implemented
before we invest time in it.
Change-Id: Ibc1321d18476c2c4a65b5b05ca3fce3467d72fc3
|
|
Add an extra parameter to llvm.py representing the environment that we
want the command to refer to. We then compute the path to the LLVM
source tree based on that and LLVM_ROOT (which is still an environment
variable for the time being).
The bash helper keeps its old interface and uses LLVM_SRC to compute the
environment that it will pass down to llvm.py. We also perform a small
number of drive-by fixes to this helper (e.g. replacing python with
python3).
Change-Id: Ie7a33103969622294e158f83be8b9f57832ef1dc
|
|
Most of the changes have been made by 2to3. Some manual fiddling was
needed to make sure we're calling python3 instead of just python in the
tests. Two of the helpers in the tests were reworked into proper methods
rather than partials because we had to convert their returned value to
str and this seems like the most straightforward/easy-to-read way.
Change-Id: I74fdc1eaade3c026ee0c3e6bcdf26f8840f259b3
|
|
This is the first step in moving all our scripts to python. For now we keep the
current content of the repo, but we'll start adding a new directory hierarchy
for the python stuff:
* modules: should contain most of the python code, organized as a package that
can be imported by the scripts
* scripts: the scripts themselves, which should be parsing the command line
and calling stuff from 'modules' to do the actual work; can be
broken down into the same categories we had before (helpers, bisect
etc), or we could just have one big pile
* tests: should contain unittests (for the stuff in modules) and command line
interface tests (for the scripts)
The code is heavily using functionality from the tcwg-release-tools repo (which
probably needs to be renamed / reorganized), so you should have that in your
PYTHONPATH when trying to run any of the scripts. To run the tests, just invoke
check.sh.
One of the important changes is that we'll be using python's argparse module to
parse command line flags, which means we'll have to stick to a more traditional
interface for the scripts. In particular, we can't have short options like "+c"
anymore. This isn't much of a problem, because we will keep the bash scripts as
they are and just make them invoke a tool written in python (scripts/llvm.py) to
do the work. The tool will have subcommands for any functionality that we want,
for instance the new interface for adding/removing subprojects is:
llvm.py projects [-a subproject subproject ... subproject]
[-r subproject ... subproject]
The -a and -r options (followed by any number of subprojects) can be used to
selectively enable/disable things. You have to pass the full name of the
subproject (e.g. llvmprojs.py -a clang lld -r libcxx) for it to work. This is
invoked by the llvm-projs bash script, which has the old, more convenient
interface.
For now the bash scripts will live in the same directories as they used to, but
after the transition is complete we will want to move them to the scripts
directory.
Note that we're also eliding any dashes in the names of the scripts, in keeping
with Python best practices for module names (i.e. using valid Python identifiers
as names).
Change-Id: I9ec08632dbb17464673240d6f6881a90f45d5371
|
