aboutsummaryrefslogtreecommitdiff
path: root/scripts
AgeCommit message (Collapse)Author
2018-02-23Require clang in stage 1 when enabling stage 2 / test-suitepythonDiana Picus
Both the test-suite and stage 2 need clang, so check early that it is built as part of stage 1. I'm not 100% certain on the former, since nowadays the test-suite contains some bitcode tests which might be able to run without clang, but it's unlikely that we'll ever be interested in such a scenario. Change-Id: I6668557d9c2ef8cbad8a309a3c73c690d24533d1
2018-02-22build-and-test: Allow flags to be read from filesDiana Picus
Use the built-in support from the argparse module to allow our subcommand to read flags from configuration files. This is achieved by specifying a prefix (in our case '@') that indicates to argparse that it should read arguments from the file ('@filepath'), one per line. They are then processed as if they had been given on the command line in the position where the filename is. This makes it possible for flags in the file to override or be overriden by flags given on the command line, depending on their position relative to the filename. Flags that accumulate all the values that they receive cannot be overriden, but will instead accumulate everything regardless of whether it came from the command line or the file. Change-Id: I22a9a92a0f5ba49dd085ef5860c37bd4fc1009e1
2018-02-21build-and-test: Make it possible to customize test suite runDiana Picus
Allow the user to pass flags to LNT when running the test-suite. Change-Id: Idb66ac25934281613f6392750b38d5c53f99761a
2018-02-21build-and-test: Make it possible to customize stage 2Diana Picus
Same possibilities as stage 1: subprojects, CMake definitions and build flags. Change-Id: Ib10be5d2f7687040979c79113e7eeb0af5903df1
2018-02-21build-and-test: Make it possible to customize stage 1Diana Picus
Add support for custom subprojects, custom CMake definitions and custom build flags/targets for the first stage of the build. The subprojects are preserved for stage 2, but the other flags aren't. They will be configured separately. We're still hardcoding the CMake generator, but we can fix that later. This also extracts a couple of helpers for processing the command line arguments, which we can use both for this subcommand and for the others. Change-Id: Id6f8773b075815eb394eddff48adc9a7123f0da9
2018-02-21build-and-test: Add required --repos-dir flagDiana Picus
This is necessary so we can add subprojects to our source config. It assumes that each subproject has a repo directly under the repos-dir. Since we now have a path to *all* the repos, we can use that to get the test-suite and LNT repos as well, instead of specifying each of them manually. This simplifies the interface a bit (we get rid of the parameters for passing the paths to the test-suite and LNT repo, and instead add a single parameter for enabling the test-suite). Change-Id: I818952969aa9335720965af7273fa8273223a64e
2018-02-20Add subcommand for running complex scenariosDiana Picus
The primary purpose of this subcommand is to help us reproduce buildbots, but I think in the long run it will be flexible enough to accommodate other goals as well (e.g. cross-compilation). There are some very important bits that are clearly missing at the moment, but we can work on them incrementally. One of those is that at the moment we take the path to an LLVM worktree as input - in the future we should be able to take an SVN revision and setup the LLVM sources as part of the script. We also want to be able to configure which subprojects to use for each stage, what flags to use for each stage and for the test-suite etc (yes, there will be lots of parameters, but that will become less of a concern when we introduce support for config files). In the future, we'll probably want to change the interface. At the moment, we pass a lot of things separately that maybe don't need all this flexibility. In this version of the patch, we would pass something like: "--source-dir <sourceDir> --stage1-build-dir <buildDir1> \ --stage2-build-dir <buildDir2> --test-suite <testSuiteDir> \ --sandbox <sandboxDir> --lnt <lntDir>" The script then knows that we want 2 stages because we have a "--stage2-build-dir", and also that we want to run the test-suite because we have a "--test-suite". Given the nature of this subcommand and how in the long run we will probably want it to setup everything on its own, we'd be better off with a single "--root-dir <rootDir>" argument, and hardcode the path to the sources as "<rootDir>/llvm", the path to the build directories as "<rootDir>/stageN" etc. We could also probably assume that the repos are in "<rootDir>/repos/test-suite" and respectively "<rootDir>/repos/lnt". We could then have a much simpler interface along the lines of: "--root-dir <rootDir> --enable-stage2 --enable-test-suite" For the other subcommands, we prefer to be very specific about everything, because they're meant to be very flexible and usable from different contexts (with or without wrapper scripts to help make things less tedious). This new subcommand is pretty different though, since it combines more than one functionality. We may even consider making it a separate script instead of a subcommand of llvm.py, but for the time being it's easier to prototype in this form. Also, we may want to change the name :) Change-Id: Ia6bf1a2f034d5b84572879215fe48f58addbfe2c
2018-02-12Move build_llvm into LLVMBuildConfigDiana Picus
This makes it possible to know what build tool to use in dry run mode when the build directory has not been configured yet. Change-Id: Ib620167c62f4c290621ace582d6ad893ad475bc3
2018-02-12Dry run-ify LLVMSourceConfigDiana Picus
Make it possible to use LLVMSourceConfig in dry run mode. This is useful because otherwise we can't have a dry run on *any* scripts that use an LLVMSourceConfig, since it might throw errors that wouldn't occur in non-dry mode. This is different from non-dry run in the following ways: * we do not perform any validation of the source directory (since in dry run mode it is reasonable to expect that the directory doesn't even exist yet) * we do not try to add or remove any subprojects on disk Instead, we do try to keep track of our updates (which projects are added and which are removed). We don't however look at what may exist on disk since that increases the complexity of the code significantly and it is very likely that scripts will create a source directory from scratch rather than use one where subprojects have already been added. We may want to fix that after a bit of refactoring. Change-Id: Ic12c292b9cdc932aa5de3fa780af5e608dcba01c
2018-01-31Support setting up the test-suiteDiana Picus
Add a subcommand 'setup-test-suite' which sets up a sandbox that can be used for running the test-suite. Change-Id: If6426acd0e4e8ed54d310d1159a545f762affe74
2018-01-31Add support for running the test-suiteDiana Picus
Add a llvm.py subcommand 'run-test-suite' which runs the test-suite in a given sandbox. This uses the 'test-suite' producer (as opposed to the old 'nt' one) since this is what the buildbots use these days. We don't update the helpers in the 'test-suite' directory because they are obsolete (and using the 'nt' producer) and hopefully nobody has been using them lately. Instead we add a really simple helper 'llvm-test-suite' which runs the test-suite with the compiler from $LLVM_BLD. We keep this helper simple for the time being, but in the future we may want to add more features here (e.g. a benchmarking mode, core/number of threads detection etc). We don't currently setup the sandbox. That will be a different commit. Change-Id: I2bc661aa483f1ba40cb9bdc01f8d297cd1fb99d0
2018-01-23Add support for building LLVMDiana Picus
Add a llvm.py subcommand for building LLVM in a given build directory. The build directory must have already been configured with CMake. The build command that is run will be either a 'ninja' command or a 'make' command, depending on whether the build directory contains a 'build.ninja' or a 'Makefile'. If it contains neither of those, an error is generated. I initially wanted to put the implementation for this in the LLVMBuildConfig, but it doesn't really need anything from there. For now I left it as a free-standing function. It doesn't have any knowledge of LLVM whatsoever, so an argument could be made for moving it into the "utils" package rather than the "llvm" package. In the future, we may want to remove the LLVMBuildConfig class and see if there's a better way to organize the code for configuring and building. One disadvantage with using python here is that we lose the nice progress bars that we could see when running 'ninja check' from the bash scripts. I'm not sure how to fix that, suggestions welcome. Change-Id: I48cf464a6412238a26eb5bcfb4723946983c86f2
2017-12-22Replace --env with --source-dirDiana Picus
Remove the --env argument, which was compulsory for all subcommands, and replace it with a --source-dir argument, only for those subcommands that need it. We do this for 2 reasons: * Not all subcommands care about the source directory (e.g. llvm.py build won't need it) * llvm.py should not have any knowledge about environments - that concept only makes sense for the helper scripts. llvm.py should instead receive very specific info about where the source and build directories are. Change-Id: Iffaeef95559e8923bd883d5c51fed8c306287280
2017-12-15Move more stuff from modules.llvm to modules.utilsDiana Picus
Move some helpers that aren't directly dealing with LLVM knowledge from modules/llvm.py to modules/utils.py. Change-Id: I792286b06a99852facc7bc77b7c39ed67988e18f
2017-12-15Add llvm.py configureDiana Picus
Add a subcommand that runs CMake in a given build directory, with a custom generator and custom CMake definitions. Also update llvm-build to use it instead of calling CMake directly, and add calls to it in llvm-projs as well to make sure we update the build directories whenever we enable or disable a project. One known issue with the current code is that the output of the CMake command is not printed out live, but rather after the command has finished execution. This is going to be more important for llvm.py build than for llvm.py configure, so we can fix it in a future commit. Change-Id: I263b2d47c2083a1778608253bbd437149375c539
2017-11-23Move --repos flag to projects subcommandDiana Picus
The --repos flag is not used by all the subcommands, so there's no reason for it to be compulsory. This commit moves it from the generic llvm.py options to the options for the projects subcommand, where it is only required when adding subprojects. Change-Id: I7ad10bda6e1b4efeb594b09e5fda200850d99c07
2017-11-03Minor fixes to scripts interfacesDiana Picus
* Remove all references to clang-tools-extra in helpers We do not support clang tools extra in the python scripts, since it's too complicated and nobody is working on it yet. Therefore, clean up all references to it in the helpers as well, to avoid any confusion. * Add a bit more info to llvm.py --help Change-Id: I5990925cebafce182b081227a4d33aecead7b4ac
2017-11-03llvm push: Print remote branch name after pushingDiana Picus
Get llvm.py push to print the name of the remote branch that it has pushed to. This is helpful so developers don't need to guess what namespaces have been prepended to the branch name, and also serves as quick visual confirmation. This required a bit of refactoring, since the name of the branch was computed when needed. We now compute it separately and pass it in to the function that pushes the branch. We rely on the fact that the branch name will be the same for all the subprojects involved. Most of the changes are mechanical, to account for this refactoring (including new names for some of the existing functions, which make more sense in this new context). Change-Id: Id7e496bcded60080803e3a850d73329428bc3bac
2017-07-07[llvmpush] Add llvm pushDiana Picus
Add support for pushing the current branch in all linked subprojects in a given config. If they are not on the same branch, an error is thrown. The branches are pushed as linaro-local/$user/$branch, where $branch is the local name of the branch and $user is the user as seen in LLVM_GITUSER (we might want to change to a more intelligent way to get the user in the future). We achieve this by adding a new method to LLVMSourceConfig: for_each_enabled, which takes an action and applies it to all the enabled subprojects. If an exception is thrown by the action, it is rethrown by the action as a RuntimeError specifying which project the action was being applied to. Note that this does not behave like a transaction - the action may have been applied to some of the projects already, and there's no way to undo it. It's also a bit difficult to tell which projects the action was applied on (in the future we should probably log that info). We also provide an action that simply pushes the current branch to origin with the updated name. This makes sure the remote branch will live in an isolated namespace where it won't conflict with other people's branches. We also update some of the internals of LLVMSourceConfig to throw exceptions if some of the subprojects that are linked are not worktrees on the same branch as LLVM. In the past they were just ignored, but that doesn't seem like a sane default anymore. Change-Id: I9c917658d65b5d0e7bad3310efce07a0fe5bce5e
2017-05-12Separate the repos from the environmentsDiana Picus
Before this patch, we had an LLVM_ROOT directory which contained the repos and a series of environments (worktrees + build dicretories). However, there is no good reason to have such a rigid layout, and in fact it prevents us from sharing the repos in any meaningful way. This patch removes LLVM_ROOT and instead introduces two command line parameters, repos and env, which represent the path to the directory containing the LLVM repositories and respectively the path to the environment that we intend to modify. The two paths may be completely independent. A side effect of this is that we can now also have different environments in different, unrelated locations (whereas previously they had to be siblings in LLVM_ROOT). For the time being, both parameters are compulsory (not only for llvm-projects, but for all subcommands). There may be situations where we can get away with only one of them (e.g. `llvm-projs <env>`, which just lists the projects enabled in the given <env> - it does not need to know where the repos are in order to do this). There may also be situations where one of them doesn't make any sense at all (e.g. `llvm-prepare`, which is not implemented yet in python, only needs the path to the repos and it would not know what to do with an environment). Making them optional would require some additional validation logic as well as more tests, and it feels a bit early on to introduce all that. It would be nice to have at least one or 2 more subcommands implemented before we invest time in it. Change-Id: Ibc1321d18476c2c4a65b5b05ca3fce3467d72fc3
2017-04-24Remove the need for LLVM_SRC in llvm.pyDiana Picus
Add an extra parameter to llvm.py representing the environment that we want the command to refer to. We then compute the path to the LLVM source tree based on that and LLVM_ROOT (which is still an environment variable for the time being). The bash helper keeps its old interface and uses LLVM_SRC to compute the environment that it will pass down to llvm.py. We also perform a small number of drive-by fixes to this helper (e.g. replacing python with python3). Change-Id: Ie7a33103969622294e158f83be8b9f57832ef1dc
2017-04-06Move to Python3Diana Picus
Most of the changes have been made by 2to3. Some manual fiddling was needed to make sure we're calling python3 instead of just python in the tests. Two of the helpers in the tests were reworked into proper methods rather than partials because we had to convert their returned value to str and this seems like the most straightforward/easy-to-read way. Change-Id: I74fdc1eaade3c026ee0c3e6bcdf26f8840f259b3
2016-12-15[llvmprojs] Rewrite llvm-projs in pythonDiana Picus
This is the first step in moving all our scripts to python. For now we keep the current content of the repo, but we'll start adding a new directory hierarchy for the python stuff: * modules: should contain most of the python code, organized as a package that can be imported by the scripts * scripts: the scripts themselves, which should be parsing the command line and calling stuff from 'modules' to do the actual work; can be broken down into the same categories we had before (helpers, bisect etc), or we could just have one big pile * tests: should contain unittests (for the stuff in modules) and command line interface tests (for the scripts) The code is heavily using functionality from the tcwg-release-tools repo (which probably needs to be renamed / reorganized), so you should have that in your PYTHONPATH when trying to run any of the scripts. To run the tests, just invoke check.sh. One of the important changes is that we'll be using python's argparse module to parse command line flags, which means we'll have to stick to a more traditional interface for the scripts. In particular, we can't have short options like "+c" anymore. This isn't much of a problem, because we will keep the bash scripts as they are and just make them invoke a tool written in python (scripts/llvm.py) to do the work. The tool will have subcommands for any functionality that we want, for instance the new interface for adding/removing subprojects is: llvm.py projects [-a subproject subproject ... subproject] [-r subproject ... subproject] The -a and -r options (followed by any number of subprojects) can be used to selectively enable/disable things. You have to pass the full name of the subproject (e.g. llvmprojs.py -a clang lld -r libcxx) for it to work. This is invoked by the llvm-projs bash script, which has the old, more convenient interface. For now the bash scripts will live in the same directories as they used to, but after the transition is complete we will want to move them to the scripts directory. Note that we're also eliding any dashes in the names of the scripts, in keeping with Python best practices for module names (i.e. using valid Python identifiers as names). Change-Id: I9ec08632dbb17464673240d6f6881a90f45d5371