AgeCommit message (Collapse)Author
2018-02-23Require clang in stage 1 when enabling stage 2 / test-suitepythonDiana Picus
Both the test-suite and stage 2 need clang, so check early that it is built as part of stage 1. I'm not 100% certain on the former, since nowadays the test-suite contains some bitcode tests which might be able to run without clang, but it's unlikely that we'll ever be interested in such a scenario. Change-Id: I6668557d9c2ef8cbad8a309a3c73c690d24533d1
2018-02-23Test compulsory args. NFCDiana Picus
Change-Id: Iabd058daf6039c77577608f6998f31170f876150
2018-02-22build-and-test: Allow flags to be read from filesDiana Picus
Use the built-in support from the argparse module to allow our subcommand to read flags from configuration files. This is achieved by specifying a prefix (in our case '@') that indicates to argparse that it should read arguments from the file ('@filepath'), one per line. They are then processed as if they had been given on the command line in the position where the filename is. This makes it possible for flags in the file to override or be overriden by flags given on the command line, depending on their position relative to the filename. Flags that accumulate all the values that they receive cannot be overriden, but will instead accumulate everything regardless of whether it came from the command line or the file. Change-Id: I22a9a92a0f5ba49dd085ef5860c37bd4fc1009e1
2018-02-21build-and-test: Make it possible to customize test suite runDiana Picus
Allow the user to pass flags to LNT when running the test-suite. Change-Id: Idb66ac25934281613f6392750b38d5c53f99761a
2018-02-21build-and-test: Make it possible to customize stage 2Diana Picus
Same possibilities as stage 1: subprojects, CMake definitions and build flags. Change-Id: Ib10be5d2f7687040979c79113e7eeb0af5903df1
2018-02-21build-and-test: Make it possible to customize stage 1Diana Picus
Add support for custom subprojects, custom CMake definitions and custom build flags/targets for the first stage of the build. The subprojects are preserved for stage 2, but the other flags aren't. They will be configured separately. We're still hardcoding the CMake generator, but we can fix that later. This also extracts a couple of helpers for processing the command line arguments, which we can use both for this subcommand and for the others. Change-Id: Id6f8773b075815eb394eddff48adc9a7123f0da9
2018-02-21build-and-test: Add required --repos-dir flagDiana Picus
This is necessary so we can add subprojects to our source config. It assumes that each subproject has a repo directly under the repos-dir. Since we now have a path to *all* the repos, we can use that to get the test-suite and LNT repos as well, instead of specifying each of them manually. This simplifies the interface a bit (we get rid of the parameters for passing the paths to the test-suite and LNT repo, and instead add a single parameter for enabling the test-suite). Change-Id: I818952969aa9335720965af7273fa8273223a64e
2018-02-20Add subcommand for running complex scenariosDiana Picus
The primary purpose of this subcommand is to help us reproduce buildbots, but I think in the long run it will be flexible enough to accommodate other goals as well (e.g. cross-compilation). There are some very important bits that are clearly missing at the moment, but we can work on them incrementally. One of those is that at the moment we take the path to an LLVM worktree as input - in the future we should be able to take an SVN revision and setup the LLVM sources as part of the script. We also want to be able to configure which subprojects to use for each stage, what flags to use for each stage and for the test-suite etc (yes, there will be lots of parameters, but that will become less of a concern when we introduce support for config files). In the future, we'll probably want to change the interface. At the moment, we pass a lot of things separately that maybe don't need all this flexibility. In this version of the patch, we would pass something like: "--source-dir <sourceDir> --stage1-build-dir <buildDir1> \ --stage2-build-dir <buildDir2> --test-suite <testSuiteDir> \ --sandbox <sandboxDir> --lnt <lntDir>" The script then knows that we want 2 stages because we have a "--stage2-build-dir", and also that we want to run the test-suite because we have a "--test-suite". Given the nature of this subcommand and how in the long run we will probably want it to setup everything on its own, we'd be better off with a single "--root-dir <rootDir>" argument, and hardcode the path to the sources as "<rootDir>/llvm", the path to the build directories as "<rootDir>/stageN" etc. We could also probably assume that the repos are in "<rootDir>/repos/test-suite" and respectively "<rootDir>/repos/lnt". We could then have a much simpler interface along the lines of: "--root-dir <rootDir> --enable-stage2 --enable-test-suite" For the other subcommands, we prefer to be very specific about everything, because they're meant to be very flexible and usable from different contexts (with or without wrapper scripts to help make things less tedious). This new subcommand is pretty different though, since it combines more than one functionality. We may even consider making it a separate script instead of a subcommand of llvm.py, but for the time being it's easier to prototype in this form. Also, we may want to change the name :) Change-Id: Ia6bf1a2f034d5b84572879215fe48f58addbfe2c
2018-02-12Move build_llvm into LLVMBuildConfigDiana Picus
This makes it possible to know what build tool to use in dry run mode when the build directory has not been configured yet. Change-Id: Ib620167c62f4c290621ace582d6ad893ad475bc3
2018-02-12Dry run-ify LLVMSourceConfigDiana Picus
Make it possible to use LLVMSourceConfig in dry run mode. This is useful because otherwise we can't have a dry run on *any* scripts that use an LLVMSourceConfig, since it might throw errors that wouldn't occur in non-dry mode. This is different from non-dry run in the following ways: * we do not perform any validation of the source directory (since in dry run mode it is reasonable to expect that the directory doesn't even exist yet) * we do not try to add or remove any subprojects on disk Instead, we do try to keep track of our updates (which projects are added and which are removed). We don't however look at what may exist on disk since that increases the complexity of the code significantly and it is very likely that scripts will create a source directory from scratch rather than use one where subprojects have already been added. We may want to fix that after a bit of refactoring. Change-Id: Ic12c292b9cdc932aa5de3fa780af5e608dcba01c
2018-02-09Remove unnecessary import. NFCDiana Picus
Change-Id: Id8c677e1e6a4193714d2664ddcfeb1eb87ca65fe
2018-01-31Support setting up the test-suiteDiana Picus
Add a subcommand 'setup-test-suite' which sets up a sandbox that can be used for running the test-suite. Change-Id: If6426acd0e4e8ed54d310d1159a545f762affe74
2018-01-31Add support for running the test-suiteDiana Picus
Add a llvm.py subcommand 'run-test-suite' which runs the test-suite in a given sandbox. This uses the 'test-suite' producer (as opposed to the old 'nt' one) since this is what the buildbots use these days. We don't update the helpers in the 'test-suite' directory because they are obsolete (and using the 'nt' producer) and hopefully nobody has been using them lately. Instead we add a really simple helper 'llvm-test-suite' which runs the test-suite with the compiler from $LLVM_BLD. We keep this helper simple for the time being, but in the future we may want to add more features here (e.g. a benchmarking mode, core/number of threads detection etc). We don't currently setup the sandbox. That will be a different commit. Change-Id: I2bc661aa483f1ba40cb9bdc01f8d297cd1fb99d0
2018-01-31Remove some duplication from the testsDiana Picus
Get rid of some of the duplication from the command line interface tests by extracting a decorator that checks that the proper exception is thrown when we run a command without one of its required arguments. Change-Id: Ifd2b93b7ff1bb249c002f1b03c0a242feb1f39f9
2018-01-26Clarify function interface. NFCDiana Picus
Update comment to describe what happens if one of the parameters is None. Change-Id: Ide41de345c085a59f1b15c522c8c4cec67edfc69
2018-01-25Set maxDiff to None in the debug decorator. NFCDiana Picus
This makes it possible for us to see large diffs between the actual and expected values when debugging a test. Change-Id: Iad81fb4c3b3a3306f3ad2ce842c47363e302cb5f
2018-01-23Add support for building LLVMDiana Picus
Add a llvm.py subcommand for building LLVM in a given build directory. The build directory must have already been configured with CMake. The build command that is run will be either a 'ninja' command or a 'make' command, depending on whether the build directory contains a 'build.ninja' or a 'Makefile'. If it contains neither of those, an error is generated. I initially wanted to put the implementation for this in the LLVMBuildConfig, but it doesn't really need anything from there. For now I left it as a free-standing function. It doesn't have any knowledge of LLVM whatsoever, so an argument could be made for moving it into the "utils" package rather than the "llvm" package. In the future, we may want to remove the LLVMBuildConfig class and see if there's a better way to organize the code for configuring and building. One disadvantage with using python here is that we lose the nice progress bars that we could see when running 'ninja check' from the bash scripts. I'm not sure how to fix that, suggestions welcome. Change-Id: I48cf464a6412238a26eb5bcfb4723946983c86f2
2018-01-12Collect live output from subprocessesDiana Picus
When launching a command, we now collect its output during execution and print it as soon as it becomes available. This is necessary in order to be able to see progress in long-running commands such as CMake or Ninja. This requires some changes to the way in which we use the Subprocess module, since we need to use the Popen constructor. This does not throw an exception if the launched process returns an error - we now need to manually check the return code and throw an exception ourselves. Change-Id: I973d8bdec0bd77449672830476cb2eef11c6633a
2017-12-22Replace --env with --source-dirDiana Picus
Remove the --env argument, which was compulsory for all subcommands, and replace it with a --source-dir argument, only for those subcommands that need it. We do this for 2 reasons: * Not all subcommands care about the source directory (e.g. llvm.py build won't need it) * llvm.py should not have any knowledge about environments - that concept only makes sense for the helper scripts. llvm.py should instead receive very specific info about where the source and build directories are. Change-Id: Iffaeef95559e8923bd883d5c51fed8c306287280
2017-12-22Add tests that I forgot in a previous commitDiana Picus
Forgot to git add this file when adding support for llvm.py configure. Change-Id: I0b900014cebc41ae57e910bf14fa762e29befcb9
2017-12-19Fixup: Add some imports that got lost in the previous commitDiana Picus
Change-Id: I33017c4dbfa29b87dad791001cf5c0bd6ee3f5db
2017-12-15Move more stuff from modules.llvm to modules.utilsDiana Picus
Move some helpers that aren't directly dealing with LLVM knowledge from modules/llvm.py to modules/utils.py. Change-Id: I792286b06a99852facc7bc77b7c39ed67988e18f
2017-12-15Add llvm.py configureDiana Picus
Add a subcommand that runs CMake in a given build directory, with a custom generator and custom CMake definitions. Also update llvm-build to use it instead of calling CMake directly, and add calls to it in llvm-projs as well to make sure we update the build directories whenever we enable or disable a project. One known issue with the current code is that the output of the CMake command is not printed out live, but rather after the command has finished execution. This is going to be more important for llvm.py build than for llvm.py configure, so we can fix it in a future commit. Change-Id: I263b2d47c2083a1778608253bbd437149375c539
2017-12-15Add for_each_subproj in LLVMSourceConfigDiana Picus
Add a method that runs an action on every subproject known by the source config. The action will be passed the subproject and a boolean indicating whether or not it is enabled. Change-Id: Ief7b67364143e40de103880fe1fefb4bdb4e93cf
2017-11-23Move --repos flag to projects subcommandDiana Picus
The --repos flag is not used by all the subcommands, so there's no reason for it to be compulsory. This commit moves it from the generic llvm.py options to the options for the projects subcommand, where it is only required when adding subprojects. Change-Id: I7ad10bda6e1b4efeb594b09e5fda200850d99c07
2017-11-03Minor fixes to scripts interfacesDiana Picus
* Remove all references to clang-tools-extra in helpers We do not support clang tools extra in the python scripts, since it's too complicated and nobody is working on it yet. Therefore, clean up all references to it in the helpers as well, to avoid any confusion. * Add a bit more info to llvm.py --help Change-Id: I5990925cebafce182b081227a4d33aecead7b4ac
2017-11-03llvm push: Print remote branch name after pushingDiana Picus
Get llvm.py push to print the name of the remote branch that it has pushed to. This is helpful so developers don't need to guess what namespaces have been prepended to the branch name, and also serves as quick visual confirmation. This required a bit of refactoring, since the name of the branch was computed when needed. We now compute it separately and pass it in to the function that pushes the branch. We rely on the fact that the branch name will be the same for all the subprojects involved. Most of the changes are mechanical, to account for this refactoring (including new names for some of the existing functions, which make more sense in this new context). Change-Id: Id7e496bcded60080803e3a850d73329428bc3bac
2017-10-31Replace git-push with llvm-pushDiana Picus
git-push is uninteresting and nobody uses it. Instead, we could use a wrapper over llvm.py push, which pushes the current branch for all enabled subprojects. Change-Id: I558bc41362d3ed14efdbe269041b467f5789f28e
2017-07-07[llvmpush] Add llvm pushDiana Picus
Add support for pushing the current branch in all linked subprojects in a given config. If they are not on the same branch, an error is thrown. The branches are pushed as linaro-local/$user/$branch, where $branch is the local name of the branch and $user is the user as seen in LLVM_GITUSER (we might want to change to a more intelligent way to get the user in the future). We achieve this by adding a new method to LLVMSourceConfig: for_each_enabled, which takes an action and applies it to all the enabled subprojects. If an exception is thrown by the action, it is rethrown by the action as a RuntimeError specifying which project the action was being applied to. Note that this does not behave like a transaction - the action may have been applied to some of the projects already, and there's no way to undo it. It's also a bit difficult to tell which projects the action was applied on (in the future we should probably log that info). We also provide an action that simply pushes the current branch to origin with the updated name. This makes sure the remote branch will live in an isolated namespace where it won't conflict with other people's branches. We also update some of the internals of LLVMSourceConfig to throw exceptions if some of the subprojects that are linked are not worktrees on the same branch as LLVM. In the past they were just ignored, but that doesn't seem like a sane default anymore. Change-Id: I9c917658d65b5d0e7bad3310efce07a0fe5bce5e
2017-06-12Factor out common test primitives. NFCI.Diana Picus
Create an LLVMTestCase class that will contain common helpers for writing tests for llvm.py subcommands. Change-Id: Ie1fe9dad9c9aeb9eb673726c80077d26739c0697
2017-05-12Set defaultbranch to python in .gitreview.Charles Baylis
Avoid reviews being accidentally submitted on the wrong branch by setting the default branch in the .gitreview file. Change-Id: I2b7571920ec03bdadce3e352b53a0e632a509342
2017-05-12Separate the repos from the environmentsDiana Picus
Before this patch, we had an LLVM_ROOT directory which contained the repos and a series of environments (worktrees + build dicretories). However, there is no good reason to have such a rigid layout, and in fact it prevents us from sharing the repos in any meaningful way. This patch removes LLVM_ROOT and instead introduces two command line parameters, repos and env, which represent the path to the directory containing the LLVM repositories and respectively the path to the environment that we intend to modify. The two paths may be completely independent. A side effect of this is that we can now also have different environments in different, unrelated locations (whereas previously they had to be siblings in LLVM_ROOT). For the time being, both parameters are compulsory (not only for llvm-projects, but for all subcommands). There may be situations where we can get away with only one of them (e.g. `llvm-projs <env>`, which just lists the projects enabled in the given <env> - it does not need to know where the repos are in order to do this). There may also be situations where one of them doesn't make any sense at all (e.g. `llvm-prepare`, which is not implemented yet in python, only needs the path to the repos and it would not know what to do with an environment). Making them optional would require some additional validation logic as well as more tests, and it feels a bit early on to introduce all that. It would be nice to have at least one or 2 more subcommands implemented before we invest time in it. Change-Id: Ibc1321d18476c2c4a65b5b05ca3fce3467d72fc3
2017-04-24Remove the need for LLVM_SRC in llvm.pyDiana Picus
Add an extra parameter to llvm.py representing the environment that we want the command to refer to. We then compute the path to the LLVM source tree based on that and LLVM_ROOT (which is still an environment variable for the time being). The bash helper keeps its old interface and uses LLVM_SRC to compute the environment that it will pass down to llvm.py. We also perform a small number of drive-by fixes to this helper (e.g. replacing python with python3). Change-Id: Ie7a33103969622294e158f83be8b9f57832ef1dc
2017-04-06Move to Python3Diana Picus
Most of the changes have been made by 2to3. Some manual fiddling was needed to make sure we're calling python3 instead of just python in the tests. Two of the helpers in the tests were reworked into proper methods rather than partials because we had to convert their returned value to str and this seems like the most straightforward/easy-to-read way. Change-Id: I74fdc1eaade3c026ee0c3e6bcdf26f8840f259b3
2016-12-15[llvmprojs] Rewrite llvm-projs in pythonDiana Picus
This is the first step in moving all our scripts to python. For now we keep the current content of the repo, but we'll start adding a new directory hierarchy for the python stuff: * modules: should contain most of the python code, organized as a package that can be imported by the scripts * scripts: the scripts themselves, which should be parsing the command line and calling stuff from 'modules' to do the actual work; can be broken down into the same categories we had before (helpers, bisect etc), or we could just have one big pile * tests: should contain unittests (for the stuff in modules) and command line interface tests (for the scripts) The code is heavily using functionality from the tcwg-release-tools repo (which probably needs to be renamed / reorganized), so you should have that in your PYTHONPATH when trying to run any of the scripts. To run the tests, just invoke check.sh. One of the important changes is that we'll be using python's argparse module to parse command line flags, which means we'll have to stick to a more traditional interface for the scripts. In particular, we can't have short options like "+c" anymore. This isn't much of a problem, because we will keep the bash scripts as they are and just make them invoke a tool written in python (scripts/llvm.py) to do the work. The tool will have subcommands for any functionality that we want, for instance the new interface for adding/removing subprojects is: llvm.py projects [-a subproject subproject ... subproject] [-r subproject ... subproject] The -a and -r options (followed by any number of subprojects) can be used to selectively enable/disable things. You have to pass the full name of the subproject (e.g. llvmprojs.py -a clang lld -r libcxx) for it to work. This is invoked by the llvm-projs bash script, which has the old, more convenient interface. For now the bash scripts will live in the same directories as they used to, but after the transition is complete we will want to move them to the scripts directory. Note that we're also eliding any dashes in the names of the scripts, in keeping with Python best practices for module names (i.e. using valid Python identifiers as names). Change-Id: I9ec08632dbb17464673240d6f6881a90f45d5371
2016-09-17Add LINK_JOBS logic to all three builder scriptsRenato Golin
The stress, helper and bisect build scripts needed the LLVM_LINK_JOBS option to make sure it doesn't run out of memory while linking, especially in SoCs that have a lot more cores than RAM (ex. HiKey). This patch calculates the jobs by seeing how much RAM the target has and adding +1, as link jobs rarely take more than 1GB, but the final ones do use several hundred. Also making sure we don't exceed the number of cores. A few other small changes: * Replace "else if" by "elif" * Build only minimal targets on ARM (even o full build) * Fixed a -j$PARALLEL bug introduced by a prvious commit * Converted a few backticks into $() Change-Id: If4f2189623338d2924357ecd3d4ee18feb522c32
2016-09-15[monitor] Add direct link to buildsRenato Golin
Adding a direct link to the buil on the build number, to avoid having to click on the bot page *then* the link. With the buildmaster *that* slow, this saves several seconds per lookup, which sometimes can be several dozens every day. Change-Id: If8cd783990042d9c51c318152b35d5b317cb5bf6
2016-09-15[llvm-build] Force lit tests to run in parallelDiana Picus
This is achieved by setting the CMake option LLVM_LIT_ARGS. If we don't do this, lit may decide that it only needs to run on one core. While we're at it, also update the run.sh that we use for bisects to force ninja to use the right number of cores while building. Change-Id: I099697f4a377541f907d8f13f8f807c2a4f901d6
2016-08-02[helpers] Teach llvm-env to self-hostRenato Golin
In addition to debug builds, this teaches llvm-env to create self-hosted bots that will use the "build" directory of the branch to compile a self-hosted version of the toolchain. It'll use clang and also lld, if available. Review changes: * Check for clang, not just build dir * Use -x instead of -f * Always export LLVM_SELFHOST * Quote strings for good measure * Using LLVM_CMAKE_FLAGS Change-Id: I55af0c213ae06cb8b99a33d1a80a05f09fe64c85
2016-07-22[llvm-build] Add -j optionDiana Picus
Allow the user to override the number of CPUs used for building LLVM. The default remains unchanged, so if you're not going to use the -j flag it won't affect you. Change-Id: I2491274a629a505744a80625381fb2c47bafd4da
2016-07-21Separate install directory for each worktreeDiana Picus
The install directory will be <Worktree>/install, right next to the build and source dirs, and it will be exported in LLVM_INSTALL. Note that this won't affect existing worktree directories, you should either nuke the build directories or manually update CMakeCache.txt for them. Change-Id: I8bb80f8d86d9d2f8d02940fbbd8a2c8b5b1a05e3
2016-06-29[llvm-env] List current worktree directories - ErrataDiana Picus
Forgot to add a patch set before merging (I don't know how I didn't notice that...). Errata for Change-Id: I787594ce1ecea0de68ceb0e518194b783da408b3 Change-Id: Ic3fc33cd643fd5d5f9a6d3280af9ab7c898227bc
2016-06-29[llvm-env] List current worktree directoriesDiana Picus
When running llvm-env without any arguments, it now dumps the current worktrees, as well as the branch that is currently checked out in them (only for llvm, to make it easy to spot places where this is different from the environment name). This is a very hacky implementation, because newer versions of git will come with git worktree list, which does exactly what we want. We can nuke this when we're all ready to move to newer gits. Change-Id: I787594ce1ecea0de68ceb0e518194b783da408b3
2016-06-16Allow people to clone from the RO repoDiana Picus
This is necessary in order to allow people outside TCWG to use our scripts Change-Id: Ia1855be8f5a45d6c8b070c782548b6e7bfb11335
2016-06-16Add option to build a certain target (and -h option)Diana Picus
This should allow us to run make/ninja on a given target, e.g. `llvm-build llc` will build only the llc target `llvm-build check-llvm-codegen-aarch64` will only run a subset of the tests This also adds a -h option. Change-Id: I4b2d195ed7f7f4b23f84438068dd1e13a094e947
2016-06-15Use git worktree in the LLVM helper scriptsDiana Picus
This commit adds a new script, llvm-env, which sets up the environment for working with LLVM. It looks for an environment variable LLVM_ROOT and tries to create the following hierarchy: $LLVM_ROOT `- repos | `- llvm | `- clang | `- compiler-rt | [...] `- <branch1> | `- llvm | `- build | `- debug `- <branch2> | `- llvm | `- build | `- debug [...] The $LLVM_ROOT/repos directory contains all the repositories, as checked out by llvm-prepare, and will always track master. For other branches, llvm-env <branch_name> will create a new directory, $LLVM_ROOT/<branch_name>, and will add an llvm worktree directory there. If -d is passed, it will also create a debug directory there, otherwise it will create a build directory. Notice that these 2 can live in parallel, and we can switch between them at any time by invoking llvm-env. It will set LLVM_SRC and LLVM_BLD accordingly, and also modify the path to point to the binaries in LLVM_BLD. The other scripts will now work with the LLVM_SRC and LLVM_BLD set by llvm-env in the current shell. Because llvm-env controls whether or not we're doing a debug build, llvm-build will no longer take a -d flag (it will instead look after a LLVM_DEBUG environment variable, also set by llvm-env). There are changes in llvm-projs, too, because now it no longer creates links - instead it creates worktree directories in the corresponding $LLVM_ROOT/<branch>/llvm. Other scripts have also been updated accordingly. To make things easier, here are some of the changes that I had to make that are not particularly important for the review (pretty mechanical stuff): * Moved function has() from llvm-branch to llvm-common, so I could reuse it * Because of this, I had to rename the has() function in llvm-projs to has_link(), which is actually a better name for it anyway * Disable the checks for LLVM_SRC and LLVM_BLD in llvm-common Change-Id: I9e02f6d8e0c803e79838845013b81331dffba99c
2016-06-15[monitor] moving 42vma bot back upstreamv1.0Renato Golin
2016-06-13[helpers] YCM helperRenato Golin
YouCompleteMe helper, updates the YCM config file once YCM-Generator creates one for you. The steps are: * Install YCM / Generator * Inside Vim, on $LLVM_SRC, run :YcmGenerateConfig * Outside vim, with llvm-env on the same workdir run: llvm-ycm * Re-start vim, all is good Why is this needed? 1. YCM relies on .ycm_extra_conf.py to be precise and correct 2. LLVM includes headers and libraries from both source and build dirs 3. YCM-Generator has no idea about the build dir, so it uses /tmp Note: * You will need to do this for each workdir, as the path has to be absolute * Don't use CMake's compilation_database.json, or header files won't work Change-Id: If14008d96d7198060cb403439db55c234693c837
2016-06-10[monitor] JSON documentationRenato Golin
Change-Id: Ifc143115fa3f6387af27ac4fddbc4bdaa1fca917
2016-06-08[release] Add -jN to test-release.shRenato Golin
Adding the number of cores to the -j option for test-release. Also, fix a bunch of shellcheck warnings. Change-Id: I54ee89534de265e764e95f5ebf094fc6cd1922c2