aboutsummaryrefslogtreecommitdiff
path: root/Makefile.programs
AgeCommit message (Collapse)Author
2015-09-30[test-suite] Don't allow FP_TOLERANCE and HASH_PROGRAM_OUTPUT to occur ↵Charlie Turner
toghether. Summary: LNT's benchmarks produce their output into files which are then compared in various ways. The benchmarks output numerical scores, which can then be compared by a custom program in the test-suite tree which does a floating-point comparison with a given tolerance (fpcmp). The test-suite system allows allows users to do a hash of the program output for a subsequently speedier comparison. This is used for benchmarks that generate a significant amount of output (currently defined as >100k), The problem is that some benchmarks that chose this feature also chose a floating-point comparison of these hashes, which at best is nonsense, and at worse causes an infinite loop. The fpcmp program is not resilient when it's input is weird like this. For a particular case of an observed mafft failure, it caused fpcmp to go into an infinite loop. Reviewers: kristof.beyls, ddunbar, rengolin Subscribers: rengolin, llvm-commits Differential Revision: http://reviews.llvm.org/D13283 git-svn-id: https://llvm.org/svn/llvm-project/test-suite/trunk@248898 91177308-0d34-0410-b5e6-96231b3b80d8
2015-08-26Add CC_Hash to simple report, containing a hash of the binary.Kristof Beyls
Now also with special casing to work around difference of strip command line syntax between linux and Darwin platforms. git-svn-id: https://llvm.org/svn/llvm-project/test-suite/trunk@246035 91177308-0d34-0410-b5e6-96231b3b80d8
2015-07-28Revert r243400.Adam Nemet
It break LNT on Darwin because strip does not support --remove-section. git-svn-id: https://llvm.org/svn/llvm-project/test-suite/trunk@243481 91177308-0d34-0410-b5e6-96231b3b80d8
2015-07-28Add CC_Hash to simple report, containing a hash of the binary.Kristof Beyls
git-svn-id: https://llvm.org/svn/llvm-project/test-suite/trunk@243400 91177308-0d34-0410-b5e6-96231b3b80d8
2014-10-16Update for llvm change.Rafael Espindola
git-svn-id: https://llvm.org/svn/llvm-project/test-suite/trunk@219968 91177308-0d34-0410-b5e6-96231b3b80d8
2014-05-13XCore target: Add target specific flags to root Makefiles.Robert Lytton
The default value of XCORE_TARGET_NEEDS_MEMORY is 32 (MB). It should be noted that the XCore emulator's speed is affected by the size of memory. Also the amount of host memory required is several times XCore-image memory size. Thus running several large xcore-images in parallel may result in thrashing! The default of 32 will require ~300MB of host memory. As Makefile.config.in is included multiple times (viz: include $(LEVEL)/Makefile.config), the addition of XCORE_TARGET_FLAGS to X_TARGET_FLAGS is done in Makefile.test RUNTIMELIMIT is set to a default of 1 hour for the XCore emulator. Test that take around or more than 1 hour to run have their RUNTIMELIMIT extended. git-svn-id: https://llvm.org/svn/llvm-project/test-suite/trunk@208667 91177308-0d34-0410-b5e6-96231b3b80d8
2014-02-14Add AArch64 port for test-suite.Jiangning Liu
git-svn-id: https://llvm.org/svn/llvm-project/test-suite/trunk@201382 91177308-0d34-0410-b5e6-96231b3b80d8
2013-10-11[projects/test-suite] Fix RUNSAFELY make variable definition.Jack Carter
To allow the RUNUNDER script to be run with parameters, the the definition of RUNSAFELY needs to quote the interpolation of RUNUNDER. Patch by Doug Gilmore git-svn-id: https://llvm.org/svn/llvm-project/test-suite/trunk@192485 91177308-0d34-0410-b5e6-96231b3b80d8
2013-08-02Allow for test suite to be run under Qemu and other host emulators.Reed Kotler
Patch by Doug Gilmore of Imagination technologies. git-svn-id: https://llvm.org/svn/llvm-project/test-suite/trunk@187691 91177308-0d34-0410-b5e6-96231b3b80d8
2013-05-09Fix use of EXECUTION_ENVIRONMENT_OVERRIDES.Daniel Dunbar
git-svn-id: https://llvm.org/svn/llvm-project/test-suite/trunk@181499 91177308-0d34-0410-b5e6-96231b3b80d8
2013-05-06Always enable hashed program output when using reference outputsHal Finkel
Because several of the reference outputs require hashed output, running with reference outputs but without hashed program output yields anomalous failures. git-svn-id: https://llvm.org/svn/llvm-project/test-suite/trunk@181260 91177308-0d34-0410-b5e6-96231b3b80d8
2013-04-26Add support for endian-specific reference outputs.Daniel Dunbar
- Use naming convention such as ".reference_output.big-endian" or, for type specific outputs, something like ".reference_output.big-endian.small". git-svn-id: https://llvm.org/svn/llvm-project/test-suite/trunk@180647 91177308-0d34-0410-b5e6-96231b3b80d8
2013-04-02Add a missing open-paren for $(TARGET_OS) reference.Will Schmidt
git-svn-id: https://llvm.org/svn/llvm-project/test-suite/trunk@178560 91177308-0d34-0410-b5e6-96231b3b80d8
2013-03-19Disable the darwin -force_cpusubtype_ALL parameter for Linux targets.Will Schmidt
git-svn-id: https://llvm.org/svn/llvm-project/test-suite/trunk@177405 91177308-0d34-0410-b5e6-96231b3b80d8
2012-11-02PGO: provide a Makefile and report generator for projects/test-suite toManman Ren
build programs with profiling instrumentation, run the program to generate profile data and then recompile while loading the profiling data. Patch by Alastair Murray with minor modifications from Manman to make it work with SPEC. git-svn-id: https://llvm.org/svn/llvm-project/test-suite/trunk@167270 91177308-0d34-0410-b5e6-96231b3b80d8
2012-10-29test-suite: Eliminate the EXIT_OK RunSafely parameter.Daniel Dunbar
- Exit codes already get compared for tests, and only a few tests actually bother to set this. git-svn-id: https://llvm.org/svn/llvm-project/test-suite/trunk@166905 91177308-0d34-0410-b5e6-96231b3b80d8
2012-10-29test-suite: Replace RunToolSafely with just another use of RunSafely (local).Daniel Dunbar
- We don't need multiple ways to run sandboxed tools. git-svn-id: https://llvm.org/svn/llvm-project/test-suite/trunk@166904 91177308-0d34-0410-b5e6-96231b3b80d8
2012-10-27test-suite: Update TEST=simple to show compile errors in log.Daniel Dunbar
git-svn-id: https://llvm.org/svn/llvm-project/test-suite/trunk@166845 91177308-0d34-0410-b5e6-96231b3b80d8
2012-04-19Remove llvm-ld from the test suite. This will be followed by a patch to LLVM toMichael J. Spencer
remove llvm-ld. git-svn-id: https://llvm.org/svn/llvm-project/test-suite/trunk@155145 91177308-0d34-0410-b5e6-96231b3b80d8
2012-04-16Remove vestiges of the C Backend from the test-suite.David Blaikie
Signed off by Eric Christopher. git-svn-id: https://llvm.org/svn/llvm-project/test-suite/trunk@154856 91177308-0d34-0410-b5e6-96231b3b80d8
2012-03-17[test-suite] Make report targets automatically build tools, too.Daniel Dunbar
git-svn-id: https://llvm.org/svn/llvm-project/test-suite/trunk@152996 91177308-0d34-0410-b5e6-96231b3b80d8
2012-03-17[test-suite] Change RunSafely to use the timeit --timeout option, instead ofDaniel Dunbar
TimedExec.sh. - Also eliminate NO_WATCHDOG option, and TimedExec.sh script itself. git-svn-id: https://llvm.org/svn/llvm-project/test-suite/trunk@152995 91177308-0d34-0410-b5e6-96231b3b80d8
2012-03-17[test-suite] Change to always use test-suite built fpcmp and timeit tools.Daniel Dunbar
git-svn-id: https://llvm.org/svn/llvm-project/test-suite/trunk@152992 91177308-0d34-0410-b5e6-96231b3b80d8
2012-03-17[test-suite] Eliminate GET_STABLE_NUMBERS and RunSafelyAndStable; this is ↵Daniel Dunbar
just a hack, LNT does this better. git-svn-id: https://llvm.org/svn/llvm-project/test-suite/trunk@152989 91177308-0d34-0410-b5e6-96231b3b80d8
2012-03-17[test-suite] We don't need to run DiffOutput under RunToolSafely anymore.Daniel Dunbar
git-svn-id: https://llvm.org/svn/llvm-project/test-suite/trunk@152988 91177308-0d34-0410-b5e6-96231b3b80d8
2012-03-17[test-suite] Remove unused TimeProgram.sh.Daniel Dunbar
git-svn-id: https://llvm.org/svn/llvm-project/test-suite/trunk@152987 91177308-0d34-0410-b5e6-96231b3b80d8
2012-02-23The nightly test scripts get confused by tests like siod that makeDuncan Sands
use of reference output because there is no accompanying time info for how long the native run took. Make such tools happy by creating fake timing info for such tests. git-svn-id: https://llvm.org/svn/llvm-project/test-suite/trunk@151258 91177308-0d34-0410-b5e6-96231b3b80d8
2012-02-23Add a *temporary* NO_WATCHDOG option to disable TimedExec.sh.Andrew Trick
This is not the right fix. TimedExec.sh needs to be replaced by adding a timeout option to the timeit tool. But, this has been sitting in my copy of the test-suite for months and I would like to be able to tell others to use the option. Disabling TimedExec.sh is the only way to get meaningful run times for short benchmarks. git-svn-id: https://llvm.org/svn/llvm-project/test-suite/trunk@151237 91177308-0d34-0410-b5e6-96231b3b80d8
2011-11-01On Debian testing and recent Ubuntu systems llvm-ld fails to find libc,Duncan Sands
causing every test in the nightly testsuite to fail. But llvm-ld is only using libc to work out which symbols to internalize, not for linking with it. How can libc effect which symbols are internalized? I can see two possibilities: either libc makes use of a function defined in the bitcode (which it clearly doesn't) or a weak function occurs both in the bitcode and in libc (in which case using the bitcode's copy should be OK). Thus there doesn't seem to any point in looking at libc and comparable system libraries (eg: libstdc++) when internalizing, and this is confirmed by my testing: not passing -lc or $(LIBS) when invoking llvm-ld doesn't cause any failures. So a simple fix for the Debian issue would be simply not to pass them to llvm-ld. But then the question arises: why use llvm-ld at all rather than "opt -std-link-opts"? This patch goes all the way and drops llvm-ld in favour of "opt -std-link-opts". It also removes some apparently pointless Makefile rules, which I suspect exist because llvm-ld produces two output files, X (a script) and X.bc (the bitcode). With this llvm-ld is not used by the nightly testsuite at all. It works for me, but since I don't know the nightly testsuite infrastructure well I may have broken something, so please review. git-svn-id: https://llvm.org/svn/llvm-project/test-suite/trunk@143460 91177308-0d34-0410-b5e6-96231b3b80d8
2011-04-30build: Add support for an EXECUTION_ENVIRONMENT_OVERRIDES variable, which canDaniel Dunbar
used to override various environment variables for test execution. Useful for tweaking things like dyld library paths, for example. git-svn-id: https://llvm.org/svn/llvm-project/test-suite/trunk@130578 91177308-0d34-0410-b5e6-96231b3b80d8
2011-04-25Update to use timeit-target when running with --without-llvm.Daniel Dunbar
git-svn-id: https://llvm.org/svn/llvm-project/test-suite/trunk@130143 91177308-0d34-0410-b5e6-96231b3b80d8
2011-04-14Add a "make build" target to the test-suite.Andrew Trick
I use this as follows make -j8 build TEST=simple ... make TEST=simple ... This is the only reasonable way I've come up with for manual benchmarking. I'm checking in since others have asked for the feature. git-svn-id: https://llvm.org/svn/llvm-project/test-suite/trunk@129530 91177308-0d34-0410-b5e6-96231b3b80d8
2011-04-14Proper whitespace removal.Andrew Trick
git-svn-id: https://llvm.org/svn/llvm-project/test-suite/trunk@129529 91177308-0d34-0410-b5e6-96231b3b80d8
2011-03-28Tidy up TEST=dbg reports.Devang Patel
git-svn-id: https://llvm.org/svn/llvm-project/test-suite/trunk@128423 91177308-0d34-0410-b5e6-96231b3b80d8
2011-02-23Adopt clang as the compiler.Devang Patel
Stay within 80 cols. git-svn-id: https://llvm.org/svn/llvm-project/test-suite/trunk@126329 91177308-0d34-0410-b5e6-96231b3b80d8
2011-02-09test-suite: Start sketching some tools.Daniel Dunbar
- timeit is just a simple replacement for /usr/bin/time which prints a whopping four digits of precision. That's right, FOUR! - fpcmp is just stubbed out at the moment. Yes, I will be reimplementing fpcmp in C. git-svn-id: https://llvm.org/svn/llvm-project/test-suite/trunk@125147 91177308-0d34-0410-b5e6-96231b3b80d8
2011-02-05test-suite: Allow fpcmp tool path to be specified.Daniel Dunbar
git-svn-id: https://llvm.org/svn/llvm-project/test-suite/trunk@124923 91177308-0d34-0410-b5e6-96231b3b80d8
2011-01-31Add support to generate TEST=dbg report.Devang Patel
git-svn-id: https://llvm.org/svn/llvm-project/test-suite/trunk@124606 91177308-0d34-0410-b5e6-96231b3b80d8
2011-01-28Add support to generate simple report for TEST=dbgopt.Devang Patel
git-svn-id: https://llvm.org/svn/llvm-project/test-suite/trunk@124493 91177308-0d34-0410-b5e6-96231b3b80d8
2010-12-06Add LLCOPTION to make it easier to test different llc options for llc and ↵Evan Cheng
llc-beta. git-svn-id: https://llvm.org/svn/llvm-project/test-suite/trunk@120983 91177308-0d34-0410-b5e6-96231b3b80d8
2010-12-03Add/implement "default" reference output, to make it easier to add tests whichDaniel Dunbar
just "exit 0". git-svn-id: https://llvm.org/svn/llvm-project/test-suite/trunk@120805 91177308-0d34-0410-b5e6-96231b3b80d8
2010-08-10Enable comparison optimizations for nightly beta tests for ARM.Bill Wendling
git-svn-id: https://llvm.org/svn/llvm-project/test-suite/trunk@110639 91177308-0d34-0410-b5e6-96231b3b80d8
2010-06-26Set -arm-code-placement as arm llcbeta option.Evan Cheng
git-svn-id: https://llvm.org/svn/llvm-project/test-suite/trunk@106936 91177308-0d34-0410-b5e6-96231b3b80d8
2010-06-22Tweak USE_REFERENCE_OUTPUT=1 implementation to not fall over when reference ↵Daniel Dunbar
outputs are missing. git-svn-id: https://llvm.org/svn/llvm-project/test-suite/trunk@106558 91177308-0d34-0410-b5e6-96231b3b80d8
2010-06-16Sigh, fix a shell quoting issue when calling DiffOutput under RunToolSafely.Daniel Dunbar
git-svn-id: https://llvm.org/svn/llvm-project/test-suite/trunk@106077 91177308-0d34-0410-b5e6-96231b3b80d8
2010-06-15Run DiffOutput/fpcmp under RunToolSafely, to allow testing old revisions whichDaniel Dunbar
unfortunately hit the fpcmp infinite loop when testing with Clang. git-svn-id: https://llvm.org/svn/llvm-project/test-suite/trunk@106035 91177308-0d34-0410-b5e6-96231b3b80d8
2010-06-02Allow setting of TARGET_FLAGS on the make command line, as is done for theJim Grosbach
nightly test target. Allows using other test targets for ARM more effectively. git-svn-id: https://llvm.org/svn/llvm-project/test-suite/trunk@105334 91177308-0d34-0410-b5e6-96231b3b80d8
2010-05-31Rename UPDATE_REFERENCE_OUTPUTS to UPDATE_REFERENCE_OUTPUT (for consistency ↵Daniel Dunbar
with USE_REFERENCE_OUTPUT). git-svn-id: https://llvm.org/svn/llvm-project/test-suite/trunk@105236 91177308-0d34-0410-b5e6-96231b3b80d8
2010-05-31Add support for having distinct reference outputs for SMALL_PROBLEM_SIZE.Daniel Dunbar
Also, add USE_REFERENCE_OUTPUT and HASH_PROGRAM_OUTPUT support for SPEC. Also, add an UPDATE_REFERENCE_OUTPUTS make variable, which copies the out-nat outputs to the reference_output files. git-svn-id: https://llvm.org/svn/llvm-project/test-suite/trunk@105210 91177308-0d34-0410-b5e6-96231b3b80d8
2010-05-31Fix a Makefile problem with TEST=simple.Daniel Dunbar
- What a wonderful language for a test harness. git-svn-id: https://llvm.org/svn/llvm-project/test-suite/trunk@105196 91177308-0d34-0410-b5e6-96231b3b80d8
generated by cgit v1.2.3 (git 2.25.1) at 2024-06-19 08:18:18 +0000