summaryrefslogtreecommitdiff
path: root/docs/TestingGuide.rst
diff options
context:
space:
mode:
authorGreg Parker <gparker@apple.com>2017-01-24 08:45:50 +0000
committerGreg Parker <gparker@apple.com>2017-01-24 08:45:50 +0000
commitebc14fff5261b02bba8525db3eea2af2ba0a5754 (patch)
tree31f2aced2941f51a62667afec2c0aa55ded39309 /docs/TestingGuide.rst
parentaeb6f08dc22af684eec3c868e042e6674644a859 (diff)
[lit] Allow boolean expressions in REQUIRES and XFAIL and UNSUPPORTED
A `lit` condition line is now a comma-separated list of boolean expressions. Comma-separated expressions act as if each expression were on its own condition line: For REQUIRES, if every expression is true then the test will run. For UNSUPPORTED, if every expression is false then the test will run. For XFAIL, if every expression is false then the test is expected to succeed. As a special case "XFAIL: *" expects the test to fail. Examples: # Test is expected fail on 64-bit Apple simulators and pass everywhere else XFAIL: x86_64 && apple && !macosx # Test is unsupported on Windows and on non-Ubuntu Linux # and supported everywhere else UNSUPPORTED: linux && !ubuntu, system-windows Syntax: * '&&', '||', '!', '(', ')'. 'true' is true. 'false' is false. * Each test feature is a true identifier. * Substrings of the target triple are true identifiers for UNSUPPORTED and XFAIL, but not for REQUIRES. (This matches the current behavior.) * All other identifiers are false. * Identifiers are [-+=._a-zA-Z0-9]+ Differential Revision: https://reviews.llvm.org/D18185 git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@292896 91177308-0d34-0410-b5e6-96231b3b80d8
Diffstat (limited to 'docs/TestingGuide.rst')
-rw-r--r--docs/TestingGuide.rst64
1 files changed, 36 insertions, 28 deletions
diff --git a/docs/TestingGuide.rst b/docs/TestingGuide.rst
index 26143febd07..99616770d8e 100644
--- a/docs/TestingGuide.rst
+++ b/docs/TestingGuide.rst
@@ -387,23 +387,49 @@ depends on special features of sub-architectures, you must add the specific
triple, test with the specific FileCheck and put it into the specific
directory that will filter out all other architectures.
-REQUIRES and REQUIRES-ANY directive
-~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
-Some tests can be enabled only in specific situation - like having
-debug build. Use ``REQUIRES`` directive to specify those requirements.
+Constraining test execution
+---------------------------
+
+Some tests can be run only in specific configurations, such as
+with debug builds or on particular platforms. Use ``REQUIRES``
+and ``UNSUPPORTED`` to control when the test is enabled.
+
+Some tests are expected to fail. For example, there may be a known bug
+that the test detect. Use ``XFAIL`` to mark a test as an expected failure.
+An ``XFAIL`` test will be successful if its execution fails, and
+will be a failure if its execution succeeds.
.. code-block:: llvm
- ; This test will be only enabled in the build with asserts
+ ; This test will be only enabled in the build with asserts.
; REQUIRES: asserts
+ ; This test is disabled on Linux.
+ ; UNSUPPORTED: -linux-
+ ; This test is expected to fail on PowerPC.
+ ; XFAIL: powerpc
+
+``REQUIRES`` and ``UNSUPPORTED`` and ``XFAIL`` all accept a comma-separated
+list of boolean expressions. The values in each expression may be:
-You can separate requirements by a comma.
-``REQUIRES`` means all listed requirements must be satisfied.
-``REQUIRES-ANY`` means at least one must be satisfied.
+- Features added to ``config.available_features`` by
+ configuration files such as ``lit.cfg``.
+- Substrings of the target triple (``UNSUPPORTED`` and ``XFAIL`` only).
+
+| ``REQUIRES`` enables the test if all expressions are true.
+| ``UNSUPPORTED`` disables the test if any expression is true.
+| ``XFAIL`` expects the test to fail if any expression is true.
+
+As a special case, ``XFAIL: *`` is expected to fail everywhere.
+
+.. code-block:: llvm
+
+ ; This test is disabled on Windows,
+ ; and is disabled on Linux, except for Android Linux.
+ ; UNSUPPORTED: windows, linux && !android
+ ; This test is expected to fail on both PowerPC and ARM.
+ ; XFAIL: powerpc || arm
-List of features that can be used in ``REQUIRES`` and ``REQUIRES-ANY`` can be
-found in lit.cfg files.
Substitutions
-------------
@@ -520,24 +546,6 @@ their name. For example:
This program runs its arguments and then inverts the result code from it.
Zero result codes become 1. Non-zero result codes become 0.
-Sometimes it is necessary to mark a test case as "expected fail" or
-XFAIL. You can easily mark a test as XFAIL just by including ``XFAIL:``
-on a line near the top of the file. This signals that the test case
-should succeed if the test fails. Such test cases are counted separately
-by the testing tool. To specify an expected fail, use the XFAIL keyword
-in the comments of the test program followed by a colon and one or more
-failure patterns. Each failure pattern can be either ``*`` (to specify
-fail everywhere), or a part of a target triple (indicating the test
-should fail on that platform), or the name of a configurable feature
-(for example, ``loadable_module``). If there is a match, the test is
-expected to fail. If not, the test is expected to succeed. To XFAIL
-everywhere just specify ``XFAIL: *``. Here is an example of an ``XFAIL``
-line:
-
-.. code-block:: llvm
-
- ; XFAIL: darwin,sun
-
To make the output more useful, :program:`lit` will scan
the lines of the test case for ones that contain a pattern that matches
``PR[0-9]+``. This is the syntax for specifying a PR (Problem Report) number