mirror of
https://git.lyx.org/repos/lyx.git
synced 2025-01-11 19:14:51 +00:00
ctest autotest documentation update
This commit is contained in:
Günter Milde
parent
26a3f46388
commit
3ec76c472e
@ -1462,6 +1462,12 @@ reference "subsec:Interpreting-export-tests"
|
|||||||
\end_layout
|
\end_layout
|
||||||
|
|
||||||
\begin_layout Paragraph
|
\begin_layout Paragraph
|
||||||
|
\begin_inset CommandInset label
|
||||||
|
LatexCommand label
|
||||||
|
name "par:ctest-options"
|
||||||
|
|
||||||
|
\end_inset
|
||||||
|
|
||||||
Running the tests
|
Running the tests
|
||||||
\end_layout
|
\end_layout
|
||||||
|
|
||||||
@ -1556,6 +1562,17 @@ status collapsed
|
|||||||
\end_inset
|
\end_inset
|
||||||
|
|
||||||
Exclude the tests whose labels match the given regular expression.
|
Exclude the tests whose labels match the given regular expression.
|
||||||
|
Cannot be combined with
|
||||||
|
\begin_inset Flex Code
|
||||||
|
status collapsed
|
||||||
|
|
||||||
|
\begin_layout Plain Layout
|
||||||
|
-L
|
||||||
|
\end_layout
|
||||||
|
|
||||||
|
\end_inset
|
||||||
|
|
||||||
|
.
|
||||||
\end_layout
|
\end_layout
|
||||||
|
|
||||||
\begin_layout Standard
|
\begin_layout Standard
|
||||||
@ -1613,7 +1630,7 @@ print-labels
|
|||||||
\end_layout
|
\end_layout
|
||||||
|
|
||||||
\begin_layout Standard
|
\begin_layout Standard
|
||||||
Other relevant options are:
|
Other useful options are:
|
||||||
\end_layout
|
\end_layout
|
||||||
|
|
||||||
\begin_layout Labeling
|
\begin_layout Labeling
|
||||||
@ -1913,45 +1930,6 @@ A good question is why do we enable the tests for non-default formats? The
|
|||||||
|
|
||||||
\end_layout
|
\end_layout
|
||||||
|
|
||||||
\begin_layout Standard
|
|
||||||
What action should you take if a test fails? This depends:
|
|
||||||
\end_layout
|
|
||||||
|
|
||||||
\begin_layout Standard
|
|
||||||
Generally, if a change breaks compilation for the target format (for the
|
|
||||||
manuals pdf2) without solving some important other issue, fix or revert
|
|
||||||
the commit that led to failure.
|
|
||||||
If a change breaks compilation for some non-target format (for the manuals
|
|
||||||
everything except pdf2), invert the test.
|
|
||||||
|
|
||||||
\end_layout
|
|
||||||
|
|
||||||
\begin_layout Standard
|
|
||||||
A special case is given, if the export was succeeding before but showing
|
|
||||||
garbled text in the PDF output.
|
|
||||||
Try to establish, that when the compilation succeeded before the resulting
|
|
||||||
PDF was good.
|
|
||||||
Otherwise, it is in fact an improvement when a test fails.
|
|
||||||
Now it might fail with a clear message of "language xyz not supported".
|
|
||||||
It is always good to check manually why something fails and if it passes
|
|
||||||
if the PDF output is good.
|
|
||||||
\end_layout
|
|
||||||
|
|
||||||
\begin_layout Standard
|
|
||||||
Sometimes a test is fixed as side-effect of some change.
|
|
||||||
We should uninvert a test (remove it from the
|
|
||||||
\begin_inset Flex Code
|
|
||||||
status collapsed
|
|
||||||
|
|
||||||
\begin_layout Plain Layout
|
|
||||||
suspiciousTests
|
|
||||||
\end_layout
|
|
||||||
|
|
||||||
\end_inset
|
|
||||||
|
|
||||||
file) in order to preserve the fix.
|
|
||||||
\end_layout
|
|
||||||
|
|
||||||
\begin_layout Standard
|
\begin_layout Standard
|
||||||
When a test or several tests fail, consider checking the files in the
|
When a test or several tests fail, consider checking the files in the
|
||||||
\begin_inset Flex Code
|
\begin_inset Flex Code
|
||||||
@ -2009,27 +1987,258 @@ CTestCostData.txt
|
|||||||
\end_layout
|
\end_layout
|
||||||
|
|
||||||
\begin_layout Paragraph
|
\begin_layout Paragraph
|
||||||
|
What action should you take if a test fails?
|
||||||
|
\end_layout
|
||||||
|
|
||||||
|
\begin_layout Standard
|
||||||
|
\paragraph_spacing single
|
||||||
|
It is always good to check manually why something fails and if it passes
|
||||||
|
if the PDF output is good.
|
||||||
|
\end_layout
|
||||||
|
|
||||||
|
\begin_layout Itemize
|
||||||
|
Generally, if a change breaks compilation for the target format (for the
|
||||||
|
manuals pdf2) without solving some important other issue,
|
||||||
|
\emph on
|
||||||
|
fix or revert the commit
|
||||||
|
\emph default
|
||||||
|
that led to failure.
|
||||||
|
\end_layout
|
||||||
|
|
||||||
|
\begin_layout Itemize
|
||||||
|
If it is not possible to (immediately) fix the failure but there are reasons
|
||||||
|
not to revert the commit (e.g.
|
||||||
|
it fixes another more important issue),
|
||||||
|
\emph on
|
||||||
|
invert
|
||||||
|
\emph default
|
||||||
|
the failing test case (see
|
||||||
|
\begin_inset CommandInset ref
|
||||||
|
LatexCommand ref
|
||||||
|
reference "par:Inverted-tests"
|
||||||
|
|
||||||
|
\end_inset
|
||||||
|
|
||||||
|
).
|
||||||
|
\end_layout
|
||||||
|
|
||||||
|
\begin_layout Itemize
|
||||||
|
If an
|
||||||
|
\emph on
|
||||||
|
inverted
|
||||||
|
\emph default
|
||||||
|
test case fails because the export now works,
|
||||||
|
\emph on
|
||||||
|
uninvert
|
||||||
|
\emph default
|
||||||
|
the test by removing the labeling pattern from
|
||||||
|
\begin_inset Quotes eld
|
||||||
|
\end_inset
|
||||||
|
|
||||||
|
suspiciousTests
|
||||||
|
\begin_inset Quotes erd
|
||||||
|
\end_inset
|
||||||
|
|
||||||
|
) (see
|
||||||
|
\begin_inset CommandInset ref
|
||||||
|
LatexCommand ref
|
||||||
|
reference "par:Inverted-tests"
|
||||||
|
|
||||||
|
\end_inset
|
||||||
|
|
||||||
|
).
|
||||||
|
\end_layout
|
||||||
|
|
||||||
|
\begin_layout Itemize
|
||||||
|
If the export did not fail previously but led to wrong output (PDF, say),
|
||||||
|
it is in fact an improvement when the test now fails, label it as
|
||||||
|
\begin_inset Quotes eld
|
||||||
|
\end_inset
|
||||||
|
|
||||||
|
unreliable:wrong:output
|
||||||
|
\begin_inset Quotes erd
|
||||||
|
\end_inset
|
||||||
|
|
||||||
|
(
|
||||||
|
\begin_inset CommandInset ref
|
||||||
|
LatexCommand ref
|
||||||
|
reference "par:Unreliable-tests"
|
||||||
|
|
||||||
|
\end_inset
|
||||||
|
|
||||||
|
).
|
||||||
|
\end_layout
|
||||||
|
|
||||||
|
\begin_layout Paragraph
|
||||||
|
\begin_inset CommandInset label
|
||||||
|
LatexCommand label
|
||||||
|
name "par:Inverted-tests"
|
||||||
|
|
||||||
|
\end_inset
|
||||||
|
|
||||||
Inverted tests
|
Inverted tests
|
||||||
\end_layout
|
\end_layout
|
||||||
|
|
||||||
\begin_layout Standard
|
\begin_layout Standard
|
||||||
These tests fail if the export does
|
Test cases whose name matches a pattern in the file
|
||||||
|
\begin_inset Flex Code
|
||||||
|
status collapsed
|
||||||
|
|
||||||
|
\begin_layout Plain Layout
|
||||||
|
development/autotests/suspiciousTests
|
||||||
|
\end_layout
|
||||||
|
|
||||||
|
\end_inset
|
||||||
|
|
||||||
|
get the label
|
||||||
\emph on
|
\emph on
|
||||||
not
|
inverted
|
||||||
\emph default
|
\emph default
|
||||||
return an error.
|
.
|
||||||
|
They get also the test property
|
||||||
|
\begin_inset Flex Code
|
||||||
|
status collapsed
|
||||||
|
|
||||||
|
\begin_layout Plain Layout
|
||||||
|
WILL_FAIL
|
||||||
|
\end_layout
|
||||||
|
|
||||||
|
\end_inset
|
||||||
|
|
||||||
|
, i.e.
|
||||||
|
they are reported as failing if the export works without error
|
||||||
|
\begin_inset Flex URL
|
||||||
|
status collapsed
|
||||||
|
|
||||||
|
\begin_layout Plain Layout
|
||||||
|
|
||||||
|
https://cmake.org/cmake/help/v3.0/command/set_tests_properties.html
|
||||||
|
\end_layout
|
||||||
|
|
||||||
|
\end_inset
|
||||||
|
|
||||||
|
.
|
||||||
|
\end_layout
|
||||||
|
|
||||||
|
\begin_layout Standard
|
||||||
|
Add failing cases to this file, if they cannot be solved
|
||||||
|
\begin_inset Quotes eld
|
||||||
|
\end_inset
|
||||||
|
|
||||||
|
immediately
|
||||||
|
\begin_inset Quotes erd
|
||||||
|
\end_inset
|
||||||
|
|
||||||
|
but it is expected that the export will work in a foreseeable future, e.g.
|
||||||
|
low priority issues like failures to export to a non-target format (for
|
||||||
|
the manuals everything except pdf2).
|
||||||
|
\end_layout
|
||||||
|
|
||||||
|
\begin_layout Standard
|
||||||
|
The following sublabels are currently present in
|
||||||
|
\begin_inset Flex Code
|
||||||
|
status collapsed
|
||||||
|
|
||||||
|
\begin_layout Plain Layout
|
||||||
|
suspiciousTests
|
||||||
|
\end_layout
|
||||||
|
|
||||||
|
\end_inset
|
||||||
|
|
||||||
|
:
|
||||||
\end_layout
|
\end_layout
|
||||||
|
|
||||||
\begin_layout Description
|
\begin_layout Description
|
||||||
inverted Export fails, but the test cases are subject to be examined.
|
todo test failures that require attention:
|
||||||
It is expected that the export will work in a foreseeable future.
|
\end_layout
|
||||||
They are labeled 'inverted'.
|
|
||||||
|
\begin_deeper
|
||||||
|
\begin_layout Itemize
|
||||||
|
minor issues to explore and properly sort later,
|
||||||
|
\end_layout
|
||||||
|
|
||||||
|
\begin_layout Itemize
|
||||||
|
easyfix issues,
|
||||||
|
\end_layout
|
||||||
|
|
||||||
|
\begin_layout Itemize
|
||||||
|
LyX bugs to report at trac (move pattern to section "lyxbugs" once done).
|
||||||
|
\end_layout
|
||||||
|
|
||||||
|
\end_deeper
|
||||||
|
\begin_layout Description
|
||||||
|
lyxbugs LyX bugs with a Trac number.
|
||||||
\end_layout
|
\end_layout
|
||||||
|
|
||||||
\begin_layout Description
|
\begin_layout Description
|
||||||
suspended Some inverted tests are labeled 'suspended'.
|
ert Export failures due to "raw" LaTeX use in ERT or preamble code.
|
||||||
This means, they are not executed using
|
\end_layout
|
||||||
|
|
||||||
|
\begin_deeper
|
||||||
|
\begin_layout Standard
|
||||||
|
"Wontfix" if demonstrating correct use and OK in the default output format.
|
||||||
|
\end_layout
|
||||||
|
|
||||||
|
\end_deeper
|
||||||
|
\begin_layout Description
|
||||||
|
texissues Export fails due to LaTeX limitations like non-ASCII characters
|
||||||
|
in verbatim or listings, incompatible packages, ...
|
||||||
|
\end_layout
|
||||||
|
|
||||||
|
\begin_deeper
|
||||||
|
\begin_layout Standard
|
||||||
|
"Wontfix" if documents demonstrate correct use in the default output format:
|
||||||
|
\end_layout
|
||||||
|
|
||||||
|
\begin_layout Itemize
|
||||||
|
If the source can be made more robust without becoming "hackish", fix the
|
||||||
|
source,
|
||||||
|
\end_layout
|
||||||
|
|
||||||
|
\begin_layout Itemize
|
||||||
|
if LyX could be enhanced to care for a permanent TeX limitation, file a
|
||||||
|
ticket at trac and add a pattern under lyxbugs,
|
||||||
|
\end_layout
|
||||||
|
|
||||||
|
\begin_layout Itemize
|
||||||
|
otherwise, add a pattern here.
|
||||||
|
\end_layout
|
||||||
|
|
||||||
|
\end_deeper
|
||||||
|
\begin_layout Description
|
||||||
|
attic Documents in the attic.
|
||||||
|
(Kept for reference and format conversion test.)
|
||||||
|
\end_layout
|
||||||
|
|
||||||
|
\begin_layout Subparagraph
|
||||||
|
suspended
|
||||||
|
\end_layout
|
||||||
|
|
||||||
|
\begin_layout Standard
|
||||||
|
Test cases whose name additionally matches a pattern in the file
|
||||||
|
\begin_inset Flex Code
|
||||||
|
status collapsed
|
||||||
|
|
||||||
|
\begin_layout Plain Layout
|
||||||
|
development/autotests/suspendedTests
|
||||||
|
\end_layout
|
||||||
|
|
||||||
|
\end_inset
|
||||||
|
|
||||||
|
get the label
|
||||||
|
\emph on
|
||||||
|
suspended
|
||||||
|
\emph default
|
||||||
|
(instead of
|
||||||
|
\emph on
|
||||||
|
export
|
||||||
|
\emph default
|
||||||
|
and
|
||||||
|
\emph on
|
||||||
|
inverted
|
||||||
|
\emph default
|
||||||
|
).
|
||||||
|
This means they are not executed using
|
||||||
\begin_inset Flex Code
|
\begin_inset Flex Code
|
||||||
status collapsed
|
status collapsed
|
||||||
|
|
||||||
@ -2050,6 +2259,18 @@ ctest -L inverted
|
|||||||
\end_inset
|
\end_inset
|
||||||
|
|
||||||
.
|
.
|
||||||
|
However, they also get the test property
|
||||||
|
\begin_inset Flex Code
|
||||||
|
status collapsed
|
||||||
|
|
||||||
|
\begin_layout Plain Layout
|
||||||
|
WILL_FAIL
|
||||||
|
\end_layout
|
||||||
|
|
||||||
|
\end_inset
|
||||||
|
|
||||||
|
, i.e.
|
||||||
|
they are reported as failing if the export works without error.
|
||||||
From time to time they still have to be checked using
|
From time to time they still have to be checked using
|
||||||
\begin_inset Flex Code
|
\begin_inset Flex Code
|
||||||
status collapsed
|
status collapsed
|
||||||
@ -2063,7 +2284,6 @@ ctest -L suspended
|
|||||||
.
|
.
|
||||||
\end_layout
|
\end_layout
|
||||||
|
|
||||||
\begin_deeper
|
|
||||||
\begin_layout Standard
|
\begin_layout Standard
|
||||||
These tests are suspended, because the export fails for known reasons which
|
These tests are suspended, because the export fails for known reasons which
|
||||||
cannot ATM be resolved.
|
cannot ATM be resolved.
|
||||||
@ -2084,15 +2304,45 @@ status collapsed
|
|||||||
\end_inset
|
\end_inset
|
||||||
|
|
||||||
parameter nothing changes.
|
parameter nothing changes.
|
||||||
Suspended or not, tests will be executed depending only on the regexes
|
Suspended or not, tests will be executed depending only on the selectiong
|
||||||
parameters given to the ctest command.
|
regular expression given to the ctest command (see
|
||||||
|
\begin_inset CommandInset ref
|
||||||
|
LatexCommand ref
|
||||||
|
reference "par:ctest-options"
|
||||||
|
|
||||||
|
\end_inset
|
||||||
|
|
||||||
|
).
|
||||||
\end_layout
|
\end_layout
|
||||||
|
|
||||||
\end_deeper
|
|
||||||
\begin_layout Paragraph
|
\begin_layout Paragraph
|
||||||
|
\begin_inset CommandInset label
|
||||||
|
LatexCommand label
|
||||||
|
name "par:Unreliable-tests"
|
||||||
|
|
||||||
|
\end_inset
|
||||||
|
|
||||||
Unreliable tests
|
Unreliable tests
|
||||||
\end_layout
|
\end_layout
|
||||||
|
|
||||||
|
\begin_layout Standard
|
||||||
|
Test cases whose name matches a pattern in the file
|
||||||
|
\begin_inset Flex Code
|
||||||
|
status collapsed
|
||||||
|
|
||||||
|
\begin_layout Plain Layout
|
||||||
|
development/autotests/unreliableTests
|
||||||
|
\end_layout
|
||||||
|
|
||||||
|
\end_inset
|
||||||
|
|
||||||
|
get the label
|
||||||
|
\emph on
|
||||||
|
unreliable
|
||||||
|
\emph default
|
||||||
|
.
|
||||||
|
\end_layout
|
||||||
|
|
||||||
\begin_layout Standard
|
\begin_layout Standard
|
||||||
These tests are not executed using
|
These tests are not executed using
|
||||||
\begin_inset Flex Code
|
\begin_inset Flex Code
|
||||||
@ -2121,11 +2371,24 @@ ctest -L inverted
|
|||||||
\begin_layout Standard
|
\begin_layout Standard
|
||||||
They pass or fail for various reasons not related to LyX (nonstandard, erratic)
|
They pass or fail for various reasons not related to LyX (nonstandard, erratic)
|
||||||
or pass but should rather fail (wrong output).
|
or pass but should rather fail (wrong output).
|
||||||
|
|
||||||
|
\begin_inset Note Note
|
||||||
|
status collapsed
|
||||||
|
|
||||||
|
\begin_layout Plain Layout
|
||||||
|
*invalid* tests (wrong output) are not *unreliable*.
|
||||||
|
# Use "unfit" or "unapplicable" as better label and name of pattern file?
|
||||||
|
|
||||||
|
\end_layout
|
||||||
|
|
||||||
|
\end_inset
|
||||||
|
|
||||||
|
|
||||||
\end_layout
|
\end_layout
|
||||||
|
|
||||||
\begin_layout Description
|
\begin_layout Description
|
||||||
nonstandard Documents with additional requirements, e.g.
|
nonstandard Documents with additional requirements, e.g.
|
||||||
a class or package file not on CTAN.
|
a class or package file not in TeXLive.
|
||||||
|
|
||||||
\begin_inset Note Note
|
\begin_inset Note Note
|
||||||
status open
|
status open
|
||||||
@ -2159,7 +2422,7 @@ TODO: use
|
|||||||
\emph on
|
\emph on
|
||||||
erratic
|
erratic
|
||||||
\emph default
|
\emph default
|
||||||
only for the phase-of-moon dependency.
|
only for the phase-of-moon dependency?
|
||||||
\end_layout
|
\end_layout
|
||||||
|
|
||||||
\end_inset
|
\end_inset
|
||||||
@ -2186,6 +2449,11 @@ output Export does not fail but the resulting document has errors.
|
|||||||
\begin_deeper
|
\begin_deeper
|
||||||
\begin_layout Standard
|
\begin_layout Standard
|
||||||
\paragraph_spacing single
|
\paragraph_spacing single
|
||||||
|
\begin_inset Note Note
|
||||||
|
status open
|
||||||
|
|
||||||
|
\begin_layout Plain Layout
|
||||||
|
\paragraph_spacing single
|
||||||
These tests are actually not
|
These tests are actually not
|
||||||
\emph on
|
\emph on
|
||||||
unreliable
|
unreliable
|
||||||
@ -2197,6 +2465,11 @@ invalid
|
|||||||
(not measuring what they should measure).
|
(not measuring what they should measure).
|
||||||
\end_layout
|
\end_layout
|
||||||
|
|
||||||
|
\end_inset
|
||||||
|
|
||||||
|
|
||||||
|
\end_layout
|
||||||
|
|
||||||
\end_deeper
|
\end_deeper
|
||||||
\begin_layout Paragraph
|
\begin_layout Paragraph
|
||||||
Export test filtering
|
Export test filtering
|
||||||
@ -3223,40 +3496,6 @@ Associated test results can be examined in ctest-log directory in files
|
|||||||
of the form 'LastFailed.*URLS.log'
|
of the form 'LastFailed.*URLS.log'
|
||||||
\end_layout
|
\end_layout
|
||||||
|
|
||||||
\begin_layout Subsubsection
|
|
||||||
Test labels
|
|
||||||
\end_layout
|
|
||||||
|
|
||||||
\begin_layout Standard
|
|
||||||
ctest label commands:
|
|
||||||
\end_layout
|
|
||||||
|
|
||||||
\begin_layout Description
|
|
||||||
\SpecialChar nobreakdash
|
|
||||||
\SpecialChar nobreakdash
|
|
||||||
print-labels shows all assigned labels
|
|
||||||
\end_layout
|
|
||||||
|
|
||||||
\begin_layout Description
|
|
||||||
\SpecialChar nobreakdash
|
|
||||||
L
|
|
||||||
\begin_inset space ~
|
|
||||||
\end_inset
|
|
||||||
|
|
||||||
labelname executes all tests to which this label is assigned to.
|
|
||||||
A test may have more that one label.
|
|
||||||
\end_layout
|
|
||||||
|
|
||||||
\begin_layout Description
|
|
||||||
\SpecialChar nobreakdash
|
|
||||||
j
|
|
||||||
\begin_inset space ~
|
|
||||||
\end_inset
|
|
||||||
|
|
||||||
number executes tests in parallel using 'number' simultaneously processes.
|
|
||||||
Some tests are marked as 'sequencial', for them this parameter has no effect.
|
|
||||||
\end_layout
|
|
||||||
|
|
||||||
\begin_layout Section
|
\begin_layout Section
|
||||||
Development policies
|
Development policies
|
||||||
\end_layout
|
\end_layout
|
||||||
|
|||||||
Loading…
Reference in New Issue
Block a user