Move them to a subdir, ignore this subdir for other tests.
Dedicated test samples for LaTeX-specific problems don't give additional value if tested for loading, conversion, or other exports.
This led to errors when compiling with polyglossia (and non-TeX fonts).
A minimal (currently non-compiling) test sample is kept in autotests/export/
and inverted in suspiciousTests.
europeCV and modernCV examples can now be exported to PDF using
LuaTeX. For the specific output that was fixed, look at the diff and
see the description in suspiciousTests that was removed by this
commit. The output was checked manually and appears fine. These
tests are thus "uninverted".
Exporting those examples to DVI with LuaTeX does not exit with
error, but the output drops characters with accents. Thus, these
tests are now marked with the "wrong_output" label. I reported this
issue on the LuaTeX mailing list at [1], but since DVI export is not
given high priority, I don't expect much action.
Note that these changes reflect an updated TeX Live 2016
installation.
[1]
https://www.mail-archive.com/search?l=mid&q=20160831134006.4fewxothddqfeyw4%40steph
As of 0b1cf133 we now warn in the GUI of this issue, but there is a
discussion about whether we should change our LaTeX output and allow
for the workflow of mixing inTitle layouts. For more information,
see #10347.
The keytests were previously enabled by default if the necessary
dependencies were found. They require a GUI and mouse so can
sometimes be annoying. Further, they are not currently reliable.
They are thus now disabled by default.
XeTeX with TeX fonts is only safe with ASCII input encoding (see #9740)
and we therefore force "ascii" when exporting with XeTeX and 8-bit TeX-fonts.
However, "utf8-plain" is a "power-user" option, which allows to switch off LyX's
encoding of the LaTeX file:
keep this also for "XeTeX with TeX fonts".
The user is responsible to ensure all characters can be processed and are
correctly shown in the output. The provided test sample shows the problems
with this encoding without special measures (like loading fontspec in the
user-preamble or a document class).
In collaboration with Günter Milde:
1.) Allow char ':' be part of a ctest-label
2.) Eliminate redundant label naming and directory names
(The testnames should not repeat the directory name)
When updating to the latest TeX Live revision, these exports now
succeed.
Inspecting the differences between the "good" PDF and the "bad" PDF
(where the test failed) there are three differences, which can be
found on the printed page numbers 81 and 82 (PDF page numbers 91 and
92). The accented i in "lím" was printed incorrectly (and the
missing glyph was correctly detected). After the update the
character is printed correctly and there is no longer an error.
The package that caused the change is likely babel or babel-spanish.
You can see the log from the TeX Live update that caused the fix by
seeing the attachment to the message here:
https://www.mail-archive.com/search?l=mid&q=20160313015902.lny3g5aujh4c4aps%40scott-Za1510
It could happen that the variable was set in creating a previous test-case.
Some combinations in the controlling files (suspiciousTests, unrelibleTests, ...)
did not set this variable.
* sort and comment remaining unsorted filter patterns in suspiciousTests
(mostly just move to TODO and state what needs to be done).
* comment the patterns for achemso and Math,
(LuTeX incompatibility fixed in TeXLive 2016-01-07).
Simplify the logic for language package selection and make it more consistent:
Use polyglossia with non-TeX fonts (system fonts/Unicode fonts) for all
export flavours (XeTeX, LuaTeX, DVI-LuaTeX), if the language package setting
is "auto" and there is no language not supported by Babel and no package
providing Babel.
This solves some Babel-related autotest cases and leads to some new failures
due to the polyglossia language nesting problem.
* missing characters in linguistics example with system fonts
* language nesting problem in fr/linguistics example
* 110f505b63 solved one failure (Basque example file)
Works with system-fonts due to font substitution in the test-script.
TODO: set system fonts that work in the lyx source(s) to allow compilation
"by hand".
testcases_speed_lyx21 added to suspiciousTests
Handling unknown body token: `\begin_manifest'at line ...
Literate_lyx16 and noweb2lyx_lyx16 added to ignoredTests
Plenty of warnings (~ 390)
New sublabels LyXBug, ERT.
Renamed sublabel TeXLimit to TeXissue.
More sorting and unification of regular expressions.
Removing double entries.
Fixes for some expressions (eu_UserGuide_pdf, examples/ja/(knitr|lilypond|sweave))
Since the stop condition is that the last two consecutive created
files are identical, we do not need to check the last file.
Also checking for load of the created files is not needed, because
exporting the previous file implicitly loads too.
Exporting to some previous lyx format:
Inside the loop
Stop at export with errors/warnings
Stop if exported file not loadable
Stop if exported file identical to previous exported file (OK case)
This is done through creating MD5SUM
We test here, if any \end_{somethig} matches \begin_{something}.
Exeptions are \end_index and \end_branch ATM, they should
match \index and \branch respectively.
Also added new testfile.
Remove patterns matching tests now ignored (CJK/Korean, latex8).
Move patterns matching documents in the attic under new sublabel.
Add more verbose comments.
Comment by Günter Milde:
Actually, *all* Spanish manuals either fail or have wrong output with
Unicode TeX engines and 8-bit fonts. The reason is known: a bug in Babel
that uses utf8 strings whenever Xe- or LuaTeX is detected.
Always-Babel now set in the example document.
Adapted the autotest categorization:
* fails for some developers (why?)
* wrong output with pdflatex/LuaTeX and DVI (missing landscape slides).
Previously all labels got the depth '7' while processing 'suspiciousTestss'.
The depth is used to sort how our labels are used to build a test-label.
Say a test gets label a, b, and c, with depth 3, 8 and 2.
The constructed test-label will be "b🅰️c"
For new (to be implemented) 'reason-labels' it is more convenient to assign them higher values.
suspiciousTests: Remove Tutorial from the regex
suspendedTests: Explicit list of languages needed
ignoredTests: Language-string is separated from other strings only by '/'. '|', '_', '(' and ')'
LyXMacros.cmake: Overseen the macro 'setmarkedtestlabel()' which added it automatically
if the test was to invert the test result.
ExportTests.cmake: Correct label handling
This works around a limitation of the test machinery, which never switches
TeX fonts on for format that need that, it only switches TeX fonts off for
formats needing it.
Thanks to Kornel we do now have the infrastructure for running dedicated
export tests. This is the first one, showing a language nesting bug which is
already in 2.1. It is inverted for now, but this will hopefully change soon.
and introduce sublabels
Sublabels section in *.Tests starts with 'Sublabel: name'
and is valid until start of a new sublabel.
'name' contains only ascii characters [a-z]+
To test all export use 'ctest -L export'. This is unchanged.
The following lists directories and assigned sub-label
lib/doc: manuals
lib/templates: templates
lib/examples: examples
development/mathmacro: mathmacros
autotests: autotests
Now
'ctest -L export' should be without errors
If there _are_ errors, the appropriate test should go to nonstandardTests
'ctest -L reverted' should be without errors
If there _are_ errors, the appropriate test should go to nonstandardTests
'ctest -L nonstandard'
Tests here may, or may not fail. Depends of installed tex extensions
All comments in revertedTests comes from Günter Milde
Non standard test is new. It should collect all tests
which may not work because of some missing non-standard
tex package or some exotic system font.
All dvi_texF and pdf5_texF testcases are nov candidates for being suspended.
After commit 279d084, now export/.*/Math_(dvi3|pdf5)_texF fails,
therefore the tests are also inverted.
This file holds regular expression to select which of the inverted
export test cases will be suspended.
These tests will not be executed with the call 'ctest -L export'.
These exports correctly fail now that we've switched to polyglossia.
Although they compiled without error with babel, the resulting PDFs
had gibberish.
I believe these tests were fixed by TeX Live updates.
Comparing the log files for a system where the tests fail with a
system where the tests pass, below are some of the differences
between the "good" and "bad" logs:
bad:
LaTeX2e <2015/01/01>
Babel <3.9l>
Package: fontspec 2015/03/14 v2.4c
Package: expl3 2015/03/01 v5547 L3 programming layer
good:
LaTeX2e <2015/01/01> patch level 2
Babel <3.9m>
Package: fontspec 2015/07/22 v2.4d
Package: expl3 2015/07/30 v5724 L3 programming layer
Many of our documents have babel-specific preamble code. By putting
this code in a \@ifpackageloaded{babel}{}{} conditional, XeTeX and
LuaTeX compilation with polyglossia now works. This fixes some
LuaTeX tests that were broken by edd37de8 and also allows us to
uninvert some XeTeX tests.
Note that in some of the files although the preambles were fixed to
allow for polyglossia, they still do not compile without errors:
es/Math.lyx
es/Customization.lyx
de/Customization.lyx
Similar fixes might be desired in other manuals but these at least
fix regressions in the tests.
PolyglossiaOpts are case-sensitive so "latin" must be changed to
"Latin". Without this change, compiling examples/sr/Braille.lyx
with LuaTeX and system fonts gives the following error:
Package Polyglossia Error: Unknown script `latin' for Serbian
language
Many of our Spanish documents use babel-specific features in the
documents, e.g. to write "sin" in Spanish ("sen"). Because babel
seems to have good support for Spanish, I am setting the "Always
babel" for the manuals.
This fixes several LuaTeX tests with non-TeX fonts. A XeTeX test is
also reverted accordingly.
We do not currently have any Arabic documents, but when we do, these
fonts will be used for the tests that use system fonts.
They can be installed on Debian-type systems from the
'fonts-sil-scheherazade' package.
The bug that caused these tests to enter into infinite loops
(and have a memory leak) has been fixed. These tests now
pass on an updated TeX Live 2014 as well as the current TeX
Live 2015 pretest.
One of the tests is also disabled for es/Math.lyx. However,
the other test passes for es/Math.lyx. To reproduce,
open fr/Math.lyx, click "Use non-TeX fonts", and choose
e.g. "FreeSans" for the three fonts. Then view as PDF (LuaTeX).
I put a note to look into why this one fails.
According to fix#9432, Scott found a new crash
1. start a new LyX document, english GUI
2. alt+m f to create a fraction
3. alt+m r to insert a root
4. undo
This should prevent possible errors for tests running in parallel.
We now also check for created export file.
complainer/feature requester: Scott Kostyshak
If the LyX window does not appear after a certain (conservative)
time, the keytest now fails. One reason that caused the LyX window
to never show up was when DISPLAY was not set. This particular case
has been fixed in <commit1hash>. This commit is meant for the more general
scenario.
(Collaboration with Scott Kostyshak)
Before, the keytests would run infinitely if the
tests were run, e.g. through SSH without X forwarding.
To reproduce that problem, run
DISPLAY='' ctest -R hello
This commit causes the test to correctly fail in such situations.
(Collaboration with Scott Kostyshak)
The configuration time suffers mostly on checking, which of the export tests
is to be reverted.
1.) There is a new configuration flag now, "LYX_ENABLE_EXOPRT_TESTS.
If not set (default) no export tests are created.
2.) If set, then the optimization halves the time needed for creation of tests.
The effect on my side:
a.) Until now the time was: ~ 26 seconds
b.) The optimized time is now: ~ 16 seconds
c.) With not enabled export tests: ~ 5 seconds
In collaboration with Scott Kostyshak:
With Qt 4 we could use Q_WS_X11, defined by FindQt4.cmake. In Qt 5,
there is no FindQt5.cmake. Instead we now define our own variable,
QT_USES_X11 by using class QX11Info available only on X11.
(http://qt-project.org/doc/qt-5/QX11Info.html)
The main consequence of this improved support is that now the keytests
can be run (ctest -R "keytest") when LyX is compiled with Qt 5.
Before, with Qt 5 we did not know if X11 was available, which is
needed by xvkbd, so the tests were not enabled. Note, however, that
many tests fail with Qt version 5.2.1 because there was a change in
the event handling mechanism in Qt that causes xvkbd to be unable to
pass capital letters (so case sensitive greps in the tests fail). This
needs to be investigated and reported.
These started failing after we implemented tests for formats
that are in the complement set to the default format (7ecbb068).
It might be worth it in the future to take a look at each individually
and see whether they are supposed to fail or if there is something LyX
can do to add support for exporting them.
Define new policy of what export formats to check.
Depending on default_output_format
default or unset xhtml, dvi*, pdf*
xhtml xhtml
any of pfd variants xhtml, pdf*
any of dvi variants xhtml, dvi*
each other (like eps3) as specified
This test was broken after wrap-around search was implemented. From
what I understand, the point of the test is that search should _not_
match in this case because "ignore format" is checked by default.
Thus, a new way of testing that there was no match is to check whether
the wrap around message shows up, because the cursor is initially at
the beginning so that would mean that the entire buffer was checked.
I confirmed that the test now works by checking that it passes and
checking that it fails if "ignore format" is unchecked. Although this
test is still fragile, it is fragile in a conservative way (i.e. it
will give false failures rather than false passes). It will fail if
the Italian translation changes, if "ignore format" is unchecked by
default, or if the wrap-around mechanism changes (e.g. does not ask if
the cursor started at the beginning of the buffer).
The JASATeX class is currently unmaintained. Also, this
commit moves the system font tests from inverted to ignored
(otherwise lualatex and xelatex run in infinite loops).
For many of these XeTeX or LuaTeX does not yet support
using TeX fonts for certain languages. The others fail
because, as Jürgen explains, they have excessive preamble
code that is only targeted at (pdf)latex.
Citing Scott:
In our current set up, we are currently testing XeTeX and LuaTeX
either with system fonts or with TeX fonts but never both. We should
test with both in my opinion. We will have to ignore/invert many tests
but it still seems useful. For example Günter fixed babel-greek so
that it works now with TeX fonts; and Jürgen found some errors in LyX
that were causing some of the English docs to fail with system fonts.
Currently we only test greek documents with system fonts and we only
test English documents with TeX fonts.
This change adds the missing test-cases.
See (thanks to Uwe for the link):
ccb0e9e2c6
We thus invert the LuaTeX Farsi tests.
All inverted tests now have explanations for why they are not
currently expected to work.
We can now test for regressions in the Greek docs for
XeTeX and LuaTeX export with FreeSans.
(Also some Indonesian tests were missing from revertedTests.)
As Kornel has explained:
There is an incompatibility between luainputenc.sty and ectaart.cls.
luainputenc.sty loads luatex.sty. Both files (luatex.sty and
ectaart.cls) define the latex-command \setattribute.
These tests should be inverted because the conflict may one day be
resolved at which point we can begin testing for regressions.
(This commit also rearranges the europeCV lines to below the
corresponding explanation comment.)
Export with XeTeX and LuaTeX (with either non-tex fonts or 8-bit
compatibility mode) does not work because the loading of inputenc with
utf8x is hardcoded in europecv.cls at this time.
This commit adds a note to es/europeCV.lyx explaining the problem and
inverts the XeTeX and LuaTeX tests.
See
http://comments.gmane.org/gmane.editors.lyx.devel/145896
Thanks to Günter Milde for the advice and to Ignacio García
for the translation of the note.
This is a manual lyx2lyx fix. Some of the problematic chunks
are not correctly converted with lyx2lyx so this commit manually
converts them to ERT.
For more details, see:
http://marc.info/?t=137702744100010&r=1&w=2
Similar to Indonesian LuaTeX support (see 7d705438),
"magyar" is not supported. After 2.1 is released,
the babel name of "magyar" should be changed to
"hungarian" (if others disagree then these tests
still should stay inverted).
The babel name of Indonesian should be changed from "bahasa"
to "indonesian". This should be done after 2.1 is released so as
to minimize potential problems such as Jürgen's example of
\addto\captionsbahasa{\renewcommand\chaptername{Foo}}
For more info, see:
http://www.mail-archive.com/lyx-devel@lists.lyx.org/msg181595.html
From Jürgen:
LuaTeX support in polyglossia is still very new and experimental.
I have this enhancement on the agenda for 2.2, but not for 2.1.
For more details, see:
http://marc.info/?l=lyx-devel&m=138506346715339&w=2
In successful cooperation with Scott Kostyshak.
We provide many lyx-documents which are not compilable with luatex or xetex.
But some of them compile, if we change the font use to non_tex_fonts.
Since this would change the appropriate source, we have to convert
it first into a save location. To make it there compilable,
we have to convert also all file references.
languages used font
he|el|ru|uk 'FreeSans'
fa 'FreeFarsi'
zh_CN 'WenQuanYi Micro Hei'
The whole job is done with a perl script.
The tests are now more robust if ctest uses '-j' (number of threads) parameter.
a.) keytests are running in sequence and in one thread only, no other thread running
b.) tex2lyx tests are locking in respect to each other
c.) Test gets one or more labels, so that we can select
ctest -L url
to run tests labeled url
d.) New macro settestlabel() to add labels to a test
Files, which contain names of ctest-tests controling
whether they should be inverted, or ignored.
We have two such files ATM: revertedTests + ignoredTests
These tests load the .lyx files to see if there are
any parsing errors or other LyX errors/warnings. The
tests fail if LyX writes a line to STDERR that does not
match any of the regexes stored in filterCheckWarnings.
Thanks to Kornel for the CMake expertise and corrections.
This patch was written jointly with him.
a) test export/templates/ectaart_pdf5 removed, because
of incompatibility between ectart.cls and luatex.sty
b) renamed some tests to reflect their purpose
tex2lyx2lyx/roundtrip/ => tex2lyx/cmplyx/
# the tests are comparing files written by lyx (not by tex2lyx)
autotests/ => keytest/
# we are testing reactions of lyx to some key strokes
c) add tests for lyx2lyx. This is rather rudimental, in that it only
checks for created output, error codes and error output.
d) use correct environment variable to set up the userdir for testing
Some export test are known to fail. Checking them every time is cumbersome.
Now, the tests marked in file revertedTests will invert the success condition
and mark the test as INVERTED_SEE-README.ctest.
If some of the tests in revertedTests fails, this is good, and the testname
can be removed from it.
If no libraries are found, XVFBDLIBS is not initialized and the command "list(REMOVE_DUPLICATES XVFBDLIBS)" fails. So, initialize XVFBDLIBS beforehand.
The xhtml format is always tested, regardless of the default format.
Without this, if the default format is xhtml, CMake gives an error
when trying to add the xhtml test because it was already added.
Now all export tests can be run with:
ctest -R "export"
Only export tests for manuals can be run with:
ctest -R "export/doc"
And only export tests for examples can be run with:
ctest -R "export/examples"
Instead of 'autotests/export/doc/' export tests are prefixed
with 'export/doc/'. This has the advantage of the name better
reflecting the test (export tests are not autotests), but has a
disadvantage in that the old name gave a better idea of where the
export test scripts are located (in the autotests directory).
keytest.py now only prints one time the command that it is executing for
an assert. The command is already printed in the function responsible
for executing it, so it is not necessary to print the command before
calling that function.
This test should fail because #7673 is not yet fixed in trunk.
Note that it is fixed in branch.
This is the first autotest that uses an existing .lyx file,
bug-7673.lyx.
If the test opens an existing .lyx file and crashes, it will leave
around a .emergency file. If a test with that same name is run again,
LyX will try to recover the .emergency file, which could throw
off the test.
This is implemented for both CMake and autotools.
To reproduce some bugs, tests need to open an existing LyX file. This
is now possible with CMake and is consistent with testing under
autotools in the sense that the .lyx file is referred to in the test
by being in the parent directory.
If the test is named bug-XXXX-in.txt and there exists a file
bug-XXXX.lyx, that file will be copied to the parent directory of
out-home in the build directory.
Many test are not running, but if changed the '-dbg xxx' to '-dbg any'
then all of sudden they pass ...
(The buffered output of lyx is often disrupted too early with 'kill -9',
so the expected data are not available for the appropriate test)
Corresponding test-case needed a fix as well and now it is passed.
git-svn-id: svn://svn.lyx.org/lyx/lyx-devel/trunk@39518 a592a061-630c-0410-9148-cb99ea01b6c8
together with the others when typing 'make', and also they can be selectively
launched by using: ./run-tests.sh filename-in.sh.
git-svn-id: svn://svn.lyx.org/lyx/lyx-devel/trunk@39241 a592a061-630c-0410-9148-cb99ea01b6c8
Added corresponding regression tests findadv-re-01 and -02 for finding these special characters.
Unfortunately, braces {} are a little bit special and don't work yet. Added findadv-re-03 regression test to not forget about this case.
This goes along the line of fixing issues reported by Andrew Parsloe and summarised as #7638.
Last note: Still there will be cases when the Advanced F&R won't work, especially if the text to be found and/or the search pattern is preceded by backslashes or others.
git-svn-id: svn://svn.lyx.org/lyx/lyx-devel/trunk@39135 a592a061-630c-0410-9148-cb99ea01b6c8
Fixed regression case (now typing twice "\" leads to "\\", i.e., the regexp matching the backslash char).
git-svn-id: svn://svn.lyx.org/lyx/lyx-devel/trunk@39019 a592a061-630c-0410-9148-cb99ea01b6c8
For example, this wasn't allowing to match '\beta\alpha' in the sequence '\alpha\beta\alpha', as
in the accompanying regression test-case (added case for ignore-format off).
git-svn-id: svn://svn.lyx.org/lyx/lyx-devel/trunk@38888 a592a061-630c-0410-9148-cb99ea01b6c8
For example, this wasn't allowing to match '\beta\alpha' in the sequence '\alpha\beta\alpha', as
in the accompanying regression test-case.
git-svn-id: svn://svn.lyx.org/lyx/lyx-devel/trunk@38883 a592a061-630c-0410-9148-cb99ea01b6c8
Added accompanying regression tests for displayed math and numbered equations.
git-svn-id: svn://svn.lyx.org/lyx/lyx-devel/trunk@38860 a592a061-630c-0410-9148-cb99ea01b6c8
For example, before this footnotes were not correctly found.
Added also accompanying regression test.
git-svn-id: svn://svn.lyx.org/lyx/lyx-devel/trunk@38846 a592a061-630c-0410-9148-cb99ea01b6c8
in Paragraph.cpp as the corresponding macros), when ignore format is off.
git-svn-id: svn://svn.lyx.org/lyx/lyx-devel/trunk@37601 a592a061-630c-0410-9148-cb99ea01b6c8
script (if any) also when LyX is not installed on your system
(using a variant of the trick suggested by Enrico, i.e., create a symlink
locale from the LyX sources root to an autotests/locale folder).
At least, it does on Linux.
git-svn-id: svn://svn.lyx.org/lyx/lyx-devel/trunk@37589 a592a061-630c-0410-9148-cb99ea01b6c8
used only if the XVKBD_HACKED environment var is defined.
Made keytest.py exploit the -wait_idle option of the hacked xvkbd.
git-svn-id: svn://svn.lyx.org/lyx/lyx-devel/trunk@37521 a592a061-630c-0410-9148-cb99ea01b6c8
the beginning of the tests (which reconfigures LyX and takes some time),
followed by all of the tests using it. This way, we get only the default
configuration of LyX as a base, without being affected by individual users
custom configurations.
git-svn-id: svn://svn.lyx.org/lyx/lyx-devel/trunk@37408 a592a061-630c-0410-9148-cb99ea01b6c8
Thunderbird while displaying LyX mails or Firefox while displaying
LyX web pages.
git-svn-id: svn://svn.lyx.org/lyx/lyx-devel/trunk@37375 a592a061-630c-0410-9148-cb99ea01b6c8
As of now, I plan to add a few regression tests for the Advanced Find & Replace using this engine.
It can be used for automated testing of other LyX subsystems as well. Just write your own *-in.txt
scripts and add them into the autotests/ folder.
TODOs:
- perhaps simplify keytest.py
- check how this behaves under various types of failure of the test
(e.g., LyX crashing, hanging, ...) just to be sure it goes on with
the other tests anyway.
git-svn-id: svn://svn.lyx.org/lyx/lyx-devel/trunk@37364 a592a061-630c-0410-9148-cb99ea01b6c8
