This fixes bug #8554 and some recently introduced busg:
- Encodings::fromLaTeXCommand() can now handle all combining characters,
not only the one letter ones
- The remainder returned from Encodings::fromLaTeXCommand() must never be
thrown away in tex2lyx, but output as ERT
- No special case for combining diacritical marks needed anymore in parse_text()
- No special cases for accents and IPA combining diacritical marks needed
anymore in parse_text()
- special tipa short cuts may only be recognized if the tipa package is loaded
- Use requirements returned by Encodings::fromLaTeXCommand() instead of
hardcoded registering of tipa and tipax
- Get rid of the name2 variable in parse_text(): We must use name, otherwise
the extra stuff that might have been put into name vanishes
tex2lyx skips LyX preamble code only if it thinks that the file was created
by LyX (i.e. special comments are found). I don't like that, but this is
how it works and therefore the special comment neds to be added if the
theorem commands are to be skipped. This header makes also the temporary
setting of in_lyx_preamble obsolete, which caused skipping of user preamble
commands, which were not re-added by LyX.
- Convert prettyref to the autopackage mechanism
- Do not load refstyle automatically if some refstyle preamble code of LyX
is found, since LyX will only load the package if an actual reference
command is used. This is needed for mixed refstyle/prettyref documents.
- Only recognize refstyle commands if refstyle was detected in the preamble
- Only recognize prettyref commands if prettyref was detected in the preamble
- Add a mixed refstyle/prettyref test case
* use verbatimStuff for parsing chunks and make try to follow closely the sweave sytax constraints.
* merge the two cases for parsing << (noweb or quote)
* \verb|ff| requires that its parameter is on a single line.
- support for Cyrillic characters
- support for \textifsymbol and \ascii (fixes bug #8556)
- support for \ding
- tex2lyx/text.cpp: correct an indentation and use "name2" because "name" is already defined in this clause
- for all TIPA characters the tipa and tipx package must always be registered because these characters are also allowed outside a TIPA inset
- include \textvertline because we again reached the maximum number of if statements allowed by MSCV (so safe at least one for now)
The LyX file format only supports one CJK font mapping per document.
Therefore all CJK environments which use a different mapping than the main
one are output in ERT. THis is now explained in the test case, and also the
converion to ERT is improved (although it is still wrong e.g. for paragraphs).
- support for \textdoublevertline, \textvertline, \textglobfall
- support \!o and \!b and \!d and \!g and \!G and \!j
- support and \*k and \*r and \*t and \*w
- register the package "tipa" if \textipa is detected in an equation
- add a \textipa equation to the testfile
(I failed to implement support for the TIPA character "\t*{ }" and the command "\=*".)
- Use the LyX name of encodings instead of the LaTeX names.
The LyX name must be unique, while the name used by LaTeX
not necessarily, e.g. different packages might implement
support for the same encoding.
- Rename koi8 to koi8-r, so that the LyX and LaTeX names match.
- Rename euc-jp-plain to euc-jp-platex, jis-plain to jis-platex
and shift-jis-plain to shift-jis-platex.
- Add utf8-platex encoding (fixes#8408).
LyX file format incremented to 463.
- support for the combining diacritical marks
- support for \texttoptiebar and \textbottomtiebar
- test-insets.lyx: add complete testcase
- TODO.txt: update what still needs to be done
This is achieved by not calling Parse::tokenize_one() anymore in
Parser::good(): The status of the input can be tested without performing the
actual tokenizing. Now there are only two methods that may prevent an encoding
change:next_token() and next_next_token().
This is an addendum to [72a44b3c/lyxgit] because depending on the environment, LyX adds a \phantomsection before \addcontentsline.
- also update the test file
This is an important part of the tests. If updating the test cases is really a
problem there are two better solutions than not testing the format: Convert
the references with lyx2lyx on the fly, or remove the hard coupling of tex2lyx
and LyX versions again. Both increase additional possible error sources, but
these errors could at least be detected. If the test machinery is made blind
for versions, file format errors are impossible to detect.
Jean-Marc discovered a possible data loss caused by Parser::getChar().
This is now fixed, and Parser::getChar() is removed, since it is no longer
needed and easy to use it in the wrong way.
- nothing needs to be done for feyn.sty - it is already recognized as simple feature and the roundtrip using the feynman example file works
- update entry for undertilde to be uniform
When switching catcodes (for example to verbatim), there are in general some tokens that have already been parsed according to the old catcodes. It is therefore needed to "forget" those token and reparse them.
The basic idea is to use idocstream::putback() to feed the characters back to the stream, but this does not work (probably because we disable buffering). Therefore, we implement a simple stream-like class that implements putback().
This is a refactoring commit, which may introduce minor regressions, but should go in the right direction.
* introduce Parser::verbatimEnvironment, which is a wrapper around verbatimStuff
* introduce output_ert, which inserts a string as ERT; this allows to factor out several code instances.
* rename handle_ert to output_ert_inset: this creates an inset and then calls output_ert
* remove handle_comment (use handle_ert instead)
* remove handle_backaspace (unused now)
* use the new methods in handle_listing
TODO:
* use for other verbatim-like cases (Sweave chunk, url...)
* maybe implement Parser::unparse to reparse tokens already parsed with the wrong catcodes (if needed)
- Implement catcode setting in Parser
- add a new Parser::verbatimStuff method that reads verbatim contents
- use this method to parse "verbatim" environment.
- use it to parse \verb too.
- rename Parser::verbatimEnvironment to ertEnvironment.
TODO:
- use for other verbatim-like cases (Sweave chunk, lstlisting...)
- factor out the function that outputs ERT (including line breaks)
- maybe implement Parser::unparse (if needed)
Provide functions for translating to the LyX name
of an encoding from either a LaTeX name or an Iconv
name, with the possibility to specify the package.
This is in anticipation of changing to use the LyX
name of the encoding in the .lyx file format and
allowing multiple lib/encodings entries to have
the same LaTeX name (but different packages!).
The tex2lyx parser needs to worry about the iconv
name of the input encoding, so store that instead
of the latex name.
The old layouts are still there (marked as deprecated). The new ones are more or less correctly reverted (polishment required), but the old ones not yet converted to the new. Once this is done, a further file format change should be made.
These encodings were not defined, since they must not be used as document
encodings (the characters {, } and \ may appear in high bytes, and latex
would be confused). However, they are supported by CJK.sty (which uses a
preprocessor to circumvent the limitations of the latex executable). These
encodings are now defined, but used for import in tex2lyx only.
The test case CJK.tex contained fake tests for shift-jis and big5 (the
japanese and chinese characters were entered using the utf8 encoding), and
therefore the wrong interpretation of these encoding looked as if it worked.
The comments about missing iconv support of shift-jis and big5 were wrong as
well (otherwise shift-jis-plain would not work either).
Actually, the test case showed several problems:
- ERT insets did use layout "Standard", not "Plain Layout"
- The font scale was read correctly, but tex2lyx claimed that it did ignore
the option "scaled=0.95"
- If a third argument of the CJK environment was given, it caused the whole
environment to be put in ERT with a broken encoding. This is now fixed for
the bug test case by using the \font_cjk header variable, but the encoding
problem still exists for unsupported encodings. I'll file a separate bug
for that.
- The CJKutf8 package was not handled in the preamble parsing. Therefore the
chinese comment in the preamble was read with a wrong encoding, and guessing
the document language did not work.
The new file CJKutf8.tex was created by copying and modifying CJK.tex, but
unfortunately it is impossible to tell git to inherit the history of CJK.tex
for the new file (search the web for git svn copy if you want to know details).
Special characters as created by latex_path() where not converted corectly
from LaTeX macros by tex2lyx. Now this is done, even for file names containing
double quotes which are not used for quoting spaces. These file names are not
legal on windows, and will causes probelms in DVI files, but if they occur
tex2lyx does not produce invalid .lyx files.
The fix is basically mechanical, the additional code for fraction like insets
with three arguments was stolen from \unitfrac. As any math package,
stackrel.sty needs a buffer parameter to switch it off.
I also added the two stackrel flavours to the toolbar.
Both problems where caused by the fact that tex2lyx did not handle
natbib/jurabib citations correctly if natbib/jurabib was loaded by the
document class. Therefore it tried to parse the standard \cite syntax, and
did not recognize \citet and \citep.
The \frametitle command is less convenient to use than the \frame argument, but it provides more options (overlay/action and short title). We thus provide this additionally to the option, like beamer itself does.
This has a list-like structure (with \onslide item commands). The previous implementation was rather useless, since it required lots of ERT. Since the new implementation is so different, we use ERT for conersion/reversion.
The lyx2lyx routines are not yet perfect, though.
The stmaryrd package adds support for lots of math symbols, using a font
designed to accompany the computer modern fonts. The changes in detail:
- Fix generate_symbols_list.py to work with stmaryrd.sty. It loooks like it
was automatically translated from a perl version and never used.
- Generate the new symbols in lib/symbols using generate_symbols_list.py and
add some manual adjustments
- Generate stmary10.ttf by a simple ttf export from stmary10.sfd with fontforge
- Add license info for stmary10.ttf
- Create a test file with all symbols from stmaryrd.sty. Actually it would be
nice to have this for the other fonts as well.
- The mechanics: lyx2lyx, tex2lyx, font machinery etc.
This patch puts all projects into subfolders (at least for MSVS). In this
way, there is a better overview (especially if the number of test projects
will be increasing).
Now that we have module support for literate programming, it is possible to do a noweb cleanup. This is basically a patch from Kayvan Sylvan:
- get rid of literate-xxx classes
- rename Scrap to Chunk, since this is the name noweb doc uses (Scrap is from nuweb)
- update lyx file format and add lyx2lyx support for gettting rid of literate-xxx classes
- update documentation
On the top of it, update tex2lyx to
- avoid creating files with literate-xxx class
- fix conflict between parsing << as a quote and parsing it as a Chunk
- create Chunk layouts instead of Scrap ones.
Actually tex2lyx can handle modules since some time (#5702), but not
theorems (#5776). Now the following issues are fixed:
- Modules that depend on other modules can be loaded, since the dependencies
are loaded first
- Default moduls of the text class are loaded correctly
- \newtheorem is recognized as a command that defines new environments and
treated similar to \newenvironment
With this new command line switch a list of modules can be loaded
unconditionally. This seems to be needed for the literate programming formats,
and it is useful to work around bug #5702 as well.
The listings inset does automatically load the color package if any parameter
contains \color. As mentioned in bug #8066 tex2lyx needs to be aware of this,
so the \usepckage{color} is automatically skipped in these cases.
Creating LYX_DATE dynamically at configure-time caused unwanted recompilation
of the whole directories (src + src/tex2lyx) because all the relevant objects
were dependent of a common file (flags.make in case of "Unix makefiles") which changed
accordingly.
There is now a new include (lyx_date.h, with only one definition)
Nothing changes for automake, since in this case LYX_DATE is defined in config.h
s is a valid horizontal position for framebox (as well as for makebox, which
was already parsed correctly). There was even a test case, but with a wrong
reference.
Instead of annoying the user with an automatically created note in the output
document which she needs to delete manually, determine the document language
automatically for documents that use CJK. This is done using a heuristic which
roughly counts the number of characters in each language and sets the one that
is used most often. This is not perfect, but it works for the two major use
cases: A document with only some CJK parts (in this case the babel language is
used), and a document which is mainly written in one CJK language. It is only
a minor problem if the heuristic is wrong, since the TeX export is still
correct, and there is no spell checking support for CJK anyway.
Now all regression tests do pass except for some relative path issues
depending on the location of the build directory.
The previous output was read correctly by LyX as well, but the new version is
more similar to the files written by LyX. Also update TODO with recent file
format changes.
Commit 7cfac95 got rid of empty lines that were created by removing \usepackage
statements. However, it added an additional newline in case the \usepackage
was not at the end of the line. This is now fixed.
The old fix was incomplete (\verb~\~ was translated to \verb~~ in roundtrip).
The real cause for this bug (and also the mistranslation of \href{...}{\}})
was the misbehaviour of Token::character() (see comment in Parser.h): This
method even returns a character if the category is catEscape, and this is not
wanted in most (all?) cases.
If this option is given, included files will be copied to the output directory.
Also -roundtrip is now allowed with given output file.
-copyfiles is useful if you want to ensure that no file (not even an included
one) is overwritten by a subsequent export from LyX. Both changes are needed
for unit tests that do not write to the source directory.
The input and output file names of tex2lyx may be relative. In this case,
getMasterFilePath() and getParentFilePath() return relative paths as well.
Now the file name translation logic for all kinds of included files can
cope with that.
If there was an encoding set by the inputenc package, it must not be ignored.
Now all tex2lyx test cases can be exported to .tex again by LyX (although
there are still some differences in the .lyx output).
It is invoked by 'make check' (automake only, it would be nice if someone
could add it to cmake as well), or by calling
python src/tex2lyx/test/runtests.py <path to tex2lyx binary>
by hand. Currently, it does not compare the output (this comes later).
The added .lyx files are from tex2lyx around mid of april, so that you
can see the regressions of the current version if you run the test
yourself (simply run git diff afterwards).
The home made test runner is quite stupid, but better than nothing.
Feel free to improve it or replace it with something better, as long
as running it stays as simple as now.
- tex2lyx/text.cpp:
- fix bug that swallowed valid braces
- the encoding is "Bg5" not "BIG5"
- add and update comments
- CJK.tex: add an example for the Bg5 encoding
- Preamble.cpp/:
- move list definition out of anon namespace
- declare lists and a function in the class
- text.cpp: remove the now already defined list
- Parser.cpp: \verb can have any character as delimiter (except of ASCII letters) not only '+', therefore partly revert [3943b887/lyxgit] and fix it for all cases
Preamble.cpp:
- support the font scaling
- improve coding style
text.cpp: - improve logic/coding style
XeTeX-polyglossia.tex:
- add scaling
- change document so that it is compilable using the Libertine font
Instead of adding the extra '\n' to tex2lyx, we should remove it from
BufferParams.cpp because it was introduced there unintendedly in d7eaaba9
(Juergen Spitzmueller; "support for CJK font argument"; Jun 5 2008).
This partly reverts commit c2810f922665c591a7c32d4c679e58e55132505a.
Fix commit [aa9fed93/lyxgit]: it is never a good idea to call substr(4) on a string without checking its length. Moreover, it is even better in this case to check that the first letters are "text" as the reminder of the code assumes...
- tex2lyx/Preamble.cpp/h: not all new header definitions since fileformat 413 have yet been added to texl2yx, so do it now; also change some new default values
- bugfix for the parsing of polyglossia's paragraph environments
- new support for polyglossia's language change commands \textxxx
- XeTeX-polyglossia.tex: add more testcases and add a missing \end
objects. The problem that led to the leak is that these objects can be held in
memory long after the Buffer that created them is gone, mostly due to their
use in the CutStack. So they were previously held in a storage facility, the
DocumentClassBundle. Unfortunately, they were now being created too often,
especially by cloning. It's not really a leak, because they're accessible, but
we weren't ever destroying them.
This new approach uses a shared_ptr instead.
Thanks to Vincent for pointing out const_pointer_cast.
While cppcheck did not turn out any suspicious error messages, using
the "performance" flag highlighted several nitpicks in three categories
* do not use it++ for iterators, ++it is better
* do not use size() to test for emptyness, empty() is here
* do not use "const T" as a function parameter, "const & T" is better
I doubt that any of these is a real performance problem, but the code is cleaner anyway.
This is a part of bug #8081: Translate cells in
\begin{sideways}...\end{sideways} to native syntax. The turn environment
comes in a later commit, since this one os suitable for backporting.
This is not needed, since LyX supports comments in math. Data loss with math
comments containing a backslash in LyX has been fixed as well.
The test case was found in bug #8104.
I introduced this bug in r40091 where support for empty dates was added.
I also noticed that the title_layout_found flag was not properly passed down
to all variants of parse_text*. Since this is rather a global variable I
placed it into the preamble class.
The test case of bug #8104 shows other problems as well, but those are no
regressions and will be fixed separately.
OK for branch?
It is broken since the latest file format update (neither did Uwe run a test
after updating the file format, nor did I after undoing my accidental change).
The reason it was broken: the bool version of write_attribute() is chosen
for "0" instead of the string version, so each table used rotate="true".
Math commands need it as well as text commands. At the same time, this
further unifies the checking for termination and fixes cases of wrong
output (e.g. for 0x2005).
If \hline is entered, do not create an unknown inset, but increase the number
of hlines of the current row if that is allowed. The same idea is applied to
copy-paste (not part of the bug report).
This is also a test for committing via git.
Setting up .gitignore or .git/info/excludes is something that should
be done. Not doing it makes it a lot harder to see actual new files
that should be added.
Signed-off-by: Lars Gullik Bjønnes <larsbj@gullik.org>
- Parser.cpp: - new function to parse verbatim environments
- test/test-structure.tex: updated example
git-svn-id: svn://svn.lyx.org/lyx/lyx-devel/trunk@40850 a592a061-630c-0410-9148-cb99ea01b6c8
The type of citation engine is determined by the citation package
being used and, in the case of natbib, its options.
git-svn-id: svn://svn.lyx.org/lyx/lyx-devel/trunk@40593 a592a061-630c-0410-9148-cb99ea01b6c8
tex2lyx does not need a -p command since it detects partial files automatically.
git-svn-id: svn://svn.lyx.org/lyx/lyx-devel/trunk@40591 a592a061-630c-0410-9148-cb99ea01b6c8
from the citation package being used or from a \bibliographystyle
command in the preamble, if any>
git-svn-id: svn://svn.lyx.org/lyx/lyx-devel/trunk@40499 a592a061-630c-0410-9148-cb99ea01b6c8
now) and move more packages to the new exclude mechanism.
The remaining ones are not so easy.
git-svn-id: svn://svn.lyx.org/lyx/lyx-devel/trunk@40442 a592a061-630c-0410-9148-cb99ea01b6c8
A complete fix is currently impossible because LyX assumes that all longtable
captions are in the first head.
git-svn-id: svn://svn.lyx.org/lyx/lyx-devel/trunk@40361 a592a061-630c-0410-9148-cb99ea01b6c8
registration mechanism, so that they are only swallowed if LyX will load them.
git-svn-id: svn://svn.lyx.org/lyx/lyx-devel/trunk@40315 a592a061-630c-0410-9148-cb99ea01b6c8
If we want to look at the token after the next token, it may be needed
to call tokenize_one() twice and not only once as done in good().
git-svn-id: svn://svn.lyx.org/lyx/lyx-devel/trunk@40234 a592a061-630c-0410-9148-cb99ea01b6c8
replace home made keyval parsing with the more robust process_keyval_opt().
git-svn-id: svn://svn.lyx.org/lyx/lyx-devel/trunk@40185 a592a061-630c-0410-9148-cb99ea01b6c8
to occur in pairs.
If you change something in tex2lyx please don't commit without running the
tests and ensuring that no warnings or errors are written on stderr.
git-svn-id: svn://svn.lyx.org/lyx/lyx-devel/trunk@40177 a592a061-630c-0410-9148-cb99ea01b6c8
- support for \makebox
- fix import of boxes without an inner box (the on-screen display was broken because of wrong default width setting)
- fix parsing of \framebox{content}, which LyX does not support yet - we have to use ERT
git-svn-id: svn://svn.lyx.org/lyx/lyx-devel/trunk@40150 a592a061-630c-0410-9148-cb99ea01b6c8
Of course this will only work if output_changes was true when creating the
.tex file.
git-svn-id: svn://svn.lyx.org/lyx/lyx-devel/trunk@40139 a592a061-630c-0410-9148-cb99ea01b6c8
Also a warning "\end{foo} does not match \begin{bar}" is gone.
git-svn-id: svn://svn.lyx.org/lyx/lyx-devel/trunk@40133 a592a061-630c-0410-9148-cb99ea01b6c8
the complete document whether LyX will load them or not.
Use the mechanism for the color package (fixes the color part of bug #7845).
git-svn-id: svn://svn.lyx.org/lyx/lyx-devel/trunk@40095 a592a061-630c-0410-9148-cb99ea01b6c8
but the title commands in the preamble are still not recognized.
git-svn-id: svn://svn.lyx.org/lyx/lyx-devel/trunk@40093 a592a061-630c-0410-9148-cb99ea01b6c8
- update preamble to the features used in the file
- add some floats
git-svn-id: svn://svn.lyx.org/lyx/lyx-devel/trunk@40024 a592a061-630c-0410-9148-cb99ea01b6c8
