When switching catcodes (for example to verbatim), there are in general some tokens that have already been parsed according to the old catcodes. It is therefore needed to "forget" those token and reparse them.
The basic idea is to use idocstream::putback() to feed the characters back to the stream, but this does not work (probably because we disable buffering). Therefore, we implement a simple stream-like class that implements putback().
This is a refactoring commit, which may introduce minor regressions, but should go in the right direction.
* introduce Parser::verbatimEnvironment, which is a wrapper around verbatimStuff
* introduce output_ert, which inserts a string as ERT; this allows to factor out several code instances.
* rename handle_ert to output_ert_inset: this creates an inset and then calls output_ert
* remove handle_comment (use handle_ert instead)
* remove handle_backaspace (unused now)
* use the new methods in handle_listing
TODO:
* use for other verbatim-like cases (Sweave chunk, url...)
* maybe implement Parser::unparse to reparse tokens already parsed with the wrong catcodes (if needed)
- Implement catcode setting in Parser
- add a new Parser::verbatimStuff method that reads verbatim contents
- use this method to parse "verbatim" environment.
- use it to parse \verb too.
- rename Parser::verbatimEnvironment to ertEnvironment.
TODO:
- use for other verbatim-like cases (Sweave chunk, lstlisting...)
- factor out the function that outputs ERT (including line breaks)
- maybe implement Parser::unparse (if needed)
Provide functions for translating to the LyX name
of an encoding from either a LaTeX name or an Iconv
name, with the possibility to specify the package.
This is in anticipation of changing to use the LyX
name of the encoding in the .lyx file format and
allowing multiple lib/encodings entries to have
the same LaTeX name (but different packages!).
The tex2lyx parser needs to worry about the iconv
name of the input encoding, so store that instead
of the latex name.
The old layouts are still there (marked as deprecated). The new ones are more or less correctly reverted (polishment required), but the old ones not yet converted to the new. Once this is done, a further file format change should be made.
These encodings were not defined, since they must not be used as document
encodings (the characters {, } and \ may appear in high bytes, and latex
would be confused). However, they are supported by CJK.sty (which uses a
preprocessor to circumvent the limitations of the latex executable). These
encodings are now defined, but used for import in tex2lyx only.
The test case CJK.tex contained fake tests for shift-jis and big5 (the
japanese and chinese characters were entered using the utf8 encoding), and
therefore the wrong interpretation of these encoding looked as if it worked.
The comments about missing iconv support of shift-jis and big5 were wrong as
well (otherwise shift-jis-plain would not work either).
