lyx_mirror/src/Lexer.h

217 lines
5.1 KiB
C
Raw Normal View History

// -*- C++ -*-
/**
* \file Lexer.h
* This file is part of LyX, the document processor.
* Licence details can be found in the file COPYING.
*
* \author Alejandro Aguilar Sierra
* \author Lars Gullik Bj<EFBFBD>nnes
*
* Full author contact details are available in file CREDITS.
*/
// Generalized simple lexical analizer.
// It can be used for simple syntax parsers, like lyxrc,
// texclass and others to come.
#ifndef LEXER_H
#define LEXER_H
#include "support/docstring.h"
#include <boost/utility.hpp>
#include <iosfwd>
namespace lyx {
namespace support { class FileName; }
///
struct keyword_item {
///
char const * tag;
///
int code;
};
/** Generalized simple lexical analizer.
Use the method isOK() to check if there is still data available
Fix bug 3293 by Bernhard Roider: This changes the semantics of isOK() and operator(), comments from Bernhard below: With the old version of lyxlex it was _impossible_ to check whether reading an integer, float, ... succeeded or not. The current solution to check for is.bad() in some cases and in other cases use is.good() does not give the desired information. Moreover the result of is.bad() depends on the stl implementation and behaves different for linux and windows. the bug was introduced by the patch that fixed the bug that crashed lyx when "inset-insert ert" was executed from the command buffer. The lexer has the method isOK() which reflects the status of the stream is. The operators void* and ! are not really well defined (they depend on the value of is.bad()). What is missing is a test if the last reading operation was successful and thus the returned value is valid. That's what i implemented in this patch. The new rule for using the lexer: if you want to know if the lexer still has data to read (either from the stream or from the pushed token) then use "lex.isOK()". If you want to test if the last reading operation was successful then use eg. "if (lex) {...}" or unsuccessful then use eg. "if (!lex) {...}" an example: int readParam(LyxLex &lex) { int param = 1; // default value if (lex.isOK()) { // the lexer has data to read int p; // temporary variable lex >> p; if (lex) param = p; // only use the input if the reading operation was successful } return param; } git-svn-id: svn://svn.lyx.org/lyx/lyx-devel/trunk@17569 a592a061-630c-0410-9148-cb99ea01b6c8
2007-03-26 13:43:49 +00:00
for lexing. Use one of the the operators void* or ! to test if
the last reading operation was successful.
Fix bug 3293 by Bernhard Roider: This changes the semantics of isOK() and operator(), comments from Bernhard below: With the old version of lyxlex it was _impossible_ to check whether reading an integer, float, ... succeeded or not. The current solution to check for is.bad() in some cases and in other cases use is.good() does not give the desired information. Moreover the result of is.bad() depends on the stl implementation and behaves different for linux and windows. the bug was introduced by the patch that fixed the bug that crashed lyx when "inset-insert ert" was executed from the command buffer. The lexer has the method isOK() which reflects the status of the stream is. The operators void* and ! are not really well defined (they depend on the value of is.bad()). What is missing is a test if the last reading operation was successful and thus the returned value is valid. That's what i implemented in this patch. The new rule for using the lexer: if you want to know if the lexer still has data to read (either from the stream or from the pushed token) then use "lex.isOK()". If you want to test if the last reading operation was successful then use eg. "if (lex) {...}" or unsuccessful then use eg. "if (!lex) {...}" an example: int readParam(LyxLex &lex) { int param = 1; // default value if (lex.isOK()) { // the lexer has data to read int p; // temporary variable lex >> p; if (lex) param = p; // only use the input if the reading operation was successful } return param; } git-svn-id: svn://svn.lyx.org/lyx/lyx-devel/trunk@17569 a592a061-630c-0410-9148-cb99ea01b6c8
2007-03-26 13:43:49 +00:00
Example:
int readParam(LyxLex &lex) {
int param = 1; // default value
if (lex.isOK()) { // the lexer has data to read
int p; // temporary variable
lex >> p;
if (lex) param = p; // only use the input if reading was successful
}
return param;
}
Fix bug 3293 by Bernhard Roider: This changes the semantics of isOK() and operator(), comments from Bernhard below: With the old version of lyxlex it was _impossible_ to check whether reading an integer, float, ... succeeded or not. The current solution to check for is.bad() in some cases and in other cases use is.good() does not give the desired information. Moreover the result of is.bad() depends on the stl implementation and behaves different for linux and windows. the bug was introduced by the patch that fixed the bug that crashed lyx when "inset-insert ert" was executed from the command buffer. The lexer has the method isOK() which reflects the status of the stream is. The operators void* and ! are not really well defined (they depend on the value of is.bad()). What is missing is a test if the last reading operation was successful and thus the returned value is valid. That's what i implemented in this patch. The new rule for using the lexer: if you want to know if the lexer still has data to read (either from the stream or from the pushed token) then use "lex.isOK()". If you want to test if the last reading operation was successful then use eg. "if (lex) {...}" or unsuccessful then use eg. "if (!lex) {...}" an example: int readParam(LyxLex &lex) { int param = 1; // default value if (lex.isOK()) { // the lexer has data to read int p; // temporary variable lex >> p; if (lex) param = p; // only use the input if the reading operation was successful } return param; } git-svn-id: svn://svn.lyx.org/lyx/lyx-devel/trunk@17569 a592a061-630c-0410-9148-cb99ea01b6c8
2007-03-26 13:43:49 +00:00
@see LyXRC.cpp for an example of usage.
*/
class Lexer : boost::noncopyable {
public:
///
Lexer(keyword_item *, int);
///
~Lexer();
/// Lex basic codes
enum {
///
LEX_UNDEF = -1,
///
LEX_FEOF = -2,
///
LEX_DATA = -3,
///
LEX_TOKEN = -4
};
/// stream is open and end of stream is not reached
Fix bug 3293 by Bernhard Roider: This changes the semantics of isOK() and operator(), comments from Bernhard below: With the old version of lyxlex it was _impossible_ to check whether reading an integer, float, ... succeeded or not. The current solution to check for is.bad() in some cases and in other cases use is.good() does not give the desired information. Moreover the result of is.bad() depends on the stl implementation and behaves different for linux and windows. the bug was introduced by the patch that fixed the bug that crashed lyx when "inset-insert ert" was executed from the command buffer. The lexer has the method isOK() which reflects the status of the stream is. The operators void* and ! are not really well defined (they depend on the value of is.bad()). What is missing is a test if the last reading operation was successful and thus the returned value is valid. That's what i implemented in this patch. The new rule for using the lexer: if you want to know if the lexer still has data to read (either from the stream or from the pushed token) then use "lex.isOK()". If you want to test if the last reading operation was successful then use eg. "if (lex) {...}" or unsuccessful then use eg. "if (!lex) {...}" an example: int readParam(LyxLex &lex) { int param = 1; // default value if (lex.isOK()) { // the lexer has data to read int p; // temporary variable lex >> p; if (lex) param = p; // only use the input if the reading operation was successful } return param; } git-svn-id: svn://svn.lyx.org/lyx/lyx-devel/trunk@17569 a592a061-630c-0410-9148-cb99ea01b6c8
2007-03-26 13:43:49 +00:00
/// FIXME: test also if pushTok is not empty
/// FIXME: the method should be renamed to something like
Fix bug 3293 by Bernhard Roider: This changes the semantics of isOK() and operator(), comments from Bernhard below: With the old version of lyxlex it was _impossible_ to check whether reading an integer, float, ... succeeded or not. The current solution to check for is.bad() in some cases and in other cases use is.good() does not give the desired information. Moreover the result of is.bad() depends on the stl implementation and behaves different for linux and windows. the bug was introduced by the patch that fixed the bug that crashed lyx when "inset-insert ert" was executed from the command buffer. The lexer has the method isOK() which reflects the status of the stream is. The operators void* and ! are not really well defined (they depend on the value of is.bad()). What is missing is a test if the last reading operation was successful and thus the returned value is valid. That's what i implemented in this patch. The new rule for using the lexer: if you want to know if the lexer still has data to read (either from the stream or from the pushed token) then use "lex.isOK()". If you want to test if the last reading operation was successful then use eg. "if (lex) {...}" or unsuccessful then use eg. "if (!lex) {...}" an example: int readParam(LyxLex &lex) { int param = 1; // default value if (lex.isOK()) { // the lexer has data to read int p; // temporary variable lex >> p; if (lex) param = p; // only use the input if the reading operation was successful } return param; } git-svn-id: svn://svn.lyx.org/lyx/lyx-devel/trunk@17569 a592a061-630c-0410-9148-cb99ea01b6c8
2007-03-26 13:43:49 +00:00
/// dataAvailable(), in order to reflect the real behavior
bool isOK() const;
Fix bug 3293 by Bernhard Roider: This changes the semantics of isOK() and operator(), comments from Bernhard below: With the old version of lyxlex it was _impossible_ to check whether reading an integer, float, ... succeeded or not. The current solution to check for is.bad() in some cases and in other cases use is.good() does not give the desired information. Moreover the result of is.bad() depends on the stl implementation and behaves different for linux and windows. the bug was introduced by the patch that fixed the bug that crashed lyx when "inset-insert ert" was executed from the command buffer. The lexer has the method isOK() which reflects the status of the stream is. The operators void* and ! are not really well defined (they depend on the value of is.bad()). What is missing is a test if the last reading operation was successful and thus the returned value is valid. That's what i implemented in this patch. The new rule for using the lexer: if you want to know if the lexer still has data to read (either from the stream or from the pushed token) then use "lex.isOK()". If you want to test if the last reading operation was successful then use eg. "if (lex) {...}" or unsuccessful then use eg. "if (!lex) {...}" an example: int readParam(LyxLex &lex) { int param = 1; // default value if (lex.isOK()) { // the lexer has data to read int p; // temporary variable lex >> p; if (lex) param = p; // only use the input if the reading operation was successful } return param; } git-svn-id: svn://svn.lyx.org/lyx/lyx-devel/trunk@17569 a592a061-630c-0410-9148-cb99ea01b6c8
2007-03-26 13:43:49 +00:00
/// FIXME: The next two operators should be replaced by one method
/// called e.g. lastReadOk(), in order to reflect the real
Fix bug 3293 by Bernhard Roider: This changes the semantics of isOK() and operator(), comments from Bernhard below: With the old version of lyxlex it was _impossible_ to check whether reading an integer, float, ... succeeded or not. The current solution to check for is.bad() in some cases and in other cases use is.good() does not give the desired information. Moreover the result of is.bad() depends on the stl implementation and behaves different for linux and windows. the bug was introduced by the patch that fixed the bug that crashed lyx when "inset-insert ert" was executed from the command buffer. The lexer has the method isOK() which reflects the status of the stream is. The operators void* and ! are not really well defined (they depend on the value of is.bad()). What is missing is a test if the last reading operation was successful and thus the returned value is valid. That's what i implemented in this patch. The new rule for using the lexer: if you want to know if the lexer still has data to read (either from the stream or from the pushed token) then use "lex.isOK()". If you want to test if the last reading operation was successful then use eg. "if (lex) {...}" or unsuccessful then use eg. "if (!lex) {...}" an example: int readParam(LyxLex &lex) { int param = 1; // default value if (lex.isOK()) { // the lexer has data to read int p; // temporary variable lex >> p; if (lex) param = p; // only use the input if the reading operation was successful } return param; } git-svn-id: svn://svn.lyx.org/lyx/lyx-devel/trunk@17569 a592a061-630c-0410-9148-cb99ea01b6c8
2007-03-26 13:43:49 +00:00
/// behavior
/// last read operation was successful.
operator void const *() const;
Fix bug 3293 by Bernhard Roider: This changes the semantics of isOK() and operator(), comments from Bernhard below: With the old version of lyxlex it was _impossible_ to check whether reading an integer, float, ... succeeded or not. The current solution to check for is.bad() in some cases and in other cases use is.good() does not give the desired information. Moreover the result of is.bad() depends on the stl implementation and behaves different for linux and windows. the bug was introduced by the patch that fixed the bug that crashed lyx when "inset-insert ert" was executed from the command buffer. The lexer has the method isOK() which reflects the status of the stream is. The operators void* and ! are not really well defined (they depend on the value of is.bad()). What is missing is a test if the last reading operation was successful and thus the returned value is valid. That's what i implemented in this patch. The new rule for using the lexer: if you want to know if the lexer still has data to read (either from the stream or from the pushed token) then use "lex.isOK()". If you want to test if the last reading operation was successful then use eg. "if (lex) {...}" or unsuccessful then use eg. "if (!lex) {...}" an example: int readParam(LyxLex &lex) { int param = 1; // default value if (lex.isOK()) { // the lexer has data to read int p; // temporary variable lex >> p; if (lex) param = p; // only use the input if the reading operation was successful } return param; } git-svn-id: svn://svn.lyx.org/lyx/lyx-devel/trunk@17569 a592a061-630c-0410-9148-cb99ea01b6c8
2007-03-26 13:43:49 +00:00
/// last read operation was not successful
bool operator!() const;
/// return true if able to open file, else false
bool setFile(support::FileName const & filename);
///
void setStream(std::istream & is);
///
std::istream & getStream();
/// Danger! Don't use it unless you know what you are doing.
void setLineNo(int l);
/// Change the character that begins a comment. Default is '#'
void setCommentChar(char c);
/// returns a lex code
int lex();
Fix bug 3293 by Bernhard Roider: This changes the semantics of isOK() and operator(), comments from Bernhard below: With the old version of lyxlex it was _impossible_ to check whether reading an integer, float, ... succeeded or not. The current solution to check for is.bad() in some cases and in other cases use is.good() does not give the desired information. Moreover the result of is.bad() depends on the stl implementation and behaves different for linux and windows. the bug was introduced by the patch that fixed the bug that crashed lyx when "inset-insert ert" was executed from the command buffer. The lexer has the method isOK() which reflects the status of the stream is. The operators void* and ! are not really well defined (they depend on the value of is.bad()). What is missing is a test if the last reading operation was successful and thus the returned value is valid. That's what i implemented in this patch. The new rule for using the lexer: if you want to know if the lexer still has data to read (either from the stream or from the pushed token) then use "lex.isOK()". If you want to test if the last reading operation was successful then use eg. "if (lex) {...}" or unsuccessful then use eg. "if (!lex) {...}" an example: int readParam(LyxLex &lex) { int param = 1; // default value if (lex.isOK()) { // the lexer has data to read int p; // temporary variable lex >> p; if (lex) param = p; // only use the input if the reading operation was successful } return param; } git-svn-id: svn://svn.lyx.org/lyx/lyx-devel/trunk@17569 a592a061-630c-0410-9148-cb99ea01b6c8
2007-03-26 13:43:49 +00:00
/** Just read the next word. If esc is true remember that
some chars might be escaped: "\ atleast
*/
bool next(bool esc = false);
/** Read next token. This one is almost the same as next,
but it will consider " as a regular character and always
split a word if it contains a backslash.
*/
bool nextToken();
/// Push a token, that next token got from lyxlex.
void pushToken(std::string const &);
///
int getLineNo() const;
///
int getInteger() const;
///
bool getBool() const;
///
double getFloat() const;
///
std::string const getString() const;
///
docstring const getDocString() const;
/** Get a long string, ended by the tag `endtag'.
This string can span several lines. The first line
serves as a template for how many spaces the lines
are indented. This much white space is skipped from
each following line. This mechanism does not work
perfectly if you use tabs.
*/
std::string const getLongString(std::string const & endtag);
///
bool eatLine();
/// Pushes a token list on a stack and replaces it with a new one.
void pushTable(keyword_item *, int);
/** Pops a token list into void and replaces it with the one now
on top of the stack.
*/
void popTable();
/** Prints an error message with the corresponding line number
and file name. If message contains the substring `$$Token',
it is replaced with the value of GetString()
*/
void printError(std::string const & message) const;
/// Prints the current token table on the supplied ostream.
void printTable(std::ostream &);
/// extract string
Lexer & operator>>(std::string &);
/// extract docstring
Lexer & operator>>(docstring &);
/// extract double
Lexer & operator>>(double &);
/// extract integer
Lexer & operator>>(int &);
/// extract unsigned integer
Lexer & operator>>(unsigned int &);
/// extract bool
Lexer & operator>>(bool &);
/// Quotes a string so that reading it again with Lexer::next(true)
/// gets the original string
static std::string const quoteString(std::string const &);
private:
class Pimpl;
///
Pimpl * pimpl_;
Fix bug 3293 by Bernhard Roider: This changes the semantics of isOK() and operator(), comments from Bernhard below: With the old version of lyxlex it was _impossible_ to check whether reading an integer, float, ... succeeded or not. The current solution to check for is.bad() in some cases and in other cases use is.good() does not give the desired information. Moreover the result of is.bad() depends on the stl implementation and behaves different for linux and windows. the bug was introduced by the patch that fixed the bug that crashed lyx when "inset-insert ert" was executed from the command buffer. The lexer has the method isOK() which reflects the status of the stream is. The operators void* and ! are not really well defined (they depend on the value of is.bad()). What is missing is a test if the last reading operation was successful and thus the returned value is valid. That's what i implemented in this patch. The new rule for using the lexer: if you want to know if the lexer still has data to read (either from the stream or from the pushed token) then use "lex.isOK()". If you want to test if the last reading operation was successful then use eg. "if (lex) {...}" or unsuccessful then use eg. "if (!lex) {...}" an example: int readParam(LyxLex &lex) { int param = 1; // default value if (lex.isOK()) { // the lexer has data to read int p; // temporary variable lex >> p; if (lex) param = p; // only use the input if the reading operation was successful } return param; } git-svn-id: svn://svn.lyx.org/lyx/lyx-devel/trunk@17569 a592a061-630c-0410-9148-cb99ea01b6c8
2007-03-26 13:43:49 +00:00
///
mutable bool lastReadOk_;
};
/** Use to enable multiple exit points.
This is needed to ensure that the pop is done upon exit from methods
with more than one exit point or that can return as a response to
exceptions.
@author Lgb
*/
class PushPopHelper {
public:
///
PushPopHelper(Lexer & lexrc, keyword_item * i, int s) : lex(lexrc) {
lex.pushTable(i, s);
}
///
~PushPopHelper() {
lex.popTable();
}
///
Lexer & lex;
};
/** Avoid wrong usage of PushPopHelper.
To avoid wrong usage:
PushPopHelper(...); // wrong
PushPopHelper pph(...); // right
*/
#define PushPopHelper(x, y, z) unnamed_PushPopHelper;
// Tip gotten from Bobby Schmidt's column in C/C++ Users Journal
} // namespace lyx
#endif