Puma Reference Manual Puma::Token Class Reference



Puma::Token Class Reference

Abstraction of a language token. More...

#include <Puma/Token.h>

Inheritance diagram for Puma::Token:
Inheritance graph

List of all members.

Public Types

enum  {
  ID_END_OF_FILE = -1,
  ID_UNKNOWN = -2,
  ID_ERROR = -3,
  ID_WARNING = -4
}
 Special token types. More...

Public Member Functions

 Token (int type=0, LanguageID lang=LanguageID(0), const char *text="")
 Constructor.
 Token (const Token &copy)
 Copy-constructor.
virtual ~Token ()
 Destructor.
void print (ostream &os) const
 Print the token text on the given stream.
virtual ListElementduplicate ()
 Duplicate this token.
Unitunit () const
 Get the unit this token belongs to.
void reset (int type=0, const char *text=0, LanguageID lang=LanguageID(0))
 Reset the token object.
void location (const Location &loc)
 Set the location of the token (usually the line and column in a file).
void cont_lines (Array< int > *cl)
 Set the continues lines appeared in this token.
int line_breaks () const
 Count the number of line breaks in the text of this token.
void macro_generated ()
 Set the token as being macro generated.
void macro_call ()
 Set the token as being a macro call.
const char * text () const
 Get the token text.
DStringdtext () const
 Get the token text.
char * get_static_text () const
 Get the static token text.
int type () const
 Get the type of the token (see Puma::CTokens).
const Locationlocation () const
 Get the location of the token (usually the line and column in a file).
bool is_macro_generated () const
 Check if the token is macro generated.
bool is_macro_call () const
 Check if the token is a macro call.
bool is_core () const
 Check if this is a core language token.
bool is_identifier () const
 Check if the token is an identifier.
bool is_keyword () const
 Check if the token is a keyword.
bool is_wildcard () const
 Check if this is a wildcard token.
bool is_preprocessor () const
 Check if this is a preprocessor token.
bool is_directive () const
 Check if this is a compiler directive.
bool is_whitespace () const
 Check if this is a white-space token.
bool is_comment () const
 Check if this is a comment.
bool is_macro_op () const
 Check if this is function-like macro operand.
bool is_open () const
 Check if this is a left parenthesis.
bool is_comma () const
 Check if this is a comma.
bool is_close () const
 Check if this is a right parenthesis.
void * operator new (size_t)
 Own operator new reusing memory.
void operator delete (void *)
 Own delete operator.

Static Public Member Functions

static char * get_static_text (int token_type)
 Get the static token text for the given token type.

Static Public Attributes

static LanguageID comment_id
 Comment tokens.
static LanguageID pre_id
 Preprocessor tokens.
static LanguageID dir_id
 Compiler directives.
static LanguageID cpp_id
 C/C++ core language tokens.
static LanguageID white_id
 White-spaces.
static LanguageID wildcard_id
 Wildcards.
static LanguageID keyword_id
 Keywords.
static LanguageID open_id
 Left parenthesis.
static LanguageID macro_op_id
 Function-like macro operands.
static LanguageID identifier_id
 Identifiers.
static LanguageID comma_id
 Comma.
static LanguageID close_id
 Right parenthesis.

Detailed Description

Abstraction of a language token.

A token is created by a scanner (see Puma::CScanner) as part a token chain (Puma::Unit). It encapsulates a lexical unit of the scanned text.

The information provided about a lexical unit is

  • the token type
  • the location in the scanned text
  • the portion of the scanned text represented by this token
  • the language of the token

Member Enumeration Documentation

anonymous enum

Special token types.

Enumerator:
ID_END_OF_FILE 

End-of-file token.

ID_UNKNOWN 

Unknown token type.

ID_ERROR 

Error token type.

ID_WARNING 

Warning token type.


Constructor & Destructor Documentation

Puma::Token::Token ( int  type = 0,
LanguageID  lang = LanguageID(0),
const char *  text = "" 
)

Constructor.

Parameters:
typeThe token type.
langThe language of the token.
textThe optional token text (defaults to the empty string).
Puma::Token::Token ( const Token copy)

Copy-constructor.

Parameters:
copyThe token to copy.
virtual Puma::Token::~Token ( ) [virtual]

Destructor.

Destroys the token text.


Member Function Documentation

void Puma::Token::cont_lines ( Array< int > *  cl) [inline]

Set the continues lines appeared in this token.

Parameters:
clThe continues line array.

Get the token text.

virtual ListElement* Puma::Token::duplicate ( ) [inline, virtual]

Duplicate this token.

Returns:
The copy of this token. Has to be destroyed by the caller.

Implements Puma::ListElement.

Get the static token text.

Only for keywords, operators, and so on with a constant token text.

Returns:
The static text or NULL for tokens with dynamic text (like identifiers).
static char* Puma::Token::get_static_text ( int  token_type) [static]

Get the static token text for the given token type.

Only for keywords, operators, and so on with a constant token text.

Returns:
The static text or NULL for tokens with dynamic text (like identifiers).
bool Puma::Token::is_close ( ) const [inline]

Check if this is a right parenthesis.

bool Puma::Token::is_comma ( ) const [inline]

Check if this is a comma.

bool Puma::Token::is_comment ( ) const [inline]

Check if this is a comment.

bool Puma::Token::is_core ( ) const [inline]

Check if this is a core language token.

bool Puma::Token::is_directive ( ) const [inline]

Check if this is a compiler directive.

bool Puma::Token::is_identifier ( ) const [inline]

Check if the token is an identifier.

bool Puma::Token::is_keyword ( ) const [inline]

Check if the token is a keyword.

bool Puma::Token::is_macro_call ( ) const [inline]

Check if the token is a macro call.

bool Puma::Token::is_macro_generated ( ) const [inline]

Check if the token is macro generated.

bool Puma::Token::is_macro_op ( ) const [inline]

Check if this is function-like macro operand.

bool Puma::Token::is_open ( ) const [inline]

Check if this is a left parenthesis.

bool Puma::Token::is_preprocessor ( ) const [inline]

Check if this is a preprocessor token.

bool Puma::Token::is_whitespace ( ) const [inline]

Check if this is a white-space token.

bool Puma::Token::is_wildcard ( ) const [inline]

Check if this is a wildcard token.

int Puma::Token::line_breaks ( ) const

Count the number of line breaks in the text of this token.

void Puma::Token::location ( const Location loc) [inline]

Set the location of the token (usually the line and column in a file).

Parameters:
locThe location.
const Location& Puma::Token::location ( ) const [inline]

Get the location of the token (usually the line and column in a file).

void Puma::Token::macro_call ( ) [inline]

Set the token as being a macro call.

void Puma::Token::macro_generated ( ) [inline]

Set the token as being macro generated.

void Puma::Token::operator delete ( void *  )

Own delete operator.

void* Puma::Token::operator new ( size_t  )

Own operator new reusing memory.

void Puma::Token::print ( ostream &  os) const

Print the token text on the given stream.

Parameters:
osThe output stream.
void Puma::Token::reset ( int  type = 0,
const char *  text = 0,
LanguageID  lang = LanguageID(0) 
)

Reset the token object.

Parameters:
typeThe new token type.
textThe new token text.
langThe new token language.
const char* Puma::Token::text ( ) const

Get the token text.

int Puma::Token::type ( ) const [inline]

Get the type of the token (see Puma::CTokens).

Unit* Puma::Token::unit ( ) const

Get the unit this token belongs to.


Member Data Documentation

Right parenthesis.

Comment tokens.

C/C++ core language tokens.

Compiler directives.

Identifiers.

Keywords.

Function-like macro operands.

Left parenthesis.

Preprocessor tokens.

White-spaces.

Wildcards.