presage  0.8.7
Public Member Functions
ForwardTokenizer Class Reference

#include <forwardTokenizer.h>

Inheritance diagram for ForwardTokenizer:
Inheritance graph
[legend]
Collaboration diagram for ForwardTokenizer:
Collaboration graph
[legend]

List of all members.

Public Member Functions

 ForwardTokenizer (std::istream &stream, const std::string blankspaces, const std::string separators)
virtual ~ForwardTokenizer ()
virtual int countTokens ()
virtual bool hasMoreTokens () const
virtual std::string nextToken ()
virtual double progress () const

Detailed Description

Tokenizes a stream from the beginning to the end. _ _ _ _ _ _ _ |_|_|_|_|_|_|_| | | | | | offend | offset offbeg

Definition at line 38 of file forwardTokenizer.h.


Constructor & Destructor Documentation

ForwardTokenizer::ForwardTokenizer ( std::istream &  stream,
const std::string  blankspaces,
const std::string  separators 
)

Definition at line 27 of file forwardTokenizer.cpp.

References Tokenizer::offbeg, and Tokenizer::offset.

Definition at line 36 of file forwardTokenizer.cpp.


Member Function Documentation

int ForwardTokenizer::countTokens ( ) [virtual]

Returns the number of tokens left.

Implements Tokenizer.

Definition at line 39 of file forwardTokenizer.cpp.

References hasMoreTokens(), nextToken(), Tokenizer::offbeg, Tokenizer::offset, and Tokenizer::stream.

Here is the call graph for this function:

bool ForwardTokenizer::hasMoreTokens ( ) const [virtual]

Tests if there are more tokens.

Implements Tokenizer.

Definition at line 61 of file forwardTokenizer.cpp.

References Tokenizer::offend, and Tokenizer::offset.

Referenced by countTokens(), main(), and ContextTracker::update().

Here is the caller graph for this function:

std::string ForwardTokenizer::nextToken ( ) [virtual]

Returns the next token.

Implements Tokenizer.

Definition at line 72 of file forwardTokenizer.cpp.

References endl(), Tokenizer::isBlankspace(), Tokenizer::isSeparator(), Tokenizer::lowercaseMode(), Tokenizer::offend, Tokenizer::offset, and Tokenizer::stream.

Referenced by countTokens(), main(), and ContextTracker::update().

Here is the call graph for this function:

Here is the caller graph for this function:

double ForwardTokenizer::progress ( ) const [virtual]

Returns progress percentage.

Implements Tokenizer.

Definition at line 151 of file forwardTokenizer.cpp.

References Tokenizer::offend, and Tokenizer::offset.

Referenced by main().

Here is the caller graph for this function:


The documentation for this class was generated from the following files: