presage
0.8.7
|
#include <forwardTokenizer.h>
Public Member Functions | |
ForwardTokenizer (std::istream &stream, const std::string blankspaces, const std::string separators) | |
virtual | ~ForwardTokenizer () |
virtual int | countTokens () |
virtual bool | hasMoreTokens () const |
virtual std::string | nextToken () |
virtual double | progress () const |
Tokenizes a stream from the beginning to the end. _ _ _ _ _ _ _ |_|_|_|_|_|_|_| | | | | | offend | offset offbeg
Definition at line 38 of file forwardTokenizer.h.
ForwardTokenizer::ForwardTokenizer | ( | std::istream & | stream, |
const std::string | blankspaces, | ||
const std::string | separators | ||
) |
Definition at line 27 of file forwardTokenizer.cpp.
References Tokenizer::offbeg, and Tokenizer::offset.
ForwardTokenizer::~ForwardTokenizer | ( | ) | [virtual] |
Definition at line 36 of file forwardTokenizer.cpp.
int ForwardTokenizer::countTokens | ( | ) | [virtual] |
Returns the number of tokens left.
Implements Tokenizer.
Definition at line 39 of file forwardTokenizer.cpp.
References hasMoreTokens(), nextToken(), Tokenizer::offbeg, Tokenizer::offset, and Tokenizer::stream.
bool ForwardTokenizer::hasMoreTokens | ( | ) | const [virtual] |
Tests if there are more tokens.
Implements Tokenizer.
Definition at line 61 of file forwardTokenizer.cpp.
References Tokenizer::offend, and Tokenizer::offset.
Referenced by countTokens(), main(), and ContextTracker::update().
std::string ForwardTokenizer::nextToken | ( | ) | [virtual] |
Returns the next token.
Implements Tokenizer.
Definition at line 72 of file forwardTokenizer.cpp.
References endl(), Tokenizer::isBlankspace(), Tokenizer::isSeparator(), Tokenizer::lowercaseMode(), Tokenizer::offend, Tokenizer::offset, and Tokenizer::stream.
Referenced by countTokens(), main(), and ContextTracker::update().
double ForwardTokenizer::progress | ( | ) | const [virtual] |
Returns progress percentage.
Implements Tokenizer.
Definition at line 151 of file forwardTokenizer.cpp.
References Tokenizer::offend, and Tokenizer::offset.
Referenced by main().