presage
0.8.7
|
#include <reverseTokenizer.h>
Public Member Functions | |
ReverseTokenizer (std::istream &stream, const std::string blanks, const std::string separs) | |
virtual | ~ReverseTokenizer () |
virtual int | countTokens () |
virtual bool | hasMoreTokens () const |
virtual std::string | nextToken () |
virtual double | progress () const |
Tokenizes a stream from the end to the beginning. _ _ _ _ _ _ _ |_|_|_|_|_|_|_| | | | | | offend | offset offbeg
Definition at line 38 of file reverseTokenizer.h.
ReverseTokenizer::ReverseTokenizer | ( | std::istream & | stream, |
const std::string | blanks, | ||
const std::string | separs | ||
) |
Definition at line 27 of file reverseTokenizer.cpp.
References Tokenizer::offend, and Tokenizer::offset.
ReverseTokenizer::~ReverseTokenizer | ( | ) | [virtual] |
Definition at line 44 of file reverseTokenizer.cpp.
int ReverseTokenizer::countTokens | ( | ) | [virtual] |
Returns the number of tokens left.
Implements Tokenizer.
Definition at line 47 of file reverseTokenizer.cpp.
References hasMoreTokens(), nextToken(), Tokenizer::offend, Tokenizer::offset, and Tokenizer::stream.
bool ReverseTokenizer::hasMoreTokens | ( | ) | const [virtual] |
Tests if there are more tokens.
Implements Tokenizer.
Definition at line 69 of file reverseTokenizer.cpp.
References Tokenizer::offbeg, and Tokenizer::offset.
Referenced by countTokens().
std::string ReverseTokenizer::nextToken | ( | ) | [virtual] |
Returns the next token.
Implements Tokenizer.
Definition at line 80 of file reverseTokenizer.cpp.
References Tokenizer::isBlankspace(), Tokenizer::isSeparator(), Tokenizer::lowercaseMode(), Tokenizer::offbeg, Tokenizer::offend, Tokenizer::offset, and Tokenizer::stream.
Referenced by countTokens(), and ContextTracker::update().
double ReverseTokenizer::progress | ( | ) | const [virtual] |
Returns progress percentage.
Implements Tokenizer.
Definition at line 174 of file reverseTokenizer.cpp.
References Tokenizer::offbeg, Tokenizer::offend, and Tokenizer::offset.