marf.nlp.util
Class NLPStreamTokenizer

java.lang.Object
  java.io.StreamTokenizer
      marf.nlp.util.NLPStreamTokenizer

public class NLPStreamTokenizer
extends java.io.StreamTokenizer

NLP Stream Tokenizer. Allows pushing back multiple tokens and has a reader reverence to be able to reset it.

$Id: NLPStreamTokenizer.java,v 1.19 2008/02/19 03:20:42 mokhov Exp $

Since:

0.3.0.2

Version:

$Revision: 1.19 $

Author:

Serguei Mokhov

Field Summary

protected int	iPushBackupLevel Default push backup level of 2.
protected java.util.Stack	oPushBackup A stack to push back tokens.
protected java.io.Reader	oReader Keep a reference to the Reader ourselves to be able to reset().
protected Token	oTopToken Reference to the token on the top of our own stack.

Fields inherited from class java.io.StreamTokenizer

nval, sval, TT_EOF, TT_EOL, TT_NUMBER, TT_WORD, ttype

Constructor Summary

NLPStreamTokenizer(java.io.InputStream poInputStream)

NLPStreamTokenizer(java.io.Reader poReader)
NLP Stream Tokenizer based on a reader.

Method Summary

static java.lang.String	getMARFSourceCodeRevision() Retrieves class' revision.
java.lang.String	getNextToken() Returns a next token from the NLP stream or the stack if any were pushed back.
int	lineno() Returns the current line number (of the latest/top token).
int	nextToken() Retrieves the next token from the stream or local stack.
void	pushBack() Overridden to place tokens back onto to stack, virtually of any number of tokens.
void	reset() Resets the internal reader's stream.

Methods inherited from class java.io.StreamTokenizer

commentChar, eolIsSignificant, lowerCaseMode, ordinaryChar, ordinaryChars, parseNumbers, quoteChar, resetSyntax, slashSlashComments, slashStarComments, toString, whitespaceChars, wordChars

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, wait, wait, wait

Field Detail

oReader

protected java.io.Reader oReader

Keep a reference to the Reader ourselves to be able to reset().

See Also:

reset()

oPushBackup

protected java.util.Stack oPushBackup

A stack to push back tokens.

iPushBackupLevel

protected int iPushBackupLevel

Default push backup level of 2.

oTopToken

protected Token oTopToken

Reference to the token on the top of our own stack.

Since:

0.3.0.5

Constructor Detail

NLPStreamTokenizer

public NLPStreamTokenizer(java.io.Reader poReader)

NLP Stream Tokenizer based on a reader.

Parameters:

poReader - reader to use to read tokens/lexemes

NLPStreamTokenizer

public NLPStreamTokenizer(java.io.InputStream poInputStream)

Parameters:

poInputStream -

Since:

0.3.0.6

Method Detail

getNextToken

public java.lang.String getNextToken()
                              throws java.io.IOException

Returns a next token from the NLP stream or the stack if any were pushed back.

Returns:

next lexeme token if there is any or null

Throws:

java.io.IOException

reset

public void reset()
           throws java.io.IOException

Resets the internal reader's stream.

Throws:

java.io.IOException

nextToken

public int nextToken()
              throws java.io.IOException

Retrieves the next token from the stream or local stack. The tokens retrieved from the local stack always if the stack is not empty. When the stack is empty the nextToken() of the parent is called.

Overrides:

nextToken in class java.io.StreamTokenizer

Returns:

the value of the token type (ttype field)

Throws:

java.io.IOException

Since:

0.3.0.5

See Also:

StreamTokenizer.ttype

pushBack

public void pushBack()

Overridden to place tokens back onto to stack, virtually of any number of tokens.

Overrides:

pushBack in class java.io.StreamTokenizer

Since:

0.3.0.5

lineno

public int lineno()

Returns the current line number (of the latest/top token). Overridden to account for tokens stored in the local stack.

Overrides:

lineno in class java.io.StreamTokenizer

Returns:

the current line number of this tokenizer

Since:

0.3.0.5

getMARFSourceCodeRevision

public static java.lang.String getMARFSourceCodeRevision()

Retrieves class' revision.

Returns:

revision string