com.Ostermiller.Syntax.Lexer
Class JavaToken

java.lang.Object
  com.Ostermiller.Syntax.Lexer.Token
      com.Ostermiller.Syntax.Lexer.JavaToken

public class JavaToken
extends Token

A JavaToken is a token that is returned by a lexer that is lexing a java source file. It has several attributes describing the token: The type of token, the text of the token, the line number on which it occurred, the number of characters into the input at which it started, and similarly, the number of characters into the input at which it ended.
The tokens should comply with the Java Language Specification.

Field Summary

static int	COMMENT_DOCUMENTATION The Java Language Specification explains this token
static int	COMMENT_END_OF_LINE The Java Language Specification explains this token
static int	COMMENT_TRADITIONAL The Java Language Specification explains this token
static int	ERROR_DOUBLE_SIZE The Java Language Specification explains this token
static int	ERROR_FLOAT The Java Language Specification explains this token
static int	ERROR_FLOAT_SIZE The Java Language Specification explains this token
static int	ERROR_IDENTIFIER The Java Language Specification explains this token
static int	ERROR_INTEGER_DECIMIAL_SIZE The Java Language Specification explains this token
static int	ERROR_INTEGER_HEXIDECIMAL_SIZE The Java Language Specification explains this token
static int	ERROR_INTEGER_OCTAL_SIZE The Java Language Specification explains this token
static int	ERROR_LONG_DECIMIAL_SIZE The Java Language Specification explains this token
static int	ERROR_LONG_HEXIDECIMAL_SIZE The Java Language Specification explains this token
static int	ERROR_LONG_OCTAL_SIZE The Java Language Specification explains this token
static int	ERROR_MALFORMED_CHARACTER The Java Language Specification explains this token
static int	ERROR_MALFORMED_STRING The Java Language Specification explains this token
static int	ERROR_MALFORMED_UNCLOSED_CHARACTER The Java Language Specification explains this token
static int	ERROR_MALFORMED_UNCLOSED_STRING The Java Language Specification explains this token
static int	ERROR_UNCLOSED_CHARACTER The Java Language Specification explains this token
static int	ERROR_UNCLOSED_COMMENT The Java Language Specification explains this token
static int	ERROR_UNCLOSED_STRING The Java Language Specification explains this token
static int	IDENTIFIER The Java Language Specification explains this token
static int	LITERAL_BOOLEAN The Java Language Specification explains this token
static int	LITERAL_CHARACTER The Java Language Specification explains this token
static int	LITERAL_DOUBLE The Java Language Specification explains this token
static int	LITERAL_FLOATING_POINT The Java Language Specification explains this token
static int	LITERAL_INTEGER_DECIMAL The Java Language Specification explains this token
static int	LITERAL_INTEGER_HEXIDECIMAL The Java Language Specification explains this token
static int	LITERAL_INTEGER_OCTAL The Java Language Specification explains this token
static int	LITERAL_LONG_DECIMAL The Java Language Specification explains this token
static int	LITERAL_LONG_HEXIDECIMAL The Java Language Specification explains this token
static int	LITERAL_LONG_OCTAL The Java Language Specification explains this token
static int	LITERAL_NULL The Java Language Specification explains this token
static int	LITERAL_STRING The Java Language Specification explains this token
static int	OPERATOR_ADD The Java Language Specification explains this token
static int	OPERATOR_ADD_ASSIGN The Java Language Specification explains this token
static int	OPERATOR_ASSIGN The Java Language Specification explains this token
static int	OPERATOR_BITWISE_AND The Java Language Specification explains this token
static int	OPERATOR_BITWISE_AND_ASSIGN The Java Language Specification explains this token
static int	OPERATOR_BITWISE_COMPLIMENT The Java Language Specification explains this token
static int	OPERATOR_BITWISE_OR The Java Language Specification explains this token
static int	OPERATOR_BITWISE_OR_ASSIGN The Java Language Specification explains this token
static int	OPERATOR_BITWISE_XOR The Java Language Specification explains this token
static int	OPERATOR_BITWISE_XOR_ASSIGN The Java Language Specification explains this token
static int	OPERATOR_COLON The Java Language Specification explains this token
static int	OPERATOR_DECREMENT The Java Language Specification explains this token
static int	OPERATOR_DIVIDE The Java Language Specification explains this token
static int	OPERATOR_DIVIDE_ASSIGN The Java Language Specification explains this token
static int	OPERATOR_EQUAL The Java Language Specification explains this token
static int	OPERATOR_GREATER_THAN The Java Language Specification explains this token
static int	OPERATOR_GREATER_THAN_OR_EQUAL The Java Language Specification explains this token
static int	OPERATOR_INCREMENT The Java Language Specification explains this token
static int	OPERATOR_LESS_THAN The Java Language Specification explains this token
static int	OPERATOR_LESS_THAN_OR_EQUAL The Java Language Specification explains this token
static int	OPERATOR_LOGICAL_AND The Java Language Specification explains this token
static int	OPERATOR_LOGICAL_NOT The Java Language Specification explains this token
static int	OPERATOR_LOGICAL_OR The Java Language Specification explains this token
static int	OPERATOR_MOD The Java Language Specification explains this token
static int	OPERATOR_MOD_ASSIGN The Java Language Specification explains this token
static int	OPERATOR_MULTIPLY The Java Language Specification explains this token
static int	OPERATOR_MULTIPLY_ASSIGN The Java Language Specification explains this token
static int	OPERATOR_NOT_EQUAL The Java Language Specification explains this token
static int	OPERATOR_QUESTION The Java Language Specification explains this token
static int	OPERATOR_SHIFT_LEFT The Java Language Specification explains this token
static int	OPERATOR_SHIFT_LEFT_ASSIGN The Java Language Specification explains this token
static int	OPERATOR_SHIFT_RIGHT The Java Language Specification explains this token
static int	OPERATOR_SHIFT_RIGHT_ASSIGN The Java Language Specification explains this token
static int	OPERATOR_SHIFT_RIGHT_UNSIGNED The Java Language Specification explains this token
static int	OPERATOR_SHIFT_RIGHT_UNSIGNED_ASSIGN The Java Language Specification explains this token
static int	OPERATOR_SUBTRACT The Java Language Specification explains this token
static int	OPERATOR_SUBTRACT_ASSIGN The Java Language Specification explains this token
static int	RESERVED_WORD_ABSTRACT The Java Language Specification explains this token
static int	RESERVED_WORD_BOOLEAN The Java Language Specification explains this token
static int	RESERVED_WORD_BREAK The Java Language Specification explains this token
static int	RESERVED_WORD_BYTE The Java Language Specification explains this token
static int	RESERVED_WORD_CASE The Java Language Specification explains this token
static int	RESERVED_WORD_CATCH The Java Language Specification explains this token
static int	RESERVED_WORD_CHAR The Java Language Specification explains this token
static int	RESERVED_WORD_CLASS The Java Language Specification explains this token
static int	RESERVED_WORD_CONST The Java Language Specification explains this token
static int	RESERVED_WORD_CONTINUE The Java Language Specification explains this token
static int	RESERVED_WORD_DEFAULT The Java Language Specification explains this token
static int	RESERVED_WORD_DO The Java Language Specification explains this token
static int	RESERVED_WORD_DOUBLE The Java Language Specification explains this token
static int	RESERVED_WORD_ELSE The Java Language Specification explains this token
static int	RESERVED_WORD_EXTENDS The Java Language Specification explains this token
static int	RESERVED_WORD_FINAL The Java Language Specification explains this token
static int	RESERVED_WORD_FINALLY The Java Language Specification explains this token
static int	RESERVED_WORD_FLOAT The Java Language Specification explains this token
static int	RESERVED_WORD_FOR The Java Language Specification explains this token
static int	RESERVED_WORD_GOTO The Java Language Specification explains this token
static int	RESERVED_WORD_IF The Java Language Specification explains this token
static int	RESERVED_WORD_IMPLEMENTS The Java Language Specification explains this token
static int	RESERVED_WORD_IMPORT The Java Language Specification explains this token
static int	RESERVED_WORD_INSTANCEOF The Java Language Specification explains this token
static int	RESERVED_WORD_INT The Java Language Specification explains this token
static int	RESERVED_WORD_INTERFACE The Java Language Specification explains this token
static int	RESERVED_WORD_LONG The Java Language Specification explains this token
static int	RESERVED_WORD_NATIVE The Java Language Specification explains this token
static int	RESERVED_WORD_NEW The Java Language Specification explains this token
static int	RESERVED_WORD_PACKAGE The Java Language Specification explains this token
static int	RESERVED_WORD_PRIVATE The Java Language Specification explains this token
static int	RESERVED_WORD_PROTECTED The Java Language Specification explains this token
static int	RESERVED_WORD_PUBLIC The Java Language Specification explains this token
static int	RESERVED_WORD_RETURN The Java Language Specification explains this token
static int	RESERVED_WORD_SHORT The Java Language Specification explains this token
static int	RESERVED_WORD_STATIC The Java Language Specification explains this token
static int	RESERVED_WORD_STRICTFP The Java Language Specification explains this token
static int	RESERVED_WORD_SUPER The Java Language Specification explains this token
static int	RESERVED_WORD_SWITCH The Java Language Specification explains this token
static int	RESERVED_WORD_SYNCHRONIZED The Java Language Specification explains this token
static int	RESERVED_WORD_THIS The Java Language Specification explains this token
static int	RESERVED_WORD_THROW The Java Language Specification explains this token
static int	RESERVED_WORD_THROWS The Java Language Specification explains this token
static int	RESERVED_WORD_TRANSIENT The Java Language Specification explains this token
static int	RESERVED_WORD_TRY The Java Language Specification explains this token
static int	RESERVED_WORD_VOID The Java Language Specification explains this token
static int	RESERVED_WORD_VOLATILE The Java Language Specification explains this token
static int	RESERVED_WORD_WHILE The Java Language Specification explains this token
static int	SEPARATOR_COMMA The Java Language Specification explains this token
static int	SEPARATOR_LBRACE The Java Language Specification explains this token
static int	SEPARATOR_LBRACKET The Java Language Specification explains this token
static int	SEPARATOR_LPAREN The Java Language Specification explains this token
static int	SEPARATOR_PERIOD The Java Language Specification explains this token
static int	SEPARATOR_RBRACE The Java Language Specification explains this token
static int	SEPARATOR_RBRACKET The Java Language Specification explains this token
static int	SEPARATOR_RPAREN The Java Language Specification explains this token
static int	SEPARATOR_SEMICOLON The Java Language Specification explains this token
static int	WHITE_SPACE The Java Language Specification explains this token

Fields inherited from class com.Ostermiller.Syntax.Lexer.Token

INITIAL_STATE, UNDEFINED_STATE

Constructor Summary

JavaToken(int ID, String contents, int lineNumber, int charBegin, int charEnd)
Create a new token.

JavaToken(int ID, String contents, int lineNumber, int charBegin, int charEnd, int state)
Create a new token.

Method Summary

String	errorString() get a String that explains the error, if this token is an error.
int	getCharBegin() get the offset into the input in characters at which this token started
int	getCharEnd() get the offset into the input in characters at which this token ended
String	getContents() get the contents of this token
String	getDescription() A description of this token.
int	getID() get the ID number of this token
int	getLineNumber() get the line number of the input on which this token started
int	getState() Get an integer representing the state the tokenizer is in after returning this token.
boolean	isComment() Checks this token to see if it is a comment.
boolean	isError() Checks this token to see if it is an Error.
boolean	isIdentifier() Checks this token to see if it is an identifier.
boolean	isLiteral() Checks this token to see if it is a literal.
boolean	isOperator() Checks this token to see if it is a Operator.
boolean	isReservedWord() Checks this token to see if it is a reserved word.
boolean	isSeparator() Checks this token to see if it is a Separator.
boolean	isWhiteSpace() Checks this token to see if it is White Space.
String	toString() get a representation of this token as a human readable string.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, wait, wait, wait

Field Detail

RESERVED_WORD_ABSTRACT

public static final int RESERVED_WORD_ABSTRACT

The Java Language Specification explains this token

See Also:

Constant Field Values

RESERVED_WORD_BOOLEAN

public static final int RESERVED_WORD_BOOLEAN

The Java Language Specification explains this token

See Also:

Constant Field Values

RESERVED_WORD_BREAK

public static final int RESERVED_WORD_BREAK

The Java Language Specification explains this token

See Also:

Constant Field Values

RESERVED_WORD_BYTE

public static final int RESERVED_WORD_BYTE

The Java Language Specification explains this token

See Also:

Constant Field Values

RESERVED_WORD_CASE

public static final int RESERVED_WORD_CASE

The Java Language Specification explains this token

See Also:

Constant Field Values

RESERVED_WORD_CATCH

public static final int RESERVED_WORD_CATCH

The Java Language Specification explains this token

See Also:

Constant Field Values

RESERVED_WORD_CHAR

public static final int RESERVED_WORD_CHAR

The Java Language Specification explains this token

See Also:

Constant Field Values

RESERVED_WORD_CLASS

public static final int RESERVED_WORD_CLASS

The Java Language Specification explains this token

See Also:

Constant Field Values

RESERVED_WORD_CONST

public static final int RESERVED_WORD_CONST

The Java Language Specification explains this token

See Also:

Constant Field Values

RESERVED_WORD_CONTINUE

public static final int RESERVED_WORD_CONTINUE

The Java Language Specification explains this token

See Also:

Constant Field Values

RESERVED_WORD_DEFAULT

public static final int RESERVED_WORD_DEFAULT

The Java Language Specification explains this token

See Also:

Constant Field Values

RESERVED_WORD_DO

public static final int RESERVED_WORD_DO

The Java Language Specification explains this token

See Also:

Constant Field Values

RESERVED_WORD_DOUBLE

public static final int RESERVED_WORD_DOUBLE

The Java Language Specification explains this token

See Also:

Constant Field Values

RESERVED_WORD_ELSE

public static final int RESERVED_WORD_ELSE

The Java Language Specification explains this token

See Also:

Constant Field Values

RESERVED_WORD_EXTENDS

public static final int RESERVED_WORD_EXTENDS

The Java Language Specification explains this token

See Also:

Constant Field Values

RESERVED_WORD_FINAL

public static final int RESERVED_WORD_FINAL

The Java Language Specification explains this token

See Also:

Constant Field Values

RESERVED_WORD_FINALLY

public static final int RESERVED_WORD_FINALLY

The Java Language Specification explains this token

See Also:

Constant Field Values

RESERVED_WORD_FLOAT

public static final int RESERVED_WORD_FLOAT

The Java Language Specification explains this token

See Also:

Constant Field Values

RESERVED_WORD_FOR

public static final int RESERVED_WORD_FOR

The Java Language Specification explains this token

See Also:

Constant Field Values

RESERVED_WORD_GOTO

public static final int RESERVED_WORD_GOTO

The Java Language Specification explains this token

See Also:

Constant Field Values

RESERVED_WORD_IF

public static final int RESERVED_WORD_IF

The Java Language Specification explains this token

See Also:

Constant Field Values

RESERVED_WORD_IMPLEMENTS

public static final int RESERVED_WORD_IMPLEMENTS

The Java Language Specification explains this token

See Also:

Constant Field Values

RESERVED_WORD_IMPORT

public static final int RESERVED_WORD_IMPORT

The Java Language Specification explains this token

See Also:

Constant Field Values

RESERVED_WORD_INSTANCEOF

public static final int RESERVED_WORD_INSTANCEOF

The Java Language Specification explains this token

See Also:

Constant Field Values

RESERVED_WORD_INT

public static final int RESERVED_WORD_INT

The Java Language Specification explains this token

See Also:

Constant Field Values

RESERVED_WORD_INTERFACE

public static final int RESERVED_WORD_INTERFACE

The Java Language Specification explains this token

See Also:

Constant Field Values

RESERVED_WORD_LONG

public static final int RESERVED_WORD_LONG

The Java Language Specification explains this token

See Also:

Constant Field Values

RESERVED_WORD_NATIVE

public static final int RESERVED_WORD_NATIVE

The Java Language Specification explains this token

See Also:

Constant Field Values

RESERVED_WORD_NEW

public static final int RESERVED_WORD_NEW

The Java Language Specification explains this token

See Also:

Constant Field Values

RESERVED_WORD_PACKAGE

public static final int RESERVED_WORD_PACKAGE

The Java Language Specification explains this token

See Also:

Constant Field Values

RESERVED_WORD_PRIVATE

public static final int RESERVED_WORD_PRIVATE

The Java Language Specification explains this token

See Also:

Constant Field Values

RESERVED_WORD_PROTECTED

public static final int RESERVED_WORD_PROTECTED

The Java Language Specification explains this token

See Also:

Constant Field Values

RESERVED_WORD_PUBLIC

public static final int RESERVED_WORD_PUBLIC

The Java Language Specification explains this token

See Also:

Constant Field Values

RESERVED_WORD_RETURN

public static final int RESERVED_WORD_RETURN

The Java Language Specification explains this token

See Also:

Constant Field Values

RESERVED_WORD_SHORT

public static final int RESERVED_WORD_SHORT

The Java Language Specification explains this token

See Also:

Constant Field Values

RESERVED_WORD_STATIC

public static final int RESERVED_WORD_STATIC

The Java Language Specification explains this token

See Also:

Constant Field Values

RESERVED_WORD_STRICTFP

public static final int RESERVED_WORD_STRICTFP

The Java Language Specification explains this token

See Also:

Constant Field Values

RESERVED_WORD_SUPER

public static final int RESERVED_WORD_SUPER

The Java Language Specification explains this token

See Also:

Constant Field Values

RESERVED_WORD_SWITCH

public static final int RESERVED_WORD_SWITCH

The Java Language Specification explains this token

See Also:

Constant Field Values

RESERVED_WORD_SYNCHRONIZED

public static final int RESERVED_WORD_SYNCHRONIZED

The Java Language Specification explains this token

See Also:

Constant Field Values

RESERVED_WORD_THIS

public static final int RESERVED_WORD_THIS

The Java Language Specification explains this token

See Also:

Constant Field Values

RESERVED_WORD_THROW

public static final int RESERVED_WORD_THROW

The Java Language Specification explains this token

See Also:

Constant Field Values

RESERVED_WORD_THROWS

public static final int RESERVED_WORD_THROWS

The Java Language Specification explains this token

See Also:

Constant Field Values

RESERVED_WORD_TRANSIENT

public static final int RESERVED_WORD_TRANSIENT

The Java Language Specification explains this token

See Also:

Constant Field Values

RESERVED_WORD_TRY

public static final int RESERVED_WORD_TRY

The Java Language Specification explains this token

See Also:

Constant Field Values

RESERVED_WORD_VOID

public static final int RESERVED_WORD_VOID

The Java Language Specification explains this token

See Also:

Constant Field Values

RESERVED_WORD_VOLATILE

public static final int RESERVED_WORD_VOLATILE

The Java Language Specification explains this token

See Also:

Constant Field Values

RESERVED_WORD_WHILE

public static final int RESERVED_WORD_WHILE

The Java Language Specification explains this token

See Also:

Constant Field Values

IDENTIFIER

public static final int IDENTIFIER

The Java Language Specification explains this token

See Also:

Constant Field Values

LITERAL_BOOLEAN

public static final int LITERAL_BOOLEAN

The Java Language Specification explains this token

See Also:

Constant Field Values

LITERAL_INTEGER_DECIMAL

public static final int LITERAL_INTEGER_DECIMAL

The Java Language Specification explains this token

See Also:

Constant Field Values

LITERAL_INTEGER_OCTAL

public static final int LITERAL_INTEGER_OCTAL

The Java Language Specification explains this token

See Also:

Constant Field Values

LITERAL_INTEGER_HEXIDECIMAL

public static final int LITERAL_INTEGER_HEXIDECIMAL

The Java Language Specification explains this token

See Also:

Constant Field Values

LITERAL_LONG_DECIMAL

public static final int LITERAL_LONG_DECIMAL

The Java Language Specification explains this token

See Also:

Constant Field Values

LITERAL_LONG_OCTAL

public static final int LITERAL_LONG_OCTAL

The Java Language Specification explains this token

See Also:

Constant Field Values

LITERAL_LONG_HEXIDECIMAL

public static final int LITERAL_LONG_HEXIDECIMAL

The Java Language Specification explains this token

See Also:

Constant Field Values

LITERAL_FLOATING_POINT

public static final int LITERAL_FLOATING_POINT

The Java Language Specification explains this token

See Also:

Constant Field Values

LITERAL_DOUBLE

public static final int LITERAL_DOUBLE

The Java Language Specification explains this token

See Also:

Constant Field Values

LITERAL_CHARACTER

public static final int LITERAL_CHARACTER

The Java Language Specification explains this token

See Also:

Constant Field Values

LITERAL_STRING

public static final int LITERAL_STRING

The Java Language Specification explains this token

See Also:

Constant Field Values

LITERAL_NULL

public static final int LITERAL_NULL

The Java Language Specification explains this token

See Also:

Constant Field Values

SEPARATOR_LPAREN

public static final int SEPARATOR_LPAREN

The Java Language Specification explains this token

See Also:

Constant Field Values

SEPARATOR_RPAREN

public static final int SEPARATOR_RPAREN

The Java Language Specification explains this token

See Also:

Constant Field Values

SEPARATOR_LBRACE

public static final int SEPARATOR_LBRACE

The Java Language Specification explains this token

See Also:

Constant Field Values

SEPARATOR_RBRACE

public static final int SEPARATOR_RBRACE

The Java Language Specification explains this token

See Also:

Constant Field Values

SEPARATOR_LBRACKET

public static final int SEPARATOR_LBRACKET

The Java Language Specification explains this token

See Also:

Constant Field Values

SEPARATOR_RBRACKET

public static final int SEPARATOR_RBRACKET

The Java Language Specification explains this token

See Also:

Constant Field Values

SEPARATOR_SEMICOLON

public static final int SEPARATOR_SEMICOLON

The Java Language Specification explains this token

See Also:

Constant Field Values

SEPARATOR_COMMA

public static final int SEPARATOR_COMMA

The Java Language Specification explains this token

See Also:

Constant Field Values

SEPARATOR_PERIOD

public static final int SEPARATOR_PERIOD

The Java Language Specification explains this token

See Also:

Constant Field Values

OPERATOR_GREATER_THAN

public static final int OPERATOR_GREATER_THAN

The Java Language Specification explains this token

See Also:

Constant Field Values

OPERATOR_LESS_THAN

public static final int OPERATOR_LESS_THAN

The Java Language Specification explains this token

See Also:

Constant Field Values

OPERATOR_LESS_THAN_OR_EQUAL

public static final int OPERATOR_LESS_THAN_OR_EQUAL

The Java Language Specification explains this token

See Also:

Constant Field Values

OPERATOR_GREATER_THAN_OR_EQUAL

public static final int OPERATOR_GREATER_THAN_OR_EQUAL

The Java Language Specification explains this token

See Also:

Constant Field Values

OPERATOR_EQUAL

public static final int OPERATOR_EQUAL

The Java Language Specification explains this token

See Also:

Constant Field Values

OPERATOR_NOT_EQUAL

public static final int OPERATOR_NOT_EQUAL

The Java Language Specification explains this token

See Also:

Constant Field Values

OPERATOR_LOGICAL_NOT

public static final int OPERATOR_LOGICAL_NOT

The Java Language Specification explains this token

See Also:

Constant Field Values

OPERATOR_LOGICAL_AND

public static final int OPERATOR_LOGICAL_AND

The Java Language Specification explains this token

See Also:

Constant Field Values

OPERATOR_LOGICAL_OR

public static final int OPERATOR_LOGICAL_OR

The Java Language Specification explains this token

See Also:

Constant Field Values

OPERATOR_ADD

public static final int OPERATOR_ADD

The Java Language Specification explains this token

See Also:

Constant Field Values

OPERATOR_SUBTRACT

public static final int OPERATOR_SUBTRACT

The Java Language Specification explains this token

See Also:

Constant Field Values

OPERATOR_MULTIPLY

public static final int OPERATOR_MULTIPLY

The Java Language Specification explains this token

See Also:

Constant Field Values

OPERATOR_DIVIDE

public static final int OPERATOR_DIVIDE

The Java Language Specification explains this token

See Also:

Constant Field Values

OPERATOR_MOD

public static final int OPERATOR_MOD

The Java Language Specification explains this token

See Also:

Constant Field Values

OPERATOR_BITWISE_COMPLIMENT

public static final int OPERATOR_BITWISE_COMPLIMENT

The Java Language Specification explains this token

See Also:

Constant Field Values

OPERATOR_BITWISE_AND

public static final int OPERATOR_BITWISE_AND

The Java Language Specification explains this token

See Also:

Constant Field Values

OPERATOR_BITWISE_OR

public static final int OPERATOR_BITWISE_OR

The Java Language Specification explains this token

See Also:

Constant Field Values

OPERATOR_BITWISE_XOR

public static final int OPERATOR_BITWISE_XOR

The Java Language Specification explains this token

See Also:

Constant Field Values

OPERATOR_SHIFT_LEFT

public static final int OPERATOR_SHIFT_LEFT

The Java Language Specification explains this token

See Also:

Constant Field Values

OPERATOR_SHIFT_RIGHT

public static final int OPERATOR_SHIFT_RIGHT

The Java Language Specification explains this token

See Also:

Constant Field Values

OPERATOR_SHIFT_RIGHT_UNSIGNED

public static final int OPERATOR_SHIFT_RIGHT_UNSIGNED

The Java Language Specification explains this token

See Also:

Constant Field Values

OPERATOR_ASSIGN

public static final int OPERATOR_ASSIGN

The Java Language Specification explains this token

See Also:

Constant Field Values

OPERATOR_ADD_ASSIGN

public static final int OPERATOR_ADD_ASSIGN

The Java Language Specification explains this token

See Also:

Constant Field Values

OPERATOR_SUBTRACT_ASSIGN

public static final int OPERATOR_SUBTRACT_ASSIGN

The Java Language Specification explains this token

See Also:

Constant Field Values

OPERATOR_MULTIPLY_ASSIGN

public static final int OPERATOR_MULTIPLY_ASSIGN

The Java Language Specification explains this token

See Also:

Constant Field Values

OPERATOR_DIVIDE_ASSIGN

public static final int OPERATOR_DIVIDE_ASSIGN

The Java Language Specification explains this token

See Also:

Constant Field Values

OPERATOR_MOD_ASSIGN

public static final int OPERATOR_MOD_ASSIGN

The Java Language Specification explains this token

See Also:

Constant Field Values

OPERATOR_BITWISE_AND_ASSIGN

public static final int OPERATOR_BITWISE_AND_ASSIGN

The Java Language Specification explains this token

See Also:

Constant Field Values

OPERATOR_BITWISE_OR_ASSIGN

public static final int OPERATOR_BITWISE_OR_ASSIGN

The Java Language Specification explains this token

See Also:

Constant Field Values

OPERATOR_BITWISE_XOR_ASSIGN

public static final int OPERATOR_BITWISE_XOR_ASSIGN

The Java Language Specification explains this token

See Also:

Constant Field Values

OPERATOR_SHIFT_LEFT_ASSIGN

public static final int OPERATOR_SHIFT_LEFT_ASSIGN

The Java Language Specification explains this token

See Also:

Constant Field Values

OPERATOR_SHIFT_RIGHT_ASSIGN

public static final int OPERATOR_SHIFT_RIGHT_ASSIGN

The Java Language Specification explains this token

See Also:

Constant Field Values

OPERATOR_SHIFT_RIGHT_UNSIGNED_ASSIGN

public static final int OPERATOR_SHIFT_RIGHT_UNSIGNED_ASSIGN

The Java Language Specification explains this token

See Also:

Constant Field Values

OPERATOR_INCREMENT

public static final int OPERATOR_INCREMENT

The Java Language Specification explains this token

See Also:

Constant Field Values

OPERATOR_DECREMENT

public static final int OPERATOR_DECREMENT

The Java Language Specification explains this token

See Also:

Constant Field Values

OPERATOR_QUESTION

public static final int OPERATOR_QUESTION

The Java Language Specification explains this token

See Also:

Constant Field Values

OPERATOR_COLON

public static final int OPERATOR_COLON

The Java Language Specification explains this token

See Also:

Constant Field Values

COMMENT_TRADITIONAL

public static final int COMMENT_TRADITIONAL

The Java Language Specification explains this token

See Also:

Constant Field Values

COMMENT_END_OF_LINE

public static final int COMMENT_END_OF_LINE

The Java Language Specification explains this token

See Also:

Constant Field Values

COMMENT_DOCUMENTATION

public static final int COMMENT_DOCUMENTATION

The Java Language Specification explains this token

See Also:

Constant Field Values

WHITE_SPACE

public static final int WHITE_SPACE

The Java Language Specification explains this token

See Also:

Constant Field Values

ERROR_IDENTIFIER

public static final int ERROR_IDENTIFIER

The Java Language Specification explains this token

See Also:

Constant Field Values

ERROR_UNCLOSED_STRING

public static final int ERROR_UNCLOSED_STRING

The Java Language Specification explains this token

See Also:

Constant Field Values

ERROR_MALFORMED_STRING

public static final int ERROR_MALFORMED_STRING

The Java Language Specification explains this token

See Also:

Constant Field Values

ERROR_MALFORMED_UNCLOSED_STRING

public static final int ERROR_MALFORMED_UNCLOSED_STRING

The Java Language Specification explains this token

See Also:

Constant Field Values

ERROR_UNCLOSED_CHARACTER

public static final int ERROR_UNCLOSED_CHARACTER

The Java Language Specification explains this token

See Also:

Constant Field Values

ERROR_MALFORMED_CHARACTER

public static final int ERROR_MALFORMED_CHARACTER

The Java Language Specification explains this token

See Also:

Constant Field Values

ERROR_MALFORMED_UNCLOSED_CHARACTER

public static final int ERROR_MALFORMED_UNCLOSED_CHARACTER

The Java Language Specification explains this token

See Also:

Constant Field Values

ERROR_INTEGER_DECIMIAL_SIZE

public static final int ERROR_INTEGER_DECIMIAL_SIZE

The Java Language Specification explains this token

See Also:

Constant Field Values

ERROR_INTEGER_OCTAL_SIZE

public static final int ERROR_INTEGER_OCTAL_SIZE

The Java Language Specification explains this token

See Also:

Constant Field Values

ERROR_INTEGER_HEXIDECIMAL_SIZE

public static final int ERROR_INTEGER_HEXIDECIMAL_SIZE

The Java Language Specification explains this token

See Also:

Constant Field Values

ERROR_LONG_DECIMIAL_SIZE

public static final int ERROR_LONG_DECIMIAL_SIZE

The Java Language Specification explains this token

See Also:

Constant Field Values

ERROR_LONG_OCTAL_SIZE

public static final int ERROR_LONG_OCTAL_SIZE

The Java Language Specification explains this token

See Also:

Constant Field Values

ERROR_LONG_HEXIDECIMAL_SIZE

public static final int ERROR_LONG_HEXIDECIMAL_SIZE

The Java Language Specification explains this token

See Also:

Constant Field Values

ERROR_FLOAT_SIZE

public static final int ERROR_FLOAT_SIZE

The Java Language Specification explains this token

See Also:

Constant Field Values

ERROR_DOUBLE_SIZE

public static final int ERROR_DOUBLE_SIZE

The Java Language Specification explains this token

See Also:

Constant Field Values

ERROR_FLOAT

public static final int ERROR_FLOAT

The Java Language Specification explains this token

See Also:

Constant Field Values

ERROR_UNCLOSED_COMMENT

public static final int ERROR_UNCLOSED_COMMENT

The Java Language Specification explains this token

See Also:

Constant Field Values

Constructor Detail

JavaToken

public JavaToken(int ID,
                 String contents,
                 int lineNumber,
                 int charBegin,
                 int charEnd)

Create a new token. The constructor is typically called by the lexer

Parameters:

ID - the id number of the token

contents - A string representing the text of the token

lineNumber - the line number of the input on which this token started

charBegin - the offset into the input in characters at which this token started

charEnd - the offset into the input in characters at which this token ended

JavaToken

public JavaToken(int ID,
                 String contents,
                 int lineNumber,
                 int charBegin,
                 int charEnd,
                 int state)

Create a new token. The constructor is typically called by the lexer

Parameters:

ID - the id number of the token

contents - A string representing the text of the token

lineNumber - the line number of the input on which this token started

charBegin - the offset into the input in characters at which this token started

charEnd - the offset into the input in characters at which this token ended

state - the state the tokenizer is in after returning this token.

Method Detail

getState

public int getState()

Get an integer representing the state the tokenizer is in after returning this token. Those who are interested in incremental tokenizing for performance reasons will want to use this method to figure out where the tokenizer may be restarted. The tokenizer starts in Token.INITIAL_STATE, so any time that it reports that it has returned to this state, the tokenizer may be restarted from there.

Specified by:

getState in class Token

getID

public int getID()

get the ID number of this token

Specified by:

getID in class Token

Returns:

the id number of the token

getContents

public String getContents()

get the contents of this token

Specified by:

getContents in class Token

Returns:

A string representing the text of the token

getLineNumber

public int getLineNumber()

get the line number of the input on which this token started

Specified by:

getLineNumber in class Token

Returns:

the line number of the input on which this token started

getCharBegin

public int getCharBegin()

get the offset into the input in characters at which this token started

Specified by:

getCharBegin in class Token

Returns:

the offset into the input in characters at which this token started

getCharEnd

public int getCharEnd()

get the offset into the input in characters at which this token ended

Specified by:

getCharEnd in class Token

Returns:

the offset into the input in characters at which this token ended

isReservedWord

public boolean isReservedWord()

Checks this token to see if it is a reserved word. Reserved words are explained in Java Language Specification.

Returns:

true if this token is a reserved word, false otherwise

isIdentifier

public boolean isIdentifier()

Checks this token to see if it is an identifier. Identifiers are explained in Java Language Specification.

Returns:

true if this token is an identifier, false otherwise

isLiteral

public boolean isLiteral()

Checks this token to see if it is a literal. Literals are explained in Java Language Specification.

Returns:

true if this token is a literal, false otherwise

isSeparator

public boolean isSeparator()

Checks this token to see if it is a Separator. Separators are explained in Java Language Specification.

Returns:

true if this token is a Separator, false otherwise

isOperator

public boolean isOperator()

Checks this token to see if it is a Operator. Operators are explained in Java Language Specification.

Returns:

true if this token is a Operator, false otherwise

isComment

public boolean isComment()

Checks this token to see if it is a comment.

Specified by:

isComment in class Token

Returns:

true if this token is a comment, false otherwise

isWhiteSpace

public boolean isWhiteSpace()

Checks this token to see if it is White Space. Usually tabs, line breaks, form feed, spaces, etc.

Specified by:

isWhiteSpace in class Token

Returns:

true if this token is White Space, false otherwise

isError

public boolean isError()

Checks this token to see if it is an Error. Unfinished comments, numbers that are too big, unclosed strings, etc.

Specified by:

isError in class Token

Returns:

true if this token is an Error, false otherwise

getDescription

public String getDescription()

A description of this token. The description should be appropriate for syntax highlighting. For example "comment" is returned for a comment.

Specified by:

getDescription in class Token

Returns:

a description of this token.

errorString

public String errorString()

get a String that explains the error, if this token is an error.

Specified by:

errorString in class Token

Returns:

a String that explains the error, if this token is an error, null otherwise.

toString

public String toString()

get a representation of this token as a human readable string. The format of this string is subject to change and should only be used for debugging purposes.

Overrides:

toString in class Object

Returns:

a string representation of this token