Class: Tokenizer

ProtoBuf.DotProto. Tokenizer

prototype tokenizer

Constructor

new Tokenizer(proto)

Constructs a new Tokenizer.

Parameters:
Name Type Description
proto string

Proto to tokenize

Source:

Members

index :number

Current index.

Type:
  • number
Source:

line :number

Current line.

Type:
  • number
Source:

source :string

Source to parse.

Type:
  • string
Source:

(non-null) stack :Array.<string>

Token stack.

Type:
  • Array.<string>
Source:

Methods

next() → (nullable) {string}

Gets the next token and advances by one.

Source:
Returns:

Token or null on EOF

Type
string

omit(expected) → {boolean}

Omits an optional token.

Parameters:
Name Type Description
expected string

Expected optional token

Source:
Returns:

true if the token exists

Type
boolean

peek() → (nullable) {string}

Peeks for the next token.

Source:
Returns:

Token or null on EOF

Type
string

skip(expected)

Skips a specific token and throws if it differs.

Parameters:
Name Type Description
expected string

Expected token

Source:
Throws:

If the actual token differs

Type
Error

toString() → {string}

Returns a string representation of this object.

Source:
Returns:

String representation as of "Tokenizer(index/length)"

Type
string