Token
property Token: Char read;
Example
procedure InspectCurrentToken(Parser: TParser);
begin
if Parser = nil then Exit;
if Parser.Token = toEOF then Exit;
if Parser.Token = toSymbol then
begin
// Read Parser.TokenString before calling NextToken.
end;
end;
Usage
Returns the character code for the parser's current identifier, string, number, punctuation or end-of-file token.
Additional Technical Info
Token identifies the parser's current lexical item. Create reads the first token before returning, and each NextToken replaces the property with the newly scanned token code. Reading the property does not advance.
| Code | Meaning |
|---|---|
toEOF / 0 | End of input |
toSymbol / 1 | Identifier/symbol |
toString / 2 | Ordinary decoded string |
toInteger / 3 | Integer-shaped source text |
toFloat / 4 | Float-shaped source text |
native toWString / 5 | Wide numeric-hash string; its intended getter is not exposed by Velox |
| any other Char | Literal punctuation character, such as =, { or ] |
Velox registers toEOF through toFloat as hidden compiler constants: scripts can use those five names even though they do not appear as public Code Library entries. Native toWString is not registered. A numeric token code describes how text was scanned, not proof that it converts successfully; use TokenInt or TokenFloat and handle conversion errors.
TokenString retrieves text for the current state. CheckToken asserts a code without consuming it. TokenComponentIdent is unusual: it can consume a dotted suffix while Token remains toSymbol.
HexToBinary also performs internal source consumption and finishes with Token set to the closing } rather than the token after it. Account for these method-specific state transitions before calling NextToken.
The property is unsynchronized mutable state. Do not inspect it concurrently with parser navigation from another execution path.
The example is source-reviewed only; no token was inspected.
External references
- Embarcadero:
TParser.Token- native current-token property. - Free Pascal:
TParser.Token- compatible token-code property.