Linux Certif

Toute la documentation sur la certification Linux LPI

HCT::Lang::Token.3pm

NAME

HCT::Lang::Token - Definition of tokens.

DESCRIPTION

A token is a categorized block of text. The block of text corresponding to the token is known as a lexeme. A lexical analyzer divides a sequence of characters into tokens (tokenization) and categorizes them according to function, giving them meaning. A token can look like anything; it just needs to be a useful part of the structured text.

The object of token created by class "HCT::Lang::Token" has such features: name, type and list of patterns. The lexemes will be gotten by lexer by using patterns.

The token name is an unique name that will be returned by lexer to parser.

The patterns are a list of patterns, that will be used for definition of the token name and token lexeme. All patterns described by regular expressions.

The token type is a number, that means position/preority, where the token have to be used. For instance, tokens can be divided on keywords, operators, numbers and identifiers, and the order in which lexer should use them will be: operators, numbers, keywords and identifiers. According to this the order can be: operators - 3, numbers - 2, keywrods - 1, identifiers - 0.

FUNCTIONS

The functions above will be exported by default.

EMPTY_TOKEN

When the lexer reach the end of input, it returns the '' empty token with an "undef" value.

CONSTRUCTOR

new ()

Creates a new "HCT::Language::Token" object.

METHODS

get_patterns ()

Returns a list of token patterns.

get_name ()

Returns token name.

get_type ()

Returns token type.