Plucene::Analysis::Standard::StandardTokenizer.3pm

Langue: en

Version: 2008-03-01 (debian - 07/07/09)

Section: 3 (Bibliothèques de fonctions)

Sommaire

NAME

Plucene::Analysis::Standard::StandardTokenizer - standard tokenizer

SYNOPSIS

         # isa Plucene::Analysis::CharTokenizer

DESCRIPTION

This is the standard tokenizer.

This should be a good tokenizer for most European-language documents.

METHODS


token_re

The regular expression for tokenising.

normalize

Remove 's and .