ROCBOT

昨天决定使用PyLucene做indexer, 主要考虑效率的问题,还有和其他robot配合的问题, 所以放弃了用Zend Framework中的Lucene做indexer. 大概在网上search了一下. 没有发现具体关于PyLucene的的中文Analyzer的一些资料, 不过还好在PyLucene的Readme.txt中, 对如何做customer analyzer进行描述. 其实很简单 ,

只要在你自定义的Anaylzer中实现一个tokenStream的方法, 而tokenStream的方法返回的你定义的_tokenStream的object. 然后需要在_tokenStream实现next()方法, next()方法返回的是Token.这样, 你的Analyzer就可以被indexer使用, 并通过next()方法,得到不同Token().

今天抽空把逆向最大匹配的中文分词算法做到一个自定Analyzer里面试一下, 字典先用sogou的.

原文如下:

Technically, the PyLucene programmer is not providing an ‘extension’
but a Python implementation of a set of methods encapsulated by a
Python class whose instances are wrapped by the Java proxies provided
by PyLucene.

For example, the code below, extracted from a PyLucene unit test,
defines a custom analyzer using a custom token stream that returns the
tokens ‘1′, ‘2′, ‘3′, ‘4′, ‘5′ for any document it is called on.

All that is needed in order to provide a custom analyzer in Python is
defining a class that implements a method called ‘tokenStream’. The
presence of the ‘tokenStream’ method is detected by the corresponding
SWIG type handler and the python instance passed in is wrapped by a new
Java PythonAnalyzer instance that extends Lucene’s abstract Analyzer
class.

In other words, SWIG in reverse.


class _analyzer(object):
def tokenStream(self, fieldName, reader):
class _tokenStream(object):
def __init__(self):
self.tokens = ['1', '2', '3', '4', '5']
self.increments = [1, 2, 1, 0, 1]
self.i = 0
def next(self):
if self.i == len(self.tokens):
return None
t = Token(self.tokens[self.i], self.i, self.i)
t.setPositionIncrement(self.increments[self.i])
self.i += 1
return t
return _tokenStream()

analyzer = _analyzer()

store = RAMDirectory()
writer = IndexWriter(store, analyzer, True)

d = Document()
d.add(Field.Text(”field”, “bogus”))
writer.addDocument(d)
writer.optimize()
writer.close()

最近在看Zend_Lucene_Search.查阅了一些资料. 这篇文章写的不错, 推荐一下.估计已经有很多人看过了.但是,其中的例子有点小问题, 需要修改一下才能跑起来, 我这里简单修改了一下, 放到附件中, 供以后需要的朋友下载 analyzer code.

附录代码(请不要直接使用, 请下载上面的zip中的code使用):

require_once 'Zend/Search/Lucene/Analysis/Analyzer.php'; 

class Simple_Chinese_Analyzer extends Zend_Search_Lucene_Analysis_Analyzer_Common
{

private $_position;

private $_cnStopWords = array();

public function setCnStopWords($cnStopWords){
$this->_cnStopWords = $cnStopWords;
}

public function reset()
{
$this->_position = 0;
$search = array(
",", "/", "\\", ".", ";", ":", "\"", "!", "~", "`", "^",
"(", ")", "?", "-", "t", "n", "'", "<", ">", "\r", "\r\n",
"$", "&", "%", "#", "@", "+", "=", "{", "}", "[", "]", "：",
"）", "（", "．", "。", "，", "！", "；", "“", "”", "‘", "’",
"［", "］", "、", "—", "　", "《", "》", "－", "…", "【", "】");
$this->_input = str_replace($search,' ',$this->_input);
$this->_input = str_replace($this->_cnStopWords,' ',$this->_input);
}

public function nextToken()
{
if ($this->_input === null) {
return null;
}
while ($this->_position < strlen($this->_input)) {
while ($this->_position < strlen($this->_input) &&
$this->_input[$this->_position]==' ' ) {
$this->_position++;
}
$termStartPosition = $this->_position;
$temp_char = $this->_input[$this->_position];
$isCnWord = false;
if(ord($temp_char)>127){
$i = 0;
while ($this->_position < strlen($this->_input) &&
ord( $this->_input[$this->_position] )>127) {
$this->_position = $this->_position + 3;
$i ++;
if($i==2){
$isCnWord = true;
break;
}
}
if($i==1)continue;
}else{
while ($this->_position < strlen($this->_input) &&
ctype_alnum( $this->_input[$this->_position] )) {
$this->_position++;
}
}
if ($this->_position == $termStartPosition) {
return null;
}

$token = new Zend_Search_Lucene_Analysis_Token(
substr($this->_input,
$termStartPosition,
$this->_position - $termStartPosition),
$termStartPosition,
$this->_position);
$token = $this->normalize($token);
if($isCnWord)$this->_position = $this->_position - 3;
if ($token !== null) {
return $token;
}
}
return null;
}

}

$stopWords = array('a', 'an', 'at', 'the', 'and', 'or', 'is', 'am');

$stopWordsFilter = new Zend_Search_Lucene_Analysis_TokenFilter_StopWords($stopWords);
$analyzer = new Simple_Chinese_Analyzer();
$cnStopWords = array('的');
$analyzer->setCnStopWords($cnStopWords);
$analyzer->addFilter($stopWordsFilter);
$value = 'this is  (a test)【中文】的测试, 长春市长春药店';
$analyzer->setInput($value, 'utf-8');

$position     = 0;
$tokenCounter = 0;
while (($token = $analyzer->nextToken()) !== null)
{
$tokenCounter++;
$tokens[] = $token;
}
print_r($tokens);

?>