Java 如何从 Lucene TokenStream 获取令牌？
声明：本页面是StackOverFlow热门问题的中英对照翻译，遵循CC BY-SA 4.0协议，如果您需要使用它，必须同样遵循CC BY-SA许可，注明原文地址和作者信息，同时你必须将它归于原作者（不是我）：StackOverFlow
原文地址: http://stackoverflow.com/questions/2638200/
Warning: these are provided under cc-by-sa 4.0 license. You are free to use/share it, But you must attribute it to the original authors (not me):
StackOverFlow
How to get a Token from a Lucene TokenStream?
提问by Eric Wilson
I'm trying to use Apache Lucene for tokenizing, and I am baffled at the process to obtain Tokens from a TokenStream.
我正在尝试使用 Apache Lucene 进行标记化，但我对从TokenStream.
The worst part is that I'm looking at the comments in the JavaDocs that address my question.
最糟糕的是，我正在查看解决我的问题的 JavaDocs 中的评论。
Somehow, an AttributeSourceis supposed to be used, rather than Tokens. I'm totally at a loss.
不知何故，AttributeSource应该使用an ，而不是Tokens。我完全不知所措。
Can anyone explain how to get token-like information from a TokenStream?
谁能解释一下如何从 TokenStream 中获取类似令牌的信息？
采纳答案by Adam Paynter
Yeah, it's a little convoluted (compared to the good ol' way), but this should do it:
是的，这有点令人费解（与好的方式相比），但这应该可以：
TokenStream tokenStream = analyzer.tokenStream(fieldName, reader);
OffsetAttribute offsetAttribute = tokenStream.getAttribute(OffsetAttribute.class);
TermAttribute termAttribute = tokenStream.getAttribute(TermAttribute.class);
while (tokenStream.incrementToken()) {
int startOffset = offsetAttribute.startOffset();
int endOffset = offsetAttribute.endOffset();
String term = termAttribute.term();
}
Edit: The newway
编辑：新方法
According to Donotello, TermAttributehas been deprecated in favor of CharTermAttribute. According to jpountz (and Lucene's documentation), addAttributeis more desirable than getAttribute.
根据 Donotello 的说法，TermAttribute已弃用CharTermAttribute. 根据 jpountz（和 Lucene 的文档），addAttribute比getAttribute.
TokenStream tokenStream = analyzer.tokenStream(fieldName, reader);
OffsetAttribute offsetAttribute = tokenStream.addAttribute(OffsetAttribute.class);
CharTermAttribute charTermAttribute = tokenStream.addAttribute(CharTermAttribute.class);
tokenStream.reset();
while (tokenStream.incrementToken()) {
int startOffset = offsetAttribute.startOffset();
int endOffset = offsetAttribute.endOffset();
String term = charTermAttribute.toString();
}
回答by yegor256
This is how it should be (a clean version of Adam's answer):
这应该是这样的（亚当答案的干净版本）：
TokenStream stream = analyzer.tokenStream(null, new StringReader(text));
CharTermAttribute cattr = stream.addAttribute(CharTermAttribute.class);
stream.reset();
while (stream.incrementToken()) {
System.out.println(cattr.toString());
}
stream.end();
stream.close();
回答by William Price
There are two variations in the OP question:
OP问题有两种变体：
- What is "the process to obtain Tokens from a TokenStream"?
- "Can anyone explain how to get token-like information from a TokenStream?"
- 什么是“从 TokenStream 获取令牌的过程”？
- “谁能解释一下如何从 TokenStream 中获取类似令牌的信息？”
Recent versions of the Lucene documentation for Tokensay (emphasis added):
Lucene 文档的Token最新版本说（强调添加）：
NOTE: As of 2.9 ... it is not necessaryto use Token anymore, with the new TokenStream API it can be used as convenience class that implements all Attributes, which is especially useful to easily switch from the old to the new TokenStream API.
注意：从 2.9 开始……不再需要使用 Token，有了新的 TokenStream API，它可以用作实现所有属性的便利类，这对于轻松从旧的 TokenStream API 切换到新的 TokenStream API 尤其有用。
And TokenStreamsays its API:
并TokenStream说它的API：
... has moved from being Token-based to Attribute-based ... the preferred way to store the information of a Token is to use AttributeImpls.
...已从基于 Token 转变为基于 Attribute ... 存储 Token 信息的首选方式是使用 AttributeImpls。
The other answers to this question cover #2 above: how to get token-likeinformation from a TokenStreamin the "new" recommended way using attributes. Reading through the documentation, the Lucene developers suggest that this change was made, in part, to reduce the number of individual objects created at a time.
这个问题的其他答案涵盖了上面的#2：如何使用属性以“新”推荐的方式从 a获取类似令牌的信息TokenStream。通过阅读文档，Lucene 开发人员建议进行此更改，部分原因是为了减少一次创建的单个对象的数量。
But as some people have pointed out in the comments of those answers, they don't directly answer #1: how do you get a Tokenif you really want/need that type?
但正如一些人在这些答案的评论中指出的那样，他们并没有直接回答 #1：Token如果你真的想要/需要那种类型，你怎么得到一个？
With the same API change that makes TokenStreaman AttributeSource, Tokennow implements Attributeand can be used with TokenStream.addAttributejust like the other answers show for CharTermAttributeand OffsetAttribute. So they really did answer that part of the original question, they simply didn't show it.
在相同的API的变化，使TokenStream一个AttributeSource，Token现在实现Attribute并可以搭配TokenStream.addAttribute就像其他的答案显示CharTermAttribute和OffsetAttribute。所以他们确实回答了原始问题的那部分，他们只是没有表现出来。
It is important that while this approach will allow you to access Tokenwhile you're looping, it is still only a single object no matter how many logical tokens are in the stream. Every call to incrementToken()will change the state of the Tokenreturned from addAttribute; So if your goal is to build a collection of different Tokenobjects to be used outside the loop then you will need to do extra work to make a newTokenobject as a (deep?) copy.
重要的是，虽然这种方法允许您Token在循环时访问，但无论流中有多少逻辑标记，它仍然只是一个对象。每次调用incrementToken()都会改变Token从返回的状态addAttribute；因此，如果您的目标是构建Token要在循环外使用的不同对象的集合，那么您将需要做额外的工作来将新Token对象创建为（深？）副本。
回答by Flamingo
For the latest version of lucene 7.3.1
最新版lucene 7.3.1
// Test the tokenizer
Analyzer testAnalyzer = new CJKAnalyzer();
String testText = "Test Tokenizer";
TokenStream ts = testAnalyzer.tokenStream("context", new StringReader(testText));
OffsetAttribute offsetAtt = ts.addAttribute(OffsetAttribute.class);
try {
ts.reset(); // Resets this stream to the beginning. (Required)
while (ts.incrementToken()) {
// Use AttributeSource.reflectAsString(boolean)
// for token stream debugging.
System.out.println("token: " + ts.reflectAsString(true));
System.out.println("token start offset: " + offsetAtt.startOffset());
System.out.println(" token end offset: " + offsetAtt.endOffset());
}
ts.end(); // Perform end-of-stream operations, e.g. set the final offset.
} finally {
ts.close(); // Release resources associated with this stream.
}
Reference: https://lucene.apache.org/core/7_3_1/core/org/apache/lucene/analysis/package-summary.html
参考：https: //lucene.apache.org/core/7_3_1/core/org/apache/lucene/analysis/package-summary.html
