Python 如何使用gensim从语料库中提取短语
声明：本页面是StackOverFlow热门问题的中英对照翻译，遵循CC BY-SA 4.0协议，如果您需要使用它，必须同样遵循CC BY-SA许可，注明原文地址和作者信息，同时你必须将它归于原作者（不是我）：StackOverFlow
原文地址: http://stackoverflow.com/questions/35716121/
Warning: these are provided under cc-by-sa 4.0 license. You are free to use/share it, But you must attribute it to the original authors (not me):
StackOverFlow
How to extract phrases from corpus using gensim
提问by Prashant Puri
For preprocessing the corpus I was planing to extarct common phrases from the corpus, for this I tried using Phrasesmodel in gensim, I tried below code but it's not giving me desired output.
为了预处理语料库，我计划从语料库中提取常用短语，为此我尝试在 gensim 中使用短语模型，我尝试了下面的代码，但它没有给我想要的输出。
My code
我的代码
from gensim.models import Phrases
documents = ["the mayor of new york was there", "machine learning can be useful sometimes"]
sentence_stream = [doc.split(" ") for doc in documents]
bigram = Phrases(sentence_stream)
sent = [u'the', u'mayor', u'of', u'new', u'york', u'was', u'there']
print(bigram[sent])
Output
输出
[u'the', u'mayor', u'of', u'new', u'york', u'was', u'there']
But it should come as
但它应该是
[u'the', u'mayor', u'of', u'new_york', u'was', u'there']
But when I tried to print vocab of train data, I can see bigram, but its not working with test data, where I am going wrong?
但是当我尝试打印火车数据的词汇时，我可以看到 bigram，但它不适用于测试数据，我哪里出错了？
print bigram.vocab
defaultdict(<type 'int'>, {'useful': 1, 'was_there': 1, 'learning_can': 1, 'learning': 1, 'of_new': 1, 'can_be': 1, 'mayor': 1, 'there': 1, 'machine': 1, 'new': 1, 'was': 1, 'useful_sometimes': 1, 'be': 1, 'mayor_of': 1, 'york_was': 1, 'york': 1, 'machine_learning': 1, 'the_mayor': 1, 'new_york': 1, 'of': 1, 'sometimes': 1, 'can': 1, 'be_useful': 1, 'the': 1})
回答by Prashant Puri
I got the solution for the problem , There was two parameters I didn't take care of it which should be passed to Phrases() model, those are
我得到了问题的解决方案，有两个参数我没有注意，应该传递给Phrases() 模型，它们是
min_countignore all words and bigrams with total collected count lower than this. Bydefault it value is 5
thresholdrepresents a threshold for forming the phrases (higher means fewer phrases). A phrase of words a and b is accepted if (cnt(a, b) - min_count) * N / (cnt(a) * cnt(b)) > threshold, where N is the total vocabulary size. Bydefault it value is 10.0
min_count忽略总收集计数低于此值的所有单词和二元组。默认值为 5
阈值表示形成词组的阈值（越高表示词组越少）。如果 (cnt(a, b) - min_count) * N / (cnt(a) * cnt(b)) > threshold，则接受单词 a 和 b 的短语，其中 N 是总词汇量。默认值为 10.0
With my above train data with two statements, threshold value was 0, so I change train datasets and add those two parameters.
对于我上面带有两个语句的训练数据，阈值为0，因此我更改了训练数据集并添加了这两个参数。
My New code
我的新代码
from gensim.models import Phrases
documents = ["the mayor of new york was there", "machine learning can be useful sometimes","new york mayor was present"]
sentence_stream = [doc.split(" ") for doc in documents]
bigram = Phrases(sentence_stream, min_count=1, threshold=2)
sent = [u'the', u'mayor', u'of', u'new', u'york', u'was', u'there']
print(bigram[sent])
Output
输出
[u'the', u'mayor', u'of', u'new_york', u'was', u'there']
Gensim is really awesome :)
Gensim 真的很棒:)
