Python Sklearn SGDClassifier 部分拟合
声明：本页面是StackOverFlow热门问题的中英对照翻译，遵循CC BY-SA 4.0协议，如果您需要使用它，必须同样遵循CC BY-SA许可，注明原文地址和作者信息，同时你必须将它归于原作者（不是我）：StackOverFlow
原文地址: http://stackoverflow.com/questions/24617356/
Warning: these are provided under cc-by-sa 4.0 license. You are free to use/share it, But you must attribute it to the original authors (not me):
StackOverFlow
Sklearn SGDClassifier partial fit
提问by David M.
I'm trying to use SGD to classify a large dataset. As the data is too large to fit into memory, I'd like to use the partial_fitmethod to train the classifier. I have selected a sample of the dataset (100,000 rows) that fits into memory to test fitvs. partial_fit:
我正在尝试使用 SGD 对大型数据集进行分类。由于数据太大而无法放入内存，我想使用partial_fit方法来训练分类器。我选择了一个适合内存的数据集样本（100,000 行）来测试fit与partial_fit：
from sklearn.linear_model import SGDClassifier
def batches(l, n):
for i in xrange(0, len(l), n):
yield l[i:i+n]
clf1 = SGDClassifier(shuffle=True, loss='log')
clf1.fit(X, Y)
clf2 = SGDClassifier(shuffle=True, loss='log')
n_iter = 60
for n in range(n_iter):
for batch in batches(range(len(X)), 10000):
clf2.partial_fit(X[batch[0]:batch[-1]+1], Y[batch[0]:batch[-1]+1], classes=numpy.unique(Y))
I then test both classifiers with an identical test set. In the first case I get an accuracy of 100%. As I understand it, SGD by default passes 5 times over the training data (n_iter = 5).
然后我用相同的测试集测试两个分类器。在第一种情况下，我的准确度为 100%。据我了解，SGD 默认通过 5 次训练数据 (n_iter = 5)。
In the second case, I have to pass 60 times over the data to reach the same accuracy.
在第二种情况下，我必须通过数据 60 次才能达到相同的精度。
Why this difference (5 vs. 60)? Or am I doing something wrong?
为什么会有这种差异（5 对 60）？还是我做错了什么？
采纳答案by David M.
I have finally found the answer. You need to shuffle the training data between each iteration, as setting shuffle=Truewhen instantiating the model will NOT shuffle the data when using partial_fit(it only applies to fit). Note: it would have been helpful to find this information on the sklearn.linear_model.SGDClassifier page.
我终于找到了答案。您需要在每次迭代之间打乱训练数据，因为在实例化模型时设置shuffle=True不会在使用partial_fit时打乱数据（它仅适用于fit）。注意：在sklearn.linear_model.SGDClassifier 页面上找到此信息会很有帮助。
The amended code reads as follows:
修改后的代码如下：
from sklearn.linear_model import SGDClassifier
import random
clf2 = SGDClassifier(loss='log') # shuffle=True is useless here
shuffledRange = range(len(X))
n_iter = 5
for n in range(n_iter):
random.shuffle(shuffledRange)
shuffledX = [X[i] for i in shuffledRange]
shuffledY = [Y[i] for i in shuffledRange]
for batch in batches(range(len(shuffledX)), 10000):
clf2.partial_fit(shuffledX[batch[0]:batch[-1]+1], shuffledY[batch[0]:batch[-1]+1], classes=numpy.unique(Y))
