如何以块的形式迭代两个 Pandas 数据帧
声明：本页面是StackOverFlow热门问题的中英对照翻译，遵循CC BY-SA 4.0协议，如果您需要使用它，必须同样遵循CC BY-SA许可，注明原文地址和作者信息，同时你必须将它归于原作者（不是我）：StackOverFlow
原文地址: http://stackoverflow.com/questions/20432328/
Warning: these are provided under cc-by-sa 4.0 license. You are free to use/share it, But you must attribute it to the original authors (not me):
StackOverFlow
How to iterate over two pandas dataframes in chunks
提问by Learner
For a machine learning task I need to deal with data sets that are too big to fit in my memory all at once, so I need to break it down into chunk. Fortunately, pandas.read_csv has a parameter chunk_size in which you can specify the amount of data that you want to use for analysis and then loop over the data set in chunks with a for loop, which looks like this:
对于机器学习任务，我需要处理太大而无法一次性全部放入我的记忆中的数据集，因此我需要将其分解为块。幸运的是，pandas.read_csv 有一个参数 chunk_size，您可以在其中指定要用于分析的数据量，然后使用 for 循环在块中循环数据集，如下所示：
#This example can be found at http://pandas.pydata.org/pandas-docs/dev/io.html
In [120]: reader = pd.read_table('tmp.sv', sep='|', chunksize=4)
In [121]: reader
<pandas.io.parsers.TextFileReader at 0xaa94ad0>
In [122]: for chunk in reader:
.....: print(chunk)
.....:
Unnamed: 0 0 1 2 3
0 0 0.469112 -0.282863 -1.509059 -1.135632
1 1 1.212112 -0.173215 0.119209 -1.044236
2 2 -0.861849 -2.104569 -0.494929 1.071804
3 3 0.721555 -0.706771 -1.039575 0.271860
[4 rows x 5 columns]
Unnamed: 0 0 1 2 3
0 4 -0.424972 0.567020 0.276232 -1.087401
1 5 -0.673690 0.113648 -1.478427 0.524988
2 6 0.404705 0.577046 -1.715002 -1.039268
3 7 -0.370647 -1.157892 -1.344312 0.844885
[4 rows x 5 columns]
Unnamed: 0 0 1 2 3
0 8 1.075770 -0.10905 1.643563 -1.469388
1 9 0.357021 -0.67460 -1.776904 -0.968914
[2 rows x 5 columns].
But I need both the train and test sets in the for loop for my machine learning algorithm to make predictions on the chunks of data, and I don't know how I could do that. I am basically looking for this:
但是我的机器学习算法需要 for 循环中的训练集和测试集来对数据块进行预测，但我不知道如何做到这一点。我基本上是在寻找这个：
#pseudo code
result = []
train = pd.read('train_set',chunksize = some_number)
test = pd.read('test_set',chunksize = some_number)
for chunk in train and test:
result.append(do_machine_learning(train,test))
save_result(result)
update:So I tried Any Hayden's solution but it gave me a new error when I try to access specific parts of the data:
更新：所以我尝试了 Any Hayden 的解决方案，但是当我尝试访问数据的特定部分时，它给了我一个新错误：
print("getting train set")
train = pd.read_csv(os.path.join(dir,"Train.csv"),chunksize = 200000)
print("getting test set")
test = pd.read_csv(os.path.join(dir,"Test.csv"),chunksize = 200000)
result = []
for chunk in train:
print("transforming train,test,labels into numpy arrays")
labels = np.array(train)[:,3]
train = np.array(train)[:,2]
test = np.array(test)[:,2]
print("getting estimator and predictions")
result.append(stochastic_gradient(train,test))
print("got everything")
result = np.array(result)
traceback:
追溯：
Traceback (most recent call last):
File "C:\Users\Ano\workspace\final_submission\src\rf.py", line 38, in <module>
main()
File "C:\Users\Ano\workspace\final_submission\src\rf.py", line 18, in main
labels = np.array(train)[:,3]
IndexError: 0-d arrays can only use a single () or a list of newaxes (and a single ...) as an index
采纳答案by Andy Hayden
In a for loop you have access to the variables in the current scope:
在 for 循环中，您可以访问当前范围内的变量：
In [11]: a = [1, 2, 3]
In [12]: b = 4
In [13]: for L in a: # no need to "and b"
print L, b
1 4
2 4
3 4
Be careful, this means assigning in a for loop overwrites variables:
小心，这意味着在 for 循环中赋值会覆盖变量：
In [14]: for b in a:
print b
1
2
3
In [15]: b
Out[15]: 3
To iterate through two iterables at the same time use zip:
要同时遍历两个可迭代对象，请使用 zip：
In [21]: c = [4, 5, 6]
In [22]: zip(a, c)
Out[22]: [(1, 4), (2, 5), (3, 6)]
In python 2 this is a list, so evaluated in memory (not so in python 3). You can use izip, it's iterator accomplice.
在 python 2 中，这是一个列表，因此在内存中进行评估（在 python 3 中不是这样）。你可以使用 izip，它是迭代器的帮凶。
In [23]: from itertools import izip # in python 3, just use zip
In [24]: for La, Lc in izip(a, c):
print La, Lb
1 4
2 5
3 6
