pandas 熊猫中csv的条件行读取
声明：本页面是StackOverFlow热门问题的中英对照翻译，遵循CC BY-SA 4.0协议，如果您需要使用它，必须同样遵循CC BY-SA许可，注明原文地址和作者信息，同时你必须将它归于原作者（不是我）：StackOverFlow
原文地址: http://stackoverflow.com/questions/28239529/
Warning: these are provided under cc-by-sa 4.0 license. You are free to use/share it, But you must attribute it to the original authors (not me):
StackOverFlow
conditional row read of csv in pandas
提问by eretmochelys
I have large CSVs where I'm only interested in a subset of the rows. In particular, I'd like to read in all the rows which occur before a particular condition is met.
我有大型 CSV，我只对行的子集感兴趣。特别是，我想读入在满足特定条件之前发生的所有行。
For example, if read_csvwould yield the dataframe:
例如，如果read_csv会产生数据帧：
A B C
1 34 3.20 'b'
2 24 9.21 'b'
3 34 3.32 'c'
4 24 24.3 'c'
5 35 1.12 'a'
...
1e9 42 2.15 'd'
is there some way to read all the rows in the csv until col B exceeds 10. In the above example, I'd like to read in:
有什么方法可以读取 csv 中的所有行，直到 col B 超过 10。在上面的示例中，我想读入：
A B C
1 34 3.20 'b'
2 24 9.21 'b'
3 34 3.32 'c'
4 24 24.3 'c'
I know how to throw these rows out once I've read the dataframe in, but at this point I've already spent all that computation reading them in. I do not have access to the index of the final row before reading the csv (no skipfooterplease)
我知道如何在读入数据帧后将这些行扔掉，但此时我已经花费了所有计算来读入它们。在读取 csv 之前我无法访问最后一行的索引（请不要跳过页脚）
回答by unutbu
You could read the csv in chunks. Since pd.read_csvwill return an iterator when the chunksizeparameter is specified, you can use itertools.takewhileto read only as many chunks as you need, without reading the whole file.
您可以分块读取 csv。由于pd.read_csv在chunksize指定参数时将返回迭代器，因此您可以使用itertools.takewhile仅读取所需数量的块，而无需读取整个文件。
import itertools as IT
import pandas as pd
chunksize = 10 ** 5
chunks = pd.read_csv(filename, chunksize=chunksize, header=None)
chunks = IT.takewhile(lambda chunk: chunk['B'].iloc[-1] < 10, chunks)
df = pd.concat(chunks)
mask = df['B'] < 10
df = df.loc[mask]
Or, to avoid having to use df.loc[mask]to remove unwanted rows from the last chunk, perhaps a cleaner solution would be to define a custom generator:
或者，为了避免使用df.loc[mask]从最后一个块中删除不需要的行，也许一个更简洁的解决方案是定义一个自定义生成器：
import itertools as IT
import pandas as pd
def valid(chunks):
for chunk in chunks:
mask = chunk['B'] < 10
if mask.all():
yield chunk
else:
yield chunk.loc[mask]
break
chunksize = 10 ** 5
chunks = pd.read_csv(filename, chunksize=chunksize, header=None)
df = pd.concat(valid(chunks))
回答by joanwa
I would go the easy route described here:
我会走这里描述的简单路线：
http://pandas.pydata.org/pandas-docs/stable/indexing.html#boolean-indexing
http://pandas.pydata.org/pandas-docs/stable/indexing.html#boolean-indexing
df[df['B'] > 10]
回答by jpp
You can use the built-in csvmodule to calculate the appropriate row number. Then use pd.read_csvwith the nrowsargument:
您可以使用内置csv模块来计算适当的行号。然后pd.read_csv与nrows参数一起使用：
from io import StringIO
import pandas as pd
import csv, copy
mycsv = StringIO(""" A B C
34 3.20 'b'
24 9.21 'b'
34 3.32 'c'
24 24.3 'c'
35 1.12 'a'""")
mycsv2 = copy.copy(mycsv) # copying StringIO object [for demonstration purposes]
with mycsv as fin:
reader = csv.reader(fin, delimiter=' ', skipinitialspace=True)
header = next(reader)
counter = next(idx for idx, row in enumerate(reader) if float(row[1]) > 10)
df = pd.read_csv(mycsv2, delim_whitespace=True, nrows=counter+1)
print(df)
A B C
0 34 3.20 'b'
1 24 9.21 'b'
2 34 3.32 'c'
3 24 24.30 'c'
回答by rdmolony
Building on @joanwa answer:
以@joanwa 回答为基础：
df = (pd.read_csv("filename.csv")
[lambda x: x['B'] > 10])
From Wes McKinney's "Python for Data Analysis" chapter on "Advanced pandas":
来自 Wes McKinney 关于“Advanced pandas”的“Python for Data Analysis”一章：
We cannot refer to the result of load_data until it has been assigned to the temporary variable df. To help with this, assign and many other pandas functions accept function-like arguments, also known as callables.
在将 load_data 的结果分配给临时变量 df 之前，我们无法引用它。为了解决这个问题，assign 和许多其他 Pandas 函数接受类似函数的参数，也称为可调用参数。
To show callables in action, consider ...
要在操作中显示可调用对象，请考虑...
df = load_data()
df2 = df[df['col2'] < 0]
Can be rewritten as:
可以改写为：
df = (load_data()
[lambda x: x['col2'] < 0])
