删除 Python Pandas 中的所有重复行
声明：本页面是StackOverFlow热门问题的中英对照翻译，遵循CC BY-SA 4.0协议，如果您需要使用它，必须同样遵循CC BY-SA许可，注明原文地址和作者信息，同时你必须将它归于原作者（不是我）：StackOverFlow
原文地址: http://stackoverflow.com/questions/23667369/
Warning: these are provided under cc-by-sa 4.0 license. You are free to use/share it, But you must attribute it to the original authors (not me):
StackOverFlow
Drop all duplicate rows in Python Pandas
提问by Jamie Bull
The pandasdrop_duplicatesfunction is great for "uniquifying" a dataframe. However, one of the keyword arguments to pass is take_last=Trueor take_last=False, while I would like to drop all rows which are duplicates across a subset of columns. Is this possible?
该pandasdrop_duplicates功能非常适合“统一”数据帧。但是，要传递的关键字参数之一是take_last=Trueor take_last=False，而我想删除跨列子集重复的所有行。这可能吗？
A B C
0 foo 0 A
1 foo 1 A
2 foo 1 B
3 bar 1 A
As an example, I would like to drop rows which match on columns Aand Cso this should drop rows 0 and 1.
作为一个例子，我想下降匹配列的行A和C所以这应该丢弃的行0和1。
采纳答案by Ben
This is much easier in pandas now with drop_duplicatesand the keep parameter.
现在有了drop_duplicates和 keep 参数，这在Pandas 中要容易得多。
import pandas as pd
df = pd.DataFrame({"A":["foo", "foo", "foo", "bar"], "B":[0,1,1,1], "C":["A","A","B","A"]})
df.drop_duplicates(subset=['A', 'C'], keep=False)
回答by CT Zhu
Actually, drop rows 0 and 1 only requires (any observations containing matched A and C is kept.):
实际上，删除第 0 行和第 1 行只需要（保留包含匹配的 A 和 C 的任何观察值。）：
In [335]:
df['AC']=df.A+df.C
In [336]:
print df.drop_duplicates('C', take_last=True) #this dataset is a special case, in general, one may need to first drop_duplicates by 'c' and then by 'a'.
A B C AC
2 foo 1 B fooB
3 bar 1 A barA
[2 rows x 4 columns]
But I suspect what you really want is this (one observation containing matched A and C is kept.):
但我怀疑你真正想要的是这个（保留一个包含匹配 A 和 C 的观察结果。）：
In [337]:
print df.drop_duplicates('AC')
A B C AC
0 foo 0 A fooA
2 foo 1 B fooB
3 bar 1 A barA
[3 rows x 4 columns]
Edit:
编辑：
Now it is much clearer, therefore:
现在更清楚了，因此：
In [352]:
DG=df.groupby(['A', 'C'])
print pd.concat([DG.get_group(item) for item, value in DG.groups.items() if len(value)==1])
A B C
2 foo 1 B
3 bar 1 A
[2 rows x 3 columns]
回答by HYRY
use groupbyand filter
使用groupby和filter
import pandas as pd
df = pd.DataFrame({"A":["foo", "foo", "foo", "bar"], "B":[0,1,1,1], "C":["A","A","B","A"]})
df.groupby(["A", "C"]).filter(lambda df:df.shape[0] == 1)
回答by Jake
Just want to add to Ben's answer on drop_duplicates:
只想添加到本关于drop_duplicates的回答中：
keep: {‘first', ‘last', False}, default ‘first'
keep: {'first', 'last', False}, 默认为 'first'
first : Drop duplicates except for the first occurrence.
last : Drop duplicates except for the last occurrence.
False : Drop all duplicates.
first : 除第一次出现外，删除重复项。
last ：删除除最后一次出现的重复项。
False ：删除所有重复项。
So setting keepto False will give you desired answer.
所以设置keep为 False 会给你想要的答案。
DataFrame.drop_duplicates(*args, **kwargs) Return DataFrame with duplicate rows removed, optionally only considering certain columns
Parameters: subset : column label or sequence of labels, optional Only consider certain columns for identifying duplicates, by default use all of the columns keep : {‘first', ‘last', False}, default ‘first' first : Drop duplicates except for the first occurrence. last : Drop duplicates except for the last occurrence. False : Drop all duplicates. take_last : deprecated inplace : boolean, default False Whether to drop duplicates in place or to return a copy cols : kwargs only argument of subset [deprecated] Returns: deduplicated : DataFrame
DataFrame.drop_duplicates(*args, **kwargs) 返回删除重复行的 DataFrame，可选择仅考虑某些列
参数：子集：列标签或标签序列，可选仅考虑某些列来识别重复项，默认情况下使用所有列保留：{'first', 'last', False}，默认'first' first：删除重复项，除了对于第一次出现。last ：删除除最后一次出现的重复项。False ：删除所有重复项。take_last : 不推荐使用的地方 : 布尔值，默认为 False 是否删除重复项或返回副本 cols : kwargs 子集的唯一参数 [已弃用] 返回：重复数据：DataFrame
回答by Ramanujam Allam
If you want result to be stored in another dataset:
如果要将结果存储在另一个数据集中：
df.drop_duplicates(keep=False)
or
或者
df.drop_duplicates(keep=False, inplace=False)
If same dataset needs to be updated:
如果需要更新相同的数据集：
df.drop_duplicates(keep=False, inplace=True)
Above examples will remove all duplicates and keep one, similar to DISTINCT *in SQL
上面的例子将删除所有重复项并保留一个，类似于DISTINCT *SQL
回答by Priyansh gupta
Try these various things
尝试这些不同的东西
df = pd.DataFrame({"A":["foo", "foo", "foo", "bar","foo"], "B":[0,1,1,1,1], "C":["A","A","B","A","A"]})
>>>df.drop_duplicates( "A" , keep='first')
or
或者
>>>df.drop_duplicates( keep='first')
or
或者
>>>df.drop_duplicates( keep='last')
