scala-spark：如何在groupby之后过滤RDD
声明：本页面是StackOverFlow热门问题的中英对照翻译，遵循CC BY-SA 4.0协议，如果您需要使用它，必须同样遵循CC BY-SA许可，注明原文地址和作者信息，同时你必须将它归于原作者（不是我）：StackOverFlow
原文地址: http://stackoverflow.com/questions/31412527/
Warning: these are provided under cc-by-sa 4.0 license. You are free to use/share it, But you must attribute it to the original authors (not me):
StackOverFlow
scala-spark: How to filter RDD after groupby
提问by add-semi-colons
I have started with a RDD that has pipe separated string. I have processed the data and gotten into following format:
我已经开始使用具有管道分隔字符串的 RDD。我已经处理了数据并得到了以下格式：
((0001F46468,239394055),(7665710590658745,-414963169),0,1420276980302)
((0001F46468,239394055),(8016905020647641,183812619),1,1420347885727)
((0001F46468,239394055),(6633110906332136,294201185),1,1420398323110)
((0001F46468,239394055),(6633110906332136,294201185),0,1420451687525)
((0001F46468,239394055),(7722056727387069,1396896294),1,1420537469065)
((0001F46468,239394055),(7722056727387069,1396896294),1,1420623297340)
((0001F46468,239394055),(8045651092287275,-4814845),1,1420720722185)
((0001F46468,239394055),(5170029699836178,-1332814297),0,1420750531018)
((0001F46468,239394055),(7722056727387069,1396896294),0,1420807545137)
((0001F46468,239394055),(4784119468604853,1287554938),1,1421050087824)
Just to give an high level view on description of the data. You can think first element in the main tuple (first tuple) as a user identification, second tuple as a product identification, and third element is user's preference on the product. (for future reference I am going to mark above data set as val userData)
只是为了对数据的描述给出一个高层次的看法。您可以将主元组（第一元组）中的第一个元素视为用户标识，第二个元组作为产品标识，第三个元素是用户对产品的偏好。（为了将来参考，我将上面的数据集标记为val userData）
My goal is that if user has casted both positive (1) and negative (0) preference for a product only take the record with positive. For example:
我的目标是，如果用户对产品同时投了正面 (1) 和负面 (0) 偏好，则只记录正面的记录。例如：
((0001F46468,239394055),(6633110906332136,294201185),1,1420398323110)
((0001F46468,239394055),(6633110906332136,294201185),0,1420451687525)
I only want to keep
我只想保留
((0001F46468,239394055),(6633110906332136,294201185),1,1420398323110)
So I grouped the users by user-product tuple
(0001F46468,239394055),(6633110906332136,294201185
所以我按用户产品元组对用户进行分组
(0001F46468,239394055),(6633110906332136,294201185
val groupedFiltered = userData.groupBy(x => (x._1, x._2)).map(u => {
for(k <- u._2) {
if(k._3 > 0)
u
}
})
But that return empty tuples.
但这会返回空元组。
So I took the following approach:
所以我采取了以下方法：
val groupedFiltered = userData. groupBy(x => (x._1, x._2)).flatMap(u => u._2).filter(m => m._3 > 0)
((47734739656882457,-1782798434),(7585453414177905,-461779195),1,1422013413082)
((47734739656882457,-1782798434),(7585453414177905,-461779195),1,1422533237758)
((55218449094787901,-1374432022),(6227831620534109,1195766703),1,1420410603596)
((71212122719822610,-807015489),(6769904840922490,1642054117),1,1422549467554)
((75414197560031509,1830213715),(6724015489416254,-1389654186),1,1420196951100)
((60422797294995441,734266951),(6335216393920738,1528026712),1,1421161253600)
((35091051395844216,451349158),(8135854751464083,-1751839326),1,1422083101033)
((16647193023519619,990937787),(5384884550662007,-910998857),1,1420659873572)
((43355867025936022,-945669937),(7336240855866885,518993644),1,1420880078266)
((12188366927481231,-2007889717),(5336507724485344,363519858),1,1420827788022)
This was promising but it looks like its taking all the records that has zero where I only want if the user has 1 and 0 for same item only keep the one with 1.
这是有希望的，但它看起来像是在我想要的所有记录为零的情况下，如果用户对同一项目有 1 和 0，则只保留带有 1 的记录。
回答by Peter Neyens
You could only keep the maximum user preference from the grouped results.
您只能从分组结果中保留最大用户偏好。
userData
// group by user and product
.groupBy(x => (x._1, x._2))
// only keep the maximum user preference per user/product
.mapValues(_.maxBy(_._3))
// only keep the values
.values
