java hadoop:1 映射多个减速器,每个减速器具有不同的功能?可能的?
声明:本页面是StackOverFlow热门问题的中英对照翻译,遵循CC BY-SA 4.0协议,如果您需要使用它,必须同样遵循CC BY-SA许可,注明原文地址和作者信息,同时你必须将它归于原作者(不是我):StackOverFlow
原文地址: http://stackoverflow.com/questions/7019546/
Warning: these are provided under cc-by-sa 4.0 license. You are free to use/share it, But you must attribute it to the original authors (not me):
StackOverFlow
hadoop : 1 map multiple reducers with each reducer having different functionality? possible?
提问by daydreamer
Here is an example:
下面是一个例子:
Is it possible to have same mapper run against multiple reducers at the same time? like
是否可以同时针对多个减速器运行相同的映射器?喜欢
map output : {1:[1,2,3,4,5,4,3,2], 4:[5,4,6,7,8,9,5,3,3,2], 3:[1,5,4,3,5,6,7,8,9,1], so on}
reducer1 : sum of all numbers
reducer2 : average of all numbers
reducer3 : mode of all numbers
act on the the same key like
reducer1 output: {1:sum of values, 2:sum of values, and so on}
reducer2 output: {1:avg of values, 2: avg of values and so on}
reducer3 output: {1:mode of values, 2: mode of values, and so on}
and so on..Please let me know.
等等..请让我知道。
采纳答案by Andrew T Finnell
I really wanted to answer this for you but it's already been asked. Hadoop one Map and multiple Reduce
我真的很想为你回答这个问题,但已经有人问过了。Hadoop 一个 Map 和多个 Reduce
回答by Thomas Jungblut
No it is not possible. But you can implement your own reducer that will calculate all the stuff for you and output it.
不,这是不可能的。但是你可以实现你自己的减速器,它会为你计算所有的东西并输出它。
You can make an custom writable for that.
您可以为此自定义可写。
回答by thuang513
http://www.cloudera.com/blog/2011/04/simple-moving-average-secondary-sort-and-mapreduce-part-3/
http://www.cloudera.com/blog/2011/04/simple-moving-average-secondary-sort-and-mapreduce-part-3/
This actually does pretty much what you are looking for.
这实际上几乎可以满足您的要求。