MapReduce

hadoop入门手册1：hadoop【2.7.1】【多节点】集群配置【必知配置知识1】

由 两盒软妹~` 提交于 2019-12-18 04:24:17
问题导读 1.说说你对集群配置的认识？ 2.集群配置的配置项你了解多少？ 3.下面内容让你对集群的配置有了什么新的认识？ 目的 目的1： 这个文档描述了如何安装配置hadoop集群，从几个节点到上千节点。为了学习hadoop，你可能先从单节点入手 (查看 Single Node Setup).这里有中文版 hadoop2.7【单节点】单机、伪分布、分布式安装指导 http://www.aboutyun.com/thread-12798-1-1.html 这个文档不包括：hadoop在安全模式下配置和HA【高可用配置】，后面在更新 目的2： 我们看了很多集群配置文档，你是否静下心来，想集群配置到底是怎么一回事。 准备 1.安装Java 2.下载hadoop包 ################################## 包集合： hadoop家族、strom、spark、Linux、flume等jar包、安装包汇总下载(持续更新) http://www.aboutyun.com/thread-8178-1-1.html ################################## 安装 安装hadoop集群包括：解压包，配置hadoop，划分主节点和子节点。 集群中可以将namenode和ResourceManager分布在不同的机器上，这些称之为 master

hadoop method to send output to multiple directories

由 风格不统一 提交于 2019-12-18 03:45:34
问题 My MapReduce job processes data by dates and needs to write output to a certain folder structure. Current expectation is to generate out put in following structure: 2013 01 02 .. 2012 01 02 .. etc. At any time, I get only upto 12 months of data, So, I am using MultipleOutputs class to create 12 outputs using the following function in the driver: public void createOutputs(){ Calendar c = Calendar.getInstance(); String monthStr, pathStr; // Create multiple outputs for last 12 months // TODO

Java 8 grouping by from one-to-many

由 ＊爱你＆永不变心＊ 提交于 2019-12-18 03:35:17
问题 I want to learn how to use the Java 8 syntax with streams and got a bit stuck. It's easy enough to groupingBy when you have one key for every value. But what if I have a List of keys for every value and still want to categorise them with groupingBy? Do I have to break it into several statements or is there possibly a little stream magic that can be done to make it simpler. This is the basic code: List<Album> albums = new ArrayList<>(); Map<Artist, List<Album>> map = albums.stream().collect

Chaining multiple mapreduce tasks in Hadoop streaming

由 落爺英雄遲暮 提交于 2019-12-18 02:50:58
问题 I am in scenario where I have two mapreduce jobs. I am more comfortable with python and planning to use it for writing mapreduce scripts and use hadoop streaming for the same. is there a convenient to chain both the jobs following form when hadoop streaming is used? Map1 -> Reduce1 -> Map2 -> Reduce2 I've heard a lot of methods to accomplish this in java, But i need something for Hadoop streaming. 回答1: Here is a great blog post on how to use Cascading and Streaming. http://www.xcombinator.com

Computing median in map reduce

由 若如初见. 提交于 2019-12-17 23:43:04
问题 Can someone example the computation of median/quantiles in map reduce? My understanding of Datafu's median is that the 'n' mappers sort the data and send the data to "1" reducer which is responsible for sorting all the data from n mappers and finding the median(middle value) Is my understanding correct?, if so, does this approach scale for massive amounts of data as i can clearly see the one single reducer struggling to do the final task. Thanks 回答1: Trying to find the median (middle number)

Can apache spark run without hadoop?

由 ♀尐吖头ヾ 提交于 2019-12-17 21:39:50
问题 Are there any dependencies between Spark and Hadoop ? If not, are there any features I'll miss when I run Spark without Hadoop ? 回答1: Spark can run without Hadoop but some of its functionality relies on Hadoop's code (e.g. handling of Parquet files). We're running Spark on Mesos and S3 which was a little tricky to set up but works really well once done (you can read a summary of what needed to properly set it here). (Edit) Note: since version 2.3.0 Spark also added native support for

Hadoop 1.2.1 - multinode cluster - Reducer phase hangs for Wordcount program?

由 好久不见. 提交于 2019-12-17 21:33:40
问题 My question may sound redundant here but the solution to the earlier questions were all ad-hoc. few I have tried but no luck yet. Acutally, I am working on hadoop-1.2.1(on ubuntu 14), Initially I had single node set-up and there I ran the WordCount program succesfully. Then I added one more node to it according to this tutorial. It started successfully, without any errors, But now when I am running the same WordCount program it is hanging in reduce phase. I looked at task-tracker logs, they

Can apache spark run without hadoop?

由 强颜欢笑 提交于 2019-12-17 21:28:57
问题 Are there any dependencies between Spark and Hadoop ? If not, are there any features I'll miss when I run Spark without Hadoop ? 回答1: Spark can run without Hadoop but some of its functionality relies on Hadoop's code (e.g. handling of Parquet files). We're running Spark on Mesos and S3 which was a little tricky to set up but works really well once done (you can read a summary of what needed to properly set it here). (Edit) Note: since version 2.3.0 Spark also added native support for

How to Group mongodb - mapReduce output?

由 邮差的信 提交于 2019-12-17 21:06:23
问题 i have a query regarding the mapReduce framework in mongodb, so i have a result of key value pair from mapReduce function , now i want to run the query on this output of mapReduce. So i am using mapReduce to find out the stats of user like this db.order.mapReduce(function() { emit (this.customer,{count:1,orderDate:this.orderDate.interval_start}) }, function(key,values){ var sum =0 ; var lastOrderDate; values.forEach(function(value) { if(value['orderDate']){ lastOrderDate=value['orderDate']; }

Hadoop Streaming - Unable to find file error

由 China☆狼群 提交于 2019-12-17 20:35:24
问题 I am trying to run a hadoop-streaming python job. bin/hadoop jar contrib/streaming/hadoop-0.20.1-streaming.jar -D stream.non.zero.exit.is.failure=true -input /ixml -output /oxml -mapper scripts/mapper.py -file scripts/mapper.py -inputreader "StreamXmlRecordReader,begin=channel,end=/channel" -jobconf mapred.reduce.tasks=0 I made sure mapper.py has all the permissions. It errors out saying Caused by: java.io.IOException: Cannot run program "mapper.py": error=2, No such file or directory at java
订阅 MapReduce