MapReduce

问题 I am learning Hadoop and was learning about Hive in recent past. Its like a SQL query which can be used instead of MR java code. My tutor said that Hive produces MR java code behind the scene, which in turn is exported as jar file and then it is ran on top of HDFS filesystem. But, he was not sure if we can get the MR code produced by Hive. I was reading Definitive guide and nothing was discussed about it. I wanted to know is it really so? As I think if we can get the MR code then that code

问题 This is a data structure implementation in Hadoop. I want to implement indexing in Hadoop using map-reduce programming. Part 1 = I want to store this text file each word using index number in a table. [Able to complete] Part 2 = Now I want to perform the hashing for this newly created table [not able to complete] 1st part I am able to complete but 2nd part I m facing difficulty  Suppose if I have a text file containing 3 lines: how is your job how is your family hi how are you I want to

问题 since amazon web service need to pay, so just wanna ask ppl who had worked on it before i jump into it, and confirm some knowledge about it. Question one: In Amazon auto scaling service, it says can scale up and down instances. that does this mean? does it mean changing the type of instance? or can start/stop more/less instance base on the condition define? Question two: can the auto scaling framework work with map reduce? for example, if i have a extreme case, i will have endless tasks, and

问题 Please excuse me for this basic question. But I wonder why mapreduce job don't get launched when we try to load some file having size more than the block size. Somewhere I learnt that MapReduce will take care of loading the datasets from LFS to HDFS. Then why I am not able to see mapreduce logs on the console when I give hadoop fs -put command? thanks in Advance. 回答1: You're thinking of hadoop distcp which will spawn a MapReduce job. https://hadoop.apache.org/docs/stable/hadoop-distcp/DistCp

问题 I am working on hadoop2.7.0 single node cluster with 4GB Ram and 40 GB HDD. While executing a word count example on Map reduce , it stopped after Running job...I've tried increasing the memory for container in yarn-site but still no luck. Error 16/11/20 17:05:03 INFO client.RMProxy: Connecting to ResourceManager at /0.0.0.0:8032 16/11/20 17:05:07 INFO input.FileInputFormat: Total input paths to process : 1 16/11/20 17:05:08 INFO mapreduce.JobSubmitter: number of splits:1 16/11/20 17:05:08

问题 Good afternoon. I am trying to get a User Event script to call or use a Map Reduce script. I am really new to the concept of a Map Reduce script and am not having much luck finding resources. Essentially, what I want to do is call a Map Reduce script that finds open transactions with the same Item Name and sets the Class on that item to the new item set by the User. The Map Reduce script would need to the Item Name and Class from the current record. Here is my User Event: /** * @NApiVersion 2

问题 My hbase table looks like this: hbase(main):040:0> scan 'TEST' ROW COLUMN+CELL 4 column=data:108, timestamp=1399972960190, value=-240.0 4 column=data:112, timestamp=1399972960138, value=-160.0 4 column=data:12, timestamp=1399972922979, value=2 4 column=data:120, timestamp=1399972960124, value=-152.0 4 column=data:144, timestamp=1399972960171, value=-240.0 4 column=data:148, timestamp=1399972960152, value=-240.0 4 column=data:16, timestamp=1399972909606, value=9 4 column=data:8, timestamp

问题 I had a view map/reduce defined as following,as most of documents have no doc.emails[i].userTypecode elements in couchdb, so it is running the view takes too long causing couch to give up / time out: the error is Error: os_preocess_error, OS process time out, can some one help me how to figure out this issue and how to optimize the map/reduce? thank you I checked there are similar issue happened , but no idea how that were fixed?https://issues.apache.org/jira/browse/COUCHDB-1333 couchdb views

问题 I am trying to put into HBase (version 1.1.X) some XML files stored locally. My goal is to store the content of those XMLs in my HBase Table as string using MapReduce ( no reduce stage ) without loading them to HDFS. Here is my pseudo-code: fetchXMLs(path); XML2OneLineFile(); configureHBase(); // + establishing connection Map(input, output); //input: one XML file in one line; output : is the Put() of HBase; closeConnection(); Is this way of tackling the problem correct, or there are better

问题 I have a quick Hadoop Streaming question. If I'm using Python streaming and I have Python packages that my mappers/reducers require but aren't installed by default do I need to install those on all the Hadoop machines as well or is there some sort of serialization that sends them to the remote machines? 回答1: If they're not installed on your task boxes, you can send them with -file. If you need a package or other directory structure, you can send a zipfile, which will be unpacked for you. Here