MapReduce

问题 I am new to Hadoop. I need to parse a small xml file using mapreduce program in java. I am using hadoop 1.0.4 say my xml file is <configuration> <property> <name>adv</name> <value>a</value> <dup>school</dup> </property> <property> <name>aghy</name> <value>a</value> <dup>bk</dup> </property> </configuration> i need an output like this adv 1 a 2 aghy 1 school 1 bk 1 how can i edit the code https://github.com/studhadoop/xmlparsing-hadoop/blob/master/XmlParser11.java .Any working idea. pls help

问题 My first go at Map/Reduce using PHP/MongoDB. I am stuck with an error when running the MapReduce command. My code: $map = "function () {". "emit(this.topic_id, {re_date:this.date_posted}); " . "}"; $reduce = "function (key, values) {" . "var result = {last_post: 0};" . "var max = ISODate('1970-01-01T00:00:00Z');" . "values.forEach(function (value) {" . "if (max == ISODate('1970-01-01T00:00:00Z')) {" . "max = value.re_date;}" . "if (max < value.re_date) {max = value.re_date;}});" . "result

问题 I installed Hadoop in my Ubuntu 12.04 by following the procedure in the below link. http://www.bogotobogo.com/Hadoop/BigData_hadoop_Install_on_ubuntu_single_node_cluster.php Everything is installed successfully and when I run the start-all.sh only some of the services are running. wanderer@wanderer-Lenovo-IdeaPad-S510p:~$ su - hduse Password: hduse@wanderer-Lenovo-IdeaPad-S510p:~$ cd /usr/local/hadoop/sbin hduse@wanderer-Lenovo-IdeaPad-S510p:/usr/local/hadoop/sbin$ start-all.sh This script is

问题 I've created two MapReduce Pipelines for uploading CSVs files to create Categories and Products in bulk. Each product is gets tied to a Category through a KeyProperty. The Category and Product models are built on ndb.Model, so based on the documentation, I would think they'd be automatically cached in Memcache when retrieved from the Datastore. I've run these scripts on the server to upload 30 categories and, afterward, 3000 products. All the data appears in the Datastore as expected. However

问题 I have a lot of input files and I want to process selected ones based on the date that has been appended in the end. I am now confused on where do I use the globStatus method to filter out the files. I have a custom RecordReader class and I was trying to use globStatus in its next method but it didn't work out. public boolean next(Text key, Text value) throws IOException { Path filePath = fileSplit.getPath(); if (!processed) { key.set(filePath.getName()); byte[] contents = new byte[(int)

问题 I'd like to implement a MultithreadMapper for my MapReduce job. For this I replaced Mapper with MultithreadMapper in a working code. Here's the exeption I'm getting: java.io.IOException: Type mismatch in key from map: expected org.apache.hadoop.io.IntWritable, recieved org.apache.hadoop.io.LongWritable at org.apache.hadoop.mapred.MapTask$MapOutputBuffer.collect(MapTask.java:862) at org.apache.hadoop.mapred.MapTask$NewOutputCollector.write(MapTask.java:549) at org.apache.hadoop.mapreduce

问题 I have a five node cluster, three nodes of which contain DataNodes and TaskTrackers. I've imported around 10million rows from Oracle via Sqoop and process it via MapReduce in an Oozie workflow. The MapReduce job takes about 30 minutes and is only using one reducer. Edit - If I run the MapReduce code on its own, separate from Oozie, the job.setNumReduceTasks(4) correctly establishes 4 reducers. I have tried the following methods to manually set the number of reducers to four, with no success:

问题 In order to join two datasets I tried to translate this example to C# the following way: I would be very thankful if anyone of you could suggest the appropriate code modification in order to achieve the same result as the example. 回答1: The solution which produces the same results as this example is the following: class Program { static void Main(string[] args) { var connectionString = "mongodb://localhost"; var client = new MongoClient(connectionString); var server = client.GetServer(); var

问题 Here is my code: import java.io.DataInput; import java.io.DataOutput; import java.io.IOException; import java.util.Iterator; import org.apache.hadoop.conf.Configuration; import org.apache.hadoop.conf.Configured; import org.apache.hadoop.fs.Path; import org.apache.hadoop.io.DoubleWritable; import org.apache.hadoop.io.LongWritable; import org.apache.hadoop.io.Text; import org.apache.hadoop.io.WritableComparable; import org.apache.hadoop.io.WritableUtils; import org.apache.hadoop.mapred.*;

问题 I am doing following to do compression of o/p files from reducer: OutputStream out = ipFs.create( new Path( opDir + "/" + fileName ) ); CompressionCodec codec = new GzipCodec(); OutputStream cs = codec.createOutputStream( out ); BufferedWriter cout = new BufferedWriter( new OutputStreamWriter( cs ) ); cout.write( ... ) But got null pointer exception in line 3: java.lang.NullPointerException at org.apache.hadoop.io.compress.zlib.ZlibFactory.isNativeZlibLoaded(ZlibFactory.java:63) at org.apache