bigdata

问题 We are reading a Hbase table with Pyspark using the following commands. from pyspark.sql.types import * host=<Host Name> port=<Port Number> keyConv = "org.apache.spark.examples.pythonconverters.ImmutableBytesWritableToStringConverter" valueConv = "org.apache.spark.examples.pythonconverters.HBaseResultToStringConverter" cmdata_conf = {"hbase.zookeeper.property.clientPort":port, "hbase.zookeeper.quorum": host, "hbase.mapreduce.inputtable": "CMData", "hbase.mapreduce.scan.columns": "info:Tenure

问题 I need to store output of map-reduce program into database, so is there any way? If so, is it possible to store output into multiple columns & tables based on requirement?? please suggest me some solutions. Thank you.. 回答1: The great example is shown on this blog, I tried it and it goes really well. I quote the most important parts of the code. At first, you must create a class representing data you would like to store. The class must implement DBWritable interface: public class

问题 I am trying to open a file (present in the HDFS location: /user/input/Summary.txt) in my browser using the URL: hdfs://localhost:8020/user/input/Summary.txt but I am getting an error in my firefox browser: Firefox doesn't know how to open this address, because the protocol (hdfs) isn't associated with any program. If I change the protocol from hdfs to http (which ideally should not work) then I am getting the message: It looks like you are making an HTTP request to a Hadoop IPC port. This is

问题 I am new to Big data. From some of the answers to What's the difference between Flume and Sqoop?, both Flume and Sqoop can pull data from source and push to Hadoop. Can anyone please specify exaclty where flume is used and where sqoop is? Can both be used for the same tasks? 回答1: Flume and Sqoop are both designed to work with different kind of data sources. Sqoop works with any kind of RDBMS system that supports JDBC connectivity. Flume on the other hand works well with streaming data sources

问题 I have a bunch of binary files compressed into *gz format. These are generated on a remote node and must be transferred to HDFS located one of the datacenter's server. I'm exploring the option of sending the files with Flume; I explore the option of doing this with a Spooling Directory configuration, but apparently this only works when the file's directory is located locally on the same HDFS node. Any suggestions how to tackle this problem? 回答1: There is no out-of-box solution for such case.

问题 Closed . This question is opinion-based. It is not currently accepting answers. Want to improve this question? Update the question so it can be answered with facts and citations by editing this post. Closed 3 years ago . I have went through a lot of articles but I dont seem to get a perfectly clear answer on what exactly a BIG DATA is. In one page I saw "any data which is bigger for your usage, is big data i.e. 100 MB is considered big data for your mailbox but not your hard disc". Whereas

问题 I have a JavaRDD which looks like this., [ [A,8] [B,3] [C,5] [A,2] [B,8] ... ... ] I want my result to be Mean [ [A,5] [B,5.5] [C,5] ] How do I do this using Java RDDs only. P.S : I want to avoid groupBy operation so I am not using DataFrames. 回答1: Here you go : import org.apache.spark.SparkConf; import org.apache.spark.api.java.JavaPairRDD; import org.apache.spark.api.java.JavaRDD; import org.apache.spark.api.java.JavaSparkContext; import org.apache.spark.util.StatCounter; import scala

问题 I am looking for a way to shuffle a large amount of data which does not fit into memory (approx. 40GB). I have around 30 millions entries, of variable length, stored in one large file. I know the starting and ending positions of each entry in that file. I need to shuffle this data which does not fit in the RAM. The only solution I thought of is to shuffle an array containing the numbers from 1 to N , where N is the number of entries, with the Fisher-Yates algorithm and then copy the entries

问题 I have decided to use memmapfile because my data (typically 30Gb to 60Gb) is too big to fit in a computer's memory. My data files consist two columns of data that correspond to the outputs of two sensors and I have them in both .bin and .txt formats. m=memmapfile('G:\E-Stress Research\Data\2013-12-18\LD101_3\EPS/LD101_3.bin','format','int32') m.data(1) I used the above code to memory map my data to a variable "m" but I have no idea what data format to use (int8', 'int16', 'int32', 'int64',

问题 I'm using rmongodb to get every document in a a particular collection. It works but I'm working with millions of small documents, potentially 100M or more. I'm using the method suggested by the author on the website: cnub.org/rmongodb.ashx count <- mongo.count(mongo, ns, query) cursor <- mongo.find(mongo, query) name <- vector("character", count) age <- vector("numeric", count) i <- 1 while (mongo.cursor.next(cursor)) { b <- mongo.cursor.value(cursor) name[i] <- mongo.bson.value(b, "name")