bigdata

I am getting the fixed width .txt source file from which I need to extract the 20K columns. As lack of libraries to process fixed width files using spark, I have developed the code which extracts the fields from fixed width text files. Code read the text file as RDD with sparkContext.textFile("abc.txt") then reads JSON schema and gets the column names and width of each column. In the function I read the fixed length string and using the start and end position we use substring function to create the Array . Map the function to RDD. Convert the above RDD to DF and map colnames and write to

Matrix M is the signatures matrix, which is produced via Minhashing of the actual data, has documents as columns and words as rows. So a column represents a document. Now it says that every stripe ( b in number, r in length) has its columns hashed, so that a column falls in a bucket. If two columns fall in the same bucket, for >= 1 stripes, then they are potentially similar. So that means that I should create b hashtables and find b independent hash functions? Or just one is enough and every stripe sends its columns to the same collections of buckets (but wouldn't this cancel the stripes)?

I am trying to rewrite some R code in Python and cannot get past one particular bit of code. I've found the foverlaps function in R to be very useful when performing a time-based join, but haven't found anything that works as well in Python3. What I am doing is joining two data tables where the time in one table falls between the start_time and end_time in another table. The periodicity of the two tables is not the same - table_A occurs on a per second basis and can have multiple entries at each interval, while table_B will have one entry every 0 - 10 minutes at irregular intervals. This

I have a three nodes Cassandra Cluster and I have created one table which has more than 2,000,000 rows. When I execute this ( select count(*) from userdetails ) query in cqlsh, I got this error: OperationTimedOut: errors={}, last_host=192.168.1.2 When I run count function for less row or with limit 50,000 it works fine. count(*) actually pages through all the data. So a select count(*) from userdetails without a limit would be expected to timeout with that many rows. Some details here: http://planetcassandra.org/blog/counting-key-in-cassandra/ You may want to consider maintaining the count

I am using sqoop 1.4.2 version. I am trying to change the sqoop metastore from default hsqldb to mysql. I have configured following properties in sqoop-site.xml file. <property> <name>sqoop.metastore.client.enable.autoconnect</name> <value>false</value> <description>If true, Sqoop will connect to a local metastore for job management when no other metastore arguments are provided. </description> </property> <property> <name>sqoop.metastore.client.autoconnect.url</name> <value>jdbc:mysql://ip:3206/sqoop?createDatabaseIfNotExist=true</value> </property> <property> <name>sqoop.metastore.client

The csv file to be processed does not fit into the memory. How can one read ~20K random lines of it to do basic statistics on the selected data frame? You can also just do it in the terminal with perl. perl -ne 'print if (rand() < .01)' biglist.txt > subset.txt This won't necessarily get you exactly 20,000 lines. (Here it'll grab about .01 or 1% of the total lines.) It will, however, be really really fast, and you'll have a nice copy of both files in your directory. You can then load the smaller file into R however you want. Try this based on examples 6e and 6f on the sqldf github home page :