bigdata

This is what my data looks like (10, 'ACCOUNTING', 'NEW YORK') (20, 'RESEARCH', 'DALLAS') (30, 'SALES', 'CHICAGO') (40, 'OPERATIONS', 'BOSTON') I want to remove ( , ) and ' from this data using Pig Script. I want my data to look like this- 10, ACCOUNTING, NEW YORK 20, RESEARCH, DALLAS 30, SALES, CHICAGO 40, OPERATIONS, BOSTON I am stuck on this from quite long time. Please help. Thanks in advance. Can you try REPLACE function with the below regex? Explanation: In Regex there are few characters have special meanings \ ^ $ . , | ? * + ( ) [ { . These special characters are called as "

I have a piece of code when I create a map like: val map = gtfLineArr(8).split(";").map(_ split "\"").collect { case Array(k, v) => (k, v) }.toMap Then I use this map to create my object: case class MyObject(val attribute1: String, val attribute2: Map[String:String]) I'm reading millions of lines and converting to MyObjects using an iterator. Like MyObject("1", map) When I do it is really slow, more than 1h for 2'000'000 entries. I remove the map from the object creation, but still I do the split process (section 1): val map = gtfLineArr(8).split(";").map(_ split "\"").collect { case Array(k,

In Hive, you can create two kinds of tables: Managed and External In case of managed table, you own the data and hence when you drop the table the data is deleted. In case of external table, you don't have ownership of the data and hence when you delete such a table, the underlying data is not deleted. Only metadata is deleted. Now, recently i have observed that you can not create an external table over a location on which you don't have write (modification) permissions in HDFS. I completely fail to understand this. Use case: It is quite common that the data you are churning is huge and read

Loading any of Amazon's listed public data sets (http://aws.amazon.com/datasets) would take a lot of resources and bandwidth. What's the best way to import them into AWS so you start working with them quickly? bardiir You will need to create a new EBS Instance using the Snapshot-ID for the public dataset. That way you won't need to pay for transfer. But be careful, some data sets are only available in one region, most likely denoted by a note similar to this. You should register your EC2 instance in the same region then. These datasets are hosted in the us-east-1 region. If you process these