amazon-emr

I am creating clusters on EMR and configure Zeppelin to read the notebooks from S3. To do that I am using a json object that looks like that: [ { "Classification": "zeppelin-env", "Properties": { }, "Configurations": [ { "Classification": "export", "Properties": { "ZEPPELIN_NOTEBOOK_STORAGE":"org.apache.zeppelin.notebook.repo.S3NotebookRepo", "ZEPPELIN_NOTEBOOK_S3_BUCKET":"hs-zeppelin-notebooks", "ZEPPELIN_NOTEBOOK_USER":"user" }, "Configurations": [ ] } ] } ] I am pasting this object in the Stoftware configuration page of EMR: My question is, how/where I can configure the Spark interpreter

I am running a spark-job on EMR cluster ,The issue i am facing is all the EMR jobs triggered are executing in steps (in queue) Is there any way to make them run parallel if not is there any alteration for that Elastic MapReduce comes by default with a YARN setup very "step" oriented, with a single CapacityScheduler queue with the 100% of the cluster resources assigned. Because of this configuration, any time you submit a job to an EMR cluster, YARN maximizes the cluster usage for that single job, granting all available resources to it until it finishes. Running multiple concurrent jobs in an

This is very close to this question, but I have added a few details specific to my question: Matplotlib Plotting using AWS-EMR jupyter notebook I would like to find a way to use matplotlib inside my Jupyter notebook. Here is the code-snippet in error, it's fairly simple: notebook import matplotlib matplotlib.use("agg") import matplotlib.pyplot as plt plt.plot([1,2,3,4]) plt.show() I chose this snippet because this line alone fails as it tries to use TKinter (which is not installed on an AWS EMR cluster): import matplotlib.pyplot as plt When I run the full notebook snippet, the result is no

Is there a way to have a whole file sent to a mapper without being split? I have read this but I am wondering if there is another way of doing the same thing without having to generate an intermediate file. Ideally, I would like an existing option on the command line to Hadoop. I am using the streaming facility with Python scripts on Amazon EMR. Just set the configuration property mapred.min.split.size to something huge (10G): -D mapred.min.split.size=10737418240 Or compress the input file using a codec that isn't splittable (Gzip). With the .gz extension, TextInputFormat will return false to

I am developing a code to read data and write it into HDFS using mapreduce . However when I have multiple files I don't understand how it is processed . The input path to the mapper is the name of the directory as evident from the output of String filename = conf1.get("map.input.file"); So how does it process the files in the directory ? In order to get the input file path you can use the context object, like this: FileSplit fileSplit = (FileSplit) context.getInputSplit(); String inputFilePath = fileSplit.getPath().toString(); And as for how it multiple files are processed: Several instances

I have tried the following combinations of bootstrap actions to increase the heap size of my job but none of them seem to work: --mapred-key-value mapred.child.java.opts=-Xmx1024m --mapred-key-value mapred.child.ulimit=unlimited --mapred-key-value mapred.map.child.java.opts=-Xmx1024m --mapred-key-value mapred.map.child.ulimit=unlimited -m mapred.map.child.java.opts=-Xmx1024m -m mapred.map.child.ulimit=unlimited -m mapred.child.java.opts=-Xmx1024m -m mapred.child.ulimit=unlimited What is the right syntax? You have two options to achieve this: Custom JVM Settings In order to apply custom

I am running this code on EMR 4.6.0 + Spark 1.6.1 : val sqlContext = SQLContext.getOrCreate(sc) val inputRDD = sqlContext.read.json(input) try { inputRDD.filter("`first_field` is not null OR `second_field` is not null").toJSON.coalesce(10).saveAsTextFile(output) logger.info("DONE!") } catch { case e : Throwable => logger.error("ERROR" + e.getMessage) } In the last stage of saveAsTextFile , it fails with this error: 16/07/15 08:27:45 ERROR codegen.GenerateUnsafeProjection: failed to compile: org.codehaus.janino.JaninoRuntimeException: Constant pool has grown past JVM limit of 0xFFFF /* 001 */ /

I want to read the file from the Hadoop File System. In order to achieve the correct path of the file, I need host name and port address of the hdfs . so finally my path of the file will look something like Path path = new Path("hdfs://123.23.12.4344:9000/user/filename.txt") Now I want to know to extract the HostName = "123.23.12.4344" & port: 9000? Basically, I want to access the FileSystem on Amazon EMR but, when I use FileSystem fs = FileSystem.get(getConf()); I get You possibly called FileSystem.get(conf) when you should have called FileSystem.get(uri, conf) to obtain a file system