yarn

Spark runs in local but can't find file when running in YARN

由 青春壹個敷衍的年華 提交于 2019-12-10 23:39:20
问题 I've been trying to submit a simple python script to run it in a cluster with YARN. When I execute the job in local, there's no problem, everything works fine but when I run it in the cluster it fails. I executed the submit with the following command: spark-submit --master yarn --deploy-mode cluster test.py The log error I'm receiving is the following one: 17/11/07 13:02:48 INFO yarn.Client: Application report for application_1510046813642_0010 (state: ACCEPTED) 17/11/07 13:02:49 INFO yarn

How concurrent # mappers and # reducers are calculated in Hadoop 2 + YARN?

由 ε祈祈猫儿з 提交于 2019-12-10 23:15:54
问题 I've searched by sometime and I've found that a MapReduce cluster using hadoop2 + yarn has the following number of concurrent maps and reduces per node: Concurrent Maps # = yarn.nodemanager.resource.memory-mb / mapreduce.map.memory.mb Concurrent Reduces # = yarn.nodemanager.resource.memory-mb / mapreduce.reduce.memory.mb However, I've set up a cluster with 10 machines, with these configurations: 'yarn_site' => { 'yarn.nodemanager.resource.cpu-vcores' => '32', 'yarn.nodemanager.resource.memory

Spark on YARN: Less executor memory than set via spark-submit

由 北慕城南 提交于 2019-12-10 23:00:40
问题 I'm using Spark in a YARN cluster (HDP 2.4) with the following settings: 1 Masternode 64 GB RAM (48 GB usable) 12 cores (8 cores usable) 5 Slavenodes 64 GB RAM (48 GB usable) each 12 cores (8 cores usable) each YARN settings memory of all containers (of one host): 48 GB minimum container size = maximum container size = 6 GB vcores in cluster = 40 (5 x 8 cores of workers) minimum #vcores/container = maximum #vcores/container = 1 When I run my spark application with the command spark-submit -

How do I get the YARN ContainerId from inside the container?

由 拥有回忆 提交于 2019-12-10 22:36:28
问题 I'm running a Spark job on YARN and would like to get the YARN container ID (as part of a requirement to generate unique IDs across a set of Spark jobs). I can see the Container.getId() method to get the ContainerId but no idea how to get a reference to the current running container from YARN. Is this even possible? How does a YARN container get it's own information? 回答1: The only way that I could get something was to use the logging directory. The following works in a spark shell. import org

MapReduce On Yarn的执行流程

由 只愿长相守 提交于 2019-12-10 21:07:33
1、概述 Yarn是一个资源调度平台，负责为运算程序提供服务器运算资源，相当于一个分布式的操作系统平台，而MapReduce等运算程序则相当于运行于操作系统之上的应用程序。 Yarn的架构如下图所示： 从Yarn的架构图来看，他主要由ResourceManager、NodeManager、ApplicationMaster和Container等一下几个组件构成。 1）ResourceManager Yarn分层结构的本质是ResourceManager，这个实体控制整个集群并管理应用程序向基础计算资源的分配。Resourcemanager将各个资源（计算，内存，带宽等）精心安排给基础NodeManager。ResourceManager还与ApplicationMaster一起分配资源，与NodeManager一起启动和监视他们的基础应用程序。 总的来说，RM有以下功能： （1）处理客户端的请求 （2）启动和监控ApplicationMaster （3）监控NodeManager （4）资源分配与调度 2）ApplicationMaster ApplicationMaster管理在Yarn内运行的每个应用程序。负责协调来自RM的资源，并通过NodeManager监控容器的执行和资源的使用（CPU、内存等的资源分配）。总体来说，AM有以下作用 （1）负责数据的切分 （2

spark2 + yarn - nullpointerexception while preparing AM container

由 血红的双手。 提交于 2019-12-10 21:07:16
问题 I'm trying to run pyspark --master yarn Spark version: 2.0.0 Hadoop version: 2.7.2 Hadoop yarn web interface is successfully started This is what happens: 16/08/15 10:00:12 DEBUG Client: Using the default MR application classpath: $HADOOP_MAPRED_HOME/share/hadoop/mapreduce/*,$HADOOP_MAPRED_HOME/share/hadoop/mapreduce/lib/* 16/08/15 10:00:12 INFO Client: Preparing resources for our AM container 16/08/15 10:00:12 DEBUG Client: 16/08/15 10:00:12 DEBUG DFSClient: /user/mispp/.sparkStaging

Spark Streaming application fails with KafkaException: String exceeds the maximum size or with IllegalArgumentException

由 一世执手 提交于 2019-12-10 19:46:15
问题 TL;DR: My very simple Spark Streaming application fails in the driver with the "KafkaException: String exceeds the maximum size". I see the same exception in the executor but I also found somewhere down the executor's logs an IllegalArgumentException with no other information in it Full problem: I'm using Spark Streaming to read some messages from a Kafka topic. This is what I'm doing: val conf = new SparkConf().setAppName("testName") val streamingContext = new StreamingContext(new

Spark on Yarn Container Failure

由 萝らか妹 提交于 2019-12-10 18:05:13
问题 For reference: I solved this issue by adding Netty 4.1.17 in hadoop/share/hadoop/common No matter what jar I try and run (including the example from https://spark.apache.org/docs/latest/running-on-yarn.html), I keep getting an error regarding container failure when running Spark on Yarn. I get this error in the command prompt: Diagnostics: Exception from container-launch. Container id: container_1530118456145_0001_02_000001 Exit code: 1 Stack trace: ExitCodeException exitCode=1: at org.apache

How to prevent EMR Spark step from retrying?

由 穿精又带淫゛_ 提交于 2019-12-10 15:59:51
问题 I have an AWS EMR cluster (emr-4.2.0, Spark 1.5.2), where I am submitting steps from aws cli. My problem is, that if the Spark application fails, then YARN is trying to run the application again (under the same EMR step). How can I prevent this? I was trying to set --conf spark.yarn.maxAppAttempts=1 , which is correctly set in Environment/Spark Properties, but it doesn't prevent YARN from restarting the application. 回答1: You should try to set spark.task.maxFailures to 1 (4 by default).

YARN: Containers and JVM

由 假如想象 提交于 2019-12-10 14:56:47
问题 Can someone help me understand the relation between JVM and containers in YARN? How JVMs are created, is it one JVM for each task? can multiple tasks run in the same JVM at the same time? (I'm aware of ubertasking where many tasks (maps/reduce) can run in same JVM one after the other). Is it one JVM for each container? or multiple containers in a single JVM? or there is no relation between JVM and containers? when a resource manager allocates containers for a job, does multiple tasks inside
订阅 yarn