yarn | 易学教程

How to increase the number of containers in nodemanager in YARN

阅读更多关于 How to increase the number of containers in nodemanager in YARN

问题 A node in my YARN cluster has 64GB memory and 24 cores. I set the following properties in the yarn-site.xml: <property> <name>yarn.nodemanager.resource.memory-mb</name> <value>32768</value> </property> <property> <name>yarn.nodemanager.resource.cpu-vcores</name> <value>16</value> </property> But I found still the nodemanager in the node has only 7 containers. What are other properties I need to set? 回答1: You need to tell YARN how to break down the memory to containers so for instance if you

How to execute Spark programs with Dynamic Resource Allocation?

阅读更多关于 How to execute Spark programs with Dynamic Resource Allocation?

问题 I am using spark-summit command for executing Spark jobs with parameters such as: spark-submit --master yarn-cluster --driver-cores 2 \ --driver-memory 2G --num-executors 10 \ --executor-cores 5 --executor-memory 2G \ --class com.spark.sql.jdbc.SparkDFtoOracle2 \ Spark-hive-sql-Dataframe-0.0.1-SNAPSHOT-jar-with-dependencies.jar Now i want to execute the same program using Spark's Dynamic Resource allocation. Could you please help with the usage of Dynamic Resource Allocation in executing

Spark on YARN两种运行模式介绍

阅读更多关于 Spark on YARN两种运行模式介绍

本文出自：Spark on YARN两种运行模式介绍 http://www.aboutyun.com/thread-12294-1-1.html (出处: about云开发) 问题导读 1.Spark在YARN中有几种模式？ 2.Yarn Cluster模式，Driver程序在YARN中运行，应用的运行结果在什么地方可以查看？ 3.由client向ResourceManager提交请求，并上传jar到HDFS上包含哪些步骤？ 4.传递给app的参数应该通过什么来指定？ 5.什么模式下最后将结果输出到terminal中？ Spark在YARN中有yarn-cluster和yarn-client两种运行模式: 1.Yarn Cluster Spark Driver首选作为一个ApplicationMaster在Yarn集群中启动，客户端提交给ResourceManager的每一个job都会在集群的worker节点上分配一个唯一的ApplicationMaster, 由该ApplicationMaster管理全生命周期的应用。因为Driver程序在YARN中运行，所以事先不用启动Spark Master/Client，应用的运行结果不能再客户端显示(可以在history server中查看) ，所以最好将结果保存在HDFS而非stdout输出

Spark History Server behind Load Balancer is redirecting to HTTP

阅读更多关于 Spark History Server behind Load Balancer is redirecting to HTTP

问题 I am currently using Spark on AWS EMR, but when this is behind a Load Balancer (AWS ELB), it is redirecting the traffic from https to http, which then ends up getting denied because I don't allow http traffic through the load balancer for the given port. It appears that this might derive from Yarn being a proxy as well, but I have no idea. 来源： https://stackoverflow.com/questions/56412083/spark-history-server-behind-load-balancer-is-redirecting-to-http

Spark History Server behind Load Balancer is redirecting to HTTP

阅读更多关于 Spark History Server behind Load Balancer is redirecting to HTTP

Dataproc set number of vcores per executor container

阅读更多关于 Dataproc set number of vcores per executor container

问题 I'm building a spark application which will run on Dataproc. I plan to use ephemeral clusters, and spin a new one up for each execution of the application. So I basically want my job to eat up as much of the cluster resources as possible, and I have a very good idea of the requirements. I've been playing around with turning off dynamic allocation and setting up the executor instances and cores myself. Currently I'm using 6 instances and 30 cores a pop. Perhaps it's more of a yarn question,

Oozie with Hadoop 2, Job hangs in “RUNNING”

阅读更多关于 Oozie with Hadoop 2, Job hangs in “RUNNING”

问题 I have workflow job with a java action node. Run with Hadoop 2.1.0.2.0.4.0-38 and Oozie 3.3.2.2.0.4.0 When I submit the job I see 2 lines in Hadoop Resource Manager screen. 1. with original job name 2. with Oozie job name. The task with Oozie job name is hanging in "RUNNING" state The task with original name is in "Accepted" state. All that I see in logs is: >>> Invoking Main class now >>> Heart beat Heart beat Heart beat Heart beat ... Thank you 回答1: It seems that number of maptasks that can

EMR Spark - TransportClient: Failed to send RPC

阅读更多关于 EMR Spark - TransportClient: Failed to send RPC

问题 I'm getting this error, I tried to increase memory on cluster instances and in the executor and driver parameters without success. 17/05/07 23:17:07 ERROR TransportClient: Failed to send RPC 6465703946954088562 to ip-172-30-12-164.eu-central-1.compute.internal/172.30.12.164:34706: java.nio.channels.ClosedChannelException Does anyone have any clue to fix this error? BTW I'm using YARN as cluster manager Thanks in advance 回答1: Finally I resolved the problem. It was due to insufficient disk

Spark Hive reporting pyspark.sql.utils.AnalysisException: u'Table not found: XXX' when run on yarn cluster

阅读更多关于 Spark Hive reporting pyspark.sql.utils.AnalysisException: u'Table not found: XXX' when run on yarn cluster

问题 I'm attempting to run a pyspark script on BigInsights on Cloud 4.2 Enterprise that accesses a Hive table. First I create the hive table: [biadmin@bi4c-xxxxx-mastermanager ~]$ hive hive> CREATE TABLE pokes (foo INT, bar STRING); OK Time taken: 2.147 seconds hive> LOAD DATA LOCAL INPATH '/usr/iop/4.2.0.0/hive/doc/examples/files/kv1.txt' OVERWRITE INTO TABLE pokes; Loading data to table default.pokes Table default.pokes stats: [numFiles=1, numRows=0, totalSize=5812, rawDataSize=0] OK Time taken:

org.apache.spark.rpc.RpcTimeoutException: Futures timed out after [120 seconds]. This timeout is controlled by spark.rpc.lookupTimeout

阅读更多关于 org.apache.spark.rpc.RpcTimeoutException: Futures timed out after [120 seconds]. This timeout is controlled by spark.rpc.lookupTimeout

问题 Getting the below error with respect to the container while submitting an spark application to YARN. The HADOOP(2.7.3)/SPARK (2.1) environment is running a pseudo-distributed mode in a single node cluster. The application works perfectly when made to run in local model however trying to check its correctness in a cluster mode using YARN as RM and hit some roadblock. New to this world hence looking for help. --- Applications logs 2017-04-11 07:13:28 INFO Client:58 - Submitting application 1 to