yarn

问题 when i run "yarn logs -applicationId application_1438080928000_6932", appear this exception: Exception in thread "main" java.io.IOException: Not a valid BCFile. at org.apache.hadoop.io.file.tfile.BCFile$Magic.readAndVerify(BCFile.java:927) at org.apache.hadoop.io.file.tfile.BCFile$Reader.<init>(BCFile.java:628) at org.apache.hadoop.io.file.tfile.TFile$Reader.<init>(TFile.java:804) at org.apache.hadoop.yarn.logaggregation.AggregatedLogFormat$LogReader.<init>(AggregatedLogFormat.java:358) at

问题 Can anyone please let me know the must have properties for Core-site.xml, hdfs-site.xml, mapred-site.xml and yarn-site.xml without which hadoop can not start? 回答1: Below settings is for Hadoop 2.x.x for Standalone and Pseudo node setup. core-site.xml <configuration> <property> <name>fs.default.name</name> <value>hdfs://localhost:9000</value> </property> </configuration> hdfs-site.xml <configuration> <property> <name>dfs.replication</name> <value>1</value> </property> <property> <name>dfs.name

问题 I enable the Kerberos on the cluster and it is working fine. But due to some issue mapred user is not able to read and display log over JobHistory server. I check the logs of job history server and it giving access error as: org.apache.hadoop.security.AccessControlException: Permission denied:user=mapred, access=READ_EXECUTE, inode="/user/history/done_intermediate/prakul":prakul:hadoop:drwxrwx--- as we can see the directory have access to hadoop group and mapred is in hadoop group, even then

问题 I am using Apache Spark with Yarn client. I have 4 worker PCs with 8 vcpus each and 30 GB of ram in my spark cluster. Im set my executor memory to 2G and number of instances to 33. My job is taking 10 hours to run and all machines are about 80% idle. I dont understand the correlation between executor memory and executor instances. Should I have an instance per Vcpu? Should I set the executor memory to be memory of machine/#executors per machine? 回答1: I believe that you have to use the

问题 I have this command line to dispaly the YARN policy: The result is: { "id": 131, "guid": "4d9c3257-0998-42ea-8506-f773a368430d", "isEnabled": true, "version": 2, "service": "Namecluster_yarn", } }, "policyItems": [ { "accesses": [ { "type": "submit-app", "isAllowed": true } ], "users": [], "groups": [ "Application_Team_1" ], "conditions": [], "delegateAdmin": false } ], "denyPolicyItems": [], "allowExceptions": [], "denyExceptions": [], "dataMaskPolicyItems": [], "rowFilterPolicyItems": [] }

问题 hi when i am writing this code >sbt And after seeing this result beyhan@beyhan:~/sparksample$ sbt Starting sbt: invoke with -help for other options [info] Set current project to Spark Sample (in build file:/home/beyhan/sparksample/) And after i am writing this code >compile And i am getting this error [error] {file:/home/beyhan/sparksample/}default-f390c8/*:update: sbt.ResolveException: unresolved dependency: org.apache.hadoop#hadoop-yarn-common;1.0.4: not found [error] unresolved dependency:

问题 I am posting jobs to a spark cluster using livy APIs. I want to increase the spark.network.timeout value and passing the same value ( 600s ) with the conf field in livy post call. How can I verify that it is getting correctly honoured and getting applied to the jobs posted? 来源： https://stackoverflow.com/questions/55690915/how-to-check-spark-config-for-an-application-in-ambari-ui-posted-with-livy