hdp

问题 Hi I'm Saving My dataframe as hive table using spark-sql. mydf.write().format("orc").saveAsTable("myTableName") I'm able to see that table is getting created using hadoop fs -ls /apps/hive/warehouse\dbname.db Also able to see data using spark-shell spark.sql(use dbname) spark.sql(show tables).show(false) but same tables I'm not able to see using hive shell. I have place my hive-site.xml file using. sudo cp /etc/hive/conf.dist/hive-site.xml /etc/spark/conf/ but still not able to see. can

问题 In MapR hadoop, in order for a user to be able to access HDFS or use YARN for programs, they needed to exist across all nodes in the cluster (with same uid and gid), this includes client nodes that don't act as either data nodes or control nodes (MapR does not really have the concept of namenodes). Is this the same for Hortonworks HDP? 回答1: Found this answer on the Hortonworks community site: User should not have account on all the nodes of the cluster. He should only have account on edge

问题 In MapR hadoop, in order for a user to be able to access HDFS or use YARN for programs, they needed to exist across all nodes in the cluster (with same uid and gid), this includes client nodes that don't act as either data nodes or control nodes (MapR does not really have the concept of namenodes). Is this the same for Hortonworks HDP? 回答1: Found this answer on the Hortonworks community site: User should not have account on all the nodes of the cluster. He should only have account on edge

问题 In MapR hadoop, in order for a user to be able to access HDFS or use YARN for programs, they needed to exist across all nodes in the cluster (with same uid and gid), this includes client nodes that don't act as either data nodes or control nodes (MapR does not really have the concept of namenodes). Is this the same for Hortonworks HDP? 回答1: Found this answer on the Hortonworks community site: User should not have account on all the nodes of the cluster. He should only have account on edge

问题 I'm trying to use hive warehouse connector and spark on hdp 3.1 and getting exception even with simplest example (below). The class causing problems: JaninoRuntimeException - is in org.codehaus.janino:janino:jar:3.0.8 (dependency of spark_sql) and in com.hortonworks.hive:hive-warehouse-connector_2.11:jar . I've tried to exclude janino library from spark_sql, but this resulted in missing other classes from janino. And I need hwc to for the new functionality. Anyone had same error? Any ideas

问题 I'm trying to use hive warehouse connector and spark on hdp 3.1 and getting exception even with simplest example (below). The class causing problems: JaninoRuntimeException - is in org.codehaus.janino:janino:jar:3.0.8 (dependency of spark_sql) and in com.hortonworks.hive:hive-warehouse-connector_2.11:jar . I've tried to exclude janino library from spark_sql, but this resulted in missing other classes from janino. And I need hwc to for the new functionality. Anyone had same error? Any ideas

问题 Getting confused when trying to run a YARN process and getting errors. Looking in ambari UI YARN section, seeing... (note it says 60GB available). Yet, when trying to run an YARN process, getting errors indicating that there are less resources available than is being reported in ambari, see... ➜ h2o-3.26.0.2-hdp3.1 hadoop jar h2odriver.jar -nodes 4 -mapperXmx 5g -output /home/ml1/hdfsOutputDir Determining driver host interface for mapper->driver callback... [Possible callback IP address: 192

问题 Have users on local machines that have HDFS /user dirs that do not show up as possible users when setting Ranger policies I can see that Ranger already have a place where you can see and add users in the settings menu of the ranger UI, but not sure where this is getting populated from. So my question then is what determines if Ranger can see cluster users for setting policies (and is there an easy way to manage this via ambari)? 回答1: The problem was that I had thought, looking at a answer on

问题 After upgrading from HDP 2.7 to HDP 3.1, I manually uninstalled many services, such as spark2,hive,hbase and Knox, for some reason. When I tried to install Knox, the installation failed. env: ambari 2.7 HDP 3.1 enabled kerberos used openldap stderr: /var/lib/ambari-agent/data/errors-15457.txt Traceback (most recent call last): File "/var/lib/ambari-agent/cache/stacks/HDP/3.0/services/KNOX/package/scripts/knox_gateway.py", line 215, in <module> KnoxGateway().execute() File "/usr/lib/ambari

问题 Can a docker image access hadoop resources? Eg. submit YARN jobs and access HDFS; something like MapR's Datasci. Refinery, but for Hortonworks HDP 3.1. (May assume that the image will be launched on a hadoop cluster node). Saw the hadoop docs for launching docker applications from hadoop nodes, but was interested in whether could go the "other way" (ie. being able to start a docker image with the conventional docker -ti ... command and have that application be able to run hadoop jars etc.