I have a Spark (version 1.4.1) application on HDP 2.3. It works fine when running it in YARN-Client mode. However, when running it on YARN-Cluster mode none of my Hive table
If you are able to fetch data using Hive CLI, then use the same hive-site.xml in your Spark job.
The only reason could be the location of metastore defined in hive-site.xml.