datastax-enterprise

Read Files from S3 bucket to Spark Dataframe using Scala in Datastax Spark Submit giving AWS Error Message: Bad Request

こ雲淡風輕ζ 提交于 2019-12-08 10:47:19
问题 I'm trying to read CSV files which are on s3 bucket which is located in Mumbai Region.I'm trying to read the files using datastax dse spark-submit. I tried changing hadoop-aws version to various other versions. Currently, hadoop-aws version is 2.7.3 spark.sparkContext.hadoopConfiguration.set("com.amazonaws.services.s3.enableV4", "true") spark.sparkContext.hadoopConfiguration.set("fs.s3a.endpoint", "s3.ap-south-1.amazonaws.com") spark.sparkContext.hadoopConfiguration.set("fs.s3a.access.key",

DSE 4.6 to DSE 4.7 Failed to find Spark assembly

自闭症网瘾萝莉.ら 提交于 2019-12-08 10:08:42
问题 I have a problem with job-server-0.5.0 after upgraded DSE 4.6 to 4.7. If I run server_start.sh I'll get error "Failed to find Spark assembly in /usr/share/dse/spark/assembly/target/scala-2.10 You need to build Spark before running this program." I found in /usr/share/dse/spark/bin/compute-classpath.sh this code raises error for f in ${assembly_folder}/spark-assembly*hadoop*.jar; do if [[ ! -e "$f" ]]; then echo "Failed to find Spark assembly in $assembly_folder" 1>&2 echo "You need to build

What are best practices for deleting/altering cassandra columns of collection data-type?

£可爱£侵袭症+ 提交于 2019-12-08 09:45:58
问题 In our Cassandra table, every time we change data-types of "collection-type" columns it start causing issue. For example: For changing datatype from text to Map<text,float> we do this: drop existing column wait for cassandra to assimilate this change. add column (same name) but different data-type. This reflects fine in all nodes, but Cassandra logs start complaining during compaction with: RuntimeException: 6d6...73 is not defined as a collection I figured out the comparator entries are not

DSE 4.6 to 4.7: 1 MUTATION messages dropped in last 5000ms

。_饼干妹妹 提交于 2019-12-08 09:41:39
问题 after upgrade our cluster(4DC, ubuntu 14.04 x64, cpp-driver 2.0.1 as client in our app) from 4.6 to 4.7, got message in logs on few nodes with small load "MessagingService.java:888 - 1 MUTATION messages dropped in last 5000ms" with 1 Pending HintedHandoff notice in thread pool dump what i try: run "nodetool truncatehints" on each running node in cluster changing openjdk to oracle jdk(1.7.0_76-b13) decommission node and rejoin it how to find this mutation/hint and drop it? side note: we do not

Access Cassandra from Spark “com.esotericsoftware.kryo.KryoException: Unable to find class: org.apache.spark.sql.cassandra.CassandraSQLRow”

半世苍凉 提交于 2019-12-08 07:04:26
问题 I am trying to run an scala program in spark that accesses Cassandra through the Cassandra connector from datastax. I am getting the following error 15/04/30 17:43:44 ERROR Executor: Exception in task 0.0 in stage 2.0 (TID 2) com.esotericsoftware.kryo.KryoException: Unable to find class: org.apache.spark.sql.cassandra.CassandraSQLRow at com.esotericsoftware.kryo.util.DefaultClassResolver.readName(DefaultClassResolver.java:138) at com.esotericsoftware.kryo.util.DefaultClassResolver.readClass

Severe degradation in Cassandra Write performance with continuous streaming data over time

六眼飞鱼酱① 提交于 2019-12-08 06:24:35
问题 I notice a severe degradation in Cassandra write performance with continuous writes over time. I am inserting time series data with time stamp (T) as the column name in a wide column that stores 24 hours worth of data in a single row. Streaming data is written from data generator (4 instances, each with 256 threads) inserting data into multiple rows in parallel. Additionally, data is also inserted into a column family that has indexes over DateType and UUIDType. CF1: Col1 | Col2 | Col3

data modeling in Cassandra with columns that can be text or numbers

[亡魂溺海] 提交于 2019-12-08 05:57:38
问题 I have table with 5 columns. 1. ID - number but it can stored as text or number 2. name - text 3. date - date value but can stored as date or text 4. time - number but it can stored as text or number 5. rating - number but it can stored as text or number I want to find which data type will make my table faster for write. How can I find. Any Cassandra stress yaml for this there? 回答1: Regarding answer that @BryceAtNetwork23 provided, it will be the same with Cassandra 2.1 or in Cassandra 2.2

Normal Query on Cassandra using DataStax Enterprise works, but not solr_query

依然范特西╮ 提交于 2019-12-08 05:20:59
问题 I am having a strange issue occur while utilizing the solr_query handler to make queries in Cassandra on my terminal. When I perform normal queries on my table, I am having no issues, but when I use solr_query I get the following error: Unable to complete request: one or more nodes were unavailable. Other individuals who have experienced this problem seem unable to do any queries on their data whatsoever, whether or not it is solr_query. My problem only persists while using that handler. Can

DSE - Cassandra : Commit Log Disk Impact on Performances

ぃ、小莉子 提交于 2019-12-08 04:50:34
问题 I'm running a DSE 4.6.5 Cluster (Cassandra 2.0.14.352). Following datastax's guidelines, on every machine, I separated the data directory from the commitlog/saved caches directories: data is on blazing fast drives commit log and saved caches are on the system drives : 2 HDD RAID1 Monitoring disks with OpsCenter while performing intensive writes, I see no issue with the first, however I see the queue size from the later (commit log) averaging around 300 to 400 with spikes up to 700 requests.

“Permanent Exception” connecting Titan 0.4.0 to Datastax AMI

依然范特西╮ 提交于 2019-12-08 04:26:10
问题 I'm new in the Titan realm so I'll keep this question as simple as possible. Using the instructions provided here, I am able to successfully connect 1 linux node running Titan 0.3.2 to the Datastax/Cassandra instance as a backend. I then downloaded Titan 0.4.0 and attempted the same, but receive a backend exception. For good measure, I changed the keyspace to in my .properties file to "titan4", which I then see reflected in the Cassandra backend. However, despite successfully seeing the