sparklyr

After I managed it to connect to our (new) cluster using sparklyr with yarn-client method, now I can show just the tables from the default scheme. How can I connect to scheme.table ? Using DBI it's working e.g. with the following line: dbGetQuery(sc, "SELECT * FROM scheme.table LIMIT 10") In HUE, I can show all tables from all schemes. ~g You can either use a fully qualified name to register temporary view: spark_session(sc) %>% invoke("table", "my_database.my_table") %>% invoke("createOrReplaceTempView", "my_view") tbl(sc, "my_view") or use sql method to switch databases spark_session(sc) %>%

I am getting heap space errors on even fairly small datasets. I can be sure that I'm not running out of system memory. For example, consider a dataset containing about 20M rows and 9 columns, and that takes up 1GB on disk. I am playing with it on a Google Compute node with 30gb of memory. Let's say that I have this data in a dataframe called df . The following works fine, albeit somewhat slowly: library(tidyverse) uniques <- search_raw_lt %>% group_by(my_key) %>% summarise() %>% ungroup() The following throws java.lang.OutOfMemoryError: Java heap space . library(tidyverse) library(sparklyr) sc

I have some data in a database, and I want to work with it in Spark, using sparklyr . I can use a DBI -based package to import the data from the database into R dbconn <- dbConnect(<some connection args>) data_in_r <- dbReadTable(dbconn, "a table") then copy the data from R to Spark using sconn <- spark_connect(<some connection args>) data_ptr <- copy_to(sconn, data_in_r) Copying twice is slow for big datasets. How can I copy data directly from the database into Spark? sparklyr has several spark_read_*() functions for import, but nothing database related. sdf_import() looks like a possibility,

Does someone have an overview with respect to advantages/disadvantages of SparkR vs sparklyr? Google does not yield any satisfactory results and both seem fairly similar. Trying both out, SparkR appears a lot more cumbersome, whereas sparklyr is pretty straight forward (both to install but also to use, especially with the dplyr inputs). Can sparklyr only be used to run dplyr functions in parallel or also "normal" R-Code? Best The biggest advantage of SparkR is the ability to run on Spark arbitrary user-defined functions written in R: https://spark.apache.org/docs/2.0.1/sparkr.html#applying

I am using sparklyr to manipulate some data. Given a, a<-tibble(id = rep(c(1,10), each = 10), attribute1 = rep(c("This", "That", 'These', 'Those', "The", "Other", "Test", "End", "Start", 'Beginning'), 2), value = rep(seq(10,100, by = 10),2), average = rep(c(50,100),each = 10), upper_bound = rep(c(80, 130), each =10), lower_bound = rep(c(20, 70), each =10)) I would like use "gather" to manipulate the data, like this: b<- a %>% gather(key = type_data, value = value_data, -c(id:attribute1)) However, "gather" is not available on sparklyr. I have seen some people using sdf_pivot to mimic "gather"