correlation

问题 I am studying pyspark in databricks. I want to generate a correlation heatmap. Let's say this is my data: myGraph=spark.createDataFrame([(1.3,2.1,3.0), (2.5,4.6,3.1), (6.5,7.2,10.0)], ['col1','col2','col3']) And this is my code: import pyspark from pyspark.sql import SparkSession import matplotlib.pyplot as plt import pandas as pd import numpy as np from ggplot import * from pyspark.ml.feature import VectorAssembler from pyspark.ml.stat import Correlation from pyspark.mllib.stat import

问题 Here is my code: import numpy as np import pandas as pd import seaborn as sns import matplotlib.pyplot as plt data = pd.read_csv('death_regression2.csv') data3 = data.replace(r'\s+', np.nan, regex = True) plt.figure(figsize=(90,90)) corr = data3.corr() print(np.shape(list(corr))) print(np.shape(data3)) (135,) (4909, 204) So before I use the correlation function, the total number of parameters was 204(number of the columns) but after using data3.corr(), some parameters go missing, reduced to

问题 Actually there are 2 questions, one is more advanced than the other. Q1: I am looking for a method that similar to corrplot() but can deal with factors. I originally tried to use chisq.test() then calculate the p-value and Cramer's V as correlation, but there too many columns to figure out. So could anyone tell me if there is a quick way to create a "corrplot" that each cell contains the value of Cramer's V , while the colour is rendered by p-value . Or any other kind of similar plot.

问题 Actually there are 2 questions, one is more advanced than the other. Q1: I am looking for a method that similar to corrplot() but can deal with factors. I originally tried to use chisq.test() then calculate the p-value and Cramer's V as correlation, but there too many columns to figure out. So could anyone tell me if there is a quick way to create a "corrplot" that each cell contains the value of Cramer's V , while the colour is rendered by p-value . Or any other kind of similar plot.

问题 Constructing a ggpairs figure in R using the following code. df is a dataframe containing 6 continuous variables and one Group variable ggpairs(df[,-1],columns = 1:ncol(df[,-1]), mapping=ggplot2::aes(colour = df$Group),legends = T,axisLabels = "show", upper = list(continuous = wrap("cor", method = "spearman", size = 2.5, hjust=0.7)))+ theme(panel.grid.major = element_blank(), panel.grid.minor = element_blank(), axis.line = element_line(colour = "black")) I am trying to add the p-value of

问题 Constructing a ggpairs figure in R using the following code. df is a dataframe containing 6 continuous variables and one Group variable ggpairs(df[,-1],columns = 1:ncol(df[,-1]), mapping=ggplot2::aes(colour = df$Group),legends = T,axisLabels = "show", upper = list(continuous = wrap("cor", method = "spearman", size = 2.5, hjust=0.7)))+ theme(panel.grid.major = element_blank(), panel.grid.minor = element_blank(), axis.line = element_line(colour = "black")) I am trying to add the p-value of

问题 I have two list of words which are ordered by the number of occurrences The ordering was generated by counting each word in two files sampled at different point in times. I would like to calculate spearman to see how well the order of the first file was found in the second file. for instance: File a: 1) is 2) went 3) work File b: 1) is 2) work 3) went Because the ordering is different I would not achieve a score of 1.0 but yet one that would suggest that these two samples are rather similar

问题 I am working on data that contains more than 300 categorical features that I have factored into 0s and 1s. Now, i need to create a matrix of the features to with frequency of joint occurrence in each cell. In the end , I am looking to create a heatmap of this frequency matrix. So, my dataframe in R looks like this: id cat1 cat2 cat3 cat4 156 0 0 1 1 465 1 1 1 0 573 0 1 1 0 The output I want is: cat1 cat2 cat3 ... cat1 0 1 0 cat2 1 0 2 cat3 1 2 0 . . where each cell value denotes the number of

问题 I understand how to calculate a rolling sum, std or average. Example: df['MA10'] = df['Asset1'].rolling(10).mean() But I don't understand the syntax to calculate the rolling correlation between two dataframes columns: df['Asset1'] and df['Asset2'] The documentation doesn't provide any example regarding the correlation. https://pandas.pydata.org/pandas-docs/stable/generated/pandas.DataFrame.rolling.html Any insights? Thanks! 回答1: It's in there, even if hidden a bit: df['Asset1'].rolling(10)

问题 all, I have a correlation matrix of 21 industry sectors. Now I want to split these 21 sectors into 4 or 5 groups, with sectors of similar behaviors grouped together. Can experts shed me some lights on how to do this in Python please? Thanks much in advance! 回答1: You might explore the use of Pandas DataFrame.corr and the scipy.cluster Hierarchical Clustering package import pandas as pd import scipy.cluster.hierarchy as spc df = pd.DataFrame(my_data) corr = df.corr().values pdist = spc.distance