Overview:
- Correlation coefficients evaluate how two variables are related to each other. The relationship could be linear, linear but in opposite direction (i.e., inversely related), or monotonic. In a monotonic relationship the variables may not change together at the same rate.
- pandas’ DataFrame class has the method corr() that computes three different correlation coefficients between two variables using any of the following methods : Pearson correlation method, Kendall Tau correlation method and Spearman correlation method. The correlation coefficients calculated using these methods vary from +1 to -1.
- While the corr() function finds the correlation coefficients between the columns of a DataFrame instance, the corrwith() function computes correlation coefficients between rows or columns of two different dataframe instances. The one dimensional collection pandas.series supports finding correlation between variables represented by two pandas.series objects.
Pearson correlation coefficient:
- Pearson correlation coefficient is defined as the covariance of two variables divided by the product of their standard deviations. It evaluates the linear relationship between two variables. Pearson correlation coefficient has a value between +1 and -1.
- The value 1 indicates that there is a linear correlation between variable x and y. The value 0 indicates that the variables x and y are not related. The value -1 indicates that there is an inverse correlation between variable x and y.
- Pearson correlation coefficient is also called as Pearson product-moment correlation coefficient.
Kendall Tau correlation coefficient:
- It quantifies the discrepancy between the number of concordant and discordant pairs of two variables.
Spearman correlation coefficient:
- Spearman correlation method is a nonparametric evaluation that finds the strength and direction of the monotonic relationship between two variables.
- This method is used when the data is not normally distributed or when the sample size is small (less than 30).
Example - Finding correlation coefficient between rows of a same DataFrame instance:
|
import pandas as pd import numpy as np import scipy as sp
values = {"X":[20, 25, 30, 35, 40, 45], "Y":[10, 9, 9, 8, 8, 7]};
dataFrame = pd.DataFrame(data=values); print("DataFrame:"); print(dataFrame);
corrrelation = dataFrame.corr(method="pearson"); print("Pearson correlation coefficient:"); print(corrrelation);
corrrelation = dataFrame.corr(method="kendall"); print("Kendall Tau correlation coefficient:"); print(corrrelation);
corrrelation = dataFrame.corr(method="spearman"); print("Spearman rank correlation:"); print(corrrelation); |
Output:
|
DataFrame: X Y 0 20 10 1 25 9 2 30 9 3 35 8 4 40 8 5 45 7 Pearson correlation coefficient: X Y X 1.00000 -0.96833 Y -0.96833 1.00000 Kendall Tau correlation coefficient: X Y X 1.000000 -0.930949 Y -0.930949 1.000000 Spearman rank correlation: X Y X 1.000000 -0.971008 Y -0.971008 1.000000 |
Example - Finding correlation coefficient between rows of a different DataFrame instances:
|
import pandas as pd dataValues1 = [(8, 9, 10, 11, 12, 13, 14, 15, 16), dataValues2 = [(2, 1.5, 1, 1.5, 3, 3, 2, 2.5, 3), print("DataFrame2:"); # Find Pearson correlation coefficient between rows of different data drames # Find Kendall Tau correlation coefficient between rows of different data drames # Find Spearman rank correlation between rows of different data drames |
Output:
|
DataFrame1: 0 1 2 3 4 5 6 7 8 0 8.0 9.0 10.0 11.0 12.0 13.0 14.0 15.0 16.0 1 8.5 9.5 10.5 11.5 12.5 13.5 14.5 15.5 16.5 DataFrame2: 0 1 2 3 4 5 6 7 8 0 2.0 1.5 1.0 1.5 3.0 3.0 2.0 2.50 3.0 1 2.1 1.5 1.2 1.4 3.2 3.1 2.2 2.53 3.2 Pearson correlation coefficient between rows of dataFrame1 and dataFrame2: 0 0.639010 1 0.645101 dtype: float64 Kendall Tau correlation coefficient between rows of dataFrame1 and dataFrame2: 0 0.449013 1 0.422577 dtype: float64 Spearman rank correlation between rows of dataFrame1 and dataFrame2: 0 0.632687 1 0.669462 dtype: float64 |