问题
I'm trying to calculate the variance inflation factor (VIF) for each column in a simple dataset in python:
a b c d
1 2 4 4
1 2 6 3
2 3 7 4
3 2 8 5
4 1 9 4
I have already done this in R using the vif function from the usdm library which gives the following results:
a <- c(1, 1, 2, 3, 4)
b <- c(2, 2, 3, 2, 1)
c <- c(4, 6, 7, 8, 9)
d <- c(4, 3, 4, 5, 4)
df <- data.frame(a, b, c, d)
vif_df <- vif(df)
print(vif_df)
Variables VIF
a 22.95
b 3.00
c 12.95
d 3.00
However, when I do the same in python using the statsmodel vif function, my results are:
a = [1, 1, 2, 3, 4]
b = [2, 2, 3, 2, 1]
c = [4, 6, 7, 8, 9]
d = [4, 3, 4, 5, 4]
ck = np.column_stack([a, b, c, d])
vif = [variance_inflation_factor(ck, i) for i in range(ck.shape[1])]
print(vif)
Variables VIF
a 47.136986301369774
b 28.931506849315081
c 80.31506849315096
d 40.438356164383549
The results are vastly different, even though the inputs are the same. In general, results from the statsmodel VIF function seem to be wrong, but I'm not sure if this is because of the way I am calling it or if it is an issue with the function itself.
I was hoping someone could help me figure out whether I was incorrectly calling the statsmodel function or explain the discrepancies in the results. If it's an issue with the function then are there any VIF alternatives in python?
回答1:
I believe the reason for this is due to a difference in Python's OLS. OLS, which is used in the python variance inflation factor calculation, does not add an intercept by default. You definitely want an intercept in there however.
What you'd want to do is add one more column to your matrix, ck, filled with ones to represent a constant. This will be the the intercept term of the equation. Once this is done, your values should match out properly.
Edited: replaced zeroes with ones
回答2:
As mentioned by others and in this post by Josef Perktold, the function's author, variance_inflation_factor expects the presence of a constant in the matrix of explanatory variables. One can use add_constant from statsmodels to add the required constant to the dataframe before passing its values to the function.
from statsmodels.stats.outliers_influence import variance_inflation_factor
from statsmodels.tools.tools import add_constant
df = pd.DataFrame(
{'a': [1, 1, 2, 3, 4],
'b': [2, 2, 3, 2, 1],
'c': [4, 6, 7, 8, 9],
'd': [4, 3, 4, 5, 4]}
)
X = add_constant(df)
>>> pd.Series([variance_inflation_factor(X.values, i)
for i in range(X.shape[1])],
index=X.columns)
const 136.875
a 22.950
b 3.000
c 12.950
d 3.000
dtype: float64
I believe you could also add the constant to the right most column of the dataframe using assign:
X = df.assign(const=1)
>>> pd.Series([variance_inflation_factor(X.values, i)
for i in range(X.shape[1])],
index=X.columns)
a 22.950
b 3.000
c 12.950
d 3.000
const 136.875
dtype: float64
The source code itself is rather concise:
def variance_inflation_factor(exog, exog_idx):
"""
exog : ndarray, (nobs, k_vars)
design matrix with all explanatory variables, as for example used in
regression
exog_idx : int
index of the exogenous variable in the columns of exog
"""
k_vars = exog.shape[1]
x_i = exog[:, exog_idx]
mask = np.arange(k_vars) != exog_idx
x_noti = exog[:, mask]
r_squared_i = OLS(x_i, x_noti).fit().rsquared
vif = 1. / (1. - r_squared_i)
return vif
It is also rather simple to modify the code to return all of the VIFs as a series:
from statsmodels.regression.linear_model import OLS
from statsmodels.tools.tools import add_constant
def variance_inflation_factors(exog_df):
'''
Parameters
----------
exog_df : dataframe, (nobs, k_vars)
design matrix with all explanatory variables, as for example used in
regression.
Returns
-------
vif : Series
variance inflation factors
'''
exog_df = add_constant(exog_df)
vifs = pd.Series(
[1 / (1. - OLS(exog_df[col].values,
exog_df.loc[:, exog_df.columns != col].values).fit().rsquared)
for col in exog_df],
index=exog_df.columns,
name='VIF'
)
return vifs
>>> variance_inflation_factors(df)
const 136.875
a 22.950
b 3.000
c 12.950
Name: VIF, dtype: float64
回答3:
For future comers to this thread (like me):
import numpy as np
import scipy as sp
a = [1, 1, 2, 3, 4]
b = [2, 2, 3, 2, 1]
c = [4, 6, 7, 8, 9]
d = [4, 3, 4, 5, 4]
ck = np.column_stack([a, b, c, d])
cc = sp.corrcoef(ck, rowvar=False)
VIF = np.linalg.inv(cc)
VIF.diagonal()
This code gives
array([22.95, 3. , 12.95, 3. ])
[EDIT]
In response to a comment, I tried to use DataFrame as much as possible (numpy is required to invert a matrix).
import pandas as pd
import numpy as np
a = [1, 1, 2, 3, 4]
b = [2, 2, 3, 2, 1]
c = [4, 6, 7, 8, 9]
d = [4, 3, 4, 5, 4]
df = pd.DataFrame({'a':a,'b':b,'c':c,'d':d})
df_cor = df.corr()
pd.DataFrame(np.linalg.inv(df.corr().values), index = df_cor.index, columns=df_cor.columns)
The code gives
a b c d
a 22.950000 6.453681 -16.301917 -6.453681
b 6.453681 3.000000 -4.080441 -2.000000
c -16.301917 -4.080441 12.950000 4.080441
d -6.453681 -2.000000 4.080441 3.000000
The diagonal elements give VIF.
回答4:
In case you don't wanna deal with variance_inflation_factor and add_constant. Please consider the following two functions.
1. Use formula in statasmodels:
import pandas as pd
import statsmodels.formula.api as smf
def get_vif(exogs, data):
'''Return VIF (variance inflation factor) DataFrame
Args:
exogs (list): list of exogenous/independent variables
data (DataFrame): the df storing all variables
Returns:
VIF and Tolerance DataFrame for each exogenous variable
Notes:
Assume we have a list of exogenous variable [X1, X2, X3, X4].
To calculate the VIF and Tolerance for each variable, we regress
each of them against other exogenous variables. For instance, the
regression model for X3 is defined as:
X3 ~ X1 + X2 + X4
And then we extract the R-squared from the model to calculate:
VIF = 1 / (1 - R-squared)
Tolerance = 1 - R-squared
The cutoff to detect multicollinearity:
VIF > 10 or Tolerance < 0.1
'''
# initialize dictionaries
vif_dict, tolerance_dict = {}, {}
# create formula for each exogenous variable
for exog in exogs:
not_exog = [i for i in exogs if i != exog]
formula = f"{exog} ~ {' + '.join(not_exog)}"
# extract r-squared from the fit
r_squared = smf.ols(formula, data=data).fit().rsquared
# calculate VIF
vif = 1/(1 - r_squared)
vif_dict[exog] = vif
# calculate tolerance
tolerance = 1 - r_squared
tolerance_dict[exog] = tolerance
# return VIF DataFrame
df_vif = pd.DataFrame({'VIF': vif_dict, 'Tolerance': tolerance_dict})
return df_vif
2. Use LinearRegression in sklearn:
# import warnings
# warnings.simplefilter(action='ignore', category=FutureWarning)
import pandas as pd
from sklearn.linear_model import LinearRegression
def sklearn_vif(exogs, data):
# initialize dictionaries
vif_dict, tolerance_dict = {}, {}
# form input data for each exogenous variable
for exog in exogs:
not_exog = [i for i in exogs if i != exog]
X, y = data[not_exog], data[exog]
# extract r-squared from the fit
r_squared = LinearRegression().fit(X, y).score(X, y)
# calculate VIF
vif = 1/(1 - r_squared)
vif_dict[exog] = vif
# calculate tolerance
tolerance = 1 - r_squared
tolerance_dict[exog] = tolerance
# return VIF DataFrame
df_vif = pd.DataFrame({'VIF': vif_dict, 'Tolerance': tolerance_dict})
return df_vif
Example:
import seaborn as sns
df = sns.load_dataset('car_crashes')
exogs = ['alcohol', 'speeding', 'no_previous', 'not_distracted']
[In] %%timeit -n 100
get_vif(exogs=exogs, data=df)
[Out]
VIF Tolerance
alcohol 3.436072 0.291030
no_previous 3.113984 0.321132
not_distracted 2.668456 0.374749
speeding 1.884340 0.530690
69.6 ms ± 8.96 ms per loop (mean ± std. dev. of 7 runs, 100 loops each)
[In] %%timeit -n 100
sklearn_vif(exogs=exogs, data=df)
[Out]
VIF Tolerance
alcohol 3.436072 0.291030
no_previous 3.113984 0.321132
not_distracted 2.668456 0.374749
speeding 1.884340 0.530690
15.7 ms ± 1.4 ms per loop (mean ± std. dev. of 7 runs, 100 loops each)
回答5:
Example for Boston Data:
VIF is calculated by auxiliary regression, so not dependent on the actual fit.
See below:
from patsy import dmatrices
from statsmodels.stats.outliers_influence import variance_inflation_factor
import statsmodels.api as sm
# Break into left and right hand side; y and X
y, X = dmatrices(formula="medv ~ crim + zn + nox + ptratio + black + rm ", data=boston, return_type="dataframe")
# For each Xi, calculate VIF
vif = [variance_inflation_factor(X.values, i) for i in range(X.shape[1])]
# Fit X to y
result = sm.OLS(y, X).fit()
回答6:
I wrote this function based on some other posts I saw on Stack and CrossValidated. It shows the features which are over the threshold and returns a new dataframe with the features removed.
from statsmodels.stats.outliers_influence import variance_inflation_factor
from statsmodels.tools.tools import add_constant
def calculate_vif_(df, thresh=5):
'''
Calculates VIF each feature in a pandas dataframe
A constant must be added to variance_inflation_factor or the results will be incorrect
:param X: the pandas dataframe
:param thresh: the max VIF value before the feature is removed from the dataframe
:return: dataframe with features removed
'''
const = add_constant(df)
cols = const.columns
variables = np.arange(const.shape[1])
vif_df = pd.Series([variance_inflation_factor(const.values, i)
for i in range(const.shape[1])],
index=const.columns).to_frame()
vif_df = vif_df.sort_values(by=0, ascending=False).rename(columns={0: 'VIF'})
vif_df = vif_df.drop('const')
vif_df = vif_df[vif_df['VIF'] > thresh]
print 'Features above VIF threshold:\n'
print vif_df[vif_df['VIF'] > thresh]
col_to_drop = list(vif_df.index)
for i in col_to_drop:
print 'Dropping: {}'.format(i)
df = df.drop(columns=i)
return df
来源：https://stackoverflow.com/questions/42658379/variance-inflation-factor-in-python