How to impute values in a column when certain conditions are fulfilled in other columns using fillna()

问题

I've calculated the counts when credit_history has NaN values.

Output when Credit_History is NaN:

Self_Employed
Yes  532
No   32

Married
No   398
Yes  21

And for the numerical values, I calculated the mean for all columns

output for non-numerical values when Credit_History is NaN:

Mean Applicant Income: 54003.1232
LoanAmount: 35435.12
Loan_Amount_Term: 360
ApplicantIncome: 30000

How do I now use fillna() in these cases:

Case 1: When Self_Employed = Y and Married = N; Credit_History should be 0

Case 2: When Self_Employed = N and ApplicantIncome > 20000; Credit_History should be 1

Case 3: When Self_Employed = Y, Married = N and ApplicantIncome > 2000; Credit_History should be 1

Also, when using fillna() is not so obvious for certain conditions, can we use a pivot table to calculate the median values and then impute them using fillna()?

Thanks in advance.

回答1:

Use numpy.select and if all condition are False, output is define by parameter default:

from  itertools import  product
c = ['Self_Employed','Married','ApplicantIncome']
df =  pd.DataFrame(list(product(list('NY'), list('NY'), [10000, 30000])), 
                   columns=c)


m1 = (df.Self_Employed == 'Y') & (df.Married == 'N')
m2 = (df.Self_Employed == 'N') & (df.ApplicantIncome > 20000)
m3 = m1 & (df.ApplicantIncome > 20000)

df['Credit_History'] = np.select([m1, m2, m3], [0,1,1], default=2)
print (df)
  Self_Employed Married  ApplicantIncome  Credit_History
0             N       N            10000               2
1             N       N            30000               1
2             N       Y            10000               2
3             N       Y            30000               1
4             Y       N            10000               0
5             Y       N            30000               0
6             Y       Y            10000               2
7             Y       Y            30000               2

---

But if want replace by conditions add fillna:

c = ['Self_Employed','Married','ApplicantIncome']
df =  pd.DataFrame(list(product(list('NY'), list('NY'), [10000, 30000])), 
                   columns=c).assign(Credit_History=[np.nan,1,0, np.nan] *2)
print (df)
  Self_Employed Married  ApplicantIncome  Credit_History
0             N       N            10000             NaN
1             N       N            30000             1.0
2             N       Y            10000             0.0
3             N       Y            30000             NaN
4             Y       N            10000             NaN
5             Y       N            30000             1.0
6             Y       Y            10000             0.0
7             Y       Y            30000             NaN

m1 = (df.Self_Employed == 'Y') & (df.Married == 'N')
m2 = (df.Self_Employed == 'N') & (df.ApplicantIncome > 20000)
m3 = m1 & (df.ApplicantIncome > 20000)

s = pd.Series(np.select([m1, m2, m3], [0,1,1], default=2), index=df.index)
df['Credit_History'] = df['Credit_History'].fillna(s)
print (df)
  Self_Employed Married  ApplicantIncome  Credit_History
0             N       N            10000             2.0
1             N       N            30000             1.0
2             N       Y            10000             0.0
3             N       Y            30000             1.0
4             Y       N            10000             0.0
5             Y       N            30000             1.0
6             Y       Y            10000             0.0
7             Y       Y            30000             2.0

来源：https://stackoverflow.com/questions/49088259/how-to-impute-values-in-a-column-when-certain-conditions-are-fulfilled-in-other