Before machine learning algorithms can be used, some data pre-processing is required, missing value treatment is one of them.
How to find missing values in Python?
The function isnull() of a pandas data frame helps to find missing values in each column.
LoanData.isnull().sum()
# Creating a sample data frame
How to treat missing values?
Once you have found the missing values in each column, then you have below options
- Deleting all the missing values
- Replacing missing values with the median for continuous columns
- Replacing missing values with the mode for categorical columns
- Interpolating values
How to delete all missing values at once?
This option is exercised only if the number of rows containing missing values is much less than the total number of rows.
The dropna() function of the pandas data frame removes all those rows that contain at least one missing value.
|
1
2
3
4
|
# Code to delete all the missing values at once
print(‘Before Deleting missing values:’, LoanData.shape)
LoanDataCleaned=LoanData.dropna()
print(‘After Deleting missing values:’, LoanDataCleaned.shape)
|
Replacing missing values using median/mode
Missing values treatment is done separately for each column in data. If the column is continuous, then its missing values will be replaced by the median of the same column. If the column is categorical, then the missing values will be replaced by the mode of the same column.
|
1
2
3
4
5
6
7
|
# Replacing with median value for a numeric variable
MedianAge=LoanData[‘AGE’].median()
LoanData[‘AGE’]=LoanData[‘AGE’].fillna(value=MedianAge)
# Replacing with mode value for a categorical variable
ModeValue=LoanData[‘APPROVE_LOAN’].mode()[0]
LoanData[‘APPROVE_LOAN’]=LoanData[‘APPROVE_LOAN’].fillna(value=ModeValue)
|
Replacing missing values using interpolation
Instead of replacing all missing values with just one value(median/mode), you can also choose to interpolate the missing values based on the data present in the nearby location. This is called interpolation.
|
1
2
3
4
5
|
# Replacing missing values by interpolation for a numeric variable
LoanData[‘AGE’]=LoanData[‘AGE’].interpolate(method=‘linear’)
# Replacing missing values by interpolation for a categorical variable
LoanData[‘APPROVE_LOAN’]=LoanData[‘APPROVE_LOAN’].interpolate(method=‘ffill’)
|