Month: September 2023

Before machine learning algorithms can be used, some data pre-processing is required, missing value treatment is one of them.

How to find missing values in Python?

The function isnull() of a pandas data frame helps to find missing values in each column.

LoanData.isnull().sum()

# Creating a sample data frame

How to treat missing values?

Once you have found the missing values in each column, then you have below options

• Deleting all the missing values
• Replacing missing values with the median for continuous columns
• Replacing missing values with the mode for categorical columns
• Interpolating values

How to delete all missing values at once?

This option is exercised only if the number of rows containing missing values is much less than the total number of rows.

The dropna() function of the pandas data frame removes all those rows that contain at least one missing value.

1 2 3 4

# Code to delete all the missing values at once print(‘Before Deleting missing values:’, LoanData.shape) LoanDataCleaned=LoanData.dropna() print(‘After Deleting missing values:’, LoanDataCleaned.shape)

Replacing missing values using median/mode

Missing values treatment is done separately for each column in data. If the column is continuous, then its missing values will be replaced by the median of the same column. If the column is categorical, then the missing values will be replaced by the mode of the same column.

Instead of replacing all missing values with just one value(median/mode), you can also choose to interpolate the missing values based on the data present in the nearby location. This is called interpolation.

 

The k-fold cross-validation procedure is a standard method for estimating the performance of a machine learning algorithm or configuration on a dataset.Repeated k-fold cross-validation provides a way to improve the estimated performance of a machine learning model. This involves simply repeating the cross-validation procedure multiple times and reporting the mean result across all folds from all runs. This mean result is expected to be a more accurate estimate of the true unknown underlying mean performance of the model on the dataset, as calculated using the standard error.

The algorithm of the k-Fold technique:

1. Pick a number of folds – k. Usually, k is 5 or 10 but you can choose any number which is less than the dataset’s length.
2. Split the dataset into k equal (if possible) parts (they are called folds)
3. Choose k – 1 folds as the training set. The remaining fold will be the test set
4. Train the model on the training set. On each iteration of cross-validation, you must train a new model independently of the model trained on the previous iteration
5. Validate on the test set
6. Save the result of the validation
7. Repeat steps 3 – 6 k times. Each time use the remaining  fold as the test set. In the end, you should have validated the model on every fold that you have.
8. To get the final score average the results that you got on step 6.

Regularization helps to solve over fitting problem in machine learning. Simple model will be a very poor generalization of data.

At the same time, complex model may not perform well in test data due to over fitting. We need to choose the right model in between simple and complex model. Regularization helps to choose preferred model complexity, so that model is better at predicting.

Regularization is nothing but adding a penalty term to the objective function and control the model complexity using that penalty term. It can be used for many machine learning algorithms.

The t-test is a statistical hypothesis that takes samples from both groups to determine if there is a significant difference between the means of the two groups.

It compares both sample mean and standard deviations while considering sample size and the degree of variability of the data.

Steps:
1) State a hypothesis. A hypothesis is classified as a null hypothesis ( H0) and an
2) alternative hypothesis (Ha) that rejects the null hypothesis. The null and alternate hypotheses are defined according to the type of test being performed.
3) Collect sample data.
4) Conduct the test.
5) Reject or fail to reject your null hypothesis H0

In linear regression when additional terms are added to the regression model to account for the possibility that the relationship between an independent variable (predictor) and the dependent variable (outcome) depends on the value of another independent variable.
In simpler terms, they represent the idea that the effect of one variable on the outcome is not constant but varies depending on the level of another variable.

Mathematically, an interaction term in a linear regression model takes the form of a product between two or more independent variables. For example, if you have two independent variables, X1 and X2, and you suspect that the effect of X1 on the outcome (Y) depends on the value of X2, you can introduce an interaction term like this:

Y = β0 + β1 * X1 + β2 * X2 + β3 * (X1 * X2) + ε

In this equation:

Y represents the dependent variable (the one you’re trying to predict).
X1 and X2 are independent variables.
β0, β1, β2, and β3 are the regression coefficients that represent the relationship between the variables.
ε represents the error term.
The coefficient β3 measures the strength and direction of the interaction effect. If β3 is statistically significant and positive, it indicates that the effect of X1 on Y increases as X2 increases. If β3 is negative, it suggests that the effect of X1 on Y decreases as X2 increases. If β3 is not statistically significant, it implies that there is no interaction effect between X1 and X2.

Interaction terms are useful when you suspect that the relationship between variables is more complex than a simple additive relationship. They allow you to capture how the relationship between two variables changes in the presence of other variables, potentially leading to a better understanding of the underlying data and improved model accuracy. However, it’s essential to be cautious when adding interaction terms, as including too many can lead to overfitting (as the equation polynomial increases, the more overfit the model), and they should be based on theoretical or domain knowledge rather than testing multiple combinations blindly.

A Scatter plot is the chart used when you want to visualize the relationship between two continuous variables in data. Typically used in Supervised ML(Regression). Where the target variable is a continuous variable. So if you want to check which continuous predictor has a clear relationship with the target variable, then you look at the scatter plots.

Consider the below scenario Here the target variable is “Weight” and we are trying to predict it based on the number of hours a person works out at the gym and the number of calories they consume in a day.

If you plot the scatter chart between weight and calories, you can see an increasing trend. We can easily deduce from this graph that, if the calory intake increases, then the weight also increases. This is known as a positive correlation. We can see a “clear trend”, hence, there is a relationship between weight and calories. In other words, the predictor variable calories can be used to predict weight.

Similarly, you can see there is a clear decreasing trend between Weight and the Hours, It means if the number of hours at the gym increases, the weight decreases. This is known as a Negative correlation. Again, there is a “clear trend”, hence there is a relationship between weight and hours. In other words, hours can be used to predict weight.

Hypothesis means assumption.

To test whether our assumption is correct based on given data is Hypothesis testing.

Take a example of a  tire factory. The radius of the ideal tire must be 16 inches. However, even if there is a deviation of 8% then it is accepted. Hence in this scenario, we can apply hypothesis testing.

1. Define the Null Hypothesis (H0): The radius of the tire= 16 Inch
2. Define the alternate Hypothesis(Ha): The radius of the tire != 16 Inch
3. Define the error tolerance limit: 8%
4. Conduct the test
5. Look at the P-value generated by the test: P-value= 0.79
6. If P-Value > 0.05 then accept the Null Hypothesis otherwise reject it. : Accept the Hypothesis, Hence, The tire produced is of good quality

P-Value is the probability of H0 being True.

The higher the P-value, the better the chances of our assumption(H0) to be true. The Textbook threshold to reject a Null Hypothesis is 5%. So, if P-Value is less than 0.05, this means there is less than 5% chance of Null Hypothesis being true, hence it is rejected. Otherwise, if P-Value is more than 0.05, then the Null Hypothesis is accepted.