Performance Matrices of Models

A deep dive into the performance matrices of Models:

I will describe here all important performance measures to choose best model for the problem. There are many such problem matrices i will explain each according to the order.

1. Accuracy

Accuracy is defined as correctly classified data points from all of the data points. In Machine Learning perspective it is the ratio of true positive ,true negative data points to the total true positive, true negative, false positive and false negatives. For them who are not known with these terms :

True Positive and True Negative : Model correctly classified positive and negative point.

False Positive and False Negative : Models incorrectly classified positive and negative points.

There are some problems of using Accuracy as a measure :

1. For Imbalanced data : If data is imbalanced,For example data is in 90 : 10 then accuracy will be 90% which is more to dumb models because of majority class.So even model is dumb accuracy gives high value which is bad.
2. If two model gives probabilistic values then there can be case that predicted values could come same .So we can not differentiate these models according to predicted data. Accuracy can not use probability scores.

2.Confusion Matrix, TPR, FPR, FNR, TNR

A confusion matrix is a table that is often used to describe the performance of classifier or model on a set of test data.These terms are quite confusing but i will tell you the hack how you can learn at you tips.First let us see how confusion matrix looks like :

Picture shows confusion matrix for binary classification. If model is sensible then a and d should be high also b and c should be small. In case of multiclass classification off diagonal elements should be low and principle diagonal elements should be high.

Lets define some bunch of terms used in Confusion matrix in case of binary classification:

We can directly learn this but people often confuse with these terms.I will give a idea to learn this.First look at below matrix:

• true positives (TP): These are cases in which we predicted yes and actual is also yes
• true negatives (TN): We predicted no, and actual is no.
• false positives (FP): We predicted yes, and actual is no.(Also known as a “Type I error”
• false negatives (FN): We predicted no, but actual is yes.(Also known as a “Type II error.”)
• FN + TP = P = sum of total number of data points which belong to positive(1) class
• TN + FP = N = sum of total number of data points which belong to negative(0) class
• n = N + P = Total number of data points

By help of these above terms we will define four important terms in Confusion matrix

TPR(true positive rate) = TP/P(Ratio of True Positive points to the total positive points

TNR(true negative rate) = TN/N(Ratio of true negative points to the total negative points

FPR(false positive rate) = FP/N(Ratio of false positive points to the total negative points)

FNR(false negative rate) = FN/P(Ratio of False Negative and Total Positive points)

Let take an example of imabalanced dataset to calculate all above terms.The datset contains 900 -(ve) points and 100 +(ve) points.And let say we already built the model,and according to it the values of matrix will like this:-

a = TN = 850

b = FN = 06

c = FP = 50

d = TP = 94

TPR = TP/P = 94/100 = 94%

TNR = TN/N = 850/900 = 94.4%

FPR = FP/N = 50/900 = 5.5%

FNR = FN/P = 6/100 = 6%

Let take same example in which the model is dumb

a = TN = 900

b = FN = 100

c = FP = 0

d = TP = 0

TPR = TP/P = 0/100 = 0%

TNR = TN/N = 900/900 = 100%

FPR = FP/N = 0/900 = 0%

FNR = FN/P = 100/100 = 100%

So as we can see that TPR and FNR are not values which we need,so confusion matrix and hence values TPR,TNR,FPR,FNR can tell the difference between a sensible model and dumb model.But accuracy can’t do that.So confusion has a big advantage over accuracy.

3).Precision,Recall and F1-Score

Precision and Recall are often used in information retrieval problems.Go through picture to understand context:

In both the cases it is revolving around TP.Precision and Recall are very useful when you only care about +ve classes.They tell how well we are doing in case of +ve classes.

Precision : Of all the points that model predicted how many are truly +ves.

Recall: Of all the actually +ve points how many of them are predicted to be positive.

F1 Score : It is the average of inverse precision and inverse recall.

4).Reciever Operating Characteristics Curve and AUC

AUC — ROC curve is a performance measurement for classification problem at various thresholds settings. ROC is a probability curve and AUC represents degree or measure of separability. It tells how much model is capable of distinguishing between classes. Higher the AUC, better the model is at predicting 0s as 0s and 1s as 1s. By analogy, Higher the AUC, better the model is at distinguishing between patients with disease and no disease.

Process:

1. Find Probability scores in decreasing order.
2. We will find TPR and FPR values by selecting threshold each time and draw a plot.
3. Area under the ROC curve is AUC.

ROC Curve

It is only used in Binary classification which is a nice idea with some brilliant properties like it does not dependent on predicted scores but depends on ordering.AUC with value 0.5 to 1 specifies model is good.