What are Loss Functions in Machine Learning? (With Examples)

As a developer, as much as it is essential to write code, it is also important to assess the quality of the code. To determine the quality of a machine learning algorithm, it is crucial to understand loss functions. Loss functions are what help machines learn. It is a metric that the model utilizes to put a number to its performance. By performance, the author means how close or far the model has made its prediction to the actual label. With the continuous use of the loss function and the learning and re-learning process, the model can significantly decrease the number of errors it generates and move closer to a percent accuracy. 

Introduction

The traditional assessment techniques that we have come across in our daily lives focus on putting a label on things as if they were either correct or incorrect; however, with machine learning algorithms, that cannot be the case which is why loss functions are stressed on so much for enhancing the learning capabilities of a machine learning algorithm. There is no “one work for all” loss function. Loss functions are customized according to the use case, and the model used. It also depends significantly on the kind of data and the count and distribution of variables being considered. 

One can think of the loss function as the penalty the model applies when it fails to produce an accurate result. That being said, it is evident that when the results deviate from the precise answer, the model will penalize itself with a number that denotes the intensity of this deviation. 

Types of Loss Functions

While working with machine learning models, the two major categories we work on are classification and regression models. Similarly, loss functions can also be classified into two categories – classification loss functions and regression loss functions. Let us recall what classification and regression mean that will enable you to understand the working of the loss function even more. A category or label is predicted for specific inputs in classification models. Let’s say that an image from Naruto has to be classified, and there are four categories – Kakashi, Itachi, Jiraya, and Naruto. The model will predict a label for the image based on the algorithm and its learnings. Problems like these are termed classification problems. Regression, on the other hand, handles values that can be quantized, like the value of a house in the famous house price prediction problem. 

Regression Type Loss Functions

1. Mean Squared Error (MSE) Loss Function – This type of error is also known as the L2 error. As the name suggests, the mean squared error loss function calculates the squared difference between the actual and predicted values and then takes the average of the same. The final number generated is termed the loss corresponding to the model. The model’s accuracy is inversely proportional to the mean squared error loss function. 
2. Mean Squared Logarithmic Error (MSLE) Loss Function. This type of error focuses not on generating a number or a difference (deviation) from the actual value but also on getting a percentage that defines the variation of the predictions made by the model from the true value. For pricing-type models, MSLE can be a good fit as it introduces an asymmetry in the error curve. 
3. Mean Absolute Error (MAE) Loss Function. Like the mean squared error loss function, the mean absolute error loss function also calculates the deviation between the actual and predicted values by the model. While the mean squared error loss function took the directions of values into account and predicted a loss value, the mean absolute error loss function does not cater to that. The significant advantage of doing so is that it becomes resilient to the outliers introduced in the dataset. Mean absolute error does not necessarily mean calculating just the difference between the actual and predicted values. It also involves several numerical programming methodologies that focus on calculating the gradients utilized in its mathematical formula. 
4. Mean Bias Error (MBE) Loss Function – Mean bias error loss function is rarely used in machine learning. It does not generate a number that measures the intensity of deviation. Still, it generates a category that signifies whether a model has a positive or negative bias toward the actual value. 

Classification Type Loss Functions 

1. Binary Cross Entropy Loss Function – The binary cross entropy loss function labels the predicted values as 0 or 1 depending upon the category that has been predicted by the model and its deviation from the actual class. 
2. Hinge Loss Function – Hinge loss is highly beneficial in classification or categorization problems. It generates a value that lies between -1 and 1 and pushes the instances to have the correct sign assigned to themselves. It is said to perform better than the cross-entropy loss functions. 
3. Squared Hinge Loss Function – This introduces a squared value instead of the values we have used in the normal hinge loss function. It is efficient where probability deviation is not of concern. The margin is calculated by the classification border generated by the squared hinge loss function. 
4. Multi-Class Cross Entropy Loss Function – The multi-class cross-entropy loss function is the ideal choice for problems involving text classification. The classes for this type of problem can have n number of possibilities. Hence, this score works well since it calculates the average difference generated between the model’s class predictions and all possibilities of different classes. 
5. Sparse Multi-Class Entropy Loss Function – The problem with the multi-class cross-entropy loss function is that to generate the deviation from n number of classes, it performs one hot vector encoding operation that won’t work well with a large number of categories or data points. The sparse multi-class entropy loss function eliminates the use of one hot vector encoding. This loss function will work well when there is a vast vocabulary set against which the model works, and its performance needs to be evaluated. 

Conclusion

In this article, we understood the intuition and the need for loss functions while working on machine learning algorithms. We also saw the kinds of loss functions and their usage criteria. The author encourages the readers to implement the loss functions while working on their models in their preferred language, Python or R, and compare the model performance after fine-tuning their models according to the loss functions. 

Read more about learning paths at codedamn here.

Happy Learning!