Root Mean Squared Error (RMSE) is a popular metric to use for regression machine learning models, but what is good value? In this post, I explain what RMSE is, what a good value is, and answer some common questions.
What is RMSE?
Root Mean Squared Error (RMSE) is the square root of the mean squared error between the predicted and actual values.
Squared error, also known as L2 loss, is a row-level error calculation where the difference between the prediction and the actual is squared. RMSE is the aggregated mean and subsequent square root of these errors, which helps us understand the model performance over the whole dataset.
A benefit of using RMSE is that the metric it produces is on the same scale as the unit being predicted. For example, calculating RMSE for a house price prediction model would give the error in terms of house price, which can help end users easily understand model performance.
The formula for calculating RMSE is:
How do you calculate RMSE?
To calculate RMSE for a set of predictions and their corresponding actual values, you need to follow these steps:
- Calculate the difference between each prediction and the actual value
- Square each of these values
- Calculate the mean of these squared values
- Calculate the square root of this mean
Example of calculating RMSE
Let’s look at an example where we calculate RMSE for a model which is predicting people’s height.
RMSE = sqrt[(100 + 25 + 4 + 9 + 16)/5] = 5.55
Calculate RMSE in Python with Numpy
We can also calculate RMSE in Python using either the scikit-learn or Numpy packages. Here is an example of how this could be done using Numpy.
import numpy as np actual = np.array([1100,2000,550,1510,2000]) prediction = np.array([1000,2000,500,1500,3000]) rmse = np.sqrt(np.mean((prediction-actual)**2))
Should RMSE be high or low?
RMSE is a metric which ranges from 0 to infinity, where the closer the score is to 0 the better performing the model is. So the RMSE value should be as low as possible.
How to interpret RMSE
RMSE is a measure of model performance given on the same scale as the prediction target. In a general sense, RMSE can be interpreted as the average error that the model’s predictions have in comparison with the actual, with extra weight added to larger prediction errors.
How can I compare RMSE values from different models?
RMSE is an absolute error value, meaning that it is only relevant for the dataset and model it is calculated for. Therefore, RMSE cannot be used to compare model performance across different datasets. One way around this issue is to calculate the Root Mean Squared Percentage Error (RMSPE) which is the RMSE calculated as a percentage, much like MAPE.
What is a good RMSE value?
The closer RMSE is to 0, the more accurate the model is. But RMSE is returned on the same scale as the target you are predicting for and therefore there isn’t a general rule for what is considered a ‘good’ value. How good your metric value is can only be evaluated within the dataset context you are working.
Let’s try to unpack this more by looking at an example.
An RMSE of 1,000 for a house price prediction model is most likely seen as good because house prices tend to be over $100,000. However, the same RMSE of 1,000 for a height prediction model is terrible as the average height is around 175cm.
So unfortunately there is no standard for what a good value is, you will have to decide what is acceptable in the context of your project.