Htykuut

I'm working on a Kaggle competition (https://www.kaggle.com/c/house-prices-advanced-regression-techniques#evaluation) and it states that my model will be evaluated by:

Submissions are evaluated on Root-Mean-Squared-Error (RMSE) between the logarithm of the predicted value and the logarithm of the observed sales price. (Taking logs means that errors in predicting expensive houses and cheap houses will affect the result equally.)

I couldn't find this in the docs (it's basically RMSE(log(truth), log(prediction)), so I went about writing a custom scorer:

def custom_loss(truth, preds):

    truth_logs = np.log(truth)

    print(truth_logs)

    preds_logs = np.log(preds)

    numerator = np.sum(np.square(truth_logs - preds_logs))

    return np.sum(np.sqrt(numerator / len(truth)))



custom_scorer = make_scorer(custom_loss, greater_is_better=False)

Two questions:

1) Should my custom loss function return a numpy array of scores (one for each (truth, prediction) pair? Or should it be the total loss over those (truth, prediction) pairs, returning a single number?

I looked into the docs but they weren't super helpful re: what my custom loss function should return.

2) When I run:

xgb_model = xgb.XGBRegressor()

params = {"max_depth": [3, 4], "learning_rate": [0.05],

         "n_estimators": [1000, 2000], "n_jobs": [8], "subsample": [0.8], "random_state": [42]}

grid_search_cv = GridSearchCV(xgb_model, params, scoring=custom_scorer,

                             n_jobs=8, cv=KFold(n_splits=10, shuffle=True, random_state=42), verbose=2)



grid_search_cv.fit(X, y)



grid_search_cv.best_score_

I get back:

-0.12137097567803554

which is very surprising. Given that my loss function is taking RMSE(log(truth) - log(prediction)), I shouldn't be able to have a negative best_score_.

Any idea why it's negative?

Thanks!

1) You should return a single number as loss, not array. GridSearchCV will sort the params accroding to the results of this scorer.

By the way instead of defining a custom metric, you can use mean_squared_log_error, which does what you want.

2) Why does it return negative? - Without your actual data and complete code we cant say.

You should be careful with the notation.

There are 2 levels of optimization here:

1. The loss function optimized when the XGBRegressor is fitted to the data.


2. The scoring function that is optimized during the grid search.

I prefer calling the second scoring function instead of loss function, since loss function usually refers to a term that is subject to optimization during the model fitting process itself.
However, your custom function only specifies 2. whilst leaving 1. untouched. In case you want to change the loss function of XGBRegressor see here. Most regression models have several criteria from which you can choose such as mean_square_error or mean_absolute_error.

Note, that passing customized loss functions is not supported at the moment (see reasons here and here).