Should the custom loss function in Keras return a single loss value for the batch or an arrary of losses for every sample in the training batch?

Question

I\'m learning keras API in tensorflow(2.3). In this guide on tensorflow website, I found an example of custom loss funciton:

    def custom_mean_squared_error         


        

Answer 1

I think the question posted by @Gödel is totally legit and is correct. The custom loss function should return a loss value per sample. And, an explanation provided by @today is also correct. In the end, it all depends on the kind of reduction used.

So if one uses class API to create a loss function, then, reduction parameter is automatically inherited in the custom class. Its default value "sum_over_batch_size" is used (which is simply averaging of all the loss values in a given batch). Other options are "sum", which computes a sum instead of averaging and the last option is "none", where an array of loss values are returned.

It is also mentioned in the Keras documentation that these differences in reduction are irreverent when one is using model.fit() because reduction is then automatically handled by TF/Keras.

And, lastly, it is also mentioned that when a custom loss function is created, then, an array of losses (individual sample losses) should be returned. Their reduction is handled by the framework.

Links:

• https://keras.io/api/losses/
• Checkout CategoricalCrossentropy Class: https://keras.io/api/losses/probabilistic_losses/#categoricalcrossentropy-class