Forcing `imshow` to properly color pixels based on `Colormap` colors when not all values are present in image?

Question

I am working on visualizing the output of semantic segmentation neural network. For this I have a set of labels(classes) and corresponding colors for pixels. I am displaying the