I generally prefer to rasterize the colorbar contents to avoid this issue using Eric Firing's advice here by adding the following line.
cbar.solids.set_rasterized(True)
This workaround supposedly fails for images with transparency, but for most normal cases it produces the desired result.