In my case, I had 2 python consoles open, both using keras/tensorflow.
As I closed the old console (forgotten from previous day),
everything started to work correctly.
So it is good to check, if you do not have multiple consoles / processes occupying GPU.