Slow training on CPU and GPU in a small network (tensorflow)

Question

Here is the original script I\'m trying to run on both CPU and GPU, I\'m expecting a much faster training on GPU however it\'s taking almost the same time. I made the follow