The reason was pretty simple.
I was calling EnsureCreated in Startup.cs which was getting conflict with migrations as working different way.
Thankfully EF Core owners made it clear for me on GitHub.
So to summarize - if you want to use Migrations, you can't use EnsureCreated.