I prefer to use SourceTree, so the solution for me was to open the submodule repo in SourceTree which shows me list of all untracked files. I then group selected them all then used "Remove".
I was able to do this because I knew all the untracked files weren't actually needed.