It sounds like you've already shared the repository with a number of other users. If this is the case, then you should probably just live with it.
If you're in control of all of the clones then you can re-write the history on top of a modified root commit with the unintended files removed. Note that you shouldn't do this if other developers have already based work off this branch.
If you're into rewriting history then you can try the following. Note that because git keeps around logs of where your HEAD commit has been recently (reflogs), the large objects won't immediately disappear from your repository, or other repositories which already have them, even if you attempt to git gc or git gc --prune. It will, however, ensure that any new clones won't end up fetching the large objects as part of the history of the master branch.
Assuming that your working directory is 'clean':
# Go back the initial commit
git checkout <SHA1_of_old_root>
# Clean up the index to remove unwanted files, e.g. using git rm <files>
# ...
# Amend the initial commit with the new tree. Note the sha1 of the new commit
git commit --amend
# Go back to the master branch
git checkout master
# Re-apply all the commits onto the new root
git rebase --onto <SHA1_of_new_root> <SHA1_of_old_root>
