First some background to my question.
It depends on the number of different security groups that are relevant in your context and how the security applies to your indexed data.
We had a similar issue which we solved the following way: When indexing we added the allowed groups to the document and when searching we added a boolean query with the groups the user was a member of. That performed well in our scenario.