Another option which is a bit influenced from @Marwan Burelle idea.
Each component will hold a sorted container of entities which have that component.
When looking for entities that match a signature you need to iterate over the component container of entities.
Adding or removing is O(nlogn) since it needs to be sorted. but you only need to add/remove it to/from a single container which will also contain fewer items.
Iterating over the items is a bit heavier since it is a factor of the amount of components and the number of entities in each component.
You still have an element of multiplying but the number of elements is again smaller.
I wrote down a simplified version as a POC.
Edit: My previous version had some bugs, now hopefully they are fixed.
// Example program
#include
#include
#include
#include