If we look at the stackoverflow website we have votes. But the question is what is the bestway to store who has voted and who has not. Lets also simplify this even more and say
I think to make another table "vote" is better. The relationship between users and votes is n to n, therefore a new table should be created. It should be something like this:
question id (int) | user id (int) | permanent (bool) | timestamp (datetime)
Permanent field can be used to make votes stay after a given time, as SO does.
Other fields may be added according to desired features.
As each row will take at least 16B, you can have up to 250M rows in the table before the table uses 4GB (fat32 limit if there is one archive per table, which is the case for MyISAM and InnoDB).
Also, as Matthew Scharley points out in a comment, don't load all votes at once into memory (as fetching all the table in a resultset). You can always use LIMIT clause to narrow your query results.