What is the best way to retrieve distinct / unique values using SPQuery?

Question

I have a list that looks like:

Movie          Year
-----          ----
Fight Club     1999
The Matrix     1999
Pulp Fiction   1994

Using CA

Answer 1

I was considering this problem earlier today, and the best solution I could think of uses the following algorithm (sorry, no code at the moment):

L is a list of known values (starts populated with the static Choice options when querying fill-in options, for example)
X is approximately the number of possible options

1. Create a query that excludes the items in L
1. Use the query to fetch X items from list (ordered as randomly as possible)
2. Add unique items to L
3. Repeat 1 - 3 until number of fetched items < X

This would reduce the total number of items returned significantly, at the cost of making more queries.

It doesn't much matter if X is entirely accurate, but the randomness is quite important. Essentially the first query is likely to include the most common options, so the second query will exclude these and is likely to include the next most common options and so on through the iterations.

In the best case, the first query includes all the options, then the second query will be empty. (X items retrieved in total, over 2 queries)

In the worst case (e.g. the query is ordered by the options we're looking for, and there are more than X items with each option) we'll make as many queries as there are options. Returning approximately X * X items in total.