I have a set S of n points in dimension d for which I can calculate all pairwise distances if need be. I need to select k points in this set so that t
Step 1: Precompute dist(pi, pj) for all pairs pi,pj
Step 2: Construct a complete graph V={p1,...,pn} with edge weights w_ij = dist(pi, pj)
Step 3: Solve the maximum edge-weighted clique (MEC) problem.
MEC is definitely NP-complete, and it is strongly related to quadratic programming. Thus, you might focus on heuristics if n is large (or even of moderate size).
Note that, by precomputing edge-weights, there are no restrictions on the distance function