Given a set of latitude and longitude points, how can I calculate the latitude and longitude of the center point of that set (aka a point that would center a view on all poi
If you want all points to be visible in the image, you'd want the extrema in latitude and longitude and make sure that your view includes those values with whatever border you want.
(From Alnitak's answer, how you calculate the extrema may be a little problematic, but if they're a few degrees on either side of the longitude that wraps around, then you'll call the shot and take the right range.)
If you don't want to distort whatever map that these points are on, then adjust the bounding box's aspect ratio so that it fits whatever pixels you've allocated to the view but still includes the extrema.
To keep the points centered at some arbitrary zooming level, calculate the center of the bounding box that "just fits" the points as above, and keep that point as the center point.